CN116993933A - Live-action map construction method, device and equipment under emergency scene and storage medium - Google Patents

Abstract

The invention provides a method, a device, equipment and a storage medium for building a live-action map in an emergency scene, which relate to the technical field of artificial intelligence, and the method comprises the following steps: acquiring live-action images of a plurality of disaster areas shot by a plurality of unmanned aerial vehicles; constructing a real-time disaster situation perception situation map based on each live-action image; acquiring a communication positioning demand grading model which is obtained based on historical grading data training under an emergency scene; based on the real-time disaster situation perception situation map and the communication positioning demand grading model, a live-action map based on communication positioning demand grading calibration and disaster point calibration is constructed. The invention can improve the real-time disaster sensing efficiency and the accuracy of communication positioning demand classification, thereby better assisting commander to sense global situation and reasonably distributing rescue resources.

Description

Method, device, equipment and storage medium for constructing real-life map in emergency scenarios

Technical Field

The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for constructing a real-scene map in an emergency scenario.

Background Art

Natural disasters usually happen unexpectedly. Due to weak infrastructure and complex terrain in disaster areas, emergency rescue is seriously delayed. Global perception, reliable communication, and accurate positioning are important components of the emergency response system and are particularly important for making timely rescue decisions.

At present, the main method used is expert judgment based on experience, that is, the commander judges the disaster situation in each emergency rescue area to guide the classification of communication positioning needs. However, the disaster situation changes rapidly, facilities are damaged, the terrain of the disaster area is complex, and expert resources are insufficient, making it difficult to obtain the overall situation in a timely and accurate manner and correctly allocate rescue resources.

Therefore, the shortcomings of the existing technology are: the efficiency of real-time disaster perception is low, and the accuracy of communication positioning demand classification is not high, which makes it difficult for commanders to obtain the overall situation in a timely and accurate manner and correctly allocate rescue resources.

Summary of the invention

The present invention provides a method, device, equipment and storage medium for constructing a real-life map in an emergency scenario, which is used to solve the defects in the prior art that the efficiency of real-time disaster perception is low and the accuracy of communication positioning demand classification is not high, resulting in difficulty for commanders to timely and accurately obtain the overall situation and correctly allocate rescue resources, thereby improving the efficiency of real-time disaster perception and the accuracy of communication positioning demand classification, thereby better assisting commanders in perceiving the overall situation and reasonably allocating rescue resources.

The present invention provides a method for constructing a real scene map in an emergency scenario, comprising:

Obtain real-life images of multiple disaster-stricken areas taken by multiple drones;

Based on the real-scene images, a real-time disaster awareness situation map is constructed;

Acquire a communication positioning demand grading model, wherein the communication positioning demand grading model is obtained by training based on historical grading data in emergency scenarios;

Based on the real-time disaster awareness situation map and the communication positioning demand grading model, a real-life map based on communication positioning demand grading calibration and disaster point calibration is constructed.

According to a method for constructing a real-scene map in an emergency scenario provided by the present invention, the real-time disaster awareness situation map is constructed based on each of the real-scene images, including:

Convert the real scene image from the pixel coordinate system to the world coordinate system to obtain a target image;

Based on the world coordinates of each of the target images, stitching the target images to obtain a stitched image;

The missing areas in the spliced image are completed to obtain the real-time disaster awareness situation map.

According to a method for constructing a real scene map in an emergency scenario provided by the present invention, the target images are spliced based on the world coordinates of the target images to obtain a spliced image, including:

For any two target images, judging whether there is a related area between the two target images according to the world coordinates of the two target images;

In the case where there are associated regions between the two target images, a scale-invariant feature point detection algorithm is used to perform feature detection on the two target images respectively to obtain a feature point set;

Calculating the descriptor of each feature point in the feature point set to obtain a descriptor set;

Based on the descriptor sets corresponding to the two target images, matching processing of similar feature points is performed on the feature point sets corresponding to the two target images to obtain a plurality of matching points;

Filtering abnormal matching points among the multiple matching points to correct the multiple matching points;

Performing perspective transformation on the two target images respectively;

Based on the corrected multiple matching points, the two target images after perspective transformation are fused to obtain the stitched image; wherein, in the process of image fusion, feathering technology is used to perform smooth image conversion, and overlapping pixels are fused by weighted average color value.

According to a method for constructing a real-scene map in an emergency scenario provided by the present invention, the method of completing the missing areas in the spliced image to obtain the real-time disaster awareness situation map includes:

Reconstructing low-dimensional visual prior information of the vacant area in the stitched image;

Based on the low-dimensional visual prior information, texture detail information is added to the reconstructed spliced image to obtain the real-time disaster awareness situation map.

According to a method for constructing a real scene map in an emergency scenario provided by the present invention, the reconstruction of low-dimensional visual prior information of the vacant area in the spliced image includes:

Downsampling the stitched image into a low-dimensional image;

Determining the low-dimensional image as a visual prior image of the spliced image;

A clustering algorithm is used to generate a color dictionary in the color space of a preset image data set;

For each pixel in the visual prior image, searching the color dictionary for an element index closest to the pixel to obtain a discrete representation of the pixel;

Inputting the discrete representation of each pixel in the visual prior image into the Transformer model for iterative calculation, replacing the elements in the discrete representation of each pixel in the vacant area in the spliced image with Gibbs sampling marks, to obtain a discrete sequence;

For each of the discrete sequences, the low-dimensional visual prior information is reconstructed by querying the color dictionary.

According to a method for constructing a real-life map in an emergency scenario provided by the present invention, the method of supplementing texture detail information to the reconstructed spliced image based on the low-dimensional visual prior information to obtain the real-time disaster awareness situation map includes:

Inputting the reconstructed image matrix of the visual prior image and the bilinear interpolation result of the image matrix into a guided upsampling network for processing to obtain a predicted value;

Adjusting the model parameters of the guided upsampling network based on a loss function between the predicted value and the true value; the loss function is a weighted sum of an L1 loss function and an adversarial loss function; the adversarial loss function is related to a discriminant value of the discriminator to the predicted value;

Jointly training the guided upsampling network and the discriminator to obtain diversified restoration results;

Determining the most robust result from the plurality of reduction results;

The texture detail information is supplemented to the reconstructed spliced image based on the result with the highest robustness to obtain the real-time disaster awareness situation map.

According to a method for constructing a real scene map in an emergency scenario provided by the present invention, a communication positioning demand grading model is obtained, including:

Acquire historical classification data under the emergency scenario; the historical classification data includes classification indicators and expert comment labels for each classification indicator;

Dividing the historical classification data into a training set and a test set;

Inputting the training set into the communication positioning demand grading model for training;

Inputting the test set into the trained communication positioning demand grading model for testing;

When the test result of the communication positioning requirement grading model fails, the model parameters of the communication positioning requirement grading model are fine-tuned until the test result of the communication positioning requirement grading model passes.

According to a method for constructing a real scene map in an emergency scenario provided by the present invention, based on the real-time disaster awareness situation map and the communication positioning demand grading model, a real scene map based on communication positioning demand grading calibration and disaster point calibration is constructed, comprising:

Determine the input indicators of the communication positioning demand grading model; the input indicators include detailed types of service objects, emergency scenarios, handling stages and objective factors of events and various levels of communication positioning demand;

Inputting the input index into the communication positioning demand grading model for processing, and outputting a communication positioning demand grading calibration result;

Determine the disaster point calibration result based on the real-time disaster awareness situation map;

The communication positioning demand classification calibration result and the disaster point calibration result are added to the real-time disaster awareness situation map to obtain the real-scene map.

The present invention also provides a real scene map construction device in an emergency scenario, comprising:

The first acquisition module is used to acquire real-life images of the disaster-stricken area taken by multiple drones;

A first construction module is used to construct a real-time disaster awareness situation map based on each of the real scene images;

A second acquisition module is used to acquire a communication positioning demand classification model, where the communication positioning demand classification model is trained based on historical classification data in emergency scenarios;

The second construction module is used to construct a real-life map based on communication positioning demand grading calibration and disaster point calibration based on the real-time disaster awareness situation map and the communication positioning demand grading model.

The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for constructing a real-life map in an emergency scenario as described in any one of the above are implemented.

The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for constructing a real-scene map in an emergency scenario as described in any one of the above.

The present invention provides a method, device, equipment and storage medium for constructing a real-life map in an emergency scenario. First, real-life images of multiple disaster-stricken areas taken by multiple drones are obtained, and a real-time disaster awareness situation map is constructed based on each real-life image, which can improve the efficiency of real-time disaster awareness. Then, a communication positioning demand grading model is obtained, and the communication positioning demand grading model is obtained by training based on historical grading data in emergency scenarios. Based on the real-time disaster awareness situation map and the communication positioning demand grading model, a real-life map based on communication positioning demand grading calibration and disaster point calibration is constructed, which can improve the accuracy of communication positioning demand grading, thereby better assisting commanders in perceiving the overall situation and reasonably allocating rescue resources.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a schematic diagram of a flow chart of a method for constructing a real scene map in an emergency scenario provided by an embodiment of the present invention;

2 is a schematic diagram of a scene of a method for constructing a real scene map in an emergency scenario provided by an embodiment of the present invention;

FIG3 is a schematic diagram of a coordinate conversion process of a drone sampling image provided by an embodiment of the present invention;

FIG4 is a schematic diagram of coordinate conversion of a drone sampling image provided by an embodiment of the present invention;

FIG5 is a schematic diagram of a UAV sampling image stitching process according to an embodiment of the present invention;

FIG6a is a schematic diagram of a target image 1 to be detected provided by an embodiment of the present invention;

FIG6b is a schematic diagram of a target image 2 to be detected provided by an embodiment of the present invention;

FIG6c is a schematic diagram of feature point detection of a target image 1 provided by an embodiment of the present invention;

FIG6d is a schematic diagram of feature point detection of a target image 2 provided by an embodiment of the present invention;

FIG7 is a schematic diagram of feature point matching provided by an embodiment of the present invention;

FIG8 is a schematic diagram of image stitching provided by an embodiment of the present invention;

FIG9 is a schematic diagram of a UAV sampling image completion process according to an embodiment of the present invention;

FIG10 is a flow chart of a Transformer-based visual prior generation algorithm according to an embodiment of the present invention;

FIG11 is a schematic diagram of a detail texture restoration process based on CNN provided by an embodiment of the present invention;

FIG12 is a schematic diagram of a grading index provided in an embodiment of the present invention;

13 is a schematic diagram of service object indicators provided in an embodiment of the present invention;

FIG14 is a schematic diagram of emergency scenario indicators provided by an embodiment of the present invention;

15 is a schematic diagram of treatment stage indicators provided by an embodiment of the present invention;

FIG16 is a schematic diagram of objective factor indicators of events provided by an embodiment of the present invention;

17 is a schematic diagram of the communication positioning requirement level provided by an embodiment of the present invention;

FIG18 is a schematic diagram of a demand classification algorithm flow chart provided by an embodiment of the present invention;

FIG19 is a drone disaster aerial photography dataset provided by an embodiment of the present invention;

FIG20 is a mosaic result of drone acquisition provided by an embodiment of the present invention;

FIG21 is a result of drone acquisition, stitching and cropping provided by an embodiment of the present invention;

FIG22 is a completion result of an image collected by a drone provided in an embodiment of the present invention;

FIG23 is a curve showing the quality change of the real scene map generated under different bandwidths and different acquisition area conditions provided by an embodiment of the present invention;

FIG24 is a diagram of disaster point detection and calibration of a real-life map provided by an embodiment of the present invention;

FIG25a is one of the confusion matrices of the communication positioning requirement classification model provided by an embodiment of the present invention;

FIG25b is a second confusion matrix of the communication positioning requirement classification model provided by an embodiment of the present invention;

26 is a real-life map based on communication positioning demand classification calibration and disaster point calibration provided by an embodiment of the present invention;

27 is a schematic diagram of the structure of a real scene map construction device for emergency scenarios provided by an embodiment of the present invention;

FIG. 28 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

The following describes the real scene map construction method for emergency scenarios of the present invention in conjunction with Figures 1 to 26.

Please refer to Figure 1, which is a flow chart of a method for constructing a real scene map in an emergency scenario provided by an embodiment of the present invention. As shown in Figure 1, the method may include the following steps:

Step 101: Acquire real-life images of multiple disaster-stricken areas taken by multiple drones;

Step 102: construct a real-time disaster awareness situation map based on each real scene image;

Step 103: Obtain a communication positioning demand classification model, where the communication positioning demand classification model is trained based on historical classification data in emergency scenarios;

Step 104: Based on the real-time disaster awareness situation map and the communication positioning demand grading model, a real-life map based on communication positioning demand grading calibration and disaster point calibration is constructed.

In step 101, as shown in FIG. 2, there are multiple disaster-stricken areas and multiple drones, and real-scene images of the multiple disaster-stricken areas taken by the multiple drones are acquired.

For example, suppose there are k drones performing perception tasks in a rectangular area of M×N, and transmitting multiple real-scene images taken back to the ground command vehicle and server. The drone set can be represented as The position of the kth UAV can be represented by the coordinates u _k ={x _k ,y _k ,z _k }, and the set of real-life images collected by the UAV can be represented as

In this step, on the one hand, due to the complex terrain of the disaster-stricken area and the wide search and rescue area, the efficiency of a single drone in shooting a global situation map is low, and it is difficult to ensure the high freshness of the real-life map required for emergency rescue. Therefore, it is necessary to introduce multiple drones for collaborative perception. On the other hand, due to the damage of ground base stations and other facilities in the disaster-stricken area, bandwidth resources are scarce, and drones only support the transmission of some high-definition pictures. Therefore, multiple drones are needed to shoot multiple real-life images for stitching.

In step 102, the real-time disaster awareness situation map can be constructed by performing coordinate conversion, splicing processing and completion processing on the real-scene images taken by multiple drones, thereby improving the efficiency of real-time disaster awareness.

In step 103, emergency scenarios may include fire, smoke, earthquake, etc., and the historical hierarchical data under emergency scenarios may include a large number of disaster data sets under emergency scenarios such as fire, smoke, earthquake, etc.

Specifically, a communication positioning demand classification model is trained based on historical classification data in emergency scenarios to obtain a trained communication positioning demand classification model.

In a possible implementation, step 103 may include the following sub-steps:

Step 1031: Obtain historical classification data in emergency scenarios; the historical classification data includes classification indicators and expert comment labels for each classification indicator;

Step 1032: Divide the historical classification data into a training set and a test set;

Step 1033: input the training set into the communication positioning demand grading model for training;

Step 1034: input the test set into the trained communication positioning requirement classification model for testing;

Step 1035: When the test result of the communication positioning requirement grading model fails, fine-tune the model parameters of the communication positioning requirement grading model until the test result of the communication positioning requirement grading model passes.

In step 104, disaster points can be calibrated through the real-time disaster awareness situation map, and communication positioning demand grading calibration can be performed through the communication positioning demand grading model, so as to obtain a real-time disaster awareness situation map and the communication positioning demand grading model, and construct a real-life map based on the communication positioning demand grading calibration and disaster point calibration, which can improve the accuracy of communication positioning demand grading.

The method for constructing a real-life map in an emergency scenario provided by an embodiment of the present invention first obtains real-life images of multiple disaster-stricken areas taken by multiple drones, and constructs a real-time disaster awareness situation map based on each real-life image, which can improve the efficiency of real-time disaster awareness; then, a communication positioning demand grading model is obtained, and the communication positioning demand grading model is trained based on historical grading data in the emergency scenario; based on the real-time disaster awareness situation map and the communication positioning demand grading model, a real-life map based on communication positioning demand grading calibration and disaster point calibration is constructed, which can improve the accuracy of communication positioning demand grading, thereby better assisting commanders in perceiving the overall situation and reasonably allocating rescue resources.

Based on the method for constructing a real scene map in an emergency scenario according to the embodiment of FIG. 1 , in an exemplary embodiment, step 102 may include the following sub-steps:

Step 1021, converting the real scene image from the pixel coordinate system to the world coordinate system to obtain a target image;

Step 1022: based on the world coordinates of each target image, stitching the target images together to obtain a stitched image;

Step 1023: Fill in the missing areas in the spliced image to obtain a real-time disaster awareness situation map.

In step 1021, as shown in FIG3, the coordinate transformation of the camera imaging system involves different coordinate systems, namely the world coordinate system, the camera coordinate system, the ideal image coordinate system, the real image coordinate system and the pixel coordinate system. The transformation and mapping between these coordinate systems are necessary steps to achieve the transformation from the world coordinate system to the pixel coordinate system.

For example, coordinate transformation is performed on the real scene image captured by the k-th drone to obtain an M×N target image. The set of t real scene images captured by the k-th drone can be expressed as For all collected real-world images, from pixel coordinates [u _t ,v _t ,1] ^T to world coordinates The conversion relationship can be expressed as:

Among them, R represents the rotation matrix, T represents the translation vector, K represents the intrinsic parameter matrix of the camera, and _Zc represents the imaging scale factor.

As shown in Figure 4, A( _Xw , _Yw , _Zw ) represents the coordinates of point A in the world coordinate system; a(x, y) represents the imaging point of A in the image, _and is expressed in coordinates (u, v) in the pixel coordinate system; Oc _{- XcYcZc} represents the camera coordinate system _, with the origin of the coordinate system as the optical center _Oc ; o-xy represents the image coordinate system, with the midpoint of the image as the optical center; uv represents the pixel coordinate system, and f represents the focal length of the camera, that is, the distance from the midpoint o of the image to the optical center _Oc of the camera:

f＝||oO _c || (2)

The principle of converting from pixel coordinate system to world coordinate system is introduced in detail below:

First, to achieve the conversion from the world to the camera coordinate system, a rigid body transformation can be used. Rigid body transformation can change the spatial position and orientation of the camera coordinate system by rotation, translation, etc. The commonly used method to represent rigid body transformation is the rotation matrix R and the translation vector T:

Secondly, to achieve the conversion from the camera coordinate system to the image coordinate system, perspective projection can be used. Perspective projection can convert point P in three-dimensional space into a point on a two-dimensional image plane. The intersection point with the image plane is point p. The process can be expressed as:

Thirdly, to correct the coordinate deviation of nonlinear imaging, distortion correction can be used. Distortion correction can accurately present the shape and size of objects in the image on a plane. At the same time, the optical signal needs to be digitized into an electrical signal to complete the conversion from the real world to the pixel coordinate system. The process can be expressed as follows:

Among them, o(u _o , _vo ) represents the coordinate in the uv coordinate system, dx and dy are the camera intrinsic parameters, which respectively represent the physical size of each pixel in the image plane coordinate system on the horizontal and vertical axes.

Therefore, the relationship between world and pixel coordinates is as follows:

Therefore, the conversion relationship from pixel coordinates to real-world coordinates can be calculated by the following expression:

Wherein, K ^-1 =(EF) ^-1 , E represents the transformation matrix between the pixel and the image plane, and F represents the camera intrinsic parameter matrix.

In step 1022, each target image is spliced according to its world coordinates to obtain a spliced image.

In a possible implementation, step 1022 may include the following sub-steps:

Step 10221: for any two target images, determine whether there is a related area between the two target images according to the world coordinates of the two target images;

Step 10222: When there are associated regions between the two target images, a scale-invariant feature point detection algorithm is used to perform feature detection on the two target images to obtain a feature point set.

Step 10223, calculating the descriptor of each feature point in the feature point set to obtain a descriptor set;

Step 10224: Based on the descriptor sets corresponding to the two target images, matching processing of similar feature points is performed on the feature point sets corresponding to the two target images to obtain multiple matching points;

Step 10225: filtering abnormal matching points among the multiple matching points to correct the multiple matching points;

Step 10226, performing perspective transformation on the two target images respectively;

Step 10227: Based on the corrected multiple matching points, the two target images after perspective transformation are fused to obtain a stitched image; wherein, in the process of image fusion, feathering technology is used to perform smooth image conversion, and overlapping pixels are fused by weighted average color value.

The above steps 10221 to 10227 are described below in conjunction with FIG. 5 :

In step 10221, for any two target images, it is determined whether the two target images are adjacent, that is, whether there is a related area, based on the world coordinates of the two target images. Specifically, it is possible to determine whether the two target images are adjacent by determining whether the four endpoint coordinates of one target image are filled in by another target image. Among them, the coverage flag corresponding to the endpoint coordinates (x _i , y _i ) can be expressed as follows: This determines whether the stitching of multiple drone images is executed or not.

It can be seen that if the coverage flag corresponding to the endpoint coordinates (x _i , y _i ) is True, it means that the two target images are adjacent and need to be stitched together; if the coverage flag corresponding to the endpoint coordinates (x _i , y _i ) is False, it means that the two target images are not adjacent and do not need to be stitched together.

In step 10222, exemplarily, target image 1 shown in FIG. 6a and target image 2 shown in FIG. 6b are taken as two target images. When there are associated areas between the two target images, a scale-invariant feature point detection algorithm is used to perform feature detection on the two target images respectively, and feature point sets as shown in FIG. 6c and FIG. 6d are obtained.

In step 10224, as shown in FIG6c and FIG6d, there are a large number of similar feature points in the two images. Similar feature points can be matched by using a linear regression model algorithm such as a brute force matching method (BF Matcher, BF) and a K-Nearest Neighbor (KNN) method to obtain multiple matching points.

Regarding the BF matching method, the two groups of descriptors with the closest Euclidean distance are iteratively solved as matching items.

Regarding the KNN algorithm, for each descriptor in one image, the K nearest neighbor points are found in the descriptor set of the other image, where the nearest neighbor is considered to be the true correspondence between the key points in the two images. Due to the high efficiency of KNN feature point matching, the result of using KNN for feature matching is shown in Figure 7.

In step 10225, in order to deal with the problem that the linear regression model is insensitive to outliers, this step needs to further filter the abnormal matching points. Therefore, it is necessary to use a transformation matrix to describe and obtain the relationship between the matching points. This transformation matrix is called a homography matrix. In the estimation of the homography matrix, unnecessary feature points that do not belong to important areas will be deleted to avoid the influence of noise. In order to eliminate the influence of outliers introduced by the linear regression model, the random sampling consensus (RANdom Sample Consensus, RANSAC) algorithm is used to calculate the homography matrix. RANSAC is a classic fitting algorithm that can estimate model parameters in a data set with noise and outliers. The basic idea is to randomly select a small part of the data, fit the model and calculate the error. After multiple iterations, the model with the smallest error is selected as the output to obtain a more robust matching result.

In step 10226 and step 10227, in order to obtain the desired output image shape, all input images are deformed by perspective transformation and fused on the transformed images. This process requires calculating the deformed image coordinate range of each input image to determine the size of the output image and mapping the four corners of each source image to the transformed image. Next, feathering and other techniques are used to achieve smooth image conversion, and the overlapping pixels are fused by weighted average color values to avoid discontinuity and gaps at the splicing. All image splicing steps are completed, and a spliced image as shown in Figure 8 is obtained.

In this embodiment, since abnormal matching points are filtered during feature matching, the influence of noise can be avoided; and, feathering technology is used to perform smooth image conversion during image fusion, and overlapping pixels are fused by weighted average color values, which can avoid discontinuities and gaps at the splicing points, thereby improving the accuracy of image stitching.

In step 1023, pixels and textures are supplemented in the missing areas that are not captured in the stitched image to obtain a real-time disaster awareness situation map.

In a possible implementation, step 1023 may include the following sub-steps:

Step 10231, reconstructing low-dimensional visual prior information of the vacant area in the spliced image;

Step 10232: Based on the low-dimensional visual prior information, texture detail information is added to the reconstructed spliced image to obtain a real-time disaster awareness situation map.

Step 10231 and step 10232 are described below in conjunction with FIG. 9 :

Before step 10231, the stitched image is first judged, that is, the position coordinates (x _i , y _i ) in the stitched image are represented by a mask to indicate the missing area, as shown in the following formula:

Then, the algorithm is used to complete the missing area, that is, the part where the mask is 0.

It can be seen that when the mask is 1, it means that there are acquisition points and there is no need to execute the completion algorithm; when the mask is 0, it means that the acquisition points are missing and the completion algorithm is executed.

In step 10231, the low-dimensional visual prior information may include a low-dimensional preliminary reconstruction structure and a coarse texture. The vacant area in the spliced image is preliminarily reconstructed to obtain a low-dimensional preliminary reconstruction structure and a coarse texture of the vacant area in the spliced image.

For example, image completion aims to stitch images with missing pixels. Convert to full image The image completion task is stochastic, so given the spliced image I _m , there is a conditional distribution p(I|I _m ). Since the rough prior information X is certain given I and I _m , p(I|I _m ) can be rewritten as:

p(I∣I _m )＝p(I∣I _m )·p(X∣I,I _m )=p(X∣I _m )·p(I∣X,I _m ) (10)

Different from sampling directly from p(I|I _m ), this step first uses the Transformer model to model the basic distribution of the visual prior given I _m and represents it as p(X|I _m ). Due to the powerful representation ability of Transformer, these reconstructed visual priors contain sufficient information of global structure and rough texture.

In step 10232, illustratively, a convolutional neural network (CNN) is used to supplement fine texture details under the guidance of low-dimensional visual prior information and unmasked pixels, and it is represented as p(I|X,I _m ).

In this implementation, low-dimensional visual prior information is firstly reconstructed for the missing areas in the stitched image, and then, based on the low-dimensional visual prior information, texture detail information is supplemented to the reconstructed stitched image to obtain a high-quality image, i.e., a real-time disaster awareness situation map, which can achieve high-fidelity multivariate image completion performance.

In this embodiment, the real-time disaster awareness situation map can be constructed by performing coordinate conversion, splicing processing and completion processing on the real-scene images taken by multiple drones, which can improve the efficiency of real-time disaster awareness.

In an exemplary embodiment, step 10231 includes the following sub-steps:

Step 102311, downsampling the spliced image into a low-dimensional image;

Step 102312, determining the low-dimensional image as a visual prior image of the spliced image;

Step 102313, using a clustering algorithm to generate a color dictionary in the color space of a preset image data set;

Step 102314: for each pixel in the visual prior image, find the element index closest to the pixel in the color dictionary to obtain a discrete representation of the pixel;

Step 102315, input the discrete representation of each pixel in the visual prior image into the Transformer model for iterative calculation, replace the elements in the discrete representation of each pixel in the vacant area in the spliced image with Gibbs sampling marks, and obtain a discrete sequence;

Step 102316: For each discrete sequence, reconstruct low-dimensional visual prior information by querying the color dictionary.

The above steps 102311 to 102316 are described below in conjunction with FIG. 10 :

In step 102311, illustratively, the stitched image is downsampled to a low-dimensional image of 32×32 or 48×48, but this embodiment is not limited thereto.

In step 102312, the low-dimensional image is determined as a visual prior image of the stitched image, and the visual prior image only contains structural information and rough texture.

In step 102313, illustratively, the clustering algorithm may be a K-Means clustering algorithm, the preset image data set may be an ImageNet data set, and the color space may be an RGB space. A color dictionary of size 512×3 is generated in the RGB space of the ImageNet data set using the K-Means clustering algorithm, but the present embodiment is not limited thereto.

In step 102315, the learning goal of the Transformer model is to replace the elements in the representation sequence of the vacant area with a special masked token, thereby converting the visual prior image into a discrete sequence.

Given each token in a discrete sequence X = {x ₁ ,x ₂ ,...,x _L }, where L is the length of the sequence, this embodiment uses a predefined embedding to map it into a d-dimensional feature vector. In order to encode position information into the input, this embodiment adds a learnable position embedding to each position The marked features make the input format of the final Transformer model

The network architecture of the model is based on the decoder-only Transformer algorithm, which mainly consists of N bidirectional self-attention Transformer layers. In each Transformer layer, the calculation formula is as follows:

Z ^l-1 =LN(BMSA(D ^l-1 ))+D ^l-1 (11)

D ^l =LN(FC(Z ^l-1 ))+Z ^l-1 (12)

Among them, LN, BMSA, and FC represent layer normalization, multi-head self-attention layer, and fully connected layer, respectively. More specifically, given the input D, BMSA can be calculated as:

BMSA＝[head ₁ ;...;head _h ]W _O (13)

Among them, head is the head, h is the number of heads, d is a hyperparameter, indicating the vector dimension, and are three learnable linear projection layers, 1≤i≤h, and _WO is a learnable fully connected layer whose goal is to fuse the output results of different heads.

The output of the final Transformer layer is mapped to 512 color dictionary elements. This mapping is done through a fully connected layer and a Softmax function. An objective function similar to the Masked Language Model (MLM) is used to optimize the Transformer model. Specifically, let Π = {π ₁ ,π ₂ ,…,π _K } represent the index of the MaskedToken in the discrete input, where K is the number of Masked Tokens. Let X _Π represent the set of Masked Tokens in X, and X _-Π represent the set of unmaskedTokens. The goal of MLM is to minimize the negative log-likelihood of the Masked Token given a known Token:

Where L _MLM is the likelihood solution of MLM, and θ is the parameter of Transformer. The MLM objective combined with bidirectional attention ensures that the Transformer model can capture the entire context information to predict the probability distribution of the missing region. In order to generate a credible distribution of the Transformer model, directly sampling the entire set of Masked positions will produce untrustworthy results because these positions are independent of each other. Instead, this embodiment uses Gibbs sampling to iteratively sample tokens at different positions. Specifically, at each iteration, the conditional distribution is sequentially sampled from The first k predicted elements are used to sample grid positions, where Represents the previously generated Token. Then, this embodiment replaces the corresponding Masked Token with the sampled Token and repeats this process until all positions are updated. Through sampling, a complete set of Token sequences can be obtained.

In step 102316, for each complete discrete sequence sampled from the Transformer model, its visual prior can be reconstructed by querying the color dictionary Thus, reasonable and diverse visual prior information is obtained, thus completing the construction of low-dimensional visual prior information.

In this embodiment, the powerful representation capability of the Transformer model is used to reconstruct low-dimensional visual prior information.

In an exemplary embodiment, step 10232 includes the following sub-steps:

Step 102321, input the image matrix of the reconstructed visual prior image and the bilinear interpolation result of the image matrix into the guided upsampling network for processing to obtain a predicted value;

Step 102322, adjusting the model parameters of the guided upsampling network based on the loss function between the predicted value and the true value; the loss function is a weighted sum of the L1 loss function and the adversarial loss function; the adversarial loss function is related to the discriminant value of the discriminator to the predicted value;

Step 102323, jointly train the guided upsampling network and the discriminator to obtain diversified restoration results;

Step 102324, determining the most robust result from the diversified restoration results;

Step 102325: based on the result with the highest robustness, add texture detail information to the reconstructed spliced image to obtain a real-time disaster awareness situation map.

The above steps 102321 to 102325 are described below in conjunction with FIG. 11 :

In step 102321, after completing the reconstruction of the low-dimensional visual prior, in order to restore the data with the original resolution of H×W×3 and maintain the boundary consistency between the masked and unmasked areas, a guided upsampling network is introduced. The network uses a neural network to model the texture pattern, and renders the reconstructed visual prior under the guidance of the masked input _Im , thereby achieving reconstruction of high-fidelity details. Among them, the guided upsampling process can be expressed as:

in, Represents the matrix after the original data X is reshaped, is the bilinear interpolation result of _It , ∩ represents the cascade operation, and _Ipred represents the predicted value of the guided upsampling network. It is the backbone of the guided upsampling network parameterized by δ, consisting of an encoder, a decoder, and multiple residual blocks.

In step 102322, the network performance is optimized by minimizing the l ₁ loss between the guided upsampling network prediction value I _pred and the true value I:

in, represents the expectation of _l1 loss.

In order to generate more realistic details during training, an additional adversarial loss is introduced:

Among them, L _adv represents the expectation of adversarial loss, is the discriminator parameterized by ω.

In step 102323, the upsampling network F and the discriminator are jointly trained by solving the following optimization problem

Among them, L _upsample represents the expectation of upsampling of l ₁ loss and adversarial loss, α1 and α2 are the l ₁ loss weight and adversarial loss weight, respectively.

In step 102324, the algorithm combines multiple factors, such as the spread of the disaster, the confidence level of the disaster site, image generation indicators, and commander evaluation, and selects the most robust result from the generated diversified restoration results for output.

In step 102325, texture detail information is added to the reconstructed spliced image based on the result with the highest robustness, so as to obtain a real-time disaster awareness situation map. FIG11 shows the best restoration effect output according to the confidence of disaster point detection.

In this embodiment, texture detail information may be supplemented into the reconstructed stitched image.

In an exemplary embodiment, step 104 may include the following sub-steps:

Step 1041: Determine input indicators of the communication positioning demand grading model; the input indicators include service objects, emergency scenarios, handling stages, detailed types of objective factors of events, and various levels of communication positioning demand;

Step 1042: input the input index into the communication positioning demand grading model for processing, and output the communication positioning demand grading calibration result;

Step 1043, determining the disaster point calibration result based on the real-time disaster awareness situation map;

Step 1044: Add the communication positioning demand classification calibration results and the disaster point calibration results to the real-time disaster awareness situation map to obtain a real-scene map.

In step 1041, the historical classification data includes classification indicators and expert comment labels for each classification indicator. Exemplarily, as shown in FIG12, the classification indicators may include: first-level indicators, second-level indicators, ..., and fifth-level indicators; wherein the first-level indicators include the priority of communication and positioning tasks, and the second-level indicators include A service objects, B emergency scenarios, C handling stages, D objective factors of events, E disaster point detection confidence, and so on.

As shown in Figure 13, the A service objects are refined, and the A service objects can include A1 personnel types and A2 vehicle types; among them, A1 personnel types can include A1-1 front-line team, A1-2 front commander, A1-3 rear commander and A1-4 support personnel; A2 vehicle types can include A2-1 communication vehicle and A2-2 command vehicle.

As shown in Figure 14, the B emergency scenario is refined and can include B1 topography and B2 weather and climate; among them, B1 topography can include B1-1 hills, B1-2 plains, B1-3 basins, B1-4 mountains, B1-5 forest areas and B1-X others; B1-5 forest areas can include B1-5-1 forest phases, B1-5-2 tree species and B1-5-X others; B2 weather and climate can include B2-1 temperature, B2-2 humidity, B2-3 wind, B2-4 rainfall and B2-X others; B2-3 wind can include B2-3-1 wind force and B2-3-2 wind direction.

As shown in Figure 15, the C handling stage is refined, and the C handling stage can include C1 before the event, C2 after the event, C3 during the event, and C4 after the event; among them, C1 before the event can include C1-1 disaster reporting and C1-2 preliminary investigation, C2 after the event can include C2-1 troop transfer, C3 during the event can include C3-1 detection stage and C3-2 handling stage, and C4 after the event can include C4-1 clearing the disaster area, C4-2 evacuation and return to the camp, and C4-3 post-disaster reconstruction.

As shown in Figure 16, the objective factors of D events are refined, and the objective factors of D events may include D1 event type, D2 disaster situation and D3 event scale; among them, D1 event type may include D1-1 peacetime and D1-2 wartime, D1-1 peacetime may include daily patrols, D1-2 wartime may include D1-2-1 fire, D1-2-2 mudslide, D1-2-3 earthquake and D1-2-X others; D2 disaster situation may include D2-1 disaster point, D2-2 potential disaster point and D2-3 non-disaster point, and D3 event scale may include D3-1 disaster-affected area and D3-2 number of affected people.

As shown in Figure 17, the various levels of communication and positioning requirements are divided into five levels. As the level increases from L1 to L5, the importance of the corresponding regional requirements also gradually increases. For example, in a relatively safe area at the rescue site, only basic voice transmission and positioning requirements are required; the closer to the disaster site, the more important the real-time information on the scene is, and a faster communication rate is required to ensure image and video transmission, and a higher positioning accuracy is required to ensure the safety of the rescue.

In step 1042, illustratively, as shown in FIG18, first, the data is entered into the disaster classification expert knowledge base based on the index data collected in the historical emergency rescue and the expert classification judgment, and the model is trained through a one-dimensional convolutional neural network. A one-dimensional convolutional network refers to a neural network that performs convolution, pooling, full connection and other operations on a one-dimensional input vector, and can infer the communication and positioning demand level based on the input disaster point situation, service object, emergency scenario, disposal stage and number of people in the area.

Before forward pass, the input data needs to be One-hot encoded. The input data can be represented as a matrix of size L×F, where each row represents a sample and each column represents a feature. Next, this matrix is input into the convolution layer for feature extraction and dimensionality reduction. The one-dimensional convolution layer can filter the input data sequence through a set of learnable convolution kernels to extract important features from the data. After the convolution operation, the one-dimensional convolution layer needs to use an activation function to perform a nonlinear transformation on the result. Among them, the ReLU activation function is a commonly used nonlinear activation function that has excellent performance in dealing with the problem of gradient disappearance. Therefore, the ReLU activation function is used for processing, which is defined as f(x)＝max(0,x). Next, the output dimension is reduced through the pooling layer, and finally the result of communication positioning demand classification is output through the fully connected layer.

During the training process, the difference between the network output and the true label needs to be measured in order to adjust the model parameters to improve performance. The categorical_crossentropy loss function can quantify the difference between the network output and the true label, and use the back propagation algorithm to calculate the gradient of each parameter so that the parameters can be updated along the direction of the gradient to reduce the cross entropy loss function. The cross entropy loss function J() can be expressed as:

Where C is the number of categories and y is a C-dimensional vector representing the true category label of the data sample. is a C-dimensional vector representing the output of the model. The value of each element is a real number between 0 and 1, and the sum of all elements is 1. _yi represents the probability that the true category is i, and It represents the probability of the model predicting that it is category i, and its value is usually calculated using the Softmax function. Input the Softmax function and convert it into a probability vector so that the sum of all elements is 1. The Softmax function σ() can be expressed as:

Substituting the calculation result of the Softmax function into the Categorical_crossentropy loss function formula, we can get the model prediction result The cross entropy loss between y and the true label y is used to measure the accuracy of the model prediction.

At the same time, in order to improve training efficiency and accuracy, this embodiment uses the Adam optimizer to update and optimize each weight parameter in the neural network. Specifically, the Adam optimizer calculates the gradient of the first-order moment estimate m _t and the second-order moment estimate v _t of each weight parameter θ _t , and performs deviation correction on these estimates to obtain and Then, an adaptive learning rate is calculated based on these estimates, and the learning rate is used to update the value of the parameter θ _t . The update rule of the Adam optimizer can be expressed as:

Among them, m _t-1 is the first-order moment estimate before updating, v _t-1 is the second-order moment estimate before updating, θ _t-1 is the weight parameter before updating, J(θ _t-1 ) is the loss function, is the gradient of the loss function with respect to the parameter θ, _θt is the updated parameter value, η is the learning rate, ∈ is a small constant to prevent the denominator from being zero, and _β1 and _β2 are the decay rates of the first-order and second-order moment meters, respectively, which aim to smooth the estimated value.

In step 1044, the communication positioning demand classification calibration results and disaster point calibration results are calibrated in the real-life situation map based on the output results and the granularity information of the regional classification, so as to assist the commanders in making more efficient and reasonable communication positioning demand classification decisions. In addition, the network model is fine-tuned using field data in each field rescue, so as to adjust and improve the model to further improve the performance of the model in actual emergency rescue.

In this embodiment, disaster points can be calibrated through the real-time disaster awareness situation map, and communication positioning demand grading calibration can be performed through the communication positioning demand grading model, so as to obtain a real-time disaster awareness situation map and the communication positioning demand grading model, and construct a real-life map based on the communication positioning demand grading calibration and disaster point calibration, which can improve the accuracy of communication positioning demand grading.

The advantages of the method for constructing a real scene map in an emergency scenario according to an embodiment of the present invention are described in detail below through experimental results.

First, build an experimental simulation platform. The experimental simulation platform is a Python platform, which is implemented using the PyTorch framework. The PyTorch framework is a deep learning framework developed by Facebook. Its advantages include dynamic computational graphs, GPU acceleration, and a rich tool library. The GPU used for simulation is RTX3090. Table 1 shows the main simulation parameters:

Table 1 Main simulation parameters

parameter Numeric Image stitching feature matching threshold 0.01 Image stitching descriptor window size 5 Image stitching RANSAC feature difference threshold 3 Number of image completion Transformer layers 14 Image completion Transformer head number 8 Image completion Embedding dimension 256 Image completion Transformer learning rate 3× ^10-4 Image completion CNN learning rate 10-4 Image completion discriminator/generator learning rate ratio 0.1 Loss weight for image completion feature matching 10 Image completion L1 loss weight 1 Image completion adversarial loss weights 0.1 Object detection confidence threshold 0.5 Demand classification Adam first-order moment estimation decay rate 0.9 Demand classification Adam second-order moment estimation decay rate 0.99 Demand-graded learning rate 4× ^10-3

As shown in Figure 19, some of the real scene information of the disaster site collected by the drone is shown. The real scene map of the global sampling of the disaster area is generated, which includes the global image information collected by the drone aerial photography and integrates the flame information created by Unity3D. After the image coordinates are converted, the collected images are filled into the map.

During the rescue process, it is crucial to obtain global real-life photos of the affected area and the details of the disaster points for effectively responding to the disaster. However, if the disaster area expands, the disaster situation will spread quickly. On the one hand, the global sampling delay is very high, and it is difficult to support the freshness requirements of the real-life situation map in the emergency rescue scenario. On the other hand, only the splicing operation requires that the two frames before and after the image have a correlation, which means that more pictures with repeated parts need to be collected, which consumes bandwidth and computing power resources in the emergency rescue scenario. In this embodiment, partial sampling operations are performed, and the collected pictures are stitched into the map according to the coordinate relationship. If they are repeated, the splicing operation is performed. After completing the above operations, the vacant area is image-filled, which can greatly reduce the latency and bandwidth and computing power consumption, and support the drone to collect pictures in real time to update the real-life map, which can maintain a high degree of freshness. According to the pictures of different features and shooting positions collected by the above-mentioned drone, this embodiment performs image splicing based on the coordinate conversion module. After careful processing and reasonable combination of these local pictures, they are successfully spliced into a complete, high-resolution panoramic image as shown in Figure 20.

As shown in Figure 20, there may be missing areas after the images are stitched. This phenomenon is usually caused by factors such as deformation caused by perspective transformation, changes in image color and brightness, and image edge processing. If the missing area is located at the edge of the real map, as shown in the black area in the figure above, these areas often have less information and can be directly cropped and aligned. The cropped result is shown in Figure 21.

If the missing area is within the real-life map area, such as the white area in Figure 21, this information is relatively important, so this embodiment uses an image completion method to predict and fill in the gaps. For image completion, this embodiment performs pre-training based on a model of the Place data set during the training process, and then the present invention performs further training for specific scenes such as forest areas and fires, so that the Transformer model can generate better visual priors. Below, we will take a disaster area in the above map as an example, as shown in Figure 22, to demonstrate the effect of image completion in the missing area.

In each set of results, the white hole part of the first row on the left is the missing area, and the right picture shows the multivariate generation result of low-resolution restoration based on Transformer's visual prior. It can be seen that through low-resolution prior restoration, the restoration result can basically restore the outline of the original image and can generate a variety of restoration effects. The left picture of the second row of each group shows the original image, and the right picture shows the high-resolution restoration result of CNN based on multivariate visual prior. As shown in the figure, under the guidance of the input mask image resolution, the model can restore better results. In addition, with the continuous accumulation of real-life emergency rescue data, the model can further improve the restoration quality. In the construction of the real-life situation map, it supports the subjective judgment of the commander and the judgment of objective indicators such as disaster point detection confidence and image generation quality, so as to determine the selection of the optimal restoration effect.

In the process of emergency rescue, although local sampling can save computing power, bandwidth and other resources. However, if the area of the missing real-life photos is too large, the restoration quality of the real-life map may not meet the requirements. Therefore, this embodiment analyzes the sampling density and bandwidth to analyze the impact of the missing area on the image generation quality. This embodiment evaluates the changes in image evaluation indicators such as the Frechet Inception Distance score (FID), structural similarity (SSIM), normalized root mean square error (NRMSE), and peak signal-to-noise ratio (PSNR) to guide the drone to adopt a more reasonable sampling strategy.

As shown in Figure 23, according to the image indicators, it can be found that the missing area of the real scene area has a significant impact on the image generation quality. Therefore, in the actual rescue process, the method of using some of the original map areas and the generated area for indicator testing can be used to more comprehensively and accurately evaluate the image generation quality. In addition to the image indicators, there are some other factors that need to be considered, such as the limitations of resources such as computing power and bandwidth, as well as the size and complexity of the disaster area. Taking bandwidth as an example, assuming that 10M bandwidth can sample global data within T time, when the available bandwidth continues to decrease, high-resolution images need to be downsampled and transmitted back. At this time, the quality of the generated map will not show a significant improvement trend as the acquisition area increases. Based on comprehensive consideration of these factors, the appropriate sampling density can be determined, thereby formulating a more reasonable and efficient drone disaster sampling strategy. This strategy can guarantee the quality of image generation to the greatest extent, while also saving resources and improving rescue efficiency, which helps to achieve better results in emergency rescue.

As shown in Figure 24, the Yolov5-based disaster detection will be performed based on the generated real-life map. In order to improve the accuracy of disaster point detection, this embodiment uses the Coco dataset for pre-training, and uses disaster datasets such as flames and smoke to fine-tune the model. Thus, a model with good performance in disaster detection is obtained. In the real-life map, this research point applies this model to disaster point detection to determine the actual location of the disaster point in the real-life map. In addition, through the coordinate conversion module, this embodiment can also obtain the real geographic coordinates of the disaster point. The calibration of the disaster point detection effect in the real-life map is shown in Figure 24.

The following will divide the communication and positioning demand levels of different regions according to indicators such as disaster point detection, service objects, disposal stage, and regional population. This embodiment uses a confusion matrix to evaluate the performance of the model trained based on a data set of historical rescue classification data. The confusion matrix is an effective tool for evaluating model performance. It can calculate performance indicators such as classification accuracy, precision, recall rate, F1 value, etc. based on the real labels and predicted labels to evaluate the classification effect of the model. Since the communication and positioning demand classification problem is a multi-classification problem, this embodiment uses the Macro-average method to calculate the indicators, as shown in the following formulas (25)-(28).

Among them, Accuracy _macro represents precision, Precision _macro represents accuracy, Recall _macro represents recall, F _macro represents F1 value, TP (True Positive) represents true positive examples, that is, the number of samples predicted to be level N and whose actual value is level N; FP (False Positive) represents false positive examples, that is, the number of samples predicted to be level N but whose actual value is non-level N; TN (True Negative) represents true negative examples, that is, the number of samples predicted to be non-level N and whose actual value is non-level N; FN (False Negative) represents false negative examples, that is, the number of samples predicted to be non-level N but whose actual value is level N.

Figure 25a and Figure 25b show the prediction confusion matrices of the models trained based on the communication and positioning requirements classification datasets, respectively. Based on the confusion matrix and its evaluation index analysis, the model performance can be effectively evaluated, as shown in Table 2.

Table 2 Performance index table of demand grading model

Table 2 shows the classification performance of the one-dimensional convolutional model trained on the communication and positioning classification dataset. In terms of communication and positioning demand classification reasoning, all indicators of the model reached more than 95%, showing excellent performance. In addition, the classification model also supports fine-tuning according to the instructions of experts in actual rescue, which can efficiently assist emergency rescue work. According to indicators such as the number of trapped people and service objects in different areas, the schematic diagram of the results of the disaster awareness situation map construction and communication positioning demand classification method can be constructed as shown in Figure 26.

In Figure 26, the communication and positioning requirements in the disaster-stricken areas are differentiated, and the requirements are graded from L1 to L5, which can serve as an important reference for the resource adaptation research in the subsequent chapters. In the resource adaptation research, the requirements are used to set parameters such as bandwidth allocation, communication rate threshold, and positioning accuracy threshold. Therefore, the division of these levels has an important guiding role in the subsequent resource adaptation research.

The following is a description of a real scene map construction device for an emergency scenario provided by the present invention. The real scene map construction device for an emergency scenario described below and the real scene map construction method for an emergency scenario described above can be referenced to each other.

Please refer to Figure 27, which is a schematic diagram of the structure of a real scene map construction device in an emergency scenario provided by an embodiment of the present invention. As shown in Figure 27, the device may include:

A first acquisition module 10 is used to acquire real-life images of the disaster-stricken area taken by multiple drones;

The first construction module 20 is used to construct a real-time disaster awareness situation map based on each real scene image;

A second acquisition module 30 is used to acquire a communication positioning demand classification model, where the communication positioning demand classification model is trained based on historical classification data in emergency scenarios;

The second construction module 40 is used to construct a real-life map based on communication positioning demand classification calibration and disaster point calibration based on the real-time disaster awareness situation map and the communication positioning demand classification model.

In an exemplary embodiment, the first building block 20 may include:

A conversion submodule is used to convert the real scene image from the pixel coordinate system to the world coordinate system to obtain the target image;

A stitching submodule, used for stitching the target images based on the world coordinates of the target images to obtain a stitched image;

The completion submodule is used to complete the missing areas in the spliced image to obtain a real-time disaster awareness situation map.

In an exemplary embodiment, the splicing submodule is specifically used for:

For any two target images, determine whether there is a related area between the two target images according to the world coordinates of the two target images;

When there are related areas between the two target images, a scale-invariant feature point detection algorithm is used to perform feature detection on the two target images respectively to obtain a feature point set;

Calculate the descriptor of each feature point in the feature point set to obtain a descriptor set;

Based on the descriptor sets corresponding to the two target images, matching processing is performed on similar feature points of the feature point sets corresponding to the two target images to obtain multiple matching points;

Filtering abnormal matching points among the multiple matching points to correct the multiple matching points;

Perform perspective transformation on the two target images respectively;

Based on the corrected multiple matching points, the two target images after perspective transformation are fused to obtain a stitched image. In the process of image fusion, feathering technology is used to perform smooth image conversion, and the overlapping pixels are fused by weighted average color value.

In an exemplary embodiment, the completion submodule includes:

A reconstruction unit, used to reconstruct low-dimensional visual prior information of the vacant area in the spliced image;

The supplement unit is used to supplement texture detail information into the reconstructed spliced image based on low-dimensional visual prior information to obtain a real-time disaster awareness situation map.

In an exemplary embodiment, the reconstruction unit is specifically configured to:

Downsample the spliced image to a low-dimensional image;

Determine the low-dimensional image as a visual prior image for the stitched image;

A clustering algorithm is used to generate a color dictionary in the color space of a preset image data set;

For each pixel in the visual prior image, find the element index closest to the pixel in the color dictionary to obtain a discrete representation of the pixel;

The discrete representation of each pixel in the visual prior image is input into the Transformer model for iterative calculation, and the elements in the discrete representation of each pixel in the vacant area of the spliced image are replaced with Gibbs sampling marks to obtain a discrete sequence;

For each discrete sequence, the low-dimensional visual prior information is reconstructed by querying the color dictionary.

In an exemplary embodiment, the supplement unit is specifically configured to:

The image matrix of the reconstructed visual prior image and the bilinear interpolation result of the image matrix are input into the guided upsampling network for processing to obtain the predicted value;

The model parameters of the guided upsampling network are adjusted based on the loss function between the predicted value and the true value; the loss function is the weighted sum of the L1 loss function and the adversarial loss function; the adversarial loss function is related to the discriminant value of the discriminator on the predicted value;

Jointly train the guided upsampling network and the discriminator to obtain diversified restoration results;

Determine the most robust result from the diversified reduction results;

Based on the most robust result, texture detail information is added to the reconstructed spliced image to obtain a real-time disaster awareness situation map.

In an exemplary embodiment, the second acquisition module 30 is specifically configured to:

Obtain historical classification data under emergency scenarios; historical classification data includes classification indicators and expert comment labels for each classification indicator;

Divide the historical classification data into training set and test set;

The training set is input into the communication positioning requirement classification model for training;

Input the test set into the trained communication positioning requirement classification model for testing;

When the test result of the communication positioning demand grading model fails, the model parameters of the communication positioning demand grading model are fine-tuned until the test result of the communication positioning demand grading model passes.

In an exemplary embodiment, the second building module 40 is specifically configured to:

Determine the input indicators of the communication positioning demand classification model; the input indicators include service objects, emergency scenarios, handling stages, detailed types of objective factors of events, and various levels of communication positioning demand;

Inputting the input indicators into the communication positioning demand grading model for processing, and outputting the communication positioning demand grading calibration results;

Determine the disaster point calibration results based on the real-time disaster awareness situation map;

The communication positioning demand classification calibration results and disaster point calibration results are added to the real-time disaster awareness situation map to obtain a real-scene map.

FIG28 illustrates a schematic diagram of the physical structure of an electronic device. As shown in FIG28 , the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a method for constructing a real-life map in an emergency scenario, the method comprising: obtaining real-life images of multiple disaster-stricken areas taken by multiple drones; constructing a real-time disaster awareness situation map based on each real-life image; obtaining a communication positioning demand grading model, the communication positioning demand grading model is obtained by training based on historical grading data in an emergency scenario; and constructing a real-life map based on communication positioning demand grading calibration and disaster point calibration based on the real-time disaster awareness situation map and the communication positioning demand grading model.

In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the real-life map construction method in emergency scenarios provided by the above-mentioned methods, and the method includes: obtaining real-life images of multiple disaster-stricken areas taken by multiple drones; based on each real-life image, constructing a real-time disaster awareness situation map; obtaining a communication positioning demand grading model, which is obtained by training based on historical grading data in emergency scenarios; based on the real-time disaster awareness situation map and the communication positioning demand grading model, constructing a real-life map based on communication positioning demand grading calibration and disaster point calibration.

On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned methods for constructing real-life maps in emergency scenarios, the method comprising: obtaining real-life images of multiple disaster-stricken areas taken by multiple drones; constructing a real-time disaster awareness situation map based on each real-life image; obtaining a communication positioning demand grading model, the communication positioning demand grading model being trained based on historical grading data in emergency scenarios; and constructing a real-life map based on communication positioning demand grading calibration and disaster point calibration based on the real-time disaster awareness situation map and the communication positioning demand grading model.

The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (11)

1. The method for constructing the live-action map in the emergency scene is characterized by comprising the following steps of:

acquiring live-action images of a plurality of disaster areas shot by a plurality of unmanned aerial vehicles;

constructing a real-time disaster situation perception situation map based on each live-action image;

acquiring a communication positioning demand grading model, wherein the communication positioning demand grading model is obtained based on historical grading data training under an emergency scene;

and constructing a real-scene map based on communication positioning demand grading calibration and disaster point calibration based on the real-time disaster situation perception situation map and the communication positioning demand grading model.

2. The method for constructing a real-time disaster situation awareness map in an emergency scene according to claim 1, wherein the constructing a real-time disaster situation awareness map based on each real-time image comprises:

Converting the live-action image from a pixel coordinate system to a world coordinate system to obtain a target image;

based on world coordinates of the target images, performing stitching processing on the target images to obtain stitched images;

and carrying out complement treatment on the vacant areas in the spliced images to obtain the real-time disaster situation perception situation map.

3. The method for building a live-action map in an emergency scene according to claim 2, wherein the stitching each of the target images based on world coordinates of each of the target images to obtain a stitched image includes:

judging whether the two target images have associated areas or not according to world coordinates of the two target images aiming at any two target images;

under the condition that the two target images have associated areas, respectively carrying out feature detection on the two target images by adopting a feature point detection algorithm with unchanged scale to obtain a feature point set;

calculating descriptors of each feature point in the feature point set to obtain a descriptor set;

based on the descriptor sets corresponding to the two target images, carrying out matching processing of similar feature points on the feature point sets corresponding to the two target images to obtain a plurality of matching points;

Filtering abnormal matching points in the plurality of matching points to correct the plurality of matching points;

respectively performing perspective transformation on the two target images;

based on the corrected multiple matching points, performing image fusion on the two target images after perspective transformation to obtain the spliced image; in the process of image fusion, a feathering technology is adopted to carry out smooth image conversion, and overlapping pixels are fused through weighted average color values.

4. The method for building the live-action map in the emergency scene according to claim 2, wherein the step of performing the completion processing on the vacant areas in the spliced image to obtain the real-time disaster sensing situation map comprises the following steps:

reconstructing low-dimensional visual priori information of a vacant area in the spliced image;

and supplementing texture detail information to the reconstructed spliced image based on the low-dimensional visual priori information to obtain the real-time disaster situation perception situation map.

5. The method for building a live-action map in an emergency scene according to claim 4, wherein the reconstructing the low-dimensional visual prior information of the empty region in the stitched image comprises:

Downsampling the stitched image into a low dimensional image;

determining the low-dimensional image as a visual prior image of the stitched image;

generating a color dictionary in a color space of a preset image data set by adopting a clustering algorithm;

for each pixel in the visual prior image, searching an element index closest to the pixel in the color dictionary to obtain a discrete representation of the pixel;

inputting the discrete representation of each pixel in the visual prior image into a transducer model for iterative computation, and replacing elements in the discrete representation of each pixel in a vacant area in the spliced image with gibbs sampling marks to obtain a discrete sequence;

for each of the discrete sequences, the low-dimensional visual priori information is reconstructed by querying the color dictionary.

6. The method for building a live-action map in an emergency scene according to claim 4, wherein the supplementing texture detail information into the reconstructed stitched image based on the low-dimensional visual prior information to obtain the real-time disaster sensing situation map comprises:

inputting the reconstructed image matrix of the visual prior image and bilinear interpolation results of the image matrix into a guide up-sampling network for processing to obtain a predicted value;

Adjusting model parameters of the guided upsampling network based on a loss function between the predicted value and a true value; the loss function is a weighted sum of an L1 loss function and an antagonistic loss function; the antagonism loss function is related to a discrimination value of the discriminator on the predicted value;

the guiding up-sampling network and the discriminator are trained in a combined mode, and a diversified reduction result is obtained;

determining the result with highest robustness from the diversified reduction results;

and supplementing the texture detail information to the reconstructed spliced image based on the result with the highest robustness to obtain the real-time disaster situation perception situation map.

7. The method for building a live-action map in an emergency scene according to any one of claims 1 to 6, wherein obtaining a communication positioning demand hierarchical model includes:

acquiring historical grading data in the emergency scene; the historical grading data comprises grading indexes and expert comment labels aiming at each grading index;

dividing the history grading data into a training set and a testing set;

inputting the training set into the communication positioning requirement grading model for training;

inputting the test set into the trained communication positioning requirement grading model for testing;

And under the condition that the test result of the communication positioning requirement grading model fails, fine tuning model parameters of the communication positioning requirement grading model until the test result of the communication positioning requirement grading model passes.

8. The method for building a real-scene map in an emergency scene according to any one of claims 1 to 6, wherein the building a real-scene map based on the communication positioning demand hierarchical calibration and disaster point calibration based on the real-time disaster situation awareness situation map and the communication positioning demand hierarchical model comprises:

determining an input index of the communication positioning demand grading model; the input indexes comprise service objects, emergency scenes, treatment stages and refinement types of event objective factors, and various grades of communication positioning requirements;

inputting the input index into the communication positioning requirement grading model for processing, and outputting a communication positioning requirement grading calibration result;

determining a disaster point calibration result based on the real-time disaster situation perception situation map;

and adding the communication positioning demand grading calibration result and the disaster point calibration result to the real-time disaster sensing situation map to obtain the live-action map.

9. The utility model provides a live-action map construction device under emergent scene which characterized in that includes:

the first acquisition module is used for acquiring live-action images of disaster areas shot by the multiple unmanned aerial vehicles;

the first construction module is used for constructing a real-time disaster situation perception situation map based on each live-action image;

the second acquisition module is used for acquiring a communication positioning demand grading model which is obtained based on historical grading data training under an emergency scene;

the second construction module is used for constructing a live-action map based on communication positioning demand grading calibration and disaster point calibration based on the real-time disaster situation perception situation map and the communication positioning demand grading model.

10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the live-action map construction method in an emergency scene according to any one of claims 1 to 8 when the program is executed.

11. A non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor, implements the steps of the live-action map construction method in an emergency scene as claimed in any one of claims 1 to 8.