CN117649551A - Airport image data processing method, system, device, medium and equipment - Google Patents
Airport image data processing method, system, device, medium and equipment Download PDFInfo
- Publication number
- CN117649551A CN117649551A CN202311611845.XA CN202311611845A CN117649551A CN 117649551 A CN117649551 A CN 117649551A CN 202311611845 A CN202311611845 A CN 202311611845A CN 117649551 A CN117649551 A CN 117649551A
- Authority
- CN
- China
- Prior art keywords
- image data
- thermal imaging
- visible light
- feature
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 16
- 238000001931 thermography Methods 0.000 claims abstract description 165
- 238000000605 extraction Methods 0.000 claims abstract description 72
- 230000004927 fusion Effects 0.000 claims abstract description 69
- 238000000034 method Methods 0.000 claims abstract description 38
- 238000013507 mapping Methods 0.000 claims description 65
- 238000001228 spectrum Methods 0.000 claims description 22
- 238000012545 processing Methods 0.000 claims description 21
- 238000010586 diagram Methods 0.000 claims description 17
- 238000004891 communication Methods 0.000 claims description 9
- 230000007613 environmental effect Effects 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 10
- 230000007123 defense Effects 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 6
- 230000001960 triggered effect Effects 0.000 description 6
- 241001465754 Metazoa Species 0.000 description 5
- 238000012549 training Methods 0.000 description 5
- 230000009466 transformation Effects 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 238000003384 imaging method Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 3
- 239000013598 vector Substances 0.000 description 3
- RZVHIXYEVGDQDX-UHFFFAOYSA-N 9,10-anthraquinone Chemical compound C1=CC=C2C(=O)C3=CC=CC=C3C(=O)C2=C1 RZVHIXYEVGDQDX-UHFFFAOYSA-N 0.000 description 2
- 230000009977 dual effect Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 239000000428 dust Substances 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000003897 fog Substances 0.000 description 1
- 230000009545 invasion Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000149 penetrating effect Effects 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 239000000779 smoke Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/09—Supervised learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/809—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of classification results, e.g. where the classifiers operate on the same input data
- G06V10/811—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of classification results, e.g. where the classifiers operate on the same input data the classifiers operating on different input data, e.g. multi-modal recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- General Engineering & Computer Science (AREA)
- Molecular Biology (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Processing (AREA)
Abstract
The application discloses an airport image data processing method, system, device, medium and equipment, wherein the method comprises the following steps: acquiring image data to be identified, which is acquired in a target area by data acquisition equipment, wherein the image data to be identified comprises visible light image data, thermal imaging image data and point cloud data; inputting the visible light image data, the thermal imaging image data and the point cloud data into a pre-trained target recognition model, and carrying out feature extraction, feature fusion and feature recognition on the visible light image data, the thermal imaging image data and the point cloud data based on the pre-trained target recognition model to obtain the category of the target object and the position information of the target object. According to the method and the device, the feature extraction, the feature fusion and the feature recognition are carried out on the visible light image data, the thermal imaging image data and the point cloud data through the pre-trained target recognition model, so that the accuracy of image data recognition is improved, and the technical problem of high manual on-site rechecking cost is solved.
Description
Technical Field
The present disclosure relates to the field of image data processing, and in particular, to a method, a system, an apparatus, a medium, and a device for processing airport image data.
Background
The airport periphery intrusion alarm system is the first defense line of airport safety precautions and plays a role in guaranteeing the safety of a flight control area. At present, an airport periphery intrusion alarm system and a video monitoring system are combined, and when the airport periphery intrusion alarm system alarms, workers can check the alarm reasons through monitoring cameras corresponding to the defense areas.
However, in severe weather, because the image recognition rate acquired by the video monitoring system is low, workers are required to review in the field, and the workload and the operation difficulty of the workers are increased.
The foregoing is merely provided to facilitate an understanding of the principles of the present application and is not admitted to be prior art.
Disclosure of Invention
The main purpose of the application is to provide an airport image data processing method, system, device, medium and equipment, which aim at carrying out feature extraction, feature fusion and feature recognition on visible light image data, thermal imaging image data and point cloud data through a pre-trained target recognition model, so that the accuracy of image data recognition is improved, and the problem of high manual on-site rechecking cost is solved.
To achieve the above object, the present application provides an airport image data processing method, the method including: acquiring image data to be identified, which is acquired in a target area by data acquisition equipment, wherein the image data to be identified comprises visible light image data, thermal imaging image data and point cloud data; inputting the visible light image data, the thermal imaging image data and the point cloud data into a pre-trained target recognition model, and carrying out feature extraction, feature fusion and feature recognition on the visible light image data, the thermal imaging image data and the point cloud data based on the pre-trained target recognition model to obtain the category of the target object and the position information of the target object.
Optionally, the inputting the visible light image data, the thermal imaging image data and the point cloud data into a pre-trained target recognition model, and performing feature extraction, feature fusion and feature recognition on the visible light image data, the thermal imaging image data and the point cloud data based on the pre-trained target recognition model to obtain the category of the target object and the position information of the target object, including: mapping the point cloud data into the visible light image data to obtain mapped visible light image data; inputting the visible light image data and the mapping visible light image data into the pre-trained target recognition model, and carrying out feature extraction, feature fusion and feature recognition on the visible light image data and the mapping visible light image data based on the pre-trained target recognition model to obtain the category of a first target object and the position information of the first target object; and/or mapping the point cloud data into the thermal imaging image data to obtain mapped thermal imaging image data; inputting the thermal imaging image data and the mapping thermal imaging image data into the pre-trained target recognition model, and carrying out feature extraction, feature fusion and feature recognition on the thermal imaging image data and the mapping thermal imaging image data based on the pre-trained target recognition model to obtain the category of the second target object and the position information of the second target object.
Optionally, the performing feature extraction, feature fusion and feature recognition on the visible light image data and the mapped visible light image data based on the pre-trained target recognition model to obtain the category of the first target object and the position information of the first target object includes: based on the pre-trained target recognition model, respectively extracting the visible light image data and the graphic features of the mapping visible light image data to obtain a first visible light feature map and a first mapping visible light feature map; fusing the first visible light characteristic diagram and the first mapping visible light characteristic diagram to obtain a third visible light characteristic diagram; and carrying out feature recognition on the third visible light feature map, and outputting the category of the first object and the position information of the first object marked in the visible light image data.
Optionally, the performing feature extraction, feature fusion and feature recognition on the thermal imaging image data and the mapped thermal imaging image data based on the pre-trained target recognition model to obtain the category of the second target object and the position information of the second target object includes: based on the pre-trained target recognition model, respectively extracting graphic features of the thermal imaging image data and the mapping thermal imaging image data to obtain a first thermal imaging feature map and a first mapping thermal imaging feature map; fusing the first thermal imaging feature map and the first mapping thermal imaging feature map to obtain a third thermal imaging feature map; and carrying out feature recognition on the third thermal imaging feature map, and outputting the category of the second target object and the position information of the second target object marked in the thermal imaging image.
Optionally, after performing feature extraction, feature fusion and feature recognition on the visible light image data, the thermal imaging image data and the point cloud data based on the pre-trained target recognition model to obtain the category of the target object and the position of the target object in the visible light image, the method further includes: acquiring the air quality grade of the day provided by a third party platform; performing weight assignment on the identified first target object and the second target object according to the current day air quality level and a preset weight assignment model; and based on the weight assignment result, taking the first target object or the second target object with the highest weight assignment as the identification result.
Optionally, before the capturing the image data to be identified acquired in the target area, the method further includes: acquiring alarm position information transmitted by an airport periphery intrusion alarm system, and determining the target area according to the alarm position information; controlling the data acquisition equipment to move to the target area, and acquiring environmental information in the target area to obtain the image data to be identified; or, acquiring the target area input by the user through the display device or the terminal input device; and controlling the data acquisition equipment to move to the target area, and acquiring the environmental information in the target area to obtain the image data to be identified.
In addition, to achieve the above object, the present application further provides an airport image data processing system, including: the data acquisition equipment comprises a robot, a double-spectrum thermal imaging holder and a radar, wherein the double-spectrum thermal imaging holder and the radar are mounted on the robot; the robot is used for moving the double-spectrum thermal imaging holder and the radar to a target area, and the double-spectrum thermal imaging holder is used for collecting visible light image data and thermal imaging image data in the target area; the radar is used for collecting point cloud data in the target area; the server is in communication connection with the data acquisition equipment; the server is used for controlling the robot to move to a target area; the server is used for acquiring image data to be identified, which is acquired by the data acquisition equipment in the target area, wherein the image data to be identified comprises visible light image data, thermal imaging image data and point cloud data; inputting the visible light image data, the thermal imaging image data and the point cloud data into a pre-trained target recognition model, and carrying out feature extraction, feature fusion and feature recognition on the visible light image data, the thermal imaging image data and the point cloud data based on the pre-trained target recognition model to obtain the category of the target object and the position information of the target object; the display device is in communication connection with the server; the display device is used for displaying the category of the target object and the position information of the target object, which are obtained by the server; and the display device is used for displaying an airport boundary map, displaying control parameters of the data acquisition device and displaying a moving picture of the data acquisition device.
In addition, to achieve the above object, the present application further provides an airport image data processing apparatus, the apparatus including: the acquisition module is used for acquiring image data to be identified, which is acquired by the data acquisition equipment in a target area, wherein the image data to be identified comprises visible light image data, thermal imaging image data and point cloud data; the identification module is used for inputting the visible light image data, the thermal imaging image data and the point cloud data into a pre-trained target identification model, and carrying out feature extraction, feature fusion and feature identification on the visible light image data, the thermal imaging image data and the point cloud data based on the pre-trained target identification model to obtain the category of the target object and the position information of the target object.
In addition, the present application also provides a computing device, including: at least one processor, memory, and input output unit; wherein the memory is configured to store a computer program and the processor is configured to invoke the computer program stored in the memory to perform the airport image data processing method of the first aspect.
Furthermore, the application provides a computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the airport image data processing method of the first aspect.
According to the airport image data processing method, system, device, medium and equipment, image data to be identified, which are acquired in a target area by the data acquisition equipment, are acquired, wherein the image data to be identified comprise visible light image data, thermal imaging image data and point cloud data; the visible light image data, the thermal imaging image data and the point cloud data are input into a pre-trained target recognition model, and based on the pre-trained target recognition model, feature extraction, feature fusion and feature recognition are carried out on the visible light image data, the thermal imaging image data and the point cloud data to obtain the category of the target object and the position information of the target object, so that the accuracy rate of image data recognition is improved, and the cost of manually rechecking the alarm reason on the spot is reduced.
Drawings
FIG. 1A is a schematic illustration of a robot in a data acquisition device;
FIG. 1B is a schematic diagram of a robot of a data acquisition device;
FIG. 2 is a schematic diagram of an airport image data processing system according to an embodiment of the present application;
FIG. 3 is a flowchart of an airport image data processing method according to an embodiment of the present application;
FIG. 4 is a schematic diagram of functional blocks of a pre-trained object recognition model;
fig. 5 is a flow chart of an airport image data processing method according to another embodiment of the present application;
fig. 6 is a schematic functional block diagram of an airport image data processing device according to an embodiment of the present application.
FIG. 7 is a schematic structural diagram of a medium according to an embodiment of the present disclosure;
fig. 8 is a schematic structural diagram of a computing device according to an embodiment of the present application.
The realization, functional characteristics and advantages of the present application will be further described with reference to the embodiments, referring to the attached drawings.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
Those skilled in the art will appreciate that embodiments of the present application may be implemented as a system, apparatus, device, method, or computer program product. Thus, the present application may be embodied in the form of: complete hardware, complete software (including firmware, resident software, micro-code, etc.), or a combination of hardware and software.
The main solutions of the embodiments of the present application are:
acquiring image data to be identified, which is acquired in a target area by data acquisition equipment, wherein the image data to be identified comprises visible light image data, thermal imaging image data and point cloud data; inputting the visible light image data, the thermal imaging image data and the point cloud data into a pre-trained target recognition model, and carrying out feature extraction, feature fusion and feature recognition on the visible light image data, the thermal imaging image data and the point cloud data based on the pre-trained target recognition model to obtain the category of the target object and the position information of the target object.
In the prior art, the perimeter of an aircraft active area is definitely provided with physical boundaries and supporting facilities, the boundaries of the first class and the second class of airports are provided with intrusion alarm systems and video monitoring systems, and the boundaries of the third class of airports are provided with intrusion alarm systems and video monitoring systems. Therefore, in order to ensure the safety of the flight control area, an airport periphery intrusion alarm system must be arranged outside the airport periphery.
Currently, in the periphery of a second class of airports, an airport periphery intrusion alarm system and a video monitoring system are generally about 100m as a defense area, each defense area is provided with a monitoring camera, and a plurality of vibration detectors are arranged in each defense area at certain intervals (for example, 6 m). When someone touches the periphery or other reasons cause invasion alarm of the periphery, the staff rechecks the alarm reasons through the monitoring cameras corresponding to the defense areas, and the operation is simple and convenient.
However, in the existing video monitoring system, the images shot under the conditions of poor light, shielding of pictures, long monitoring distance and the like are blurred, so that the staff cannot check the alarm reasons through the monitoring cameras corresponding to the defense areas, and therefore the staff is required to check the alarm reasons in the field. However, when the alarm reasons are checked in the field, especially at night or in severe weather such as heavy wind and fog, the operation amount is large and the operation difficulty is high, so that the method has a certain threat to the personal safety of staff.
The application provides a solution, which aims at carrying out feature extraction, feature fusion and feature recognition on visible light image data, thermal imaging image data and point cloud data through a pre-trained target recognition model so as to improve the accuracy of image data recognition and further reduce the cost of manually checking alarm reasons on the spot.
It should be noted that any number of elements in the figures are for illustration and not limitation, and that any naming is used for distinction only and not for limitation.
The principles and spirit of the present application are explained in detail below with reference to several representative embodiments thereof.
Referring to fig. 2, an airport image data processing system according to an embodiment of the present application includes a data acquisition device 110, a server 120, and a display device 130.
Specifically, the data acquisition device 110 may include a robot, and a dual-spectrum thermal imaging head 3 and a radar 4 mounted on the robot. The robot may be the mobile robot 1 shown in fig. 1A or the orbital robot 2 shown in fig. 1B. When the airport periphery intrusion alarm system is triggered to alarm, the robot can carry the double-spectrum thermal imaging holder 3 and the radar 4 to move to a target area, and the target area is an alarm position; the dual-spectrum thermal imaging holder 3 can be used for collecting visible light image data and thermal imaging image data in a target area; the radar 4 may be used to collect point cloud data within a target area.
The server 120 is in communication with the data acquisition device 110; when the airport periphery intrusion alarm system is touched to alarm, the server can control the robot to move to a target area; the server can acquire image data to be identified, which is acquired by the data acquisition equipment in a target area, wherein the image data to be identified can comprise visible light image data, thermal imaging image data and point cloud data; the server can further input the visible light image data, the thermal imaging image data and the point cloud data into a pre-trained target recognition model, and perform feature extraction, feature fusion and feature recognition on the visible light image data, the thermal imaging image data and the point cloud data based on the pre-trained target recognition model to obtain the category of the target object and the position information of the target object.
The display device 130 is communicatively connected to the server 120; when the airport periphery intrusion alarm system is triggered to alarm, the display equipment can be used for displaying an airport periphery map, so that a user can determine a target area, namely an alarm position, according to the airport periphery map; after the server obtains the category of the target object and the position information of the target object, the display device can be used for displaying the category of the target object and the position information of the target object obtained by the server; when the user remotely controls the robot, the double-spectrum thermal imaging holder and/or the radar, the display device can be used for displaying control parameters of the data acquisition device, so that the user can conveniently adjust the moving speed of the robot, and adjust the angle, the frequency and the like when the double-spectrum thermal imaging holder and/or the radar acquire data; and the display device is used for displaying a moving picture of the data acquisition device in the process that the server controls the robot to reach the target area.
To further explain the above system, an embodiment of the present application provides an airport image data processing method, where the airport image data processing method is applied to an airport image data processing system, and an execution subject of the airport image data processing method is the server 120 with the computing function in the above embodiment, and the server 120 may be a desktop computer, a notebook computer, a mobile phone, or the like.
Referring to fig. 3, an airport image data processing method according to an embodiment of the present application includes:
step S310, acquiring image data to be identified, which is acquired in a target area by data acquisition equipment.
In an example embodiment, the data acquisition device may be the data acquisition device 110 in the above embodiment, and the data acquisition device may include a robot, and a dual spectrum thermal imaging head and radar mounted on the robot.
The image data to be identified may include visible light image data, thermal imaging image data, and point cloud data, where the visible light image data, the thermal imaging image data may be acquired by a dual-spectrum thermal imaging cradle head, and the point cloud data may be acquired by a radar.
The target area is an alarm area when the airport periphery intrusion alarm system is triggered to alarm. The target area can be an area selected by a user on an airport periphery map according to a trigger alarm area provided by an airport periphery intrusion alarm system; the method can also be a triggering alarm area which is directly uploaded to a server by an airport periphery intrusion alarm system.
It should be noted that, in the embodiment of the present application, the radar may be a millimeter wave radar. Millimeter wave radar is a radar operating in the millimeter wave band (millimeter wave), and millimeter waves are generally electromagnetic waves with a wavelength of 1-10 mm in the frequency domain of 30-300 GHz. The millimeter wave radar has strong capability of penetrating fog, smoke and dust, and can distinguish and identify very small targets in severe weather, so that the image identification effect on the surrounding environment can be improved. And millimeter wave radar has the characteristics of light in weight and small, carries on the back on the robot for data acquisition equipment has stronger mobility, the environment in the airport enclosure that can be better adaptation, reduces the number of times that staff checked the warning reason on the spot, has reduced staff's work load and working strength to a certain extent.
Step S320, inputting the visible light image data, the thermal imaging image data and the point cloud data into a pre-trained target recognition model, and carrying out feature extraction, feature fusion and feature recognition on the visible light image data, the thermal imaging image data and the point cloud data based on the pre-trained target recognition model to obtain the category of the target object and the position information of the target object.
In an example embodiment, the pre-trained target recognition model may be a pre-trained multi-modal fusion recognition model.
Referring to fig. 4, the multimodal fusion recognition model may be a model constructed by selecting pytorch2.0 as a deep learning framework. The multi-mode fusion recognition model mainly comprises a camera image feature extraction module, a millimeter wave mapping image feature extraction module, a feature fusion extraction module and a multi-layer perceptron module.
The camera image feature extraction module performs preliminary feature extraction on an input camera image; the millimeter wave mapping image feature extraction module performs preliminary feature extraction on the input millimeter wave mapping image. Specifically, the camera image feature extraction module and the millimeter wave mapping image feature extraction module are both feature extraction modules based on a Transformer structure, and the feature extraction process is as follows: the picture is cut into N picture blocks of fixed size, and the data of each picture block is flattened by three dimensions (C, H, W) and position coded to generate an embedded image block (embedded patches) having a two-dimensional vector of (N, C x H x W). The embedded image block is input to a transform feature extraction module comprising a plurality of self-attention layers (self-attention layers) and a feed forward neural network layer (feed forward) for preliminary feature extraction, the extracted features being two-dimensional vectors of (N, C x H x W).
The feature fusion extraction module can perform feature fusion on the feature map of the camera image and the feature map of the millimeter wave mapping image in a feature splicing mode, and further performs feature extraction on the fused features. The method comprises the following specific steps: the feature map of the camera image and the feature map of the millimeter wave mapping image are added (add) to obtain a fusion feature with the structure of (N, C multiplied by H multiplied by W); the fused features are input into a feature extraction module of a transducer structure together with a detection token (DET tokens) with the structure of (100, C×H×W) for further feature extraction.
And the multi-layer perceptron (MLP) module calculates and outputs a recognition result corresponding to the camera image according to the detection token output after feature fusion and extraction.
It should be noted that, the camera image feature extraction module, the millimeter wave mapping image feature extraction module and the feature fusion extraction module may all adopt a transducer network structure. The self-attention mechanism and scalability of the Transformer network are advantageous over the CNN feature extraction module in terms of multi-modal data fusion.
The training method for the multi-mode fusion recognition model can comprise the following steps: firstly, a sample database is constructed, environmental data of airports are collected under different climatic environments of different airports through data collection equipment, and a sample database of people, animals and vehicles in an airport enclosure scene is constructed.
Next, three transducer modules are weighted for initialization based on the pre-training model of the existing Visual Transformer model and yolo target detection model. And then, performing fine tuning training on the constructed multi-mode fusion recognition model by using a sample database until the loss value is 1, and stopping fine tuning training. When the built multi-mode fusion recognition model is subjected to fine tuning training, the adopted loss function can be classified loss function cross entropy, frame regression loss function and/or L1 regression loss function.
And finally, after verifying that the recognition precision of the multi-mode fusion recognition model meets the requirement, deploying the multi-mode fusion recognition model on a server through a Docker engine, and applying the multi-mode fusion recognition model to airport image data processing.
In a specific embodiment, the step S320 may include:
step S3201, mapping the point cloud data into visible light image data to obtain mapped visible light image data; step S3202, inputting the visible light image data and the mapping visible light image data into a pre-trained target recognition model, and carrying out feature extraction, feature fusion and feature recognition on the visible light image data and the mapping visible light image data based on the pre-trained target recognition model to obtain the category of the first target object and the position information of the first target object.
In combination with the above embodiment, step S3201 may be understood that the dual-spectrum thermal imaging holder and the radar are both fixed on the robot, so that the relative positional relationship between the radar and the camera of the dual-spectrum thermal imaging holder is ensured. The server maps the point cloud data from a coordinate system of millimeter waves to a coordinate system of the camera according to the relative position relation between the radar and the camera, and projects the point cloud data from the coordinate system of the camera to the pixel coordinate system according to imaging parameters of the camera to form a mapped image. The conversion process is as follows:
first, a radar coordinate system and a camera coordinate system are converted. That is, the transformation of the radar coordinate system into the camera coordinate system conforms to the rigid transformation relationship, and the transformation can be achieved by a rigid transformation matrix (R, T), where R is a rotation matrix and T is a translation vector. If the radar measured point is (Xr, yr, zr) in the radar coordinate system, the coordinates converted into the camera coordinate system are (Xc, yc, zc), and the conversion relationship is:
because the radar and the camera are fixed in position, the parameters of the rigid transformation matrix (R, T) cannot be changed.
And secondly, converting the camera coordinate system and the two-dimensional pixel coordinate system. That is, the coordinates in the camera coordinate system are (Xc, yc, zc), the coordinates converted into the two-dimensional pixel coordinate system are (u, v), and the conversion relationship is:
Wherein f is the focal length of the camera, and the visible light camera and the thermal imaging camera should ensure not to change the focal length in the use process; dx and dy represent how many millimeters 1 pixel represents in the column and row directions, respectively, on the image; (u 0, v 0) is the camera imaging origin.
In combination with the foregoing embodiment, the step S3202 may be understood that the server performs time synchronization on the visible light image data and the mapped visible light image data according to the time stamp, inputs the time-synchronized visible light image data and the mapped visible light image data into a pre-trained multi-mode fusion recognition model, and performs feature extraction, feature fusion and feature recognition on the visible light image data and the mapped visible light image data based on the pre-trained target recognition model, so as to obtain the category of the first target object and the position information of the first target object.
Specifically, based on a pre-trained target recognition model, respectively extracting visible light image data and graphic features for mapping the visible light image data to obtain a first visible light feature map and a first mapped visible light feature map; fusing the first visible light characteristic diagram and the first mapping visible light characteristic diagram to obtain a third visible light characteristic diagram; and performing feature recognition on the third visible light feature map, and outputting the category of the first object and the position information of the first object marked in the visible light image data.
In other words, the time synchronization is carried out on the visible light image data and the mapping visible light image data according to the time stamp, the time-synchronized visible light image data and the mapping visible light image data are input into the multi-mode fusion recognition model, and the camera image feature extraction module of the multi-mode fusion recognition model carries out preliminary feature extraction on the input visible light image data to obtain a first visible light feature map; the millimeter wave mapping image feature extraction module performs preliminary feature extraction on the input mapping visible light image data to obtain a first mapping visible light feature map; then, a feature fusion extraction module of the multi-mode fusion recognition model performs feature fusion on the first visible light feature map and the first mapping visible light feature map in a feature splicing mode to obtain a third visible light feature map, and further performs feature extraction on the fused third visible light feature map; finally, a multi-mode fusion recognition model multi-layer perceptron (MLP) module calculates and outputs a recognition result corresponding to the visible light image of the camera according to the third visible light characteristic diagram, wherein the recognition result is the category of the first target object and the position information of the first target object. For example, the output recognition result is pedestrian, animal, and/or vehicle, and positional information of the pedestrian, animal, and/or vehicle in the visible light image.
In another embodiment, the step S120 may include:
step S3210, mapping the point cloud data into thermal imaging image data to obtain mapped thermal imaging image data; step S3211, inputting the thermal imaging image data and the mapping thermal imaging image data into a pre-trained target recognition model, and performing feature extraction, feature fusion and feature recognition on the thermal imaging image data and the mapping thermal imaging image data based on the pre-trained target recognition model to obtain the category of the second target object and the position information of the second target object.
In combination with the above embodiment, step S3210 may be understood that the dual-spectrum thermal imaging holder and the radar are both fixed on the robot, so as to ensure the relative positional relationship between the radar and the camera of the dual-spectrum thermal imaging holder. The server maps the point cloud data from a coordinate system of millimeter waves to a coordinate system of the camera according to the relative position relation between the radar and the camera, and projects the point cloud data from the coordinate system of the camera to the pixel coordinate system according to imaging parameters of the camera to form a mapped image.
The specific mapping process is the same as the mapping principle of the step S3201, please refer to the above-mentioned example embodiment, and details in this example embodiment are not repeated.
Step S3211 may be understood that the server performs time synchronization on the thermal imaging image data and the mapping thermal imaging image data according to the time stamp, inputs the thermal imaging image data and the mapping thermal imaging image data in a pre-trained multi-mode fusion recognition model, inputs the thermal imaging image data and the mapping thermal imaging image data in a pre-trained target recognition model, and performs feature extraction, feature fusion and feature recognition on the thermal imaging image data and the mapping thermal imaging image data based on the pre-trained target recognition model, so as to obtain the category of the second target object and the position information of the second target object.
Specifically, based on a pre-trained target recognition model, respectively extracting thermal imaging image data and graphic features of mapping thermal imaging image data to obtain a first thermal imaging feature map and a first mapping thermal imaging feature map; fusing the first thermal imaging feature map and the first mapping thermal imaging feature map to obtain a third thermal imaging feature map; and carrying out feature recognition on the third thermal imaging feature map, and outputting the category of the second target object and the position information of the second target object marked in the thermal imaging image.
In other words, the thermal imaging image data and the mapping thermal imaging image data are time-synchronized according to the time stamp, the time-synchronized thermal imaging image data and the mapping thermal imaging image data are input into the multi-mode fusion recognition model, and the camera image feature extraction module of the multi-mode fusion recognition model performs preliminary feature extraction on the input thermal imaging image data to obtain a first thermal imaging feature map; the millimeter wave mapping image feature extraction module performs preliminary feature extraction on the input mapping thermal imaging image data to obtain a first mapping thermal imaging feature map; then, a feature fusion extraction module of the multi-mode fusion recognition model performs feature fusion on the first thermal imaging feature map and the first mapping thermal imaging feature map in a feature splicing mode to obtain a third thermal imaging feature map, and further performs feature extraction on the fused third thermal imaging feature map; finally, a multi-mode fusion recognition model multi-layer perceptron (MLP) module calculates and outputs a recognition result corresponding to the visible light image of the camera according to the third thermal imaging feature map, wherein the recognition result is the category of the first target object and the position information of the first target object. For example, the output recognition result is pedestrian, animal, and/or vehicle, and positional information of the pedestrian, animal, and/or vehicle in the visible light image.
When the data to be identified is identified, under the condition of better weather light, the visible light image data can be used for identifying the target object, or the thermal imaging image data is used for identifying the target object, or the visible light image data and the thermal imaging image data are respectively used for identifying the target object; when the weather is severe, the quality of the acquired visible light image data is low, and at the moment, the thermal imaging image data can be used for identifying the target object.
When the visible light image data and the thermal imaging image data are respectively adopted for identifying the target objects, a first target object and a second target object are obtained, and if the first target object and the second target object are the same, the first target object is displayed on the visible light image data so as to be convenient for a user to view; if the first object and the second object are different, the second object identified by the thermal imaging image data is used as the reference, so that the accuracy of image data identification can be improved to a certain extent.
Implementing the steps S310 to S320, the server acquires image data to be identified acquired by the data acquisition device in the target area, where the image data to be identified may include visible light image data, thermal imaging image data and point cloud data; inputting visible light image data, thermal imaging image data and point cloud data into a pre-trained target recognition model, and carrying out feature extraction, feature fusion and feature recognition on the visible light image data, the thermal imaging image data and the point cloud data based on the pre-trained target recognition model to obtain the category of a target object and the position information of the target object; the method comprises the steps of carrying out feature extraction, feature fusion and feature recognition on visible light image data, thermal imaging image data and point cloud data through a pre-trained target recognition model so as to improve the accuracy of image data recognition, thereby reducing the cost of manually rechecking the alarm reasons on site.
In addition, the point cloud data can be mapped from a millimeter wave coordinate system to a camera coordinate system, then the point cloud data is projected from the camera coordinate system to a pixel coordinate system according to imaging parameters of a camera to respectively form mapped visible light image data and mapped thermal imaging image data, then the time synchronization is carried out on the visible light image data and the mapped visible light image data according to a time stamp, the time synchronized visible light image data and the mapped visible light image data are input into a pre-trained target recognition model, and the pre-trained target recognition model carries out feature extraction, feature fusion and feature recognition on the visible light image data and the mapped visible light image data to obtain the category of a target object and the position information of the target object; and/or performing time synchronization on the thermal imaging image data and the mapping thermal imaging image data according to the time stamp, respectively inputting the thermal imaging image data and the mapping thermal imaging image data which are subjected to time synchronization into a pre-trained target recognition model, and performing feature extraction, feature fusion and feature recognition on the thermal imaging image data and the mapping thermal imaging image data by the pre-trained target recognition model to obtain the category of the target object and the position information of the target object. On one hand, a mapping image is formed through the point cloud data, so that the image features of a pre-trained target recognition model can be conveniently extracted; on the other hand, through feature fusion, the image features can be enhanced, and the recognition rate of the target object is improved.
In another embodiment of the present application, referring to fig. 5, in order to reduce the number of times that the staff check the reason of the alarm in the field, the server controls the acquisition device to perform data acquisition on the area triggering the alarm, that is, before the step S310, the method may further include the following steps:
step S501, alarm position information transmitted by an airport periphery intrusion alarm system is obtained, and a target area is determined according to the alarm position information. Or step S502, the target area input by the user through the display device or the terminal input device is acquired.
Step S503, the data acquisition device is controlled to move to a target area, and environmental information in the target area is acquired to obtain image data to be identified.
In an example embodiment, in combination with the above embodiment, the data acquisition device includes a robot, and a dual spectrum thermal imaging head and radar on-board the robot.
It can be understood that in an embodiment, when the airport periphery intrusion alarm system is triggered to alarm, a worker checks the reason of the alarm through a display device and a terminal input device which are connected with a server, if the video collected by the video monitoring system of the airport is ambiguous, the worker selects alarm position information on an airport periphery map displayed by the display device, the server acquires the alarm position information, determines a target area according to the alarm position information, and then the server controls the robot to move to an alarm position; when the robot moves to an alarm position, the server controls the dual-spectrum thermal imaging holder and the radar to acquire environmental information in a target area, and image data to be identified are obtained.
In addition, in another embodiment, after step S501 is performed, step S503 may also be directly performed, specifically, when the airport periphery intrusion alarm system is triggered to alarm, the triggered alarm position information is sent to the server, the server obtains the alarm position information, and determines the target area according to the alarm position information, and then the server controls the robot to move to the alarm position; when the robot moves to an alarm position, the server controls the dual-spectrum thermal imaging holder and the radar to acquire environmental information in a target area, and image data to be identified are obtained.
In another embodiment of the present application, referring to fig. 5, in order to further improve the recognition rate of the data to be detected, after step S320, the method may further include the following steps:
step S510, obtaining the current day air quality level provided by the third party platform.
And step S520, carrying out weight assignment on the identified first object and the second object according to the current day air quality level and a preset weight assignment model.
Step S530, based on the weight assignment result, the first object or the second object with the highest weight assignment is used as the identification result.
In an example embodiment, the third party provided air quality levels may include, for example, primary excellent, secondary good, tertiary light pollution, and quaternary heavy pollution. The preset weight assignment model is based on an entropy weight algorithm, and the weights of the first object and the second object are corrected through the entropy weight according to the weather change, the air quality change and other variables.
And in combination with an actual application scene, if the air quality on the same day is four-level severe pollution, and under the weather of rain, snow and the like, the server carries out weight assignment on the identified first object and the second object by combining with a preset weight assignment model, the weight of the second object is larger than that of the first object, and the second object is taken as a final identification result at the moment, wherein the second object is an identification result obtained after the server carries out feature extraction, feature fusion and feature identification on the thermal imaging image data and the mapping thermal imaging image data by utilizing a pre-trained object identification model.
It should be noted that, because the thermal imaging image data is less affected by objective factors such as weather compared with the visible light image data, when the weight of the first object and the second object is assigned by using the preset weight assignment model, the weight of the second object is higher than the weight of the first object. However, in order to facilitate the user's viewing of the recognition result, visible light image data is displayed on the display device.
Having described the method of the exemplary embodiments of the present application, an airport image data processing apparatus of the exemplary embodiments of the present application, which may include an acquisition module 610 and an identification module 620, is described next with reference to fig. 6, wherein,
The acquiring module 610 may be configured to acquire image data to be identified acquired by the data acquisition device in the target area, where the image data to be identified includes visible light image data, thermal imaging image data, and point cloud data;
the recognition module 620 may be configured to input the visible light image data, the thermal imaging image data, and the point cloud data into a pre-trained target recognition model, and perform feature extraction, feature fusion, and feature recognition on the visible light image data, the thermal imaging image data, and the point cloud data based on the pre-trained target recognition model, to obtain the category of the target object and the position information of the target object.
Having described the method and apparatus of the exemplary embodiments of the present application, reference will be made to fig. 6 for describing a computer-readable storage medium of the exemplary embodiments of the present application, and reference will be made to fig. 6 for showing a computer-readable storage medium that is an optical disc 60 and has a computer program (i.e., a program product) stored thereon, which when executed by a processor, implements the steps described in the above-described method embodiments, for example, acquiring image data to be identified acquired by a data acquisition device in a target area, where the image data to be identified includes visible light image data, thermal imaging image data, and point cloud data; inputting visible light image data, thermal imaging image data and point cloud data into a pre-trained target recognition model, and carrying out feature extraction, feature fusion and feature recognition on the visible light image data, the thermal imaging image data and the point cloud data based on the pre-trained target recognition model to obtain the category of a target object and the position information of the target object; the specific implementation of each step is not repeated here.
It should be noted that examples of the computer readable storage medium may also include, but are not limited to, a phase change memory (PRAM), a Static Random Access Memory (SRAM), a Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a flash memory, or other optical or magnetic storage medium, which will not be described in detail herein.
Having described the methods, apparatus, and media of exemplary embodiments of the present application, next, a computing device for model processing of exemplary embodiments of the present application is described with reference to fig. 7.
Fig. 7 illustrates a block diagram of an exemplary computing device 70 suitable for use in implementing embodiments of the present application, the computing device 70 may be a computer system or server. The computing device 70 shown in fig. 7 is merely an example and should not be taken as limiting the functionality and scope of use of embodiments of the present application.
As shown in fig. 7, components of computing device 70 may include, but are not limited to: one or more processors or processing units 701, a system memory 702, and a bus 703 that connects the various system components (including the system memory 702 and the processing units 701).
Computing device 70 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by computing device 70 and includes both volatile and nonvolatile media, removable and non-removable media.
The system memory 702 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 7021 and/or cache memory 7022. Computing device 70 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, ROM7023 may be used to read from or write to non-removable, nonvolatile magnetic media (not shown in FIG. 7, commonly referred to as a "hard disk drive"). Although not shown in fig. 7, a magnetic disk drive for reading from and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media), may be provided. In such cases, each drive may be coupled to bus 703 through one or more data medium interfaces. The system memory 702 may include at least one program product having a set (e.g., at least one) of program modules configured to carry out the functions of the embodiments of the present application.
A program/utility 7025 having a set (at least one) of program modules 7024 may be stored, for example, in system memory 702, and such program modules 7024 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment. Program modules 7024 generally perform the functions and/or methods in the embodiments described herein.
Computing device 70 may also communicate with one or more external devices 704 (e.g., keyboard, pointing device, display, etc.). Such communication may occur through an input/output (I/O) interface 705. Moreover, the computing device 70 may also communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet, through a network adapter 706. As shown in fig. 7, the network adapter 706 communicates with other modules of the computing device 70 (e.g., processing unit 701, etc.) over bus 703. It should be appreciated that although not shown in fig. 7, other hardware and/or software modules may be used in connection with computing device 70.
The processing unit 701 executes various functional applications and data processing by running a program stored in the system memory 702, for example, acquires image data to be recognized acquired by the data acquisition device in a target area, the image data to be recognized including visible light image data, thermal imaging image data, and point cloud data; inputting the visible light image data, the thermal imaging image data and the point cloud data into a pre-trained target recognition model, and carrying out feature extraction, feature fusion and feature recognition on the visible light image data, the thermal imaging image data and the point cloud data based on the pre-trained target recognition model to obtain the category of the target object and the position information of the target object. The specific implementation of each step is not repeated here. It should be noted that although in the above detailed description several units/modules or sub-units/sub-modules of the airport image data processing device are mentioned, such a division is only exemplary and not mandatory. Indeed, the features and functionality of two or more units/modules described above may be embodied in one unit/module according to embodiments of the present application. Conversely, the features and functions of one unit/module described above may be further divided into ones that are embodied by a plurality of units/modules.
In the description of the present application, it should be noted that the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, and are not repeated herein.
In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. The above-described apparatus embodiments are merely illustrative, for example, the division of units is merely a logical function division, and there may be other manners of division in actual implementation, and for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some communication interface, device or unit indirect coupling or communication connection, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer readable storage medium executable by a processor. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Finally, it should be noted that: the foregoing examples are merely illustrative of specific embodiments of the present application, and are not intended to limit the scope of the present application, although the present application is described in detail with reference to the foregoing examples, it will be understood by those skilled in the art that: any person skilled in the art may modify or easily conceive of the technical solution described in the foregoing embodiments, or make equivalent substitutions for some of the technical features within the technical scope of the disclosure of the present application; such modifications, changes or substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Furthermore, although the operations of the methods of the present application are depicted in the drawings in a particular order, this is not required to or suggested that these operations must be performed in this particular order or that all of the illustrated operations must be performed in order to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform.
Claims (10)
1. A method of airport image data processing, the method comprising:
acquiring image data to be identified, which is acquired in a target area by data acquisition equipment, wherein the image data to be identified comprises visible light image data, thermal imaging image data and point cloud data;
inputting the visible light image data, the thermal imaging image data and the point cloud data into a pre-trained target recognition model, and carrying out feature extraction, feature fusion and feature recognition on the visible light image data, the thermal imaging image data and the point cloud data based on the pre-trained target recognition model to obtain the category of the target object and the position information of the target object.
2. The method of claim 1, wherein inputting the visible light image data, the thermal imaging image data, and the point cloud data into a pre-trained object recognition model, and performing feature extraction, feature fusion, and feature recognition on the visible light image data, the thermal imaging image data, and the point cloud data based on the pre-trained object recognition model to obtain the category of the object and the position information of the object, comprises:
Mapping the point cloud data into the visible light image data to obtain mapped visible light image data;
inputting the visible light image data and the mapping visible light image data into the pre-trained target recognition model, and carrying out feature extraction, feature fusion and feature recognition on the visible light image data and the mapping visible light image data based on the pre-trained target recognition model to obtain the category of a first target object and the position information of the first target object; and/or the number of the groups of groups,
mapping the point cloud data into the thermal imaging image data to obtain mapped thermal imaging image data;
inputting the thermal imaging image data and the mapping thermal imaging image data into the pre-trained target recognition model, and carrying out feature extraction, feature fusion and feature recognition on the thermal imaging image data and the mapping thermal imaging image data based on the pre-trained target recognition model to obtain the category of the second target object and the position information of the second target object.
3. The method of claim 2, wherein performing feature extraction, feature fusion and feature recognition on the visible light image data and the mapped visible light image data based on the pre-trained object recognition model to obtain the category of the first object and the position information of the first object comprises:
Based on the pre-trained target recognition model, respectively extracting the visible light image data and the graphic features of the mapping visible light image data to obtain a first visible light feature map and a first mapping visible light feature map;
fusing the first visible light characteristic diagram and the first mapping visible light characteristic diagram to obtain a third visible light characteristic diagram;
and carrying out feature recognition on the third visible light feature map, and outputting the category of the first object and the position information of the first object marked in the visible light image data.
4. The method of claim 2, wherein performing feature extraction, feature fusion, and feature recognition on the thermographic image data and the mapped thermographic image data based on the pre-trained object recognition model to obtain the category of the second object and the location information of the second object comprises:
based on the pre-trained target recognition model, respectively extracting graphic features of the thermal imaging image data and the mapping thermal imaging image data to obtain a first thermal imaging feature map and a first mapping thermal imaging feature map;
Fusing the first thermal imaging feature map and the first mapping thermal imaging feature map to obtain a third thermal imaging feature map;
and carrying out feature recognition on the third thermal imaging feature map, and outputting the category of the second target object and the position information of the second target object marked in the thermal imaging image.
5. The method of claim 2, wherein after performing feature extraction, feature fusion, and feature recognition on the visible light image data, the thermal imaging image data, and the point cloud data based on the pre-trained object recognition model, obtaining the category of the object and the position of the object in the visible light image, the method further comprises:
acquiring the air quality grade of the day provided by a third party platform;
performing weight assignment on the identified first target object and the second target object according to the current day air quality level and a preset weight assignment model;
and based on the weight assignment result, taking the first target object or the second target object with the highest weight assignment as the identification result.
6. The method of any of claims 1-4, wherein prior to the acquiring of the image data to be identified acquired within the target area, the method further comprises:
Acquiring alarm position information transmitted by an airport periphery intrusion alarm system, and determining the target area according to the alarm position information;
controlling the data acquisition equipment to move to the target area, and acquiring environmental information in the target area to obtain the image data to be identified; or,
acquiring the target area input by a user through a display device or a terminal input device;
and controlling the data acquisition equipment to move to the target area, and acquiring the environmental information in the target area to obtain the image data to be identified.
7. An airport image data processing system, the system comprising:
the data acquisition equipment comprises a robot, a double-spectrum thermal imaging holder and a radar, wherein the double-spectrum thermal imaging holder and the radar are mounted on the robot; the robot is used for moving the double-spectrum thermal imaging holder and the radar to a target area, and the double-spectrum thermal imaging holder is used for collecting visible light image data and thermal imaging image data in the target area; the radar is used for collecting point cloud data in the target area;
the server is in communication connection with the data acquisition equipment; the server is used for controlling the robot to move to a target area; the server is used for acquiring image data to be identified, which is acquired by the data acquisition equipment in the target area, wherein the image data to be identified comprises visible light image data, thermal imaging image data and point cloud data; inputting the visible light image data, the thermal imaging image data and the point cloud data into a pre-trained target recognition model, and carrying out feature extraction, feature fusion and feature recognition on the visible light image data, the thermal imaging image data and the point cloud data based on the pre-trained target recognition model to obtain the category of the target object and the position information of the target object; and
The display device is in communication connection with the server; the display device is used for displaying the category of the target object and the position information of the target object, which are obtained by the server; and the display device is used for displaying an airport boundary map, displaying control parameters of the data acquisition device and displaying a moving picture of the data acquisition device.
8. An airport image data processing apparatus, the apparatus comprising:
the acquisition module is used for acquiring image data to be identified, which is acquired by the data acquisition equipment in a target area, wherein the image data to be identified comprises visible light image data, thermal imaging image data and point cloud data;
the identification module is used for inputting the visible light image data, the thermal imaging image data and the point cloud data into a pre-trained target identification model, and carrying out feature extraction, feature fusion and feature identification on the visible light image data, the thermal imaging image data and the point cloud data based on the pre-trained target identification model to obtain the category of the target object and the position information of the target object.
9. A computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the airport image data processing method of any of claims 1-6.
10. A computing device, the computing device comprising:
at least one processor, memory, and input output unit;
wherein the memory is for storing a computer program and the processor is for invoking the computer program stored in the memory to perform the airport image data processing method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311611845.XA CN117649551A (en) | 2023-11-27 | 2023-11-27 | Airport image data processing method, system, device, medium and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311611845.XA CN117649551A (en) | 2023-11-27 | 2023-11-27 | Airport image data processing method, system, device, medium and equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117649551A true CN117649551A (en) | 2024-03-05 |
Family
ID=90045917
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311611845.XA Pending CN117649551A (en) | 2023-11-27 | 2023-11-27 | Airport image data processing method, system, device, medium and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117649551A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN119044444A (en) * | 2024-10-31 | 2024-11-29 | 佛山大学 | Tail coal ash content rapid detection method and system based on state space cross fusion |
-
2023
- 2023-11-27 CN CN202311611845.XA patent/CN117649551A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN119044444A (en) * | 2024-10-31 | 2024-11-29 | 佛山大学 | Tail coal ash content rapid detection method and system based on state space cross fusion |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3967972B1 (en) | Positioning method, device, and computer-readable storage medium | |
| CN109387186B (en) | Surveying and mapping information acquisition method and device, electronic equipment and storage medium | |
| US12033289B2 (en) | Method and system for visualizing overlays in virtual environments | |
| US12092468B2 (en) | Methods and systems for response vehicle deployment | |
| CN107316012B (en) | Fire detection and tracking method for small unmanned helicopter | |
| CN109523471B (en) | Method, system and device for converting ground coordinates and wide-angle camera picture coordinates | |
| CN111368615B (en) | Illegal building early warning method and device and electronic equipment | |
| Long et al. | Fusion of millimeter wave radar and RGB-depth sensors for assisted navigation of the visually impaired | |
| CN111679271B (en) | Target tracking method, target tracking device, monitoring system and storage medium | |
| KR102557769B1 (en) | System for detect accident based on cctv using digital twin virtual model | |
| US20230334849A1 (en) | Method and system for virtual area visualization | |
| CN113869231B (en) | Method and equipment for acquiring real-time image information of target object | |
| US20240257526A1 (en) | Monitoring method and apparatus, and unmanned vehicle and monitoring device | |
| KR20200067286A (en) | 3D scan and VR inspection system of exposed pipe using drone | |
| CN117649551A (en) | Airport image data processing method, system, device, medium and equipment | |
| CN112291478B (en) | Method, device and equipment for monitoring high-altitude falling object and storage medium | |
| CN116545122A (en) | A transmission line anti-breakage monitoring device and anti-breakage monitoring method | |
| CN113009505A (en) | Airborne laser radar data acquisition equipment, system and unmanned aerial vehicle aircraft | |
| US11703820B2 (en) | Monitoring management and control system based on panoramic big data | |
| Liu et al. | Vehicle video surveillance system based on image fusion and parallel computing | |
| CN112926415A (en) | Pedestrian avoiding system and pedestrian monitoring method | |
| Pozo et al. | 3D reconstruction technologies for using in dangerous environments with lack of light: a comparative analysis | |
| CN114723923B (en) | Transmission solution simulation display system and method | |
| CN115760964A (en) | Method and equipment for acquiring screen position information of target object | |
| JP2016115082A (en) | Image search system and image search method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |