Disclosure of Invention
The invention aims to solve the problem of providing a streaming processing technical scheme suitable for in-orbit real-time processing of optical satellite remote sensing images aiming at a plurality of groups of low-power-consumption embedded in-orbit processing modules.
The technical scheme of the invention provides an on-orbit real-time streaming processing method for optical satellite remote sensing images, which comprises the following steps:
step 1, constructing a processing assembly line of a single processing module, respectively measuring the speed of data inflow, data processing and data outflow, and recording corresponding time consumption;
step 2, estimating the number N of processing modules needed to achieve real-time processing performance as follows,
wherein, TIN,TProcAnd TOUTRespectively measuring the time consumed by data inflow, data processing and data outflow in the step 1;
step 3, formulating and configuring a data dynamic subpackage strategy according to the result of the step 2, realizing configuration according to different processing algorithm requirements, and including preliminarily dividing into continuous standard scenes, supporting the interior of the standard scenes to be further divided into continuous data segments according to requirements, and further dividing into parallel grids, super pixels or pixels in the data segments according to requirements;
step 4, determining a load balancing strategy according to the hardware capacity;
step 5, deploying a customized streaming processing framework to realize on-orbit real-time processing of the optical satellite remote sensing image, wherein the on-orbit real-time processing comprises deploying N processing modules based on the processing module number N obtained in the step 2; and (4) performing real-time subpackaging and distribution on continuously inflowing input according to the data subpackaging strategy formulated in the step (3) and the load balancing strategy determined in the step (4) so as to realize parallel processing among a plurality of processing modules and realize on-orbit real-time processing of the optical satellite remote sensing images.
Moreover, the configuration in step 3 is implemented according to different processing algorithm requirements, including but not limited to the strategies of dividing according to a standard scene, dividing according to a grid, dividing according to a super-pixel, and dividing according to a pixel.
And, the load balancing policy in step 4 includes, but is not limited to, a policy of sequentially and evenly distributing data packets and a policy of weighting and distributing data packets according to different processing capabilities of modules according to different hardware capabilities and processing algorithm requirements.
And the strategy of weighted distribution according to the processing capacities of different modules is to set weights according to the processing capacities of the processing modules, and the modules with strong processing capacities and large weights are distributed with more to-be-processed data packets.
Moreover, the processing capability weights of the single processing module1 to the single processing module N are respectively W1,W2,…,WNThen, there are:
generating C data packets in unit time, the number of data packets processed by each processing module is Wi·。
The invention also provides an on-orbit real-time streaming processing system for the optical satellite remote sensing image, which comprises the following modules:
the first module is used for constructing a processing assembly line of the single processing module, respectively measuring the speed of data inflow, data processing and data outflow and recording corresponding consumed time;
a second module for estimating a number of processing modules N required to achieve real-time processing performance as follows,
wherein, TIN,TProcAnd TOUTRespectively measuring the time consumed by data inflow, data processing and data outflow in the step 1;
the third module is used for formulating and configuring a data dynamic subpackage strategy according to the result obtained by the second module, realizing configuration according to different processing algorithm requirements, and comprises the steps of preliminarily dividing the data into continuous standard scenes, further dividing the interior of the standard scenes into continuous data sections according to requirements, and further dividing the interior of the data sections into parallel grids, super pixels or pixels according to requirements;
a fourth module for determining a load balancing policy according to the hardware capability;
the fifth module is used for deploying a customized streaming processing architecture to realize on-orbit real-time processing of the optical satellite remote sensing image, and comprises a processing module number N obtained based on the second module and N processing modules; and performing real-time subpackaging and distribution on continuously inflowing input according to a data subpackaging strategy formulated by the third module and a load balancing strategy determined by the fourth module so as to realize parallel processing among a plurality of processing modules and realize on-orbit real-time processing of the optical satellite remote sensing images.
Moreover, the third module implements configurations according to different processing algorithm requirements, including but not limited to strategies of dividing according to a standard scene, dividing according to a grid, dividing according to super-pixels, and dividing according to pixels.
And the load balancing strategy in the fourth module includes, but is not limited to, strategies of sequentially and evenly distributing data packets and weighting and distributing according to different module processing capacities according to different hardware capacities and processing algorithm requirements.
And the strategy of weighted distribution according to the processing capacities of different modules is to set weights according to the processing capacities of the processing modules, and the modules with strong processing capacities and large weights are distributed with more to-be-processed data packets.
Moreover, the processing capability weights of the single processing module1 to the single processing module N are respectively W1,W2,…,WNThen, there are:
generating C data packets per unit timeThen the number of data packets processed by each processing module is Wi·。
The streaming processing framework suitable for the on-orbit processing of the optical satellite remote sensing image can fully utilize the processing capacity of the on-orbit processing module and the characteristics of the processing algorithm, optimize the on-orbit data processing flow, realize the on-orbit real-time processing of the optical satellite remote sensing image of 'data inflow-data processing-data outflow', and improve the acquisition efficiency and the real-time performance of remote sensing information. The processing algorithm applicable to the invention comprises and is not limited to optical satellite sensor correction, target detection, cloud detection and the like, and has important market value.
Detailed Description
The technical scheme of the invention is explained in detail in the following by combining the drawings and the embodiment.
The invention provides an in-orbit real-time streaming processing architecture of an optical satellite remote sensing image, which comprises the steps of determining the time consumed by data inflow, data processing and data outflow in a single processing module aiming at the characteristics of an in-orbit processing algorithm and the characteristics of processing hardware of a specific optical satellite remote sensing image; further calculating the number of processing modules required for achieving the real-time processing performance; and combining the characteristics of an on-orbit processing algorithm to formulate and configure a data dynamic subpackage strategy so as to generate a customized streaming architecture combining software and hardware; and deploying a customized streaming framework to realize on-orbit real-time processing of the optical satellite remote sensing image.
As shown in fig. 1, an in-orbit real-time streaming processing method for optical satellite remote sensing images provided in an embodiment includes the following steps:
step 1, constructing a processing assembly line of a single processing module, respectively measuring the speed of data inflow, data processing and data outflow, and recording corresponding time consumption.
Here, the processing module refers to a basic data processing unit including functional modules for processing, storing, communicating, and the like, and includes but is not limited to a processing board and an SoC system using an FPGA and a DSP as core devices. Based on the processing module, the incoming data can be processed through programming, and the processing result is transmitted out. In the processing process, for a specific data block, three main steps of data inflow, data processing and data outflow must be executed in a strict sequence; for a plurality of data blocks, the three steps can be processed by constructing a pipeline stack, that is, for data A, B, C which flows in successively, data B can be processed while data C flows in, and the processing result of data a is output. Due to the inconsistency of the time consumption of the three steps of data inflow, data processing and data outflow, the execution efficiency of the whole process depends on the step (usually data processing) which takes the longest time.
The invention realizes on-track parallel real-time processing by reasonably sub-packaging the input data and uniformly distributing the input data to the multi-processing modules. Aiming at specific processing algorithms such as optical satellite sensor correction, target detection, cloud detection and the like, the ratio of the processing rate of the target algorithm to the data input and output rate needs to be measured first, and then the minimum number of processing modules needed to be used is obtained.
Combining the hardware characteristics of the processing module, the target algorithm needs to be efficiently mapped to the processing module, and the average values of the consumed time of data inflow, data processing and data outflow are obtained by executing a plurality of continuous tests on the single-processing module pipeline and are respectively marked as TIN,TProcAnd TOUT。
And 2, calculating the number of the processing modules required for achieving the real-time processing performance, namely the total number of the single processing modules. The ratio of the data processing speed to the data inflow/outflow speed (the slowest of the two) can be calculated and rounded up. The invention further proposes a preferred scheme, and the number of the processing modules is N, and the value is obtained by the following formula:
wherein T isIN,TProcAnd TOUTAre respectively asAnd executing a plurality of groups of tests on the single processing module to obtain the average value of the consumed time of data inflow, data processing and data outflow.
The step is independent mathematical calculation based on the result of the step 1, and is independent of processing environments such as specific software and hardware.
And 3, formulating and configuring a data dynamic subpackage strategy according to the result of the step 2, wherein the strategy comprises the strategies of dividing according to a standard scene, dividing according to a grid, dividing according to a super pixel, dividing according to a pixel and the like.
In the field of optical satellite remote sensing image data processing, a square or nearly square image product is output according to the imaging width of a satellite, and the image product is called as a standard scene. As shown in fig. 2, the left side is the image data to be processed, which are sequentially transmitted (flowed) from top to bottom, and are initially divided into continuous standard scenes; inside the standard scene, the standard scene can be further divided into continuous data segments (segment 0, segment 1, segment 2, segment 3, etc.); further, the data segments may be divided into parallel grids, superpixels, or pixels. The right side is a parallel multi-processing module, and the processing module is internally composed of a control core (such as an embedded CPU, a DSP and the like) suitable for processing scalar operations such as control, branching, circulation, iteration and the like and a parallel core (such as an embedded GPU, an FPGA and the like) suitable for processing single instruction multiple data vector operations. In fig. 2, the left divided data segment 0, 1, 2, 3 is distributed to modules 0, 1, 2, 3(Module0, Module1, Module2, Module3) for processing, and the data segment is further divided into grids and processed with the control core and the parallel core.
The method adopts different sub-packaging strategies for different types of optical remote sensing image processing algorithms, sub-packages the inflow data according to the whole scene granularity for the algorithm needing the whole scene processing, such as the correction of an optical satellite sensor, and distributes the data to different processing modules one by one for processing; for algorithms such as target identification and cloud detection, a standard scene needs to be further divided into a plurality of approximately equal data segments, the data segments are sequentially distributed to different data processing modules section by section for processing, the data segments are further divided into grids, superpixels or pixels in the processing modules, and a control core and a parallel core are called to perform collaborative parallel computing according to the main operation type of a specific processing algorithm (different operation cores are suitable for processing different types of operations).
Based on the processing module number N required for realizing real-time calculation obtained in the step 2, the specific data subpackaging strategy is formulated by combining the characteristics of different types of processing algorithms. The sub-packaging strategy is closely related to the characteristics of the processing algorithm, and is not related to the configuration and the computing capacity of a single processing module. For example, for an algorithm needing to consider the relation between data segments, the data packet is distributed to N processing modules for processing by taking the whole scene as a unit; for the algorithm capable of processing independently among the data segments, the data packets are distributed to N processing modules by taking the data segments as units, and the data segments are further divided into grids which are distributed to different cores in a single module for processing.
And 4, determining a load balancing strategy according to the hardware capacity.
The load balancing strategy comprises the strategies of sequentially and evenly distributing data packets and weighting and distributing according to different module processing capacities according to different hardware capacities and processing algorithm requirements without limitation.
The embodiment sets the weight according to the processing capacity of the processing module, and the module group with strong processing capacity and large weight is allocated with more data packets to be processed, so that each processing module is fully utilized, the overall execution efficiency is improved, and node overload or node starvation is avoided. The processing capability weights of the single processing module1 to the single processing module N are respectively W1,W2,…,WNThen, there are:
generating C data packets in unit time, the number of data packets processed by each processing module is Wi·。
And 5, deploying a customized streaming processing framework to realize on-orbit real-time processing of the optical satellite remote sensing image. As shown in fig. 3, based on the number N of processing modules obtained in step 2, N processing modules (1-N) are deployed; the data distribution module performs real-time sub-packaging and distribution on continuously flowing input according to the data sub-packaging strategy formulated in the step 3 and the load balancing strategy determined in the step 4 so as to realize parallel processing among a plurality of processing modules; inside the processing module, scalar operation and vector operation are cooperatively processed by the control core and the parallel core according to the characteristics of the processing algorithm, so that the on-orbit real-time processing of the optical satellite remote sensing image is realized.
Taking a time line for realizing real-time cloud detection processing production of the optical satellite remote sensing image by applying the method, from the original point, the camera spends 0.403 seconds to generate data of 'data 0 segment 0', and the data segment is distributed to the module0 for processing and consumes 1.439 seconds; while the camera continues to spend 0.403 seconds generating "data 0 segment 1" data, which is assigned to module1 for processing, taking 1.501 seconds; and so on. Therefore, through a reasonable data sub-packaging and balancing strategy, the data can be immediately processed after being generated, and the whole system does not have the phenomena of waiting and data accumulation. And 1.417 seconds after the data generation is finished, the data processing is finished, and the real-time cloud detection processing of the optical satellite remote sensing image is realized.
In specific implementation, the method provided by the invention can realize automatic operation flow based on software technology, and can also realize a corresponding system in a modularized mode.
The embodiment of the invention also provides an on-orbit real-time streaming processing system for the optical satellite remote sensing image, which comprises the following modules:
the first module is used for constructing a processing assembly line of the single processing module, respectively measuring the speed of data inflow, data processing and data outflow and recording corresponding consumed time;
a second module for estimating a number of processing modules N required to achieve real-time processing performance as follows,
wherein, TIN,TProcAnd TOUTRespectively measuring the time consumed by data inflow, data processing and data outflow in the step 1;
the third module is used for formulating and configuring a data dynamic subpackage strategy according to the result obtained by the second module, realizing configuration according to different processing algorithm requirements, and comprises the steps of preliminarily dividing the data into continuous standard scenes, further dividing the interior of the standard scenes into continuous data sections according to requirements, and further dividing the interior of the data sections into parallel grids, super pixels or pixels according to requirements;
a fourth module for determining a load balancing policy according to the hardware capability;
the fifth module is used for deploying a customized streaming processing architecture to realize on-orbit real-time processing of the optical satellite remote sensing image, and comprises a processing module number N obtained based on the second module and N processing modules; and performing real-time subpackaging and distribution on continuously inflowing input according to a data subpackaging strategy formulated by the third module and a load balancing strategy determined by the fourth module so as to realize parallel processing among a plurality of processing modules and realize on-orbit real-time processing of the optical satellite remote sensing images.
The specific implementation of each module can refer to the corresponding step, and the detailed description of the invention is omitted.
The specific embodiments described herein are merely illustrative of the spirit of the invention. Various modifications or additions may be made to the described embodiments or alternatives may be employed by those skilled in the art without departing from the spirit or ambit of the invention as defined in the appended claims.