CN119835214B

CN119835214B - Distributed data transmission method, device, equipment, medium and product

Info

Publication number: CN119835214B
Application number: CN202411870923.2A
Authority: CN
Inventors: 卢伟超
Original assignee: Shenzhen Kaihong Digital Industry Development Co Ltd
Current assignee: Shenzhen Kaihong Digital Industry Development Co Ltd
Priority date: 2024-12-18
Filing date: 2024-12-18
Publication date: 2025-10-21
Anticipated expiration: 2044-12-18
Also published as: CN119835214A

Abstract

The present application discloses a distributed data transmission method, apparatus, device, medium, and product, relating to the field of data transmission technology. The distributed data transmission method includes: determining multiple computing nodes in a distributed computing terminal; constructing a transmission matrix based on the transmission delay, bandwidth, and amount of data to be transmitted between the computing nodes, wherein the computing nodes include a source node and a target node; optimizing the path between the source node and the target node according to the transmission matrix to obtain a first path; dividing all computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure, wherein the computing nodes in each level process data in parallel; and transmitting preset task data to the target node through the multi-level aggregation structure. The present application improves the data transmission efficiency of the distributed computing terminal.

Description

Distributed data transmission method, device, equipment, medium and product

Technical Field

The present application relates to the field of data transmission technologies, and in particular, to a distributed data transmission method, apparatus, device, medium, and product.

Background

In modern computing environments, distributed computing systems are widely used to process large-scale data sets and perform complex computing tasks. Such systems typically consist of a number of heterogeneous devices, including but not limited to high performance computing units such as CPUs (Central Processing Unit, central processing units), GPUs (Graphics Processing Unit, graphics processors), FPGAs (Field-Programmable gate arrays) and the like, which are interconnected by a network to work cooperatively. A key advantage of a distributed computing system is that it can take advantage of the parallel processing capabilities of multiple devices, thereby significantly increasing computing efficiency and processing speed. However, in practical applications, the performance of distributed computing systems is often limited by bottlenecks in data transfer. In particular, in the field of deep learning hardware acceleration and high performance computing architectures, delay and bandwidth occupation of data transmission become important factors affecting overall performance. For example, in large-scale computing tasks (e.g., deep learning training, scientific simulation), frequent data exchanges between nodes can lead to communication bottlenecks, thereby significantly reducing the throughput and response speed of the system. Most conventional systems employ static routing or simple load balancing strategies, and cannot dynamically adjust the data transmission paths. The data volume is large and the data are frequently transmitted, so that the bandwidth occupation is high, the delay is uncontrollable, the problem is more remarkable particularly when the task complexity and the node number are increased, the data aggregation mode is single, and hierarchical optimization is lacking, so that the node communication pressure is concentrated at a single aggregation point, the transmission bottleneck is easy to form, the transmission efficiency between distributed computing terminals is low, and the system expansibility and the overall performance are limited. Especially, in the face of increasing data volume and complex computing demands, the conventional method is difficult to meet the requirement of high-performance computing, and a method capable of improving the data transmission efficiency of the distributed computing terminal is highly demanded.

Disclosure of Invention

The application mainly aims to provide a distributed data transmission method, a device, equipment, a medium and a product, which aim to improve the data transmission efficiency of a distributed computing terminal.

In order to achieve the above object, the present application provides a distributed data transmission method, applied to a distributed computing terminal, including:

determining a plurality of computing nodes in the distributed computing terminal;

constructing a transmission matrix according to transmission delay, bandwidth and data quantity to be transmitted among computing nodes, wherein the computing nodes comprise source nodes and target nodes;

path optimization is carried out on paths between a source node and a target node according to the transmission matrix, so that a first path is obtained;

Dividing all the computing nodes into a plurality of layers according to a first path to obtain a multi-level aggregation structure, wherein the computing nodes in each layer process data in parallel;

and transmitting preset task data to the target node through the multi-level aggregation structure.

In an embodiment, the step of obtaining the first path by performing path optimization on the path between the source node and the target node according to the transmission matrix includes:

and determining all feasible paths from the source node to the target node aiming at any source node in the transmission matrix, finding the shortest path meeting the lowest cost condition in all the feasible paths, and taking the shortest path as a first path, wherein the cost between any two adjacent nodes in the shortest path is the lowest.

In an embodiment, the step of obtaining the first path after performing path optimization on the path between the source node and the target node according to the transmission matrix includes:

When the network state is changed frequently, determining whether the cost of the first path is updated;

And if the cost of the first path is updated, re-executing the step of carrying out path optimization on the path between the source node and the target node according to the transmission matrix so as to find the latest shortest path, and updating the latest shortest path into the first path.

In one embodiment, dividing all computing nodes into a plurality of levels according to a first path, the step of obtaining a multi-level aggregate structure comprises:

taking the computing nodes with the distance smaller than a preset distance threshold value, the delay smaller than a preset delay threshold value and no dependency relationship with other computing nodes as first-level computing nodes;

For any level after the first level, selecting a computing node which is closest to the previous level and has a delay smaller than a preset delay index as a computing node of the current level according to a first path to obtain a multi-level aggregation structure, wherein the computing node of the last level is a target node, and any computing node before the computing node of the last level is a source node;

dividing all the computing nodes into a plurality of layers according to the first path, wherein the step of obtaining the multi-stage aggregation structure comprises the following steps:

If any first computing node has the delay greater than or equal to the preset delay index, the first computing node is replaced by a preset standby node.

In one embodiment, the step of transmitting the preset task data to the target node through the multi-level aggregation structure includes:

Inputting task data into a first-level computing node in a multi-level aggregation structure, and aggregating the task data through the first-level computing node to obtain an aggregation result;

And for any level of computing nodes after the first level of computing nodes, re-aggregating the aggregation results of the previous level to obtain the aggregation results of the current level until the aggregation results of the current level are aggregated to the target node in the multi-level aggregation structure, wherein the aggregation results of the current level are stored in the computing nodes of the current level nearest to the next level.

In one embodiment, the step of constructing the transmission matrix according to the transmission delay, bandwidth and amount of data to be transmitted between the computing nodes comprises:

Determining all computing node pairs, wherein the computing node pairs consist of two computing nodes;

For each computing node pair, detecting a network state to obtain a transmission delay and a bandwidth between the computing node pairs;

Determining the data quantity to be transmitted between the computing node pairs according to the task data;

calculating the ratio of the data quantity to be transmitted to the bandwidth, taking the product of the ratio and the transmission delay as the transmission cost of each calculation node pair, and taking the transmission cost as the element of the transmission matrix to obtain the transmission matrix.

In addition, in order to achieve the above object, the present application also provides a distributed data transmission device, which is disposed in a distributed computing terminal, and the distributed data transmission device includes:

a determining computing node module for determining a plurality of computing nodes in the distributed computing terminal;

A transmission matrix module is constructed, and a transmission matrix is constructed according to the transmission delay, the bandwidth and the data quantity to be transmitted between the computing nodes, wherein the computing nodes comprise source nodes and target nodes;

the path optimization module is used for carrying out path optimization on the paths between the source node and the target node according to the transmission matrix to obtain a first path;

The aggregation structure building module divides all the computing nodes into a plurality of layers according to the first path to obtain a multi-level aggregation structure, wherein the computing nodes in each layer process data in parallel;

and the aggregation transmission module is used for transmitting the preset task data to the target node through the multi-level aggregation structure.

In addition, in order to achieve the above object, the present application also proposes a distributed data transmission device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the distributed data transmission method as described above.

In addition, in order to achieve the above object, the present application also proposes a medium, which is a computer-readable storage medium, on which a computer program is stored, which when being executed by a processor implements the steps of the distributed data transmission method as described above.

Furthermore, to achieve the above object, the present application provides a product, which is a computer program product comprising a computer program which, when being executed by a processor, implements the steps of the distributed data transmission method as described above.

One or more technical schemes provided by the application have at least the following technical effects:

The method is applied to a distributed computing terminal, a plurality of computing nodes in the distributed computing terminal are determined, a transmission matrix is constructed according to transmission delay, bandwidth and data quantity to be transmitted among the computing nodes, the computing nodes comprise source nodes and target nodes, the matrix comprehensively reflects transmission cost information in a network and provides key data support for path optimization, the path between the source nodes and the target nodes is optimized according to the transmission matrix to obtain a first path, the path ensures data transmission quality, simultaneously minimizes transmission cost and improves data transmission efficiency, all the computing nodes are divided into a plurality of layers according to the first path to obtain a multi-level aggregation structure, the computing nodes in each layer parallelly process data, the parallelism of data processing is improved, the hierarchical structure of data transmission is optimized, the overall computing efficiency is further improved, the preset task data is transmitted to the target nodes through the multi-level aggregation structure, the process fully utilizes the optimized path and the hierarchical structure, high-speed low-delay transmission of the data is realized, and the defect that the transmission efficiency of the distributed computing terminal is low is effectively solved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required to be used in the description of the embodiments or the prior art will be briefly described below, and it will be obvious to those skilled in the art that other drawings can be obtained from these drawings without inventive effort.

Fig. 1 is a schematic flow chart of a distributed data transmission method according to a first embodiment of the present application;

FIG. 2 is a schematic diagram of a distributed data transmission method according to the present application;

FIG. 3 is a flow chart of a distributed data transmission method according to the present application;

fig. 4 is a schematic block diagram of a distributed data transmission device according to an embodiment of the present application;

fig. 5 is a schematic device structure diagram of a hardware operating environment related to a distributed data transmission method according to an embodiment of the present application.

The achievement of the objects, functional features and advantages of the present application will be further described with reference to the accompanying drawings, in conjunction with the embodiments.

Detailed Description

It should be understood that the specific embodiments described herein are merely illustrative of the technical solution of the present application and are not intended to limit the present application.

For a better understanding of the technical solution of the present application, the following detailed description will be given with reference to the drawings and the specific embodiments.

It should be noted that, the execution body of the embodiment may be a computing service device having functions of data processing, network communication and program running, such as a tablet computer, a personal computer, a mobile phone, or an electronic device, a terminal system, or the like capable of implementing the above functions. The present embodiment and the following embodiments will be described below with reference to a system.

Based on this, the embodiment provides a distributed data transmission method, which is applied to a distributed computing terminal.

In this embodiment, referring to fig. 1, fig. 1 is a flow chart of a distributed data transmission method according to the present application, and the distributed data transmission method includes steps S10 to S50:

Step S10, determining a plurality of computing nodes in the distributed computing terminal;

Step S20, a transmission matrix is constructed according to transmission delay, bandwidth and data quantity to be transmitted among computing nodes, wherein the computing nodes comprise source nodes and target nodes;

It should be noted that, the distributed computing terminal refers to a computing system formed by connecting a plurality of computing nodes through a network, and the computing nodes can cooperate together to complete tasks, so as to improve computing efficiency. The computing nodes are the basic units in the distributed computing terminals, which may be servers, personal computers or other computing devices, each node having data processing capabilities. The source node is a node that generates data and prepares to transmit during data transmission. The target node is a node that receives data during the data transmission process.

In a distributed computing system, it is first necessary to determine all the nodes involved in the computation, including at least one source node (the node that generated the data) and at least one target node (the node that received the data). The nodes are connected through a network to form a distributed computing terminal.

It should be noted that the transmission matrix is a mathematical model, which is used to describe key information such as transmission delay, bandwidth, and data amount to be transmitted between the source node and the target node.

The transmission matrix is constructed based on the transmission delay, bandwidth and amount of data to be transmitted between the source node and the target node. The transmission delay refers to the time required for data to be transmitted from the source node to the destination node, the bandwidth refers to the capability of the network to transmit, i.e. the amount of data that can be transmitted per second, and the amount of data to be transmitted is the amount of data that the source node needs to send to the destination node. This information is determined by monitoring the network state and computational tasks in real time and then constructing a matrix containing the transmission delay, bandwidth and amount of data to be transmitted. This matrix will serve as the basis for path optimization.

The transmission matrix is illustratively calculated as equation (one)

Where M _trans (i, j) is a transmission matrix, latency _i,j is a transmission delay from a computing node i to a computing node j, bandwidth _i,j is a Bandwidth from a computing node i to a computing node j, and data_size _i,j is an amount of Data to be transmitted from a computing node i to a computing node j.

Step S30, path optimization is carried out on the paths between the source node and the target node according to the transmission matrix, and a first path is obtained;

the path optimization is to adjust the data transmission path by an algorithm to reduce the transmission delay and improve the bandwidth utilization. The first path is the optimal data transmission path obtained after path optimization.

The purpose of path optimization is to find the best transmission path from the source node to the target node to minimize the total transmission overhead of the system. This typically involves the construction of an objective function that takes into account a number of factors, such as transmission delay, bandwidth utilization, and the amount of data to be transmitted. The shortest path between nodes is calculated using a dynamic programming algorithm (e.g., dijkstra or Floyd-Warshall) while taking into account that the data for each task must be transmitted in its entirety and the path must not exceed the available bandwidth limit. If the network state changes frequently, a dynamic optimization method based on an ant colony algorithm can be adopted to continuously adjust the transmission path. By path optimization we get an optimal path from the source node to the target node, i.e. the first path.

Step S40, dividing all the computing nodes into a plurality of layers according to a first path to obtain a multi-level aggregation structure, wherein the computing nodes in each layer process data in parallel;

it should be noted that, the multi-level aggregation structure is a data aggregation policy, and divides the computing nodes into a plurality of levels, and performs data aggregation step by step, so as to reduce the data transmission amount. Parallel processing is where multiple compute nodes process data simultaneously to increase the speed of data processing.

The design of the multi-stage aggregation structure is to further reduce the data transmission quantity and improve the system expansibility. All the computing nodes are divided into a plurality of layers, and data aggregation is completed step by step. Within each hierarchy, the nodes independently process the data in parallel to reduce the total amount of backhauls. The result of each stage of aggregation is used as the input of the next stage of aggregation until the top layer aggregation is completed to obtain a global result. The selection of the aggregation nodes is based on factors such as distance, delay and the like, and the nodes with relatively close distances and relatively low delays are preferentially selected as the first-level aggregation nodes. Meanwhile, the positions of the aggregation nodes are dynamically adjusted according to the real-time transmission matrix so as to cope with the change of the network state.

Step S50, transmitting preset task data to a target node through a multi-level aggregation structure.

The task data specifically refers to various information involved in the execution of the distributed computing task. These task data may be raw input data, intermediate calculation results, final output results, etc. In distributed computing, task data typically needs to be transferred and shared among multiple computing nodes for collaborative computing and processing. During transmission by the distributed computing terminals, task data may be in a variety of formats, including text data, image data, audio data, video data, etc., depending on the requirements of the task and the nature of the data.

After the optimal transmission path and the multi-level aggregation structure are determined, the preset task data are transmitted according to the paths and the structures. In the transmission process, data can pass through a plurality of aggregation nodes to be aggregated and compressed step by step. Each level of aggregation reduces the total amount of data, thereby avoiding excessive transmission data of the next layer. In the transmission process, data are sequentially processed and aggregated at the computing nodes in each hierarchy according to the hierarchical relationship of the multi-level aggregation structure, and finally reach the target node.

Illustratively, assuming that the active nodes A, B, C, D, and the target node E, the transmission matrix M _trans is expressed as equation (two), the amount of node data is A:100MB, B:200MB, C:150MB, D:50MB.

Path optimization is performed by using a Di Jie St algorithm to obtain a first path of A-B, B-D, C-D, D-E, all calculation nodes are divided into a plurality of levels according to the first path, and referring to FIG. 2, the first level calculation nodes are A, C, the second level calculation nodes are B, D, and the third level calculation nodes are target nodes E, so that data is aggregated from A to B, B to obtain data A+B, C to D, D to obtain data C+D, and finally data is aggregated from B and D to E by the second level calculation nodes, and the total transmission quantity is reduced by 40% by multi-level aggregation and path optimization.

Illustratively, for GPU nodes G1, G2, CPU nodes C1, C2, real-time delay monitor transmission matrix dynamics. GPU nodes preferentially execute computation intensive aggregation, CPU nodes process lightweight transmission, a transmission matrix optimization path is adjusted in real time, delay is reduced by 30% after the path is dynamically optimized, and the total transmission amount is reduced by 25%.

The method and the device are applied to a distributed computing terminal, a plurality of computing nodes in the distributed computing terminal are determined, a transmission matrix is constructed according to transmission delay, bandwidth and data quantity to be transmitted among the computing nodes, the computing nodes comprise source nodes and target nodes, the matrix comprehensively reflects transmission cost information in a network and provides key data support for path optimization, the path between the source nodes and the target nodes is optimized according to the transmission matrix to obtain a first path, the path ensures data transmission quality, simultaneously achieves minimization of transmission cost, improves data transmission efficiency, all the computing nodes are divided into a plurality of layers according to the first path to obtain a multi-level aggregation structure, the computing nodes in each layer are used for processing data in parallel, the parallelism of data processing is improved, the hierarchical structure of data transmission is optimized, the overall computing efficiency is further improved, the preset task data is transmitted to the target nodes through the multi-level aggregation structure, the process fully utilizes the optimized path and the hierarchical structure, high-speed low-delay transmission of the data is achieved, and the defect that the transmission efficiency of the distributed computing terminal is low is effectively overcome.

In one possible embodiment, step S50 may include steps T10 to T20:

Step T10, compressing data by applying a preset lightweight compression algorithm to obtain compressed data;

it should be noted that a lightweight compression algorithm is a data processing technique that aims to reduce the size of data while maintaining data integrity or acceptable loss. It is generally faster and more efficient than conventional compression algorithms, and is suitable for environments with stringent requirements on real-time or resource consumption. Compressed data is the conversion of raw data into a smaller volume data form by applying a compression algorithm. The compressed data maintains all or most of the information of the original data, but occupies less storage space or transmission bandwidth.

And compressing the original data by applying a preset lightweight compression algorithm. This process typically involves analyzing characteristics of the data, such as repeatability, redundancy, predictability, etc., and utilizing these characteristics to reduce the overall size of the data. The lightweight compression algorithm has the advantages of high efficiency and low resource consumption, and can remarkably reduce the volume of data without sacrificing too much data quality. By applying this algorithm we have obtained compressed data which, although smaller, still contains the core information of the original data.

In step T20, the compressed data is transmitted to the target node through the multi-stage aggregation structure.

The compressed data is transmitted to the target node through the multi-level aggregation structure. In this process, compressed data is divided into a plurality of data blocks or packets and transmitted in a hierarchical relationship of a multi-level aggregate structure. The compute nodes of each layer receive data from the previous layer, perform necessary processing (e.g., further compression, format conversion, etc.), and then pass the processed data to the next layer. This layer-by-layer transmission and processing ensures the reliability and integrity of the data while improving the transmission efficiency. The multistage polymeric structure plays a key role in this process. It not only provides an efficient data transmission channel, but also reduces redundancy and complexity of data by layer-by-layer aggregation. In the transmission process, the computing nodes of each layer can optimize and process the data, so that the data finally reaching the target node is ensured to be efficient and accurate. Eventually, the compressed data is successfully transmitted to the target node. At the target node, this data may be further processed, analyzed, or stored.

In the embodiment, due to the adoption of a lightweight compression algorithm and a multi-stage aggregation structure, the whole process not only improves the efficiency and reliability of data transmission, but also reduces the resource consumption and cost. This gives the method of the present embodiment significant advantages when processing large-scale data sets or real-time data streams.

In one possible embodiment, step S50 may include steps R10 to R40:

r10, determining all computing node pairs, wherein the computing node pairs consist of two computing nodes;

It should be noted that, in a distributed computing environment, a computing node pair is a pairing of two computing nodes, and there may be a need for data transmission between the two nodes.

All computing node pairs are determined. In a distributed computing environment, there are multiple computing nodes, each of which may be in data transmission with other nodes. Thus, it is necessary to pair these nodes two by two to form a computing node pair. This step is the basis for constructing the transmission matrix.

Step R20, for each computing node pair, detecting the network state to obtain the transmission delay and bandwidth between the computing node pairs;

It should be noted that the network status is various parameters and indexes describing the current operation status of the network, including but not limited to transmission delay, bandwidth, etc. Transmission delay is the time required for data to travel in a network, and is typically due to factors such as network distance, network congestion, device performance, etc. Bandwidth is the amount of data that a network can transmit in a unit time, and is an important indicator for measuring the transmission capacity of the network.

For each computing node pair, network states are detected to obtain transmission delays and bandwidths therebetween. Network state detection is a complex process involving measurement and analysis of multiple network parameters. In the present embodiment, two parameters of transmission delay and bandwidth are mainly focused. The transmission delay reflects the speed at which data is transmitted in the network, while the bandwidth determines the amount of data that the network can transmit per unit time. These two parameters together determine the efficiency and cost of data transmission.

R30, determining the data quantity to be transmitted between the computing node pairs according to the task data;

It should be noted that, the amount of data to be transmitted is the amount of data that needs to be transmitted from the source node to the target node in the distributed computing task.

And determining the data quantity to be transmitted between the computing node pairs according to the task data. Task data refers to data that needs to be processed in a distributed computing task. The amount of data that needs to be transmitted between each computing node pair needs to be considered in constructing the transmission matrix. This amount of data may vary from task to task.

And R40, calculating the ratio of the data quantity to be transmitted to the bandwidth, taking the product of the ratio and the transmission delay as the transmission cost of each calculation node pair, and taking the transmission cost as the element of the transmission matrix to obtain the transmission matrix.

It should be noted that the transmission cost is the total cost required for data transmission calculated by comprehensively considering the transmission delay and the bandwidth, and is generally measured by time or resource consumption. The transmission matrix is a two-dimensional matrix used for representing the transmission cost between each computing node pair in the distributed computing terminal.

And calculating the ratio of the data quantity to be transmitted to the bandwidth, and taking the product of the ratio and the transmission delay as the transmission cost of each calculation node pair. This step is the core of the transmission matrix construction. The transmission cost reflects the resource and time consumption required for data transmission. By calculating the ratio of the amount of data to be transmitted to the bandwidth, the time required for data transmission can be obtained (in the ideal case, i.e. without taking the transmission delay into account). This time is then added to the transmission delay to give the total transmission cost. This cost takes into account both the speed of data transmission and the additional delay caused by network congestion and other factors. And taking the transmission cost as an element of the transmission matrix to obtain the transmission matrix. The transmission matrix is a two-dimensional matrix in which each element represents the transmission cost of a pair of computing nodes. This matrix provides critical data support for subsequent path optimization. By comparing the transmission costs of different paths, an optimal transmission path can be selected to improve the efficiency and reliability of data transmission.

In this embodiment, a transmission matrix is constructed by determining a pair of computing nodes, detecting a network state, determining an amount of data to be transmitted, calculating a transmission cost, and the like. The matrix provides data support for subsequent path optimization, and is beneficial to realizing efficient transmission of data in the distributed computing terminal.

In the second embodiment of the present application, the same or similar content as that of the first embodiment can be referred to the description above, and the description is omitted. On this basis, the step of step S30 further includes step a10:

And step A10, determining all feasible paths from the source node to the target node aiming at any source node in the transmission matrix, finding the shortest path meeting the lowest cost condition in all the feasible paths, and taking the shortest path as a first path, wherein the cost between any two adjacent nodes in the shortest path is the lowest.

It should be noted that a feasible path is one or more paths that exist in the transmission matrix from the source node to the destination node, and that the nodes and connections on these paths are both active, i.e. the transmission costs between them are known. The lowest cost condition is that among a plurality of possible paths, a path with the smallest transmission cost is selected as a criterion of an optimal path. The shortest path is a path having the smallest number of nodes passing from the source node to the target node or the smallest total transmission cost on the premise that the lowest cost condition is satisfied. Adjacent nodes are two nodes arranged adjacently on a path.

This embodiment aims to find the shortest path with the lowest cost from any source node to a target node in the transmission matrix. This step is an important part of the path optimization problem in a distributed computing environment, which determines the efficiency and cost of data transfer. First, we need to determine all possible paths from the source node to the target node. This typically involves traversing the transmission matrix to find all paths from the source node that can reach the destination node. These paths may contain different combinations of nodes and connections, but all need to meet the condition that the transmission cost is known. Next, of all the possible paths, we need to find the shortest path that satisfies the lowest cost condition. This is typically accomplished by comparing the total transmission costs of the various paths. The total transmission cost is the sum of the transmission costs between all neighboring nodes on the path. To find the lowest cost path we can use graph theory algorithms such as Dijkstra algorithm (Dijkstra algorithm), bellman-Ford algorithm (Bellman-Ford algorithm) or Floyd-Warshall algorithm (Bellman-Ford algorithm). These algorithms can efficiently calculate the shortest path from the source node to all other nodes and give the corresponding transmission costs. After finding the shortest path with the lowest cost, we regard it as the first path. This path not only meets the condition of lowest cost, but also the cost between any two adjacent nodes on it is lowest. This means that on this path, each step of data transmission is optimal, thus ensuring maximization of the overall transmission efficiency.

Further, a dynamic programming algorithm may be employed to solve for the shortest path from the source node to the target node. The core of the dynamic programming algorithm is to gradually construct a global optimal solution by decomposing the complex problem of solving the shortest path into a plurality of sub-problems through a state transition equation. In this example, we consider each node in the network as a state and the path from the source node to the destination node as a sequence of states. By defining the state transition equation, we can calculate the shortest path length from the source node to each intermediate node and extend it gradually to the target node. This process involves a lot of computation, but thanks to the memorised search feature of the dynamic programming algorithm we can find the shortest path from the source node to the target node by retrospective means and take this path as the first path.

The embodiment successfully finds the optimal solution for the data transmission problem in the distributed computing environment through the steps of determining the feasible paths, calculating the total transmission cost, comparing and selecting the shortest path with the lowest cost, and the like. The solution not only improves the efficiency of data transmission, but also reduces the transmission cost, and provides powerful support for optimizing the distributed computing environment.

In a possible implementation manner, the step of step S30 further includes steps a20 to a30:

step A20, when the network state is changed frequently, determining whether the cost of the first path is updated;

And step A30, if the cost of the first path is updated, re-executing the step of optimizing the path between the source node and the target node according to the transmission matrix so as to find the latest shortest path, and updating the latest shortest path to the first path.

It should be noted that, the network state refers to a set of parameters such as a connection state, a transmission speed, a delay, a packet loss rate, and the like between nodes in the network, and these parameters jointly determine performance and stability of the network. Cost updates refer to the cost of recalculating the path due to changes in network conditions (e.g., node failure, link congestion, reduced transmission speed, etc.) that result in changes in the transmission cost on the path.

In a network environment, the network state is dynamically changing, which may result in the previously determined first path being no longer the optimal path. Therefore, this embodiment proposes a dynamic path optimization mechanism to adapt to the change of the network state. First, the system needs to monitor changes in network state and determine if these changes may result in a cost update for the first path. This typically involves continuous monitoring of the network status and real-time detection of parameters that may affect path costs (e.g., delay, packet loss rate, etc.). If it is detected that the cost of the first path may have been updated, the system needs to trigger a path optimization step. This step is similar to the previous path optimization procedure, but now an updated transmission matrix (reflecting the change in network state) is used. The system will recalculate the cost of all feasible paths between the source node and the target node based on the new transmission matrix and find the shortest path with the lowest cost. After the latest shortest path is found, the system needs to update it to the first path. This means that the subsequent data transmission will follow this new shortest path to ensure optimization of transmission efficiency and cost. It should be noted that such path optimization procedures may need to be performed frequently due to frequent changes in network state. Therefore, the system needs to have efficient path computation and optimization capabilities to quickly respond to changes in network conditions and ensure stability and efficiency of data transmission.

Furthermore, an ant colony algorithm can be introduced to dynamically optimize the path. The ant colony algorithm finds the optimal path by simulating the foraging behavior of ants. In the algorithm, each ant represents a potential path selection scheme that selects the next trend based on the "pheromone" on the current path (i.e., the quality of the path, which can be calculated based on the elements in the transmission matrix). As ants move in the network, pheromones on the paths accumulate continuously, thereby guiding more ants to select a better path. This process embodies a positive feedback mechanism so that the algorithm can gradually approach the globally optimal solution. In the ant colony algorithm, we define the update rule of pheromone and the movement rule of ants. The updating rule of the pheromone determines the updating mode of the path quality degree, and the moving rule of the ant determines how the ant selects the next trend. Through multiple iterations, the algorithm converges to a path with the highest pheromone concentration, namely a new optimal path. Notably, in the ant colony algorithm, we also introduce heuristic information to guide the search direction of ants. Heuristic information may be computed from elements in the transmission matrix that reflect the expected cost from the current node to the target node. By combining heuristic information and pheromones, we can better guide ants to move toward the optimal path.

The embodiment realizes the dynamic optimization of the data transmission path in the network environment by monitoring the change of the network state and re-executing the path optimization step when necessary. The mechanism not only improves the efficiency and stability of data transmission, but also enhances the adaptability of the network to dynamic changes.

In a possible implementation manner, the step of step S30 further includes steps B10 to B20:

step B10, constructing an objective function for minimizing transmission cost according to the transmission matrix

It should be noted that the objective function is a mathematical expression or function used to evaluate the merits of different solutions in the optimization problem. In this example, the objective function is aimed at minimizing the transmission cost.

An objective function is constructed based on the transmission matrix. A transmission matrix is a precondition that contains cost information for transmitting data between all pairs of nodes in the network. These costs may be specific values of performance metrics such as delay time, bandwidth consumption, packet loss rate, etc. The objective function is then constructed based on the cost information to quantify the total cost of transmitting data from the source node to the destination node. The form of the objective function may be linear or nonlinear depending on the way the transmission cost is calculated and the complexity of the optimization problem. This objective function is used to evaluate the transmission costs of the different paths. This typically involves extraction and computation of the relevant elements in the transmission matrix. For each possible path, we calculate the total transmission cost of the path based on the corresponding cost information in the transmission matrix. This process may require traversing multiple elements of the transmission matrix and performing corresponding mathematical operations.

And step B20, estimating the transmission cost of the path according to the objective function, and selecting the path with the lowest transmission cost as the first path.

It should be noted that the transmission cost is calculated according to the elements in the transmission matrix, and represents the total cost or cost required for transmitting the data from the source node to the destination node. The first path is the optimal path selected according to a particular criteria (e.g., lowest transmission cost) among a plurality of possible paths.

The path with the lowest transmission cost is selected as the first path. After evaluating the transmission costs of all possible paths, we can compare these cost values to find the smallest one. This path corresponding to the minimum cost is the first path we want to find. The selection of the first path is based on a quantitative comparison of the transmission costs, and is therefore a solution to the optimization problem, representing the lowest cost path for transmitting data from the source node to the destination node.

Illustratively, the way the objective function is calculated is expressed as equation (III)

Min Σ _(i,j)M_trans (i, j) formula (III)

Where i and j are positive integers, M _trans (i, j) is the cost of transmission from the source node i to the destination node j, and the minimum cost of the entire transmission path can be calculated by the formula (iii).

By constructing a minimized transmission cost objective function based on a transmission matrix, the present embodiment can precisely quantify and compare transmission costs of different paths from a source node to a target node. The method not only considers various performance indexes such as delay time, bandwidth consumption, packet loss rate and the like, but also ensures that the selected first path has the lowest transmission cost by traversing the transmission matrix and calculating the total cost of each path. The beneficial effects are that the efficiency of data transmission is improved, the overall cost is reduced, and powerful support is provided for network optimization and resource allocation.

In the third embodiment of the present application, the same or similar contents as those of the first or second embodiment can be referred to the above description, and the description thereof will be omitted. The step S40 further comprises the steps of C10-C30:

Step C10, taking the computing nodes with the distance smaller than a preset distance threshold value, the delay smaller than a preset delay threshold value and no dependency relationship with other computing nodes as first-level computing nodes;

It should be noted that, the preset distance threshold is a critical value for screening and calculating the distance between the nodes when the multi-level aggregation structure is constructed. Only computing nodes that are less than the threshold are considered potential hierarchical connection nodes. The preset delay threshold is a critical value for evaluating data transmission delay between computing nodes when constructing the multi-level aggregation structure. Only the computing nodes with delays less than the threshold are considered valid connecting nodes. No dependencies are those between computing nodes for which there is no data or computational precedence or dependency, i.e., the computation or data transfer of one node is not dependent on the completion of another node. The first-level computing nodes are first selected computing node sets which meet preset distance and delay threshold values and have no dependency relationship when the multi-level aggregation structure is built, and form the topmost layer of the multi-level aggregation structure.

And screening out the computing nodes meeting the conditions as the computing nodes of the first level according to the preset distance and the delay threshold. The distance between the nodes is smaller than a preset distance threshold, the delay is smaller than a preset delay threshold, and no dependency exists between the nodes. These nodes form the topmost layer of the multi-level aggregate structure, providing a basis for the construction of subsequent levels.

Step C20, for any level after the first level, selecting a computing node which is closest to the previous level and has a delay smaller than a preset delay index as a computing node of the level according to the first path to obtain a multi-level aggregation structure, wherein the computing node of the last level is a target node, and any computing node before the computing node of the last level is a source node;

It should be noted that, when constructing the multi-level aggregation structure, the subsequent levels are other levels except the first level, and each level is constructed based on the selection logic of the previous level. The target node is a computing node that is an endpoint of data transmission or computation in a multi-level aggregation structure.

For the subsequent hierarchy, according to the selection logic of the first path, the computing node closest to the previous hierarchy and having a delay less than the preset delay index is selected as the computing node of the next hierarchy. This process is recursive until the last level, the target node, is selected. In this process, we ensure that the connections between each hierarchy are based on optimal paths, thereby optimizing the efficiency of data transfer and computation.

And step C30, if the delay of any one of the first computing nodes is greater than or equal to the preset delay index, replacing the first computing node with a preset standby node.

It should be noted that, in this embodiment, the multi-level aggregation structure is a network structure formed by connecting a plurality of hierarchical computing nodes through optimal paths, and aims to optimize the efficiency of data transmission and computation. Updating the location is to adjust the hierarchical location of a certain computing node in the structure to optimize overall performance if the delay of the computing node does not meet the requirements during the process of constructing the multi-level aggregation structure.

If during the construction of the multi-level aggregate structure we find that the delay of a certain first computing node is greater than or equal to a preset delay index, we need to update the position of this node in the structure. This typically means that we need to move this node to a level closer to the target node or reselect another node meeting the condition to replace it. This process is dynamic and aims to continuously optimize the performance of the multi-stage polymeric structure.

In this embodiment, the preset distance and delay threshold are the basis for screening the computing nodes, the selection logic of the first path is the key for connecting the computing nodes of different levels, and the operation of updating the position is an important means for optimizing the performance of the multi-level aggregation structure. These steps together form the core of the present embodiment, and aim to build an efficient and flexible multi-level aggregation structure to optimize the efficiency of data transmission and computation.

In a possible embodiment, referring to fig. 3, the step of step S50 further includes steps D10 to D20:

step D10, inputting task data into a first-level computing node in a multi-level aggregation structure, and aggregating the task data through the first-level computing node to obtain an aggregation result;

It should be noted that, the first-level computing node is a computing node level that first receives and processes data in the multi-level aggregate structure. These nodes are typically located at the top level of the aggregate structure and are responsible for the preliminary data aggregation operations. The preliminary aggregate result is an intermediate result obtained by performing an aggregate operation on the input data at the first level of computing nodes. This result is the input for the subsequent hierarchical aggregation.

The data is transmitted to a first level computing node for preliminary aggregation. After the nodes receive the data, the data are processed according to preset aggregation logic (such as summation, average and the like), and a preliminary aggregation result is obtained. This result is the basis for subsequent hierarchical aggregation.

And step D20, for any level of computing nodes after the first level of computing nodes, re-aggregating the aggregation result of the previous level to obtain the aggregation result of the current level until the aggregation result of the current level reaches the target node in the multi-level aggregation structure, wherein the aggregation result of the current level is stored in the computing node of the current level closest to the next level.

It should be noted that, the target node is a computing node in the multi-level aggregation structure, which is responsible for receiving and storing the final aggregation result. This node is typically the last layer or designated layer of the aggregate structure. The current level aggregate result is an intermediate result obtained at each level of computing nodes from the previous level aggregate result and the aggregation logic of the present level node. This result will be input to the next level aggregation. The nearest computing node to the next level is the node in the multi-level aggregation structure that is nearest to the network connection (such as near physical distance, low network delay) of the next level computing node, and is used to store the aggregation result of the current level for fast transmission to the next level.

The preliminary polymerization results are further polymerized by a multistage polymerization structure. At each level, the computing node obtains the aggregation result of the current level according to the aggregation result of the previous level and the aggregation logic of the node of the current level. This result is a further understanding and processing result of the data by the node of the present stage. Importantly, in order to optimize data transfer efficiency and reduce latency, the aggregate results of the current tier are stored in the nearest compute node to the next tier. This ensures that data can be transferred to the next level for computation as soon as possible, thereby improving the efficiency of the overall aggregation process.

Finally, through aggregation operation of multiple layers, the final aggregation result is transmitted to the target node. This target node is the last or designated layer of the multi-level aggregation structure, which is responsible for receiving and storing the final aggregation result. The result is a comprehensive understanding and processing result of the input data, and can be used for subsequent tasks such as data analysis, decision support and the like.

The final aggregate result is illustratively calculated as equation (IV)

G (l) = Σ _i∈Level(l)Data_i formula (four)

Wherein G (l) is the total aggregation result, i and l are positive integers, level (l) refers to the first Level, i epsilon Level (l) refers to the i compute node of the l Level, and Data _i refers to the Data processed by the i compute node.

The embodiment provides an efficient data transmission and aggregation mechanism through a multi-stage aggregation structure, and reduces delay and cost of data processing by optimizing data storage and transmission paths. This gives the method of the present embodiment significant advantages when processing large-scale data sets.

It should be noted that the foregoing examples are only for understanding the present application, and are not meant to limit the distributed data transmission method of the present application, and more forms of simple transformation based on the technical concept are all within the scope of the present application.

The present application also provides a distributed data transmission device, referring to fig. 4, the distributed data transmission device includes:

a determining computing node module 10 that determines a plurality of computing nodes in the distributed computing terminal;

The transmission matrix constructing module 20 constructs a transmission matrix according to transmission delay, bandwidth and data quantity to be transmitted between computing nodes, wherein the computing nodes comprise source nodes and target nodes;

the path optimization module 30 performs path optimization on the path between the source node and the target node according to the transmission matrix to obtain a first path;

the aggregation structure constructing module 40 divides all the computing nodes into a plurality of levels according to the first path to obtain a multi-level aggregation structure, wherein the computing nodes in each level process data in parallel;

The aggregation transmission module 50 transmits preset task data to the target node through the multi-level aggregation structure.

The distributed data transmission device provided by the application adopts the distributed data transmission method in the embodiment, so that the data transmission efficiency of the distributed computing terminal can be improved. Compared with the prior art, the beneficial effects of the distributed data transmission device provided by the application are the same as those of the distributed data transmission method provided by the embodiment, and other technical features in the distributed data transmission device are the same as those disclosed by the method of the embodiment, and are not repeated herein.

The application provides a distributed data transmission device which comprises at least one processor and a memory in communication connection with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the distributed data transmission method in the first embodiment.

Referring now to fig. 5, a schematic diagram of a distributed data transmission device suitable for use in implementing embodiments of the present application is shown. The distributed data transmission apparatus in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal DIGITAL ASSISTANT: personal digital assistants), PADs (Portable Application Description: tablet computers), PMPs (Portable MEDIA PLAYER: portable multimedia players), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. The distributed data transmission apparatus shown in fig. 5 is merely an example, and should not impose any limitation on the functions and scope of use of the embodiments of the present application.

As shown in fig. 5, the distributed data transmission apparatus may include a processing device 1001 (e.g., a central processor, a graphics processor, etc.), which may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access Memory (RAM: random Access Memory) 1004. In the RAM1004, various programs and data required for the operation of the distributed data transmission apparatus are also stored. The processing device 1001, the ROM1002, and the RAM1004 are connected to each other by a bus 1005. An input/output (I/O) interface 1006 is also connected to the bus. In general, a system including an input device 1007 such as a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc., an output device 1008 including a Liquid crystal display (LCD: liquid CRYSTAL DISPLAY), a speaker, a vibrator, etc., a storage device 1003 including a magnetic tape, a hard disk, etc., and a communication device 1009 may be connected to the I/O interface 1006. The communication means 1009 may allow the distributed data transfer device to communicate wirelessly or by wire with other devices to exchange data. While a distributed data transmission device having various systems is illustrated, it should be understood that not all illustrated systems are required to be implemented or provided. More or fewer systems may alternatively be implemented or provided.

In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. The above-described functions defined in the method of the disclosed embodiment of the application are performed when the computer program is executed by the processing device 1001.

The distributed data transmission equipment provided by the application adopts the distributed data transmission method in the embodiment, so that the data transmission efficiency of the distributed computing terminal can be improved. Compared with the prior art, the beneficial effects of the distributed data transmission device provided by the application are the same as those of the distributed data transmission method provided by the above embodiment, and other technical features of the distributed data transmission device are the same as those disclosed by the method of the above embodiment, and are not repeated herein.

It is to be understood that portions of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof. In the description of the above embodiments, particular features, structures, materials, or characteristics may be combined in any suitable manner in any one or more embodiments or examples.

The foregoing is merely illustrative embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think about variations or substitutions within the technical scope of the present application, and the application should be covered. Therefore, the protection scope of the application is subject to the protection scope of the claims.

The present application provides a medium, which is a computer readable storage medium having computer readable program instructions (i.e., a computer program) stored thereon for performing the distributed data transmission method of the above embodiments.

The computer readable storage medium provided by the present application may be, for example, a USB flash disk, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system or device, or a combination of any of the foregoing. More specific examples of a computer-readable storage medium may include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access Memory (RAM: random Access Memory), a Read-Only Memory (ROM), an erasable programmable Read-Only Memory (EPROM: erasable Programmable Read Only Memory or flash Memory), an optical fiber, a portable compact disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this embodiment, the computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system or device. Program code embodied on a computer readable storage medium may be transmitted using any appropriate medium, including but not limited to electrical wiring, fiber optic cable, RF (Radio Frequency) and the like, or any suitable combination of the foregoing.

The computer readable storage medium may be included in the distributed data transmission apparatus or may exist alone without being assembled into the distributed data transmission apparatus.

The computer-readable storage medium carries one or more programs that, when executed by the distributed data transmission apparatus, cause the distributed data transmission apparatus to:

Computer program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of remote computers, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN: local Area Network) or a wide area network (WAN: wide Area Network), or may be connected to an external computer (for example, through the Internet using an Internet service provider).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules involved in the embodiments of the present application may be implemented in software or in hardware. Wherein the name of the module does not constitute a limitation of the unit itself in some cases.

The readable storage medium provided by the application is a computer readable storage medium, and the computer readable storage medium stores computer readable program instructions (namely computer programs) for executing the distributed data transmission method, so that the data transmission efficiency of the distributed computing terminal can be improved. Compared with the prior art, the beneficial effects of the computer readable storage medium provided by the application are the same as those of the distributed data transmission method provided by the above embodiment, and are not described in detail herein.

The application also provides a product, which is a computer program product comprising a computer program which, when being executed by a processor, implements the steps of the distributed data transmission method as described above.

The computer program product provided by the application can improve the data transmission efficiency of the distributed computing terminal. Compared with the prior art, the beneficial effects of the computer program product provided by the application are the same as those of the distributed data transmission method provided by the above embodiment, and are not described herein.

The foregoing is only a part of embodiments of the present application, and is not intended to limit the scope of the present application, and all the equivalent structural changes made by the description and drawings of the present application or the direct/indirect application in other related technical fields are included in the scope of the present application.

Claims

1. A distributed data transmission method, applied to a distributed computing terminal, the distributed data transmission method comprising:

Constructing a transmission matrix according to the transmission delay, the bandwidth and the data quantity to be transmitted among the computing nodes, wherein the computing nodes comprise source nodes and target nodes;

Path optimization is carried out on the paths between the source node and the target node according to the transmission matrix, so that a first path is obtained;

The step of optimizing the path between the source node and the target node according to the transmission matrix to obtain a first path includes:

For any source node in the transmission matrix, determining all feasible paths from the source node to a target node, finding a shortest path meeting the lowest cost condition in all the feasible paths, and taking the shortest path as a first path, wherein the cost between any two adjacent nodes in the shortest path is lowest;

dividing all the computing nodes into a plurality of layers according to the first path to obtain a multi-level aggregation structure, wherein the computing nodes in each layer process data in parallel;

the step of dividing all the computing nodes into a plurality of levels according to the first path to obtain a multi-level aggregation structure comprises the following steps:

For any level after the first level, selecting a computing node which is closest to the previous level and has a delay smaller than a preset delay index as a computing node of the current level according to the first path to obtain a multi-level aggregation structure, wherein the computing node of the last level is the target node, and any computing node before the computing node of the last level is the source node;

And for any level of computing nodes after the first level of computing nodes, re-aggregating the aggregation results of the previous level to obtain the aggregation results of the current level until the aggregation results of the current level are aggregated to the target nodes in the multi-level aggregation structure, wherein the aggregation results of the current level are stored in the computing nodes of the current level nearest to the next level.

2. The distributed data transmission method of claim 1, wherein the step of performing path optimization on paths between the source node and the target node according to the transmission matrix to obtain a first path comprises:

Determining whether the cost of the first path is updated when the network state is frequently changed;

and if the cost of the first path is updated, re-executing the step of path optimization on the path between the source node and the target node according to the transmission matrix so as to find the latest shortest path, and updating the latest shortest path into the first path.

3. The distributed data transmission method of claim 1, wherein the step of dividing all the computing nodes into a plurality of levels according to the first path to obtain a multi-level aggregate structure comprises:

if any first computing node has the delay greater than or equal to the preset delay index, replacing the first computing node with a preset standby node.

4. The distributed data transmission method of claim 1, wherein the constructing a transmission matrix according to the transmission delay, the bandwidth, and the amount of data to be transmitted between the computing nodes comprises:

for each computing node pair, detecting a network state to acquire transmission delay and bandwidth between the computing node pairs;

5. A distributed data transmission apparatus, disposed at a distributed computing terminal, the distributed data transmission apparatus comprising:

The path optimization module performs path optimization on the path between the source node and the target node according to the transmission matrix to obtain a first path, wherein the path optimization on the path between the source node and the target node according to the transmission matrix includes:

The aggregation structure building module divides all the computing nodes into a plurality of levels according to the first path to obtain a multi-level aggregation structure, wherein the computing nodes in each level process data in parallel, and the dividing all the computing nodes into a plurality of levels according to the first path to obtain the multi-level aggregation structure comprises the following steps:

The aggregation transmission module is used for inputting task data into a first-level computing node in a multi-level aggregation structure, and aggregating the task data through the first-level computing node to obtain an aggregation result;

6. A distributed data transmission device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the distributed data transmission method according to any one of claims 1 to 4.

7. A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the distributed data transmission method according to any of claims 1 to 4.

8. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the distributed data transmission method according to any one of claims 1 to 4.