[go: up one dir, main page]

CN113900977A - DMA data transmission optimization method in federal learning - Google Patents

DMA data transmission optimization method in federal learning Download PDF

Info

Publication number
CN113900977A
CN113900977A CN202111086414.7A CN202111086414A CN113900977A CN 113900977 A CN113900977 A CN 113900977A CN 202111086414 A CN202111086414 A CN 202111086414A CN 113900977 A CN113900977 A CN 113900977A
Authority
CN
China
Prior art keywords
data
dma
dma connection
connection
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202111086414.7A
Other languages
Chinese (zh)
Other versions
CN113900977B (en
Inventor
李杨
张翔宇
孙军欢
陈沫
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Zhixing Technology Co Ltd
Original Assignee
Shenzhen Zhixing Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Zhixing Technology Co Ltd filed Critical Shenzhen Zhixing Technology Co Ltd
Priority to CN202111086414.7A priority Critical patent/CN113900977B/en
Publication of CN113900977A publication Critical patent/CN113900977A/en
Application granted granted Critical
Publication of CN113900977B publication Critical patent/CN113900977B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F13/14—Handling requests for interconnection or transfer
    • G06F13/20—Handling requests for interconnection or transfer for access to input/output bus
    • G06F13/28—Handling requests for interconnection or transfer for access to input/output bus using burst mode transfer, e.g. direct memory access DMA, cycle steal
    • G06F13/30—Handling requests for interconnection or transfer for access to input/output bus using burst mode transfer, e.g. direct memory access DMA, cycle steal with priority control
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F13/14—Handling requests for interconnection or transfer
    • G06F13/20—Handling requests for interconnection or transfer for access to input/output bus
    • G06F13/28—Handling requests for interconnection or transfer for access to input/output bus using burst mode transfer, e.g. direct memory access DMA, cycle steal
    • G06F13/287—Multiplexed DMA
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G06N20/20—Ensemble learning
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Computer And Data Communications (AREA)

Abstract

The application discloses a DMA data transmission optimization method in federated learning, which is applied to a data sender and comprises the following steps: sending a DMA connection request to a network layer of a data receiving party, and receiving a DMA connection identifier fed back by the data receiving party based on the DMA connection request to establish DMA connection; constructing target DMA connection identification data based on the DMA connection identification and the data information of the data to be transmitted corresponding to the DMA connection identification; and sending the target DMA connection identification data to a logic layer of the data receiver so that the data receiver receives the data to be transmitted, which is transmitted by the data transmitter through the DMA connection, according to the target DMA connection identification data. The method and the device solve the technical problem that the use limitation of the DMA data transmission technology in the prior art is high.

Description

DMA data transmission optimization method in federal learning
Technical Field
The present application relates to the field of communications technologies, and in particular, to a DMA data transfer optimization method, apparatus, device, readable storage medium, and product in federated learning.
Background
With the continuous development of artificial intelligence technology, the data transmission mode is also continuously updated, and DMA (Direct Memory Access) is a network communication mode without system intervention of both parties, high throughput and low delay. At present, only one DMA connection is usually established between a data sending party and a data receiving party for DMA data transmission, and then the bottom logic layer of the data receiving party can directly receive data, and when a plurality of DMA connections are required to be established between the data sending party and the data receiving party for data transmission, the bottom logic layer of the data receiving party cannot generally know which DMA connection to receive the transmitted data, and then the DMA data transmission technology can only be limited to a scene in which one DMA connection is pre-established for data transmission, so the use limitation of the existing DMA data transmission technology is high.
Disclosure of Invention
The application mainly aims to provide a DMA data transmission optimization method, device, equipment, readable storage medium and product in federated learning, and aims to solve the technical problem that the DMA data transmission technology in the prior art is high in use limitation.
In order to achieve the above object, the present application provides a DMA data transmission optimization method in federated learning, where the DMA data transmission optimization method in federated learning is applied to a data sender, and the DMA data transmission optimization method in federated learning includes:
sending a DMA connection request to a network layer of a data receiving party, and receiving a DMA connection identifier fed back by the data receiving party based on the DMA connection request to establish DMA connection;
constructing target DMA connection identification data based on the DMA connection identification and the data information of the data to be transmitted corresponding to the DMA connection identification;
and sending the target DMA connection identification data to a logic layer of the data receiver so that the data receiver receives the data to be transmitted, which is transmitted by the data transmitter through the DMA connection, according to the target DMA connection identification data.
Optionally, before the step of sending the DMA connection request to the network layer of the data receiver, the method for optimizing DMA data transfer in federal learning further includes:
acquiring data receiver information corresponding to a data receiver of data to be transmitted;
searching a DMA connection identifier corresponding to the data receiver information in a preset DMA connection pool according to the data receiver information, wherein the preset DMA connection pool is used for storing the data receiver information of at least one data receiver and the DMA connection identifier corresponding to the data receiver information;
if the search is successful, extracting the DMA connection identification from the preset DMA connection pool;
if the search fails, executing the following steps: and sending a DMA connection request to a network layer of a data receiving party, and receiving a DMA connection identifier fed back by the data receiving party based on the DMA connection request so as to establish DMA connection.
Optionally, the step of sending the target DMA connection identification data to the logic layer of the data receiver includes:
putting the target DMA connection identification data and the data receiver information corresponding to the target DMA connection identification data into a preset blocking queue;
and sequentially sending the target DMA connection identification data to a logic layer of a data receiver corresponding to the data receiver information according to the preset blocking queue.
Optionally, the pre-set blocking queue comprises a bounded blocking queue,
after the step of sequentially sending the target DMA connection identification data to the logic layer of the data receiver corresponding to the data receiver information according to the preset blocking queue, the DMA data transmission optimization method in federal learning further includes:
judging whether the target DMA connection identification data is successfully transmitted within a preset time period;
and if the target DMA connection identification data is not successfully sent, directly sending the target DMA connection identification data to a logic layer of a data receiver corresponding to the data receiver information.
Optionally, after the step of receiving the DMA connection identifier fed back by the data receiver based on the DMA connection request, the DMA data transfer optimization method in federal learning further includes:
and storing the data receiver information corresponding to the DMA connection identifier and the DMA connection identifier to a preset DMA connection pool in a key value pair mode.
After the step of sending the target DMA connection identification data to the logic layer of the data receiver so that the data receiver receives the to-be-transmitted data transmitted by the data sender through the DMA connection according to the target DMA connection identification data, the DMA data transmission optimization method in federal learning includes:
and after the preset data to be transmitted is executed, sending a connection disconnection instruction to the data receiving party so that the data receiving party can disconnect the corresponding DMA connection according to the connection disconnection instruction.
In order to achieve the above object, the present application provides a DMA data transmission optimization method in federated learning, where the DMA data transmission optimization method in federated learning is applied to a data receiver, and the DMA data transmission optimization method in federated learning includes:
receiving a DMA connection request of a data sending party through a network layer of the data receiving party, and generating a DMA connection identifier corresponding to the DMA connection request;
transmitting the DMA connection identification to a corresponding data transmitting party through a network layer, so that the data transmitting party constructs target DMA connection identification data based on the DMA connection identification and data information of data to be transmitted corresponding to the DMA connection identification, and transmits the target DMA connection identification data to a logic layer of the data receiving party;
and receiving the data to be transmitted by the data transmitting side by using the DMA connection according to the target DMA connection identification data.
The application also provides a DMA data transmission optimizing device in federal learning, DMA data transmission optimizing device is the virtual device in federal learning, just DMA data transmission optimizing device is applied to the data sender in federal learning, DMA data transmission optimizing device includes in federal learning:
the request module is used for sending a DMA connection request to a network layer of a data receiving party and receiving a DMA connection identifier fed back by the data receiving party based on the DMA connection request so as to establish DMA connection;
the building module is used for building target DMA connection identification data based on the DMA connection identification and the data information of the data to be transmitted corresponding to the DMA connection identification;
and the sending module is used for sending the target DMA connection identification data to a logic layer of the data receiving party so that the data receiving party can receive the data to be transmitted, which is transmitted by the data sending party by using the DMA connection, according to the target DMA connection identification data.
Optionally, the DMA data transfer optimization apparatus in federal learning is further configured to:
acquiring data receiver information corresponding to a data receiver of data to be transmitted;
searching a DMA connection identifier corresponding to the data receiver information in a preset DMA connection pool according to the data receiver information, wherein the preset DMA connection pool is used for storing the data receiver information of at least one data receiver and the DMA connection identifier corresponding to the data receiver information;
if the search is successful, extracting the DMA connection identification from the preset DMA connection pool;
if the search fails, executing the following steps: and sending a DMA connection request to a network layer of a data receiving party, and receiving a DMA connection identifier fed back by the data receiving party based on the DMA connection request so as to establish DMA connection.
Optionally, the sending module is further configured to:
putting the target DMA connection identification data and the data receiver information corresponding to the target DMA connection identification data into a preset blocking queue;
and sequentially sending the target DMA connection identification data to a logic layer of a data receiver corresponding to the data receiver information according to the preset blocking queue.
Optionally, the preset blocking queue includes a bounded blocking queue, and the sending module is further configured to:
judging whether the target DMA connection identification data is successfully transmitted within a preset time period;
and if the target DMA connection identification data is not successfully sent, directly sending the target DMA connection identification data to a logic layer of a data receiver corresponding to the data receiver information.
Optionally, the DMA data transfer optimization apparatus in federal learning is further configured to:
and storing the data receiver information corresponding to the DMA connection identifier and the DMA connection identifier to a preset DMA connection pool in a key value pair mode.
Optionally, the DMA data transfer optimization apparatus in federal learning is further configured to:
and after the preset data to be transmitted is executed, sending a connection disconnection instruction to the data receiving party so that the data receiving party can disconnect the corresponding DMA connection according to the connection disconnection instruction.
The application also provides a DMA data transmission optimizing device in federal learning, DMA data transmission optimizing device is the virtual device in federal learning, just DMA data transmission optimizing device is applied to the data receiver in federal learning, DMA data transmission optimizing device includes in federal learning:
the generation module is used for receiving a DMA connection request of a data sending party through a network layer of the data receiving party and generating a DMA connection identifier corresponding to the DMA connection request;
the sending module is used for transmitting the DMA connection identifier to a corresponding data sending party through a network layer, so that the data sending party can construct target DMA connection identifier data based on the DMA connection identifier and data information of data to be transmitted corresponding to the DMA connection identifier, and send the target DMA connection identifier data to a logic layer of the data receiving party;
and the connection establishing module is used for receiving the data to be transmitted, which is transmitted by the data transmitting party by using the DMA connection, according to the target DMA connection identification data.
The application also provides DMA data transmission optimizing equipment in federated learning, DMA data transmission optimizing equipment is entity equipment in federated learning, DMA data transmission optimizing equipment includes in federated learning: the method comprises a memory, a processor and a program of the method for optimizing the DMA data transmission in federal learning stored in the memory and capable of running on the processor, wherein the program of the method for optimizing the DMA data transmission in federal learning can realize the steps of the method for optimizing the DMA data transmission in federal learning when being executed by the processor.
The application also provides a readable storage medium, wherein a program for implementing the DMA data transmission optimization method in federal learning is stored on the readable storage medium, and when the program for implementing the DMA data transmission optimization method in federal learning is executed by a processor, the steps of the DMA data transmission optimization method in federal learning are implemented.
The present application also provides a product, which is a computer program product, comprising a computer program, which when executed by a processor, implements the steps of the DMA data transfer optimization method in federal learning as described above.
The application provides a DMA data transmission optimization method, a device, equipment, a readable storage medium and a product in Federation learning, which are applied to a data transmitting party, namely, firstly, a DMA connection request is sent to a network layer of a data receiving party, and a DMA connection identifier fed back by the data receiving party based on the DMA connection request is received, so as to establish DMA connection, and then, a target DMA connection identifier data is established based on the DMA connection identifier and data information of data to be transmitted corresponding to the DMA connection identifier, so that the purpose of associating the DMA connection identifier with the corresponding data to be transmitted is realized, and then the target DMA connection identifier data is sent to a logic layer of the data receiving party, so that the data receiving party receives the data to be transmitted by the data transmitting party by utilizing the DMA connection according to the target DMA connection identifier data, and even if a plurality of DMA connections exist between the data receiving party and the data transmitting party, the bottom logic layer of the data receiver can know which DMA connection is to transmit the data to be transmitted according to the DMA connection identification, namely the bottom logic layer of the data receiver can simultaneously sense the data of which data sender needs to be received and the data information of the data to be transmitted, so that the technical defect that when a plurality of DMA connections need to be established between the data sender and the data receiver for data transmission, the bottom logic layer of the data receiver can not know which DMA connection needs to be received, and further the DMA data transmission technology can be limited to a scene in which one DMA connection is pre-established for data transmission is overcome, so that the use limitation of the DMA data transmission technology is reduced.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.
FIG. 1 is a schematic flow chart diagram illustrating a first embodiment of a DMA data transfer optimization method in Federal learning according to the present application;
fig. 2 is a schematic view of interaction between a data sender and a data receiver when the target DMA connection identifier data in the DMA data transmission optimization method in federal learning of the present application is a DMA connection identifier itself;
FIG. 3 is a flowchart illustrating a second embodiment of a DMA data transfer optimization method in Federal learning according to the present application;
fig. 4 is a schematic flow chart illustrating data interaction between the data sending side and the data receiving side based on a preset DMA connection pool and a preset blocking queue in the DMA data transmission optimization method in federal learning according to the present application;
FIG. 5 is a flowchart illustrating a DMA data transfer optimization method according to a third embodiment of the Federal study;
fig. 6 is a schematic device structure diagram of a hardware operating environment related to a DMA data transfer optimization method in federal learning in an embodiment of the present application.
The objectives, features, and advantages of the present application will be further described with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
Example one
The embodiment of the present application provides a DMA data transmission optimization method in federated learning, which is applied to a data sender, and in a first embodiment of the DMA data transmission optimization method in federated learning of the present application, referring to fig. 1, the DMA data transmission optimization method in federated learning includes:
step S10, sending a DMA connection request to a network layer of a data receiver, and receiving a DMA connection identifier fed back by the data receiver based on the DMA connection request to establish a DMA connection;
in this embodiment, it should be noted that the DMA data transmission optimization method in federal learning may be applied to a federal learning scenario, for example, the data sender may be a party in federal learning, and the data receiver may be another party in federal learning, or alternatively, the data sender may be a party in federal learning, and the data receiver may be a federal server in federal learning.
Additionally, it should be noted that the network layer is an upper layer of the data receiving party, and the data receiving party further includes a logic layer which is a bottom layer, where the network layer cannot directly communicate with the logic layer due to the different language environments of the network layer and the logic layer, for example, the language environment of the network layer may be a python language environment, and the logic layer is a C language environment.
The method comprises the steps of sending a DMA connection request to a network layer of a data receiving party, receiving a DMA connection identifier fed back by the data receiving party based on the DMA connection request to establish DMA connection, specifically, sending the DMA connection request to the network layer of the data receiving party, generating the DMA connection identifier by the data receiving party after receiving the DMA connection request, and sending the DMA connection identifier to the data receiving party to establish DMA connection between the data sending party and the data receiving party, wherein the DMA connection identifier is an identity identifier of the DMA connection.
Step S20, constructing target DMA connection identification data based on the DMA connection identification and the data information of the data to be transmitted corresponding to the DMA connection identification;
in this embodiment, it should be noted that the data information of the data to be transmitted is used to indicate the data receiving side to prepare a corresponding memory to receive the data to be transmitted, where the data information of the data to be transmitted includes size information of the data to be transmitted and/or a storage space required by the data to be transmitted, where the size information of the data to be transmitted may be a total data size of the data to be transmitted, and may also be a data size to be transmitted each time when the data to be transmitted is transmitted through the DMA connection and a number of times of data transmission through the DMA connection established this time, for example, assuming that the total data size of the data to be transmitted is 100M, the size of the data that can be transmitted each time through the DMA connection is 10M, the size of the data transmission performed each time through the DMA connection is 10M, and the number of times of data transmission is 10 times.
And constructing target DMA connection identification data based on the DMA connection identification and the data information of the data to be transmitted corresponding to the DMA connection identification, specifically, acquiring the size information of the preset data to be transmitted, and further constructing the target DMA connection identification data based on the DMA connection identification and the size information of the data to be transmitted so as to establish a temporary association relationship between the DMA connection identification and the data to be transmitted, thereby realizing the purpose of associating the DMA connection identification with the data to be transmitted.
Wherein the step of constructing target DMA connection identification data based on the DMA connection identification and the size information of the data to be transmitted comprises:
packaging the DMA connection identifier and the size information of the data to be transmitted into a data packet to be transmitted, and further taking the data packet to be transmitted as target DMA connection identifier data; or storing the DMA connection identifier and the corresponding size information of the data to be transmitted as a target key value pair, and further taking the target key value pair as target DMA connection identifier data; or directly splicing the DMA connection identifier and the size information of the data to be transmitted to obtain a splicing result, and further taking the splicing result as a target DMA connection identifier.
Additionally, it should be noted that the target DMA connection identifier data may also be a DMA connection identifier itself, for the following reasons:
before the data receiving party and the data sending party establish DMA connection, the data receiving party and the data sending party can agree on the size of data to be transmitted each time, and then the data sending party only needs to temporarily associate the DMA connection identifier with the corresponding data to be transmitted, at the moment, target DMA connection identifier data is the DMA connection identifier, and further after the DMA connection corresponding to the DMA connection identifier is established, the data sending party can directly transmit the data to be transmitted to the data receiving party through the DMA connection because the DMA connection identifier and the corresponding data to be transmitted are temporarily associated in advance, and the data receiving party and the data sending party can directly divide a storage space with the corresponding size in advance for data receiving and storage before data transmission each time, fig. 2 is a schematic view showing interaction between a data sending party and a data receiving party when the target DMA connection identifier data is a DMA connection identifier itself, where a customer service end is the data sending party, a service side is the data receiving party, and a connection identifier of a DMA connection is the DMA connection identifier.
Step S30, sending the target DMA connection identification data to the logic layer of the data receiver, so that the data receiver receives the to-be-transmitted data transmitted by the data transmitter through the DMA connection according to the target DMA connection identification data.
In this embodiment, the target DMA connection identification data is sent to the logic layer of the data receiver, so that the data receiver receives the data to be transmitted by the data transmitter through the DMA connection according to the target DMA connection identification data, specifically, the target DMA connection identification data is sent to the logic layer of the data receiver, the data receiver prepares a target memory corresponding to the size of the data to be transmitted, associates the target memory with the DMA connection identification, and the data receiver receives the data to be transmitted by the data transmitter through the DMA connection according to the DMA connection and the target memory associated with each other, so that the bottom logic layer of the data receiver can sense which DMA connection is going to perform data transmission and prepare a corresponding target memory to receive the data to be transmitted, wherein, the data receiving party prepares a target memory corresponding to the size of the data to be transmitted, associates the target memory with the DMA connection identifier, and then receives the data to be transmitted by the data transmitting party by using the DMA connection according to the mutually associated DMA connection and the target memory, wherein the step of receiving the data to be transmitted by the data transmitting party by using the DMA connection further comprises the following steps:
the data receiver firstly carries out memory application and memory registration, wherein the memory application is used for preparing a target memory corresponding to the size information of the data to be transmitted in the target DMA connection identification data, the memory registration is used to allow the network card to write the data to be transmitted received through the DMA connection directly into the corresponding memory, and the data receiver submits the target memory to the network card and specifies the DMA connection identifier corresponding to the target memory, the data receiver can inform the data sender to send data, and the data receiver can directly receive and store the data to be transmitted by the DMA connection corresponding to the DMA connection identifier according to the target memory, and because the DMA connection identifier is given to the DMA connection, the underlying logical layer of the data receiver always senses which DMA connection is making the data transfer ready for data reception.
Wherein the step of sending the target DMA connection identification data to the logic layer of the data receiver comprises:
step S31, the target DMA connection identification data and the data receiver information corresponding to the target DMA connection identification data are put into a preset blocking queue;
in this embodiment, it should be noted that the data receiving side information includes a server IP address and a service port number, where the server IP address is an IP address of a server in the data receiving side, and the service port number is an identifier of a server port used for establishing a DMA connection, where at least one server exists in the data receiving side, a service port in the server corresponds to at least one DMA connection, and the data receiving side may establish the DMA connection with at least one data sending side through at least one service port.
And putting the target DMA connection identification data and the data receiver information corresponding to the target DMA connection identification data into a preset blocking queue, specifically, putting the target DMA connection identification data and the data receiver information corresponding to the target DMA connection identification data into the preset blocking queue together as queue elements, wherein it needs to be stated that at least one queue element exists in the preset blocking queue, that is, at least one DMA connection identification exists in a data sending party. The data sender can establish DMA connection with at least one service port of one data receiver, namely, the data sender can establish corresponding DMA connection with a plurality of service ports, and the service ports can belong to the same data receiver and also belong to different data receivers, and then when the concurrency of data transmission of the data receiver is high, a large number of DMA connection identifiers are sequentially sent to the service ports of the corresponding data receivers, so that the peak pressure of the data transmission of the data receivers is reduced.
And step S32, sequentially sending the target DMA connection identification data to the logic layer of the data receiver corresponding to the data receiver information according to the preset blocking queue.
In this embodiment, the target DMA connection identification data is sequentially sent to the logic layer of the data receiving side corresponding to the data receiving side information according to the preset blocking queue, and specifically, when the queue element corresponding to the target DMA connection identification data is dequeued, the target DMA connection identification data is sequentially sent to the logic layer of the corresponding data receiving side according to the server IP address and the service port number in the queue element.
Additionally, it should be noted that, in order to better manage the DMA connection identifier, at least one preset blocking queue may be deployed in the data sending side, and the data receiving side may be set to correspond to the preset blocking queue one to one, so as to prevent the DMA connection identifiers of different data receiving sides from being mixed up in the same preset blocking queue, thereby affecting the sending efficiency of the DMA connection identifier. The preset blocking queue may send the target DMA connection identification data to a logic layer of a corresponding data receiver in a First Input First Output (FIFO) data transmission manner.
Wherein the pre-set blocking queues comprise bounded blocking queues,
after the step of sequentially sending the target DMA connection identification data to the logic layer of the data receiver corresponding to the data receiver information according to the preset blocking queue, the DMA data transmission optimization method in federal learning further includes:
step S311, judging whether the target DMA connection identification data is successfully transmitted within a preset time period;
step S312, if the sending is not successful, directly sending the target DMA connection identification data to the logic layer of the data receiver corresponding to the data receiver information.
In this embodiment, whether the target DMA connection identifier data is successfully transmitted within a preset time period is determined, and if the target DMA connection identifier data is not successfully transmitted, it is verified that the bounded blocking queue is blocked, and the target DMA connection identifier data is directly transmitted to the logic layer of the data receiver corresponding to the data receiver information, so as to prevent the target DMA connection identifier data from being delayed and unable to be transmitted to the logic layer of the corresponding data receiver due to the full blocking queue, where the reasons for the blocking of the bounded blocking queue include that the amount of data transmitted by the data transmitter is greater than the queue capacity, and/or the data transmission speed of the data transmitter is greater than the processing speed of the data receiver, and the like, for example, when the data receiver receives the DMA connection identifier at a slow speed, the data transmitter does not successfully transmit the target DMA connection identifier data, at this time, the next target DMA connection identification data cannot be sent, and the data sending side still puts the target DMA connection identification data into the bounded blocking queue, and at this time, the situation that the bounded blocking queue is blocked is easily generated, so that the target DMA connection identification data is directly sent to the data receiving side without passing through the bounded blocking queue, and the sending efficiency of the DMA connection identification can be improved.
Additionally, it should be noted that, because the bounded blocking queue is bounded, the data sending party can only send a number of DMA connection identifiers whose number is not greater than the preset number threshold to the data receiving party within the preset time period, and the number of DMA connections used by the data receiving party and at least one data receiving party within the preset time period does not exceed the preset upper limit that the network card can bear, so that it can be prevented that the device of the data sending party occupies too many device resources due to the establishment of too many DMA connections, which causes the device of the data sending party to send a jam or a malfunction, for example, if the network card can use 100 DMA connections at most for data transmission, and if the data sending party sends a DMA connection identifier to the data receiving party for data transmission completely, when the data is sent in a large amount, it is easy to cause the number of DMA connections used by the network card for data transmission to exceed 100, and then the system is stuck or down, and the DMA connection identifier is sent to the data receiving party through the bounded blocking queue, so that the DMA connection used by the network card for data transmission in a preset time period can not exceed 100 by setting the length of the bounded blocking queue or the time interval for sending the DMA connection identifier, and further the occurrence of the sticking or the down can be prevented.
After the step of sending the target DMA connection identification data to the logic layer of the data receiver so that the data receiver receives the to-be-transmitted data transmitted by the data sender through the DMA connection according to the target DMA connection identification data, the DMA data transmission optimization method in federal learning includes:
step S40, after the preset data to be transmitted is executed, sending a disconnection instruction to the data receiving side, so that the data receiving side disconnects the corresponding DMA connection according to the disconnection instruction.
After the preset data to be transmitted is executed, sending a connection and disconnection instruction to the data receiver so that the data receiver can disconnect the corresponding DMA connection according to the connection and disconnection instruction, specifically, after the preset data to be transmitted is executed, generating a connection and disconnection instruction, and sending the connection and disconnection instruction to the server in the corresponding data receiver according to the server IP address and the service port number corresponding to the preset data to be transmitted so that the data receiver can disconnect the DMA connection of the corresponding server port according to the connection and disconnection instruction so as to release the equipment resource of the data receiver and reduce the load of the data receiver.
The embodiment of the application provides a DMA data transmission optimization method in federated learning, which is applied to a data transmitting party, namely, firstly, a DMA connection request is transmitted to a network layer of a data receiving party, and a DMA connection identifier fed back by the data receiving party based on the DMA connection request is received to establish DMA connection, and then, a target DMA connection identifier data is established based on the DMA connection identifier and data information of data to be transmitted corresponding to the DMA connection identifier, so that the purpose of associating the DMA connection identifier with the corresponding data to be transmitted is realized, and the target DMA connection identifier data is transmitted to a logic layer of the data receiving party so that the data receiving party can receive the data to be transmitted by using the DMA connection according to the target DMA connection identifier data, even if a plurality of DMA connections exist between the data receiving party and the data transmitting party, the bottom logic layer of the data receiver can know which DMA connection is to transmit the data to be transmitted according to the DMA connection identification, namely the bottom logic layer of the data receiver can simultaneously sense the data of which data sender needs to be received and the data information of the data to be transmitted, so that the technical defect that when a plurality of DMA connections need to be established between the data sender and the data receiver for data transmission, the bottom logic layer of the data receiver can not know which DMA connection needs to be received, and further the DMA data transmission technology can be limited to a scene in which one DMA connection is pre-established for data transmission is overcome, so that the use limitation of the DMA data transmission technology is reduced.
Example two
Further, referring to fig. 3, according to the first embodiment of the present application, in another embodiment of the present application, before the step of sending the DMA connection request to the network layer of the data receiver, the method for optimizing DMA data transfer in federal learning further includes:
step A10, acquiring data receiver information corresponding to a data receiver of data to be transmitted;
in this embodiment, it should be noted that each data sender at least maintains a preset DMA connection pool, where the preset DMA connection pool is used to store data receiver information of at least one data receiver and a DMA connection identifier corresponding to the data receiver information, and the data receiver information and the DMA connection identifier corresponding to the data receiver information are stored in the preset DMA connection pool in a key value pair form, preferably, the data receiver information may be set as a key in a key value pair, and the DMA connection identifier may be set as a value in the key value pair, where the data receiver information includes a server IP address and a service port number, and when a data sender needs to establish a DMA connection with a service port of a server of a certain data receiver, the corresponding DMA connection identifier may be queried in the preset DMA connection pool by using the server IP address and the corresponding service port number, and sending the DMA connection identifier to a logic layer of a corresponding data receiver according to the IP address of the server and the corresponding service port number.
Step A20, according to the data receiver information, searching a DMA connection identifier corresponding to the data receiver information in a preset DMA connection pool;
in this embodiment, according to the data receiver information, a DMA connection identifier corresponding to the data receiver information is searched in a preset DMA connection pool, and specifically, with the data receiver information as an index, a DMA connection identifier corresponding to the data receiver information is searched in a preset DMA connection pool.
Step A30, if the search is successful, extracting the DMA connection identifier from the preset DMA connection pool;
step a40, if the search fails, executing the steps of: and sending a DMA connection request to a network layer of a data receiving party, and receiving a DMA connection identifier fed back by the data receiving party based on the DMA connection request so as to establish DMA connection.
In this embodiment, if the lookup is successful, the DMA connection identifier is directly extracted from the preset DMA connection pool, and if the lookup is failed, it is verified that the DMA connection identifier corresponding to the data receiver information is not backed up in the preset DMA connection pool, and then the steps are performed: and sending a DMA connection request to a network layer of a data receiving party, and receiving a DMA connection identifier fed back by the data receiving party based on the DMA connection request so as to establish DMA connection.
After the step of receiving the DMA connection identifier fed back by the data receiver based on the DMA connection request, the DMA data transmission optimization method in federal learning further includes:
and step B10, storing the data receiving party information corresponding to the DMA connection identifier and the DMA connection identifier to a preset DMA connection pool in a key value pair mode.
In this embodiment, specifically, after receiving the DMA connection identifier fed back by the data receiver based on the DMA connection request, the data receiver information corresponding to the DMA connection identifier and the DMA connection identifier are stored in a key value pair form in a preset DMA connection pool, and when the data transmitter needs to establish the DMA connection corresponding to the DMA connection identifier with the data receiver again, the data transmitter does not need to want to request the DMA identifier for the data receiver again, and the data receiver does not need to generate the DMA identifier again, so that an interaction process between the data transmitter and the data receiver is saved, and the efficiency of data transmission through DMA is improved.
Additionally, it should be noted that when data transmission is performed between a data receiving side and a plurality of data transmitting sides, the load of the data receiving side is often extremely high, and in the embodiment of the present application, the DMA connection identifier is placed on the data transmitting side for maintenance, and the data receiving side does not need to maintain the DMA identifier or generate the DMA identifier, so that the load of the data receiving side can be reduced.
In an implementable manner of the embodiment of the present application, as shown in fig. 4, a schematic flow diagram of data interaction between the data sending side and the data receiving side on the basis of a preset DMA connection pool and a preset blocking queue in the embodiment of the present application is shown, where the connection request is the DMA connection request, the DMA connection identifier is set to be a process of storing the DMA connection identifier in the preset DMA connection pool, and the DMA connection identifier is obtained to be a process of extracting the DMA connection identifier in the preset DMA connection pool.
The embodiment of the application provides a DMA data transmission optimization method in federated learning, namely, a DMA connection request is sent to a network layer of a data receiver, a DMA connection identifier fed back by the data receiver based on the DMA connection request is received to establish DMA connection, and then target DMA connection identifier data is established based on the DMA connection identifier and data information of data to be transmitted corresponding to the DMA connection identifier, and then the target DMA connection identifier data is sent to a logic layer of the data receiver so that the data receiver can receive the data to be transmitted by the data sender through the DMA connection according to the target DMA connection identifier data, wherein the DMA identifier is put on the data sender to be maintained in the embodiment of the application, when the data sender and the data receiver are required to establish DMA connection, the data sender does not need to request the data receiver for the DMA connection identifier, the data receiving party does not need to generate the DMA connection identification immediately, and even does not need to feed back the DMA identification generated immediately to the data sending party, so that the use limitation of the DMA data transmission technology is further reduced.
EXAMPLE III
Further, referring to fig. 5, based on the first embodiment of the present application, in another embodiment of the present application, the DMA data transfer optimization method in federal learning is applied to a data receiver, and the DMA data transfer optimization method in federal learning includes:
step C10, receiving a DMA connection request of a data sender through the network layer of the data receiver, and generating a DMA connection identifier corresponding to the DMA connection request;
step C20, transmitting the DMA connection identifier to a corresponding data transmitting side through a network layer, so that the data transmitting side can construct target DMA connection identifier data based on the DMA connection identifier and the data information of the data to be transmitted corresponding to the DMA connection identifier, and transmit the target DMA connection identifier data to a logic layer of the data receiving side;
in this embodiment, it should be noted that the data information of the data to be transmitted is used to instruct the data receiving side to prepare a corresponding memory to receive the data to be transmitted.
Specifically, a DMA connection request of a data sending party is received through a network layer of a data receiving party, and a DMA connection identifier corresponding to the DMA connection request is generated, and then the DMA connection identification is transmitted to a corresponding data transmitting party through a network layer, so that the data transmitting party constructs target DMA connection identification data based on the DMA connection identification and the data information of the corresponding data to be transmitted, so as to establish a temporary association relationship between the DMA connection identification and the corresponding data to be transmitted, and the data sender sends the target DMA connection identification data to the logical layer of the data receiver, the specific process of the data sender constructing the target DMA connection identifier data based on the DMA connection identifier and the data information of the data to be transmitted of the corresponding data to be transmitted may refer to the specific implementation process in step S20, which is not described herein again.
And step C30, receiving the data to be transmitted by the data sender through the DMA connection according to the target DMA connection identification data.
In this embodiment, according to the target DMA connection identification data, data to be transmitted, which is transmitted by the data transmitting side through the DMA connection, is received, specifically, size information of the data to be transmitted and a DMA connection identification in the target DMA connection identification data are obtained, a target memory corresponding to the size of the data to be transmitted is prepared, the target memory is associated with the DMA connection identification, and the data receiving side receives the data to be transmitted, which DMA connection is to be used for data transmission, according to the DMA connection and the target memory associated with each other, by the data transmitting side, and the bottom logic layer of the data receiving side can sense which DMA connection is to be used for data transmission, and prepares the corresponding target memory to receive the data to be transmitted.
The step of preparing a target memory corresponding to the size of the data to be transmitted, associating the target memory with the DMA connection identifier, and receiving the data to be transmitted by the data sender through the DMA connection according to the DMA connection and the target memory associated with each other includes:
the data receiver firstly carries out memory application and memory registration, wherein the memory application is used for preparing a target memory corresponding to the size information of the data to be transmitted in the target DMA connection identification data, the memory registration is used to allow the network card to write the data to be transmitted received through the DMA connection directly into the corresponding memory, and the data receiver submits the target memory to the network card and specifies the DMA connection identifier corresponding to the target memory, the data receiver can inform the data sender to send data, and the data receiver can directly receive and store the data to be transmitted by the DMA connection corresponding to the DMA connection identifier according to the target memory, and because the DMA connection identifier is given to the DMA connection, the underlying logical layer of the data receiver always senses which DMA connection is making the data transfer ready for data reception.
The embodiment of the application provides a DMA data transmission optimization method in federated learning, namely, a DMA connection request of a data sending party is received through a network layer of a data receiving party, a DMA connection identifier corresponding to the DMA connection request is generated, and the DMA connection identifier is transmitted to the corresponding data sending party through the network layer, so that the data sending party can construct target DMA connection identifier data based on the DMA connection identifier and data information of data to be transmitted corresponding to the DMA connection identifier, and the target DMA connection identifier data is sent to a logic layer of the data receiving party, thereby realizing the purpose of associating the DMA connection with the corresponding data to be transmitted, and further receiving the data to be transmitted by the data sending party by utilizing the DMA connection according to the target DMA connection identifier data, even if a plurality of DMA connections exist between the data receiving party and the data sending party, the bottom logic layer of the data receiver can know which DMA connection is to transmit the data to be transmitted according to the DMA connection identification, namely the bottom logic layer of the data receiver can simultaneously sense the data of which data sender needs to be received and the data information of the data to be transmitted, so that the technical defect that when a plurality of DMA connections need to be established between the data sender and the data receiver for data transmission, the bottom logic layer of the data receiver can not know which DMA connection needs to be received, and further the DMA data transmission technology can be limited to a scene in which one DMA connection is pre-established for data transmission is overcome, so that the use limitation of the DMA data transmission technology is reduced.
Referring to fig. 6, fig. 6 is a schematic device structure diagram of a hardware operating environment according to an embodiment of the present application.
As shown in fig. 6, the DMA data transfer optimization device in federal learning may include: a processor 1001, such as a CPU, a memory 1005, and a communication bus 1002. The communication bus 1002 is used for realizing connection communication between the processor 1001 and the memory 1005. The memory 1005 may be a high-speed RAM memory or a non-volatile memory (e.g., a magnetic disk memory). The memory 1005 may alternatively be a memory device separate from the processor 1001 described above.
Optionally, the DMA data transmission optimization device in federal learning may further include a rectangular user interface, a network interface, a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. The rectangular user interface may comprise a Display screen (Display), an input sub-module such as a Keyboard (Keyboard), and the optional rectangular user interface may also comprise a standard wired interface, a wireless interface. The network interface may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface).
Those skilled in the art will appreciate that the DMA data transfer optimization device architecture in Federal learning shown in FIG. 6 does not constitute a limitation of the DMA data transfer optimization device in Federal learning, and may include more or fewer components than shown, or some components in combination, or a different arrangement of components.
As shown in fig. 6, the memory 1005, which is a kind of computer storage medium, may include therein an operating system, a network communication module, and a DMA data transfer optimization program in federal learning. The operating system is a program for managing and controlling hardware and software resources of the DMA data transmission optimization equipment in the Federal learning, and supports the operation of the DMA data transmission optimization program in the Federal learning and other software and/or programs. The network communication module is used for realizing communication among components in the memory 1005 and communication with other hardware and software in a DMA data transmission optimization system in federal learning.
In the device for optimizing DMA data transfer in federal learning shown in fig. 6, the processor 1001 is configured to execute a program for optimizing DMA data transfer in federal learning stored in the memory 1005, so as to implement any of the steps of the method for optimizing DMA data transfer in federal learning described above.
The specific implementation and beneficial effects of the DMA data transmission optimization device in federal learning of the application are basically the same as those of the above embodiments of the DMA data transmission optimization method in federal learning, and are not described herein again.
Example four
The embodiment of the present application further provides a DMA data transmission optimization device in federated learning, the DMA data transmission optimization device is applied to the data sender in federated learning, the DMA data transmission optimization device includes in federated learning:
sending a DMA connection request to a network layer of a data receiving party, and receiving a DMA connection identifier fed back by the data receiving party based on the DMA connection request to establish DMA connection;
constructing target DMA connection identification data based on the DMA connection identification and the data information of the data to be transmitted corresponding to the DMA connection identification;
and sending the target DMA connection identification data to a logic layer of the data receiver so that the data receiver receives the data to be transmitted, which is transmitted by the data transmitter through the DMA connection, according to the target DMA connection identification data.
Optionally, before the step of sending the DMA connection request to the network layer of the data receiver, the method for optimizing DMA data transfer in federal learning further includes:
acquiring data receiver information corresponding to a data receiver of data to be transmitted;
searching a DMA connection identifier corresponding to the data receiver information in a preset DMA connection pool according to the data receiver information;
if the search is successful, extracting the DMA connection identification from the preset DMA connection pool;
if the search fails, executing the following steps: and sending a DMA connection request to a network layer of a data receiving party, and receiving a DMA connection identifier fed back by the data receiving party based on the DMA connection request so as to establish DMA connection.
Optionally, the step of sending the target DMA connection identification data to the logic layer of the data receiver includes:
putting the target DMA connection identification data and the data receiver information corresponding to the target DMA connection identification data into a preset blocking queue;
and sequentially sending the target DMA connection identification data to a logic layer of a data receiver corresponding to the data receiver information according to the preset blocking queue.
Optionally, the pre-set blocking queue comprises a bounded blocking queue,
after the step of sequentially sending the target DMA connection identification data to the logic layer of the data receiver corresponding to the data receiver information according to the preset blocking queue, the DMA data transmission optimization method in federal learning further includes:
judging whether the target DMA connection identification data is successfully transmitted within a preset time period;
and if the target DMA connection identification data is not successfully sent, directly sending the target DMA connection identification data to a logic layer of a data receiver corresponding to the data receiver information.
Optionally, after the step of receiving the DMA connection identifier fed back by the data receiver based on the DMA connection request, the DMA data transfer optimization method in federal learning further includes:
and storing the data receiver information corresponding to the DMA connection identifier and the DMA connection identifier to a preset DMA connection pool in a key value pair mode.
After the step of sending the target DMA connection identification data to the logic layer of the data receiver so that the data receiver receives the to-be-transmitted data transmitted by the data sender through the DMA connection according to the target DMA connection identification data, the DMA data transmission optimization method in federal learning includes:
and after the preset data to be transmitted is executed, sending a connection disconnection instruction to the data receiving party so that the data receiving party can disconnect the corresponding DMA connection according to the connection disconnection instruction.
The specific implementation and beneficial effects of the DMA data transmission optimization device in federal learning of the application are basically the same as those of the above embodiments of the DMA data transmission optimization method in federal learning, and are not described herein again.
EXAMPLE five
The embodiment of the present application further provides a DMA data transmission optimization device in federated learning, the DMA data transmission optimization device is applied to the data receiver in federated learning, the DMA data transmission optimization device includes in federated learning:
the generation module is used for receiving a DMA connection request of a data sending party through a network layer of the data receiving party and generating a DMA connection identifier corresponding to the DMA connection request;
the sending module is used for transmitting the DMA connection identifier to a corresponding data sending party through a network layer, so that the data sending party can construct target DMA connection identifier data based on the DMA connection identifier and data information of data to be transmitted corresponding to the DMA connection identifier, and send the target DMA connection identifier data to a logic layer of the data receiving party;
and the connection establishing module is used for receiving the data to be transmitted, which is transmitted by the data transmitting party by using the DMA connection, according to the target DMA connection identification data.
The specific implementation and beneficial effects of the DMA data transmission optimization device in federal learning of the application are basically the same as those of the above embodiments of the DMA data transmission optimization method in federal learning, and are not described herein again.
EXAMPLE six
The embodiment of the application provides a readable storage medium, and the readable storage medium stores one or more programs, which can be further executed by one or more processors to implement the steps of the DMA data transfer optimization method in federal learning described in any one of the above.
The specific implementation and beneficial effects of the readable storage medium of the application are substantially the same as those of the embodiments of the DMA data transmission optimization method in the federal learning, and are not described herein again.
EXAMPLE seven
The present application also provides a product, which is a computer program product, comprising a computer program, which when executed by a processor, implements the steps of the DMA data transfer optimization method in federal learning as described above.
The specific implementation and beneficial effects of the computer program product of the present application are substantially the same as those of the above embodiments of the DMA data transmission optimization method in federal learning, and are not described herein again.
The above description is only a preferred embodiment of the present application, and not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings, or which are directly or indirectly applied to other related technical fields, are included in the scope of the present application.

Claims (12)

1. The DMA data transmission optimization method in federated learning is characterized by being applied to a data sending party and comprising the following steps:
sending a DMA connection request to a network layer of a data receiving party, and receiving a DMA connection identifier fed back by the data receiving party based on the DMA connection request to establish DMA connection;
constructing target DMA connection identification data based on the DMA connection identification and the data information of the data to be transmitted corresponding to the DMA connection identification;
and sending the target DMA connection identification data to a logic layer of the data receiver so that the data receiver receives the data to be transmitted, which is transmitted by the data transmitter through the DMA connection, according to the target DMA connection identification data.
2. The method for optimizing DMA data transfer in federal learning as claimed in claim 1, wherein prior to the step of sending a DMA connection request to a network layer of a data receiver, the method for optimizing DMA data transfer in federal learning further comprises:
acquiring data receiver information corresponding to a data receiver of data to be transmitted;
searching a DMA connection identifier corresponding to the data receiver information in a preset DMA connection pool according to the data receiver information;
if the search is successful, extracting the DMA connection identification from the preset DMA connection pool;
if the search fails, executing the following steps: and sending a DMA connection request to a network layer of a data receiving party, and receiving a DMA connection identifier fed back by the data receiving party based on the DMA connection request so as to establish DMA connection.
3. The method for DMA data transfer optimization in federal learning as in claim 1, wherein the step of sending the target DMA connection identification data to the logic layer of the data receiver comprises:
putting the target DMA connection identification data and the data receiver information corresponding to the target DMA connection identification data into a preset blocking queue;
and sequentially sending the target DMA connection identification data to a logic layer of a data receiver corresponding to the data receiver information according to the preset blocking queue.
4. The method for DMA data transfer optimization in federal learning as in claim 3, wherein the preset blocked queues include bounded blocked queues,
after the step of sequentially sending the target DMA connection identification data to the logic layer of the data receiver corresponding to the data receiver information according to the preset blocking queue, the DMA data transmission optimization method in federal learning further includes:
judging whether the target DMA connection identification data is successfully transmitted within a preset time period;
and if the target DMA connection identification data is not successfully sent, directly sending the target DMA connection identification data to a logic layer of a data receiver corresponding to the data receiver information.
5. The DMA data transfer optimization method for federal learning of claim 1, wherein after the step of receiving the DMA connection identifier fed back by the data receiver based on the DMA connection request, the DMA data transfer optimization method for federal learning further comprises:
and storing the data receiver information corresponding to the DMA connection identifier and the DMA connection identifier to a preset DMA connection pool in a key value pair mode.
6. The method for optimizing DMA data transfer in federal learning of claim 1, wherein after the step of sending the target DMA connection identification data to the logic layer of the data receiver for the data receiver to receive the data to be transferred, which is transmitted by the data sender using the DMA connection, according to the target DMA connection identification data, the method for optimizing DMA data transfer in federal learning comprises:
and after the preset data to be transmitted is executed, sending a connection disconnection instruction to the data receiving party so that the data receiving party can disconnect the corresponding DMA connection according to the connection disconnection instruction.
7. The DMA data transmission optimization method in federated learning is characterized by being applied to a data receiver and comprising the following steps:
receiving a DMA connection request of a data sending party through a network layer of the data receiving party to establish DMA connection and generate a DMA connection identifier corresponding to the DMA connection request;
transmitting the DMA connection identification to a corresponding data transmitting party through a network layer, so that the data transmitting party constructs target DMA connection identification data based on the DMA connection identification and data information of data to be transmitted corresponding to the DMA connection identification, and transmits the target DMA connection identification data to a logic layer of the data receiving party;
and receiving the data to be transmitted by the data transmitting side by using the DMA connection according to the target DMA connection identification data.
8. The utility model provides a DMA data transmission optimizing apparatus in federal study, its characterized in that is applied to the data sender, DMA data transmission optimizing apparatus includes in the federal study:
the DMA connection establishing module is used for sending a DMA connection request to a network layer of a data receiving party and receiving a DMA connection identifier fed back by the data receiving party based on the DMA connection request so as to establish DMA connection;
the building module is used for building target DMA connection identification data based on the DMA connection identification and the data information of the data to be transmitted corresponding to the DMA connection identification;
and the data transmission module is used for sending the target DMA connection identification data to a logic layer of the data receiving party so that the data receiving party can receive the data to be transmitted, which is transmitted by the data transmitting party by using the DMA connection, according to the target DMA connection identification data.
9. The utility model provides a DMA data transmission optimizing apparatus in federal study, its characterized in that is applied to the data receiver, DMA data transmission optimizing apparatus includes in the federal study:
the DMA connection establishing module is used for receiving a DMA connection request of a data sending party through a network layer of the data receiving party so as to establish DMA connection and generate a DMA connection identifier corresponding to the DMA connection request;
the sending module is used for transmitting the DMA connection identifier to a corresponding data sending party through a network layer, so that the data sending party can construct target DMA connection identifier data based on the DMA connection identifier and data information of data to be transmitted corresponding to the DMA connection identifier, and send the target DMA connection identifier data to a logic layer of the data receiving party;
and the data receiving module is used for receiving the data to be transmitted, which is transmitted by the data transmitting party by using the DMA connection, according to the target DMA connection identification data.
10. The utility model provides a DMA data transmission optimizing apparatus in federal study, its characterized in that DMA data transmission optimizing apparatus includes in the federal study: a memory, a processor, and a program stored on the memory for implementing the method for optimizing DMA data transfer in Federal learning,
the memory is used for storing a program for realizing the DMA data transmission optimization method in the federal learning;
the processor is configured to execute a program for implementing the DMA data transfer optimization method in federal learning, so as to implement the steps of the DMA data transfer optimization method in federal learning according to any one of claims 1 to 6 or claim 7.
11. A readable storage medium having a program stored thereon for implementing a DMA data transfer optimization method for federal learning, the program being executed by a processor to implement the steps of the DMA data transfer optimization method for federal learning as claimed in any one of claims 1 to 6 or claim 7.
12. An article of manufacture, being a computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the steps of the method for DMA data transfer optimization in federal learning as claimed in any of claims 1 to 6 or claim 7.
CN202111086414.7A 2021-09-16 2021-09-16 DMA data transmission optimization method in federated learning Active CN113900977B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111086414.7A CN113900977B (en) 2021-09-16 2021-09-16 DMA data transmission optimization method in federated learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111086414.7A CN113900977B (en) 2021-09-16 2021-09-16 DMA data transmission optimization method in federated learning

Publications (2)

Publication Number Publication Date
CN113900977A true CN113900977A (en) 2022-01-07
CN113900977B CN113900977B (en) 2025-05-13

Family

ID=79028570

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111086414.7A Active CN113900977B (en) 2021-09-16 2021-09-16 DMA data transmission optimization method in federated learning

Country Status (1)

Country Link
CN (1) CN113900977B (en)

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103714027A (en) * 2014-01-10 2014-04-09 浪潮(北京)电子信息产业有限公司 Data transmission method and device for direct memory access controller
CN107205033A (en) * 2017-06-02 2017-09-26 南京大学 Method to share RDMA connections without lock mode
CN109391560A (en) * 2017-08-11 2019-02-26 华为技术有限公司 Notifying method, agent node and the computer equipment of network congestion
CN109936510A (en) * 2017-12-15 2019-06-25 微软技术许可有限责任公司 Multipath RDMA transmission
CN110943941A (en) * 2019-12-06 2020-03-31 北京天融信网络安全技术有限公司 Message receiving method, message sending method, network card and electronic equipment
CN111290979A (en) * 2020-03-23 2020-06-16 优刻得科技股份有限公司 Data transmission method, device and system
CN112583655A (en) * 2020-11-02 2021-03-30 北京迈格威科技有限公司 Data transmission method and device, electronic equipment and readable storage medium
CN112822299A (en) * 2021-04-19 2021-05-18 北京易捷思达科技发展有限公司 RDMA (remote direct memory Access) -based data transmission method and device and electronic equipment
CN112948129A (en) * 2021-03-30 2021-06-11 深圳致星科技有限公司 Data transmission optimization method and device based on federal learning and readable storage medium
WO2021135471A1 (en) * 2019-12-31 2021-07-08 腾讯科技（深圳）有限公司 Data transmission method and apparatus, network card and storage medium

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103714027A (en) * 2014-01-10 2014-04-09 浪潮(北京)电子信息产业有限公司 Data transmission method and device for direct memory access controller
CN107205033A (en) * 2017-06-02 2017-09-26 南京大学 Method to share RDMA connections without lock mode
CN109391560A (en) * 2017-08-11 2019-02-26 华为技术有限公司 Notifying method, agent node and the computer equipment of network congestion
CN109936510A (en) * 2017-12-15 2019-06-25 微软技术许可有限责任公司 Multipath RDMA transmission
CN110943941A (en) * 2019-12-06 2020-03-31 北京天融信网络安全技术有限公司 Message receiving method, message sending method, network card and electronic equipment
WO2021135471A1 (en) * 2019-12-31 2021-07-08 腾讯科技（深圳）有限公司 Data transmission method and apparatus, network card and storage medium
CN111290979A (en) * 2020-03-23 2020-06-16 优刻得科技股份有限公司 Data transmission method, device and system
CN112583655A (en) * 2020-11-02 2021-03-30 北京迈格威科技有限公司 Data transmission method and device, electronic equipment and readable storage medium
CN112948129A (en) * 2021-03-30 2021-06-11 深圳致星科技有限公司 Data transmission optimization method and device based on federal learning and readable storage medium
CN112822299A (en) * 2021-04-19 2021-05-18 北京易捷思达科技发展有限公司 RDMA (remote direct memory Access) -based data transmission method and device and electronic equipment

Also Published As

Publication number Publication date
CN113900977B (en) 2025-05-13

Similar Documents

Publication Publication Date Title
CN111580995B (en) Synchronous communication method and system of distributed cloud platform and Internet of things intelligent terminal based on MQTT asynchronous communication scene
CN102377685B (en) Subscription message sending system and subscription message sending method
CN102377686B (en) Message subscription system, method and device
US11500666B2 (en) Container isolation method and apparatus for netlink resource
US20190158634A1 (en) Network packet processing method and apparatus and network server
WO2023020432A1 (en) Computing power request method, computing power allocation method, computing power execution method, and terminal and network-side device
CN105141603A (en) Communication data transmission method and system
CN105635083A (en) Service processing method and service processing system based on server and client architecture
CN103312593B (en) A kind of message distributing system and method
CN116866438B (en) Cross-cluster task scheduling method and device, computer equipment and storage medium
US9319492B2 (en) Method and an apparatus for controlling messages between host and controller
CN118200253A (en) RDMA UD transmission-oriented reliable communication method, electronic equipment and readable medium
CN111190960B (en) A method, device, equipment and medium for distributed file transmission
CN107277780B (en) Broadcast message sending method and device and mobile terminal
CN116346959A (en) A DPU scenario elastic network card efficient implementation method and device
US11977935B2 (en) Method and apparatus for improving message processing efficiency of flash channel controller
US20210328887A1 (en) Method for performing task processing on common service entity, common service entity, apparatus and medium for task processing
CN105580341A (en) Data processing apparatus and data processing method
CN110008032B (en) Communication mode realization method and electronic equipment
CN113900977A (en) DMA data transmission optimization method in federal learning
CN117472844A (en) Multi-chip module and data processing method
CN109831467B (en) Data transmission method, equipment and system
CN104243358B (en) The method and device that PE device softwares load in VCF systems
CN113992740B (en) Middleware based on autonomous control and data transmission method
CN114301989B (en) Information transmission method, device and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant