CN118138597A - Multi-subnet distributed database data transmission method, device, equipment and medium - Google Patents

Abstract

The embodiments of the present disclosure disclose a multi-subnet distributed database data transmission method, device, equipment and medium. A specific implementation of the method includes: determining a seed node in a node set in a target subnet to obtain a seed node set; using the seed node set to connect to the target subnet; in response to detecting that there is a data transmission task in the target subnet, using the seed node to complete the data transmission task. This implementation can avoid single point failures, reduce the loss of monitoring information, break through single point bottlenecks, and increase the robustness of the system.

Description

Multi-subnet distributed database data transmission method, device, equipment and medium

Technical Field

Embodiments of the present disclosure relate to the field of computer technology, and in particular to a multi-subnet distributed database data transmission method, device, electronic device, and computer-readable medium.

Background technique

Distributed database systems or other distributed applications generally have multiple nodes that are linked together through the network to work together to handle daily affairs, such as SQL execution. The number of these nodes ranges from a few to thousands or even tens of thousands, and the distribution is often complex and may span multiple subnets. Due to the needs of daily monitoring and operation, it is necessary to collect monitoring indicators of each node (such as CPU, memory usage), and report them to the monitoring center for storage (usually Prometheus or other similar storage) in some way for later display and alarm.

However, the existing solution has a single point bottleneck. Monitoring information is often time-series. Each indicator, each node, and each time interval (such as 10 seconds) will generate a piece of data. Therefore, the amount of data monitored by a single node is already quite objective. If there are thousands of nodes or more, the Leader node may become a bottleneck of the network. When nodes are added or deleted within the subnet, or when the network is unstable, unstable phenomena such as brain split may occur. If the Leader node fails for some reason, it will trigger a re-election. During the re-election process, monitoring information is easily lost (when there is no cache on the node)

Summary of the invention

The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

Some embodiments of the present disclosure propose a multi-subnet distributed database data transmission method, device, electronic device and computer-readable medium to solve the technical problems mentioned in the above background technology section.

In a first aspect, some embodiments of the present disclosure provide a multi-subnet distributed database data transmission method, the method comprising: determining a seed node in a node set in a target subnet to obtain a seed node set; using the above-mentioned seed node set to connect with the above-mentioned target subnet; in response to detecting the existence of a data transmission task in the above-mentioned target subnet, using the above-mentioned seed node to complete the above-mentioned data transmission task.

In a second aspect, some embodiments of the present disclosure provide a multi-subnet distributed database data transmission device, the device comprising: a determination unit, configured to determine a seed node from a set of nodes in a target subnet to obtain a seed node set; a connection unit, configured to connect to the target subnet using the seed node set; a transmission unit, configured to complete the data transmission task using the seed node in response to detecting the presence of a data transmission task in the target subnet.

In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by one or more processors, the one or more processors implement a method as described in any implementation method in the first aspect.

In a fourth aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

One of the above-mentioned embodiments of the present disclosure has the following beneficial effects: the present invention first selects some nodes as seed nodes in each subnet through certain rules. These nodes can access the network outside the subnet, and broadcasts the seed node information to each node in the subnet. At each node that needs to report monitoring information, each time the information is reported, one of them is randomly selected from the seed node list as an agent to send the monitoring information. The seed node forwards the monitoring information after receiving it, thereby avoiding single point failures, reducing the loss of monitoring information, breaking through single point bottlenecks, and increasing the robustness of the system.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

FIG1 is a flow chart of some embodiments of a multi-subnet distributed database data transmission method according to the present disclosure;

FIG2 is a schematic diagram of the structure of some embodiments of a multi-subnet distributed database data transmission device according to the present disclosure;

FIG. 3 is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

Referring to FIG1 , a process 100 of some embodiments of the multi-subnet distributed database data transmission method according to the present disclosure is shown. The multi-subnet distributed database data transmission method comprises the following steps:

Step 101, determining a seed node in a node set in a target subnet to obtain a seed node set.

In some embodiments, the execution subject (eg, server) of the multi-subnet distributed database data transmission method can determine the seed node in the node set in the target subnet to obtain the seed node set. Specifically, the target subnet usually refers to the subnet that needs to perform data transmission.

Here, the execution subject may select nodes in the target subnet that have external communication capabilities outside the target subnet and content communication capabilities within the target subnet as seed nodes.

It is understandable that the multi-subnet distributed database data transmission method can be executed by a terminal device, or it can also be executed by a server. The execution subject of the above method can also include a device formed by integrating the above terminal device and the above server through a network, or it can also be executed by various software programs. Among them, the terminal device can be various electronic devices with information processing capabilities, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers and desktop computers, etc. The execution subject can also be embodied as a server, software, etc. When the execution subject is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here.

Step 102: Use the seed node set to connect to the target subnet.

In some embodiments, the execution entity (eg, a server) may utilize the seed node set to connect to the target subnet.

Specifically, the execution subject may perform data transmission with the nodes in the target subnet through the seed node set.

In some optional implementation methods of some embodiments, the above-mentioned execution entity may determine the remaining nodes in the above-mentioned target subnet as subnet nodes to obtain a subnet node set; for each subnet node in the above-mentioned subnet node set, use the above-mentioned seed node set to send the seed node information corresponding to the seed node set to the above-mentioned subnet node; control the subnet nodes in the above-mentioned subnet node set to transmit data with the above-mentioned seed node set according to the above-mentioned seed node information.

Here, the above seed node information generally refers to a seed node list.

Step 103, in response to detecting that there is a data transmission task in the target subnet, using the seed node to complete the data transmission task.

In some embodiments, in response to detecting that there is a data transmission task in the target subnet, the execution entity may use the seed node to complete the data transmission task.

Specifically, the above data transmission task can be unidirectional transmission or bidirectional transmission.

In some optional implementations of some embodiments, the execution subject may determine a target node in the target subnet according to the data transmission task. Specifically, the target node generally refers to a node storing target data. The target data corresponding to the data transmission task is determined. Here, the target data generally refers to data to be transmitted.

In response to determining that the target node is a subnet node, controlling the target node to send the target data to a target seed node in the seed node set, and using the target seed node to transmit the target data.

In response to determining that the target node is a seed node, the target node is controlled to transmit the target data.

In some optional implementations of some embodiments, in response to determining that the above-mentioned target node has not sent the above-mentioned target data to the target seed node in the above-mentioned seed node set, the above-mentioned execution entity may reselect a seed node from the above-mentioned seed node set as the target seed node according to the above-mentioned seed node information until the above-mentioned target node sends the above-mentioned target data to the above-mentioned target seed node.

As an example, the execution subject may execute the multi-subnet distributed database data transmission method through the following steps:

1. First, by configuring the seed nodes of each subnet, you can manually intervene to adjust the number of seed nodes. The principle is that the seed nodes can connect to the external network and send data to the monitoring center (such as the execution subject server). And send the seed node list to all nodes in this subnet.

The more available seed nodes there are, the lower the probability of network bottlenecks or single point failures, and the better the system robustness.

2. For each node that needs to report monitoring information, repeat the following steps:

3. Each time a report is submitted, a seed node is randomly selected from the list of available seed nodes.

4. Send monitoring information to the seed node.

5. If successful, wait for the next monitoring information to be sent and repeat step 3.

6. If it fails, mark the current seed node as temporarily unavailable (valid for this transmission), then randomly select the next one from the remaining seed nodes and repeat step 4.

One of the above-mentioned embodiments of the present disclosure has the following beneficial effects: the present invention first selects some nodes as seed nodes in each subnet through certain rules. These nodes can access the network outside the subnet, and broadcasts the seed node information to each node in the subnet. At each node that needs to report monitoring information, each time the information is reported, one of them is randomly selected from the seed node list as an agent to send the monitoring information. The seed node forwards the monitoring information after receiving it, thereby avoiding single point failures, reducing the loss of monitoring information, breaking through single point bottlenecks, and increasing the robustness of the system.

Further referring to FIG. 2 , as an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a multi-subnet distributed database data transmission device, which device embodiments correspond to the method embodiments shown in FIG. 2 , and the device can be specifically applied to various electronic devices.

As shown in Fig. 2, a multi-subnet distributed database data transmission device 200 of some embodiments includes: a determination unit 201, a connection unit 202 and a transmission unit 203. The determination unit 201 is configured to determine a seed node in a node set in a target subnet to obtain a seed node set; the connection unit 202 is configured to connect to the target subnet using the seed node set; the transmission unit 203 is configured to complete the data transmission task using the seed node in response to detecting that there is a data transmission task in the target subnet.

It is understandable that the units recorded in the device 200 correspond to the steps in the method described with reference to Figure 1. Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 200 and the units contained therein, and will not be repeated here.

One of the above-mentioned embodiments of the present disclosure has the following beneficial effects: the present invention first selects some nodes as seed nodes in each subnet through certain rules. These nodes can access the network outside the subnet, and broadcasts the seed node information to each node in the subnet. At each node that needs to report monitoring information, each time the information is reported, one of them is randomly selected from the seed node list as an agent to send the monitoring information. The seed node forwards the monitoring information after receiving it, thereby avoiding single point failures, reducing the loss of monitoring information, breaking through single point bottlenecks, and increasing the robustness of the system.

Referring to FIG3 below, it shows a schematic diagram of the structure of an electronic device (e.g., a server for executing a multi-subnet distributed database data transmission method) 300 suitable for implementing some embodiments of the present disclosure. The electronic device shown in FIG3 is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

As shown in FIG3 , the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input/output (I/O) interface 305 is also connected to the bus 304.

Typically, the following devices may be connected to the I/O interface 305: input devices 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 3 shows an electronic device 300 with various devices, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively. Each box shown in FIG. 3 may represent one device, or may represent multiple devices as needed.

In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

It should be noted that the computer-readable medium described above in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.

The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: determines a seed node in a node set in a target subnet to obtain a seed node set; uses the seed node set to connect to the target subnet; and in response to detecting that a data transmission task exists in the target subnet, uses the seed node to complete the data transmission task.

Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and/or flow chart, and the combination of the square boxes in the block diagram and/or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

The units described in some embodiments of the present disclosure may be implemented by software or hardware. The described units may also be set in a processor, for example, it may be described as: a processor includes a determination unit, a connection unit, and a transmission unit. Among them, the names of these units do not constitute a limitation on the unit itself in some cases. For example, the determination unit may also be described as "a unit that determines a seed node in a set of nodes in a target subnet and obtains a set of seed nodes."

The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims (10)

1. A multi-subnet distributed database data transmission method, comprising: Determine a seed node from a set of nodes in a target subnet to obtain a seed node set; Using the seed node set to connect to the target subnet; In response to detecting that there is a data transmission task in the target subnet, the seed node is used to complete the data transmission task. 2. The method according to claim 1, wherein in response to detecting that a data transmission task exists for a node in the seed node set and/or the subnet node set, using the seed node to complete the data transmission task comprises: Determine a target node in the target subnet according to the data transmission task; Determine the target data corresponding to the data transmission task; In response to determining that the target node is a subnet node, controlling the target node to send the target data to a target seed node in the seed node set, and transmitting the target data using the target seed node; In response to determining that the target node is a seed node, controlling the target node to transmit the target data. 3. The method according to claim 1, wherein the step of connecting to the target subnet using the seed node set comprises: Determine the remaining nodes in the target subnet as subnet nodes to obtain a subnet node set; For each subnet node in the subnet node set, using the seed node set to send seed node information corresponding to the seed node set to the subnet node; The subnet nodes in the subnet node set are controlled to perform data transmission with the seed node set according to the seed node information. 4. The method according to claim 2, wherein the controlling the target node to send the target data to the target seed node in the seed node set comprises: In response to determining that the target node has not sent the target data to the target seed node in the seed node set, a seed node is reselected from the seed node set as the target seed node according to the seed node information until the target node sends the target data to the target seed node. 5. A device for multi-subnet distributed database data transmission, comprising: A determination unit is configured to determine a seed node from a set of nodes in a target subnet to obtain a seed node set; a connecting unit, configured to connect to the target subnet using the seed node set; The transmission unit is configured to complete the data transmission task by using the seed node in response to detecting that there is a data transmission task in the target subnet. 6. The apparatus according to claim 5, wherein the transmission unit is further configured to: Determine a target node in the target subnet according to the data transmission task; Determine the target data corresponding to the data transmission task; In response to determining that the target node is a subnet node, controlling the target node to send the target data to a target seed node in the seed node set, and transmitting the target data using the target seed node; In response to determining that the target node is a seed node, controlling the target node to transmit the target data. 7. The apparatus according to claim 5, wherein the connecting unit is further configured to: Determine the remaining nodes in the target subnet as subnet nodes to obtain a subnet node set; For each subnet node in the subnet node set, using the seed node set to send seed node information corresponding to the seed node set to the subnet node; The subnet nodes in the subnet node set are controlled to perform data transmission with the seed node set according to the seed node information. 8. The apparatus according to claim 5, wherein the transmission unit is further configured to: In response to determining that the target node has not sent the target data to the target seed node in the seed node set, a seed node is reselected from the seed node set as the target seed node according to the seed node information until the target node sends the target data to the target seed node. 9. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4. 10. A computer readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.