CN106383665B - Date storage method and coordination memory node in data-storage system - Google Patents

Abstract

The invention discloses a data storage method and a coordinated storage node in a data storage system, belonging to the field of information technology. The method includes: coordinating the storage node to divide the data with a length of M into X data fragments, and obtaining Y verification fragments; coordinating the storage node to send the start address and length of the i-th data fragment to N The i-th storage node among the storage nodes; when X is greater than N, the coordinating storage node allocates a first storage node for each of the X-N data fragments and Y verification fragments; when X When it is not greater than N, the coordinating storage section allocates a second storage node for each of the Y verification fragments. When the present invention converts data from multi-copy storage mode to EC storage mode for storage, there is no need to send data to a special coordinating storage node, and data distribution is performed by means of a storage node that can be used as a data coordinating node, which greatly reduces the amount of data transmission. The consumption of network resources in the system is saved.

Description

Data storage method and coordinating storage nodes in data storage system

technical field

The invention relates to the field of information technology, in particular to a data storage method and coordinating storage nodes in a data storage system.

Background technique

In order to improve data security and avoid data loss during transmission, redundant methods are often used to store data in distributed storage. Currently, there are mainly two types of redundancy methods: copy redundancy and EC (Erasure code, error correction code) redundancy. Among them, multi-copy redundancy refers to copying original data into multiple copy data, and each copy data is stored on a storage node. EC redundancy is to divide the original data into X pieces of data equally, use the EC algorithm to verify the X pieces of data, obtain Y pieces of verification data, and then store the X+Y pieces in different on the storage node.

Currently the most widely used data storage method is the combination of multi-copy redundancy and EC redundancy. The specific storage process is as follows: the client copies the original data to obtain multiple copy data, stores the multiple copy data on different copy storage nodes, sends any copy data to the EC coordinator node, and the EC coordinator node stores the copy data The data is equally divided into X pieces of data, and the EC algorithm is used to verify the X pieces of data, and Y pieces of verification data are obtained. The EC coordination node stores a piece of data, and then X+Y-1 Shards are sent to different EC storage nodes for storage.

When the data storage method is converted from copy redundancy to EC redundancy, any copy data needs to be sent to the EC coordinating node first, and then the EC coordinating node will send multiple split data including fragment data and fragment verification data The slices are sent to different EC storage nodes for storage. This process transmits a large amount of data, resulting in a large consumption of network resources in the storage system.

Contents of the invention

In order to reduce the consumption of network resources in a storage system when data is stored in an EC storage mode from a multi-copy storage mode, an embodiment of the present invention provides a data storage method and a coordinated storage node in a storage system. Described technical scheme is as follows:

In a first aspect, a data storage method in a storage system is provided, the storage system includes a plurality of storage nodes, and N storage nodes among the plurality of storage nodes store data with a length of M, and the N storage nodes include A coordinated storage node, wherein N is a natural number greater than 1, the method includes:

In the storage system, due to business needs, when the data is stored in the EC storage mode from the multi-copy storage mode to the EC storage mode, the coordinating storage node divides the data with a length of M into X data fragments, and each data fragment has a Start address and length, the start address of each data fragment is the position of the data fragment in the data with length M, and the length of each data fragment is M/X. Based on the obtained X data fragments, the coordinating storage node verifies the X data fragments to obtain Y verification fragments, and the length of each verification fragment is also M/X.

For the i-th data fragment among the X data fragments, the coordinating storage node sends the start address and length of the i-th data fragment to the i-th storage node among the N storage nodes, where i is greater than 1 and a natural number less than or equal to D, where D is the smallest value among X and N.

When X is greater than N, the coordinating storage node assigns each of the X-N data fragments and Y verification fragments to a first storage node, and the first storage node is a plurality of storage nodes except N any storage node.

When X is less than or equal to N, the coordinating storage node allocates a second storage node for each of the Y verification slices, and the second storage node is any one of the multiple storage nodes except the X storage nodes. Storage nodes.

Since the data storage node as a coordinating storage node has a dual function, it can not only store complete data, but also distribute shards including data shards and check shards, so there is no need to send complete data to a dedicated The coordinated storage nodes in the storage system greatly reduce the amount of data transmission in the storage system and save the consumption of network resources in the storage system.

With reference to the first aspect, in the first possible implementation manner of the first aspect, the storage system where the coordinating node is located further includes a view management node, and based on the view management node, the coordinating storage node starts from fragmenting the i-th data When sending the start address and length to the i-th storage node among the N storage nodes, it can first receive the storage node information sent by the view management node. Including storage node ID and so on. The coordinating storage node sends the start address and length of the i-th data fragment to the i-th storage node among the N storage nodes in a targeted manner according to the received storage node information.

In combination with the first aspect, in the second possible implementation of the first aspect, the coordinating storage node will also receive a storage success message, which is sent by the storage node storing X data fragments and Y verification fragments other storage nodes except the coordinating storage node. By receiving the storage success message, the coordinating storage node can know the storage status of each fragment, so that when any fragment is not successfully stored, the fragment is sent to the corresponding storage node for storage in time to ensure data integrity.

In combination with the first possible implementation of the first aspect, in the third possible implementation of the first aspect, in order to enable the coordinating storage node to target the start address and length of the i-th data fragment Specifically, the view management node in the storage system can perform hash calculation on the data with a length of M to obtain a hash value, and according to the hash value and the pre-constructed hash space, determine that the data with a length of M is in the hash The position in the Greek space, and then starting from this position, the N storage nodes in the specified direction are used as storage nodes for storing data with a length of M and storing N pieces of X data fragments and Y checksum fragments. Shard storage nodes.

In a second aspect, a coordinating storage node is provided. The coordinating storage node is located in a storage system, and the storage system includes a plurality of storage nodes and a view management node. N storage nodes in the plurality of storage nodes store information with a length of M data, the coordinating storage node is included in N storage nodes, wherein N is a natural number greater than 1, and the coordinating storage node is used to execute the method performed by the coordinating storage node in the first aspect above, and the view management node uses In performing the method performed by the view management node in the first aspect above.

In a third aspect, a coordinating storage node is provided, the coordinating storage node is located in a storage system, the storage system includes a plurality of storage nodes, and N storage nodes in the plurality of storage nodes store data with a length of M, And the coordinating storage node is included in N storage nodes, wherein N is natural data greater than 1, the coordinating storage node includes a memory, a processor and a bus, and the memory and the processor are directly connected through the bus;

The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to execute the data storage method in the storage system described in the first aspect above.

Specifically, the coordinating storage nodes of the second aspect and the third aspect above may perform the following steps through specific structural units or processors:

Divide the data with a length of M into X data fragments, and verify the X data fragments to obtain Y verification fragments, wherein X and Y are natural numbers, and each data The length of the slice and each check slice is M/X;

Send the start address and length of the i-th data fragment to the i-th storage node among the N storage nodes, where i is a natural number greater than 1 and less than or equal to D, and D is the smallest of X and N value;

When X is greater than N, each of the X-N data fragments and each of the Y verification fragments is allocated a first storage node, and the first storage node is one of the multiple storage nodes except Any storage node other than the N storage nodes.

Specifically, the coordinating storage nodes of the second aspect and the third aspect above may perform the following steps through specific structural units or processors:

Divide the data with a length of M into X data fragments, and verify the X data fragments to obtain Y verification fragments, wherein X and Y are natural numbers, and each data The length of the slice and each check slice is M/X;

Send the start address and length of the i-th data fragment to the i-th storage node among the N storage nodes, where i is a natural number greater than 1 and less than or equal to D, and D is the smallest of X and N value;

When X is not greater than N, a second storage node is assigned to each of the Y verification slices, and the second storage node is the storage nodes except the X storage nodes among the plurality of storage nodes Any storage node other than .

The beneficial effects brought by the technical solution provided by the embodiments of the present invention are:

When the data is stored from the multi-copy storage mode to the EC storage mode, there is no need to send the data to a dedicated coordinating storage node, and the data distribution can be performed with the help of the storage node that can be used as the data coordinating node, which greatly reduces the amount of data transmission and saves Reduced network resource consumption within the system.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained based on these drawings without creative effort.

FIG. 1 is a schematic diagram of a storage system structure provided by an embodiment of the present invention;

Fig. 2 is a flow chart of a data storage method in a storage system provided by another embodiment of the present invention;

Fig. 3 is a schematic diagram of a consistent hash algorithm provided by another embodiment of the present invention;

FIG. 4 is a schematic diagram of the distribution of storage nodes in a storage system according to another embodiment of the present invention;

Fig. 5 is a schematic diagram of a data storage process provided by another embodiment of the present invention;

Fig. 6 is a schematic diagram of a data storage process provided by another embodiment of the present invention;

Fig. 7 is a schematic structural diagram of a coordinating storage node provided by another embodiment of the present invention;

Fig. 8 is a schematic structural diagram of a coordinating storage node provided by another embodiment of the present invention;

Figure 9 is an illustrative computer architecture of a coordinating storage node used in another embodiment of the invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the implementation manner of the present invention will be further described in detail below in conjunction with the accompanying drawings.

Referring to FIG. 1 , it shows a storage system, which includes a view management node 101 , a client 102 and a storage node 103 .

Wherein, the view management node 101 may be a single computing device, or may be a computing cluster composed of multiple computing devices. The view management node is configured to receive the heartbeat information reported by the storage node 103, and update the node status of the storage node 103 according to the heartbeat information reported by the storage node 103. The view management node 101 is also used to allocate storage nodes for data in the client 102 . The view management node 101 also maintains a view of the corresponding relationship between hash values and storage nodes, and the view is used by the client 102 to query data from the storage nodes.

The client 102 may be a smart phone, a tablet computer, a notebook computer, etc., and this embodiment does not specifically limit the product type of the client 102 . The client 102 is used to receive the view pushed by the view management node 101, and query data according to the view.

The storage node 103 is the physical medium for storing data in the storage system, and the storage node will report heartbeat information to the view management node every preset period of time, so that the view management node maintains the state of the storage node.

The view management node 101, the client 102 and the data node 103 may communicate through a wired network or a wireless network, and the client 102 and the data node 103 may communicate through a wired network or a wireless network.

Based on the database system shown in FIG. 1 , an embodiment of the present invention provides a data storage method. The storage system includes a plurality of storage nodes and a view management node, wherein N storage nodes among the plurality of storage nodes store length It is the data of M, and the N storage nodes include a coordinating storage node. Referring to FIG. 2, the method flow provided by the embodiment of the present invention includes:

201. The view management node allocates storage node information for data of length M.

Due to the large number of storage nodes included in the storage system, in order to store data with a length of M in a redundant manner, and to minimize the consumption of network resources in the system during data transmission, in this embodiment, The view management node will also use the storage node selection algorithm to allocate storage nodes for the data whose length is M.

Wherein, the storage node selection algorithm may include a consistent hash algorithm and the like. The principle of the consistent hashing algorithm is as follows: construct a hash space of 0 to 2 ³² -1, form a hash ring from the hash space of 0 to 2 ³² -1, and divide the hash ring into multiple equal intervals, each The difference range of the hash values in the two intervals is the same. FIG. 3 shows a hash ring composed of a hash space. Referring to FIG. 3 , the hash ring is equally divided into 20 intervals, respectively P0-P19 and the difference ranges of the hash values of P0-P19 are the same.

Limited by the performance of the storage system itself, when the storage system stores data with a length of M in a redundant manner, the number of copy data (replicated data with a length of M), the number of data fragments, and the number of data fragments The number of verification fragments obtained by verification is determined. When the storage system described in this embodiment is stored in a redundant manner, the number of copy data is N, the number of data fragments is X, and the number of check fragments is Y, wherein N is greater than A natural number of 1, X and Y are natural numbers.

Specifically, when the view management node adopts the storage node selection algorithm to allocate storage nodes for the data with a length of M, it may include the following steps 2011-2013:

2011. The view management node performs hash calculation on the data with a length of M to obtain a hash value.

The view management node can use the specified hash algorithm to perform hash calculation on the data with a length of M to obtain the hash value. Wherein, the specified hash algorithm includes an addition hash algorithm, a multiplication hash algorithm, a division hash algorithm, a bit operation hash algorithm, a table lookup hash algorithm, and the like.

2012. The view management node determines the position of the data with length M in the hash space according to the hash value.

In this embodiment, each interval in the hash space corresponds to a range of hash values. When the hash value is located in any interval of the hash space, the position of the interval can be used as the data of length M in the position in the hash space. For example, the hash value of the data KEY with a length of M is calculated as k1, and this k1 is located in the interval where the PO is located in Figure 3. Therefore, the position of the PO can be used as the data KEY with a length of M in the hash position in Greek space.

2013. The view management node takes the location as the starting point, and uses N storage nodes in the specified direction as storage nodes for storing data with a length of M.

Wherein, the specified direction may be a clockwise direction starting from the location of the hash value, or a counterclockwise direction starting from the location of the hash value. Since in this embodiment, N copies are used to redundantly store the data of length M, therefore, the view management node needs to allocate N storage nodes in the copy storage mode for the data of length M. Based on the hash value obtained by hashing the data with a length of M, the view management node starts from the location of the hash value, and uses N storage nodes in the specified direction as storage nodes for storing data with a length of M , the selection method of the N storage nodes can be continuous selection, that is, the N continuous storage nodes starting from the location of the hash value in the specified direction are used as storage nodes for storing data with a length of M, or discontinuous Select, that is, N discontinuous storage nodes starting from the location of the hash value in the specified direction as storage nodes for storing data with a length of M.

2014. The view management node takes the location as the starting point, and uses N storage nodes in the specified direction as storage nodes for storing N pieces of X data pieces and Y check pieces.

Based on the hash value obtained by hashing the data with a length of M, the view management node starts from the location of the hash value and uses N storage nodes in the specified direction as storage nodes for X data fragments and Y collations. The storage nodes of N slices in the verification slice, the selection method of N storage nodes for storing X data slices and Y check slices and the selection of N storage nodes for storing data with a length of M In the same way, if the N storage nodes that store data with a length of M are selected in a continuous manner, then the N continuous storage nodes starting from the location of the hash value will be used to store X data fragments and Y checkpoints. N pieces of storage nodes in the inspection shard, if the N storage nodes that store data with a length of M are selected in a discontinuous manner, then the N discontinuous storage nodes starting from the location of the hash value As a storage node that stores N pieces of X data pieces and Y check pieces. No matter which method is used to select, ensure that the N storage nodes storing data with a length of M overlap with the storage nodes storing N pieces of X data fragments and Y verification fragments to the greatest extent. .

Taking the redundant storage mode of 3 copies of data, 4 data fragments, and 2 parity fragments as an example, the process of allocating storage nodes for data with a length of M by the view management node can be referred to the following example.

(1) For a data KEY with a length of M, the view management node performs hash calculation on the data KEY with a length of M to obtain a hash value k1.

(2) According to the hash value k1, the view management node determines the location of the hash value k1 from the hash space, takes this location as the starting point, and uses the three storage nodes in the clockwise direction as the storage length M The storage nodes of the data, referring to Fig. 3, PO, P1, P2 are the storage nodes for storing the data of length M.

(3) According to the hash value k1, the view management node determines the location of the hash value k1 from the hash space, and takes this location as the starting point, and uses the three storage nodes in the clockwise direction as the storage of four data Shards and 3 storage nodes of 2 verification shards, see Figure 3, P0, P1, and P2 are storage nodes for 4 data shards and 3 of 2 verification shards Storage nodes.

The above-mentioned view management node allocates the storage node for the data with a length of M. In fact, it is a virtual storage node where each interval in the hash space is located. This virtual storage node cannot actually store data. Establish a mapping relationship with the physical storage node, and persist the data on the corresponding virtual storage node to the physical storage node. In order to facilitate the management of virtual storage nodes and physical storage nodes, the view management node also maintains a mapping table, which stores the mapping relationship between each interval of the hash space and the physical storage nodes. The number of nodes is smaller than the number of intervals in the hash space, so that each physical storage node actually corresponds to multiple intervals, that is, the data of multiple virtual storage nodes is stored on the same physical storage node. Referring to Figure 3, the hash space is equally divided into 20 intervals from P0 to P19. The storage system includes 4 physical storage nodes, namely physical storage node 1, physical storage node 2, physical storage node 3, and physical storage node 4. Among them, A(P0), E(P4), I(P8), M(P12), Q(P16) are mapped to physical node 1, B(P1), F(P5), J(P9), N (P13), R(P17) are mapped to physical node 2, C(P3), G(P6), K(P10), 0(P14), S(P18) are mapped to physical node 3, D( P4), H (P7), L (P11), P (P15), T (P19) are mapped to physical node 4.

Referring to Figure 4, based on the mapping table maintained by the view management node, when the client needs to query the data KEY1 from the storage system, the client can first calculate the hash value k1 of the KEY1, and determine that the hash value k1 is in the hash space Then, according to the mapping table, look up the physical storage node where the hash value is located, so as to query the data KEY1 from the physical storage node.

202. The view management node sends the storage node information to the coordinating storage node and the client.

Since the storage node for storing data with a length of M and the storage node for storing N pieces of X data fragments and Y checksum fragments are the keys for the client to perform multi-copy storage and coordinate storage nodes for EC storage , in order to enable the client and the coordinating storage node to store in a targeted manner, the view management node can allocate the storage node information of N storage nodes for data with a length of M, and store X data fragments and Y collation The storage node information of the storage nodes of the N pieces in the verification shard is sent to the client and the coordinating storage node. Wherein, the storage node information includes a storage node identifier and the like.

203. The client copies the data with a length of M according to the storage node information, and sends the copied data to N storage nodes for storage.

When receiving the storage node information sent by the view management node, the client copies N copies of the data with a length of M to obtain N data with a length of M, and then sends the N data with a length of M to the allocated N storage Nodes are stored by each storage node.

In order to improve the data access rate, the storage system can convert the data storage mode according to the access volume of the client. When the access volume of the client to any data is greater than a certain value, the storage system can trigger the coordination storage node to convert the data from the copy storage mode. Store for EC storage mode.

204. The coordinating storage node equally divides the data with a length of M into X data fragments, and verifies the X data fragments to obtain Y verification fragments.

Since the storage system in this embodiment uses X data fragments and Y verification fragments for storage in a redundant manner, therefore, in the EC storage mode, the coordinating storage node needs to save the length M The data of is divided into X data fragments equally. Based on the divided X data fragments, the coordinating storage node uses a specified verification algorithm to verify the X data fragments to obtain Y verification fragments. Wherein, the specified check algorithm includes a parity check algorithm, a CRC cyclic redundancy check algorithm, and the like.

205. The coordinating storage node sends the start address and length of the i-th data segment to the i-th storage node among the N storage nodes.

Wherein, i is a natural number greater than 1 and less than or equal to D, and D is the minimum value among X and N. Since data of length M has been stored in N storage nodes, in order to save network resources consumed by data transmission in the system, the coordinating storage node can send fragmentation information to N storage nodes, and each storage node Cut the stored data with a length of M according to the slice information to obtain a data slice and store it. Wherein, the fragmentation information includes the start address and length of the data fragmentation. For the i-th data fragment among the X data fragments, the coordinating storage node sends the start address and length of the i-th data fragment to the i-th storage node among the N storage nodes, and the i-th storage node The node cuts the stored data of length M according to the start address and length of the i-th data fragment, obtains a data fragment corresponding to the start address and length, and stores the data fragment.

206. When X is greater than N, the coordinating storage node allocates a first storage node to each of the X-N data fragments and Y verification fragments respectively.

Wherein, the first storage node is any storage node except the N storage nodes among the plurality of storage nodes. When X is greater than N, it means that the number of data fragments is greater than the number of copies of data with a length of M. At this time, the coordinating storage node needs to allocate one for each of X-N data fragments and Y verification fragments. The first storage node sends the X-N data fragments and Y verification fragments to the allocated first storage node for storage by the first storage node.

207. When X is not greater than N, the coordinating storage node allocates a second storage node for each of the Y check segments.

Wherein, the second storage node is any storage node except the X storage nodes among the plurality of storage nodes. When X is not greater than N, it means that the number of data fragments is less than or equal to the number of copies of data with a length of M. At this time, the coordination storage section needs to allocate a second storage node for each of the Y verification fragments. And send the Y verification fragments to the second storage node for storage by the second storage node.

For the above process, in order to facilitate understanding, a specific example will be used below to illustrate.

Referring to Figure 5, the data size is set to 4M, the storage nodes allocated by the view management node for data with a storage length of 4M are N1, N2, and N3, and the storage nodes allocated for 4 data fragments and 2 check fragments are N1, N2, N3, N4, N5, N6, the coordinating storage node N1 divides the data with a length of 4M into four 1M data fragments, and verifies the four data fragments to obtain 2 verification points piece. When it is necessary to convert the data with a length of M from the multi-copy storage mode to the EC storage mode for storage, the coordinating storage node N1, according to the fragmentation information with a starting address of 0 and a length of 1M, stores the data with a length of 4M The data is cut, and the data fragments are obtained and stored, and the fragmentation information with the starting address of 1M and the length of 1M is sent to the storage node N2, and the fragmentation information with the starting address of 2M and the length of 1M is sent to the storage node N3 , the storage nodes N2 and N3 cut the data with a length of 4M stored in this node to obtain data fragments and store them. For storage nodes N4, N5, and N6, the coordinating storage node N1 can fragment the data with a starting address of 3M and a length of 1M to storage node N4 for storage, and split the checksum with a starting address of 0 and a length of 1M Send to the storage node N5 for storage, and send the check fragment with the starting address of 1M and the length of 1M to the storage node N6 for storage.

In order to enable the client to know the storage request of data in the EC storage mode, the designated storage node will also send a storage success message to the coordinating storage node when it successfully stores each data segment or verification segment. Wherein, the designated storage node is a storage node other than the coordinating storage node among the storage nodes storing X data fragments and Y verification fragments.

Figure 6 shows the process diagram of data storage when the data is converted from the multi-copy storage mode to the EC storage mode. Referring to Figure 6, the view management node allocates storage nodes N1, N2, and N3 for data with a length of M, and there are 4 The storage nodes assigned to data shards and 2 check shards are N1, N2, N3, N4, N5, and N6. When changing from the multi-copy storage mode to the EC storage mode, the coordinating storage node N1 divides the stored data of length M into 4 data fragments, and checks the 4 data fragments to obtain 2 checksums Fragmentation. The coordinating storage node N1N1 cuts the data of length M stored by the node according to the fragmentation information <offset1, len1>, obtains data fragmentation and stores it; and sends the fragmentation information <offset2, len2> to the storage node N2, The storage node N2 cuts the stored data of length M according to <offset2, len2>, and obtains a data fragment and stores it; sends the fragment information <offset3, len3> to N3, and the storage node N3 according to <offset3 , len3> cut the stored data with a length of M to obtain a data fragment and store it; send the data fragment whose fragment information is <offset4, len4> to the storage node N4 for storage, and store the two The verified fragments are sent to storage nodes N5 and N6 for storage respectively.

In the method provided by the embodiment of the present invention, when the data is stored from the multi-copy storage mode to the EC storage mode, it is not necessary to send the data to a special coordinating storage node, but to distribute data by means of a storage node that can be used as a data coordinating node. It greatly reduces the amount of data transmission and saves the consumption of network resources in the system.

Referring to FIG. 7 , an embodiment of the present invention provides a coordinated storage node. The coordinated storage node is located in a storage system, and the storage system includes multiple storage nodes. data, the coordinated storage nodes are included in N storage nodes, where N is a natural number greater than 1, and the coordinated storage nodes include:

A division unit 701, configured to equally divide the data with a length of M into X data fragments, and verify the X data fragments to obtain Y verification fragments, wherein X and Y are natural numbers, and each The length of a data slice and each check slice is M/X;

The sending unit 702 is configured to send the start address and length of the i-th data fragment to the i-th storage node among the N storage nodes, wherein, i is a natural number greater than 1 and less than or equal to D, and D is X and the minimum of N;

The allocating unit 703 is configured to allocate a first storage node to each of the X-N data fragments and Y verification fragments when X is greater than N, and the first storage node is a plurality of storage nodes except Any storage node other than N storage nodes.

In another embodiment of the present invention, the allocating unit 703 is configured to allocate a second storage node for each of the Y verification slices when X is not greater than N, and the second storage node is a plurality of Any storage node except X storage nodes among the storage nodes.

Optionally, in another embodiment of the present invention, the allocation unit 703 is configured to allocate a first storage node to each of the X-N data fragments and Y verification fragments when X is greater than N , the first storage node is any storage node except N storage nodes among the plurality of storage nodes; when X is not greater than N, assign a second storage node to each of the Y verification slices, The second storage node is any storage node except the X storage nodes among the plurality of storage nodes.

Referring to FIG. 8, in another embodiment of the present invention, the storage system further includes a view management node, and the coordinating storage node further includes a receiving unit 704;

The receiving unit 704 is configured to receive storage node information sent by the view management node, where the storage node information includes storage node information that stores data with a length of M;

The sending unit 702 is specifically configured to send the start address and length of the i-th data fragment to the i-th storage node among the N storage nodes according to the storage node information.

In another embodiment of the present invention, the receiving unit 704 is also used to receive a storage success message, and the storage success message is sent by other storage nodes except the coordinating storage node that store X data fragments and Y verification fragments. Storage node sends.

In another embodiment of the present invention, the view management node is used to perform hash calculation on the data with a length of M to obtain a hash value; according to the hash value, determine the position of the data with a length of M in the hash space ;Take this location as the starting point, use the N storage nodes in the specified direction as the storage nodes for storing data with a length of M; take this location as the starting point, use the N storage nodes in the specified direction as the storage nodes for storing X data fragments and Y Verify the storage nodes of N shards in the shard.

To sum up, when the system provided by the embodiment of the present invention converts data from multi-copy storage mode to EC storage mode for storage, it does not need to send data to a dedicated coordinating storage node, but uses the storage system that can be used as a data coordinating node Nodes distribute data, which greatly reduces the amount of data transmission and saves the consumption of network resources in the system.

Referring to Figure 9, an illustrative computer architecture for a coordinating storage node 900 used in one embodiment of the invention is shown. The coordinating storage node 900 is a server or a storage device. The coordinating storage node 900 includes a processor 901 , a memory 902 and a bus 903 , where the processor 901 and the memory 902 are connected through the bus 903 . Optionally, the coordinating storage node 900 also includes a basic input/output system (I/O system) 904 that helps to transmit information between various devices in the computing device and a system for storing operating systems, application programs, and other program modules. mass storage device 905 .

Optionally, the mass storage device 905 is connected to the processor 901 through a mass storage controller (not shown) connected to the bus 903 . The mass storage device 905 and its associated computer-readable media provide non-volatile storage for the coordinating storage node 900 . That is, the mass storage device 905 may include a computer readable medium (not shown) such as a hard disk or CD-ROM drive.

Without loss of generality, such computer-readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technologies, CD-ROM, DVD or other optical storage, tape cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices. Certainly, those skilled in the art know that the computer storage medium is not limited to the above-mentioned ones.

According to various embodiments of the present invention, the coordinating storage node 900 can also run on a remote computer connected to the network through a network such as the Internet. That is, the coordinating storage node 900 can be connected to the network 907 through the network interface unit 906 connected to the bus 903, or in other words, the network interface unit 906 can also be used to connect to other types of networks or remote computer systems (not shown ). Wherein, the network interface unit 906 may be a network card (also referred to as a network interface card) or a host bus adapter or the like.

The computer equipment provided by the embodiment of the present invention, when converting data from the multi-copy storage mode to the EC storage mode for storage, does not need to send the data to a special coordinating storage node, but uses the storage node that can be used as a data coordinating node for data distribution , which greatly reduces the amount of data transmission and saves the consumption of network resources in the system.

It should be noted that when the coordinating storage node provided in the above embodiment stores data, it only uses the division of the above-mentioned functional modules as an example for illustration. The internal structure of the coordination storage node is divided into different functional modules to complete all or part of the functions described above. In addition, the coordinating storage node provided in the above embodiment is based on the same idea as the data storage method embodiment in the data storage system, and its specific implementation process is detailed in the method embodiment, and will not be repeated here.

Those of ordinary skill in the art can understand that all or part of the steps for implementing the above embodiments can be completed by hardware, and can also be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium. The above-mentioned The storage medium mentioned may be a read-only memory, a magnetic disk or an optical disk, and the like.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within range.

Claims (10)

1. A data storage method in a storage system, wherein the storage system includes a plurality of storage nodes, and N storage nodes in the plurality of storage nodes store data with a length of M, and the N storage nodes The storage nodes include a coordination storage node, wherein N is a natural number greater than 1, and the method includes: The coordinating storage node divides the data with a length of M into X data fragments, and verifies the X data fragments to obtain Y verification fragments, wherein X and Y are natural numbers , and the length of each data fragment and each check fragment is M/X; The coordinating storage node sends the start address and length of the i-th data fragment to the i-th storage node among the N storage nodes, where i is a natural number greater than 1 and less than or equal to D, and D is X and the minimum of N; When X is greater than N, the coordinating storage node allocates a first storage node for each of the X-N data fragments and the Y verification fragments, and the first storage node provides Any storage node in the storage nodes except the N storage nodes; When X is not greater than N, the coordinating storage node allocates a second storage node for each of the Y verification fragments, and the second storage node is all but one of the plurality of storage nodes. Any storage node other than the above X storage nodes. 2. The method according to claim 1, wherein the storage system further comprises a view management node, and the coordinating storage node sends the start address and length of the i-th data fragment to the N storage The i-th storage node in the node, including: The coordinating storage node receives the storage node information sent by the view management node, and the storage node information includes information of the storage node storing the data of length M; The coordinating storage node sends the start address and length of the i-th data fragment to the i-th storage node among the N storage nodes according to the storage node information. 3. The method according to claim 1, characterized in that the method further comprises: The coordinating storage node receives a storage success message, and the storage success message is sent by other storage nodes except the coordinating storage node among the storage nodes storing X data fragments and Y verification fragments. 4. The method according to claim 2, characterized in that the method further comprises: The view management node performs hash calculation on the data with a length of M to obtain a hash value; The view management node determines the position of the data with a length of M in the hash space according to the hash value; The view management node takes the position as a starting point, and uses N storage nodes in a specified direction as storage nodes for storing the data whose length is M; The view management node takes the position as a starting point, and uses N storage nodes in a specified direction as storage nodes for storing the X data fragments and the N fragments in the Y verification fragments. 5. A coordinating storage node, characterized in that the coordinating storage node is located in a storage system, and the storage system includes a plurality of storage nodes, and N storage nodes in the plurality of storage nodes store information of length M data, the coordinating storage node is included in the N storage nodes, where N is a natural number greater than 1, and the coordinating storage node includes: A division unit, configured to equally divide the data with a length of M into X data fragments, and verify the X data fragments to obtain Y verification fragments, wherein X and Y are natural numbers , and the length of each data fragment and each check fragment is M/X; A sending unit, configured to send the start address and length of the i-th data fragment to the i-th storage node among the N storage nodes, wherein, i is a natural number greater than 1 and less than or equal to D, and D is the minimum of X and N; An allocation unit, configured to allocate a first storage node to each of the X-N data fragments and the Y verification fragments when X is greater than N, and the first storage node is the Any storage node other than the N storage nodes among the N storage nodes. 6. The coordinating storage node according to claim 5, wherein the allocating unit is further configured to, when X is not greater than N, allocate a first Two storage nodes, the second storage node is any storage node in the plurality of storage nodes except the X storage nodes. 7. The coordinated storage node according to claim 5, wherein the storage system further comprises a view management node, and the coordinated storage node further comprises a receiving unit; The receiving unit is configured to receive storage node information sent by the view management node, where the storage node information includes storage node information that stores the data with a length of M; The sending unit is specifically configured to send the start address and length of the i-th data fragment to the i-th storage node among the N storage nodes according to the storage node information. 8. The coordinating storage node according to claim 5, further comprising a receiving unit configured to receive a storage success message, the storage success message is stored by storing X data fragments and Y verification fragments Other storage nodes in the nodes except the coordinating storage node send. 9. The coordinated storage node according to claim 7, wherein the view management node is configured to perform hash calculation on the data whose length is M to obtain a hash value; according to the hash value, Determine the position of the data with a length of M in the hash space; take the position as a starting point, use N storage nodes in a specified direction as storage nodes for storing the data with a length of M; use the position as a starting point , using N storage nodes in a specified direction as storage nodes for storing N pieces of the X data pieces and the Y check pieces. 10. A coordinating storage node, characterized in that the coordinating storage node is located in a storage system, the storage system includes a plurality of storage nodes, and N storage nodes in the plurality of storage nodes store information with a length of M data, the coordinating storage node is included in the N storage nodes, where N is a natural number greater than 1, the coordinating storage node includes a memory, a processor, and a bus, and the memory and the processor pass through The bus is directly connected; The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to execute the data storage method in the storage system according to any one of claims 1 to 4.