CN113835637B - Data writing method, device and equipment - Google Patents
Data writing method, device and equipment Download PDFInfo
- Publication number
- CN113835637B CN113835637B CN202111124319.1A CN202111124319A CN113835637B CN 113835637 B CN113835637 B CN 113835637B CN 202111124319 A CN202111124319 A CN 202111124319A CN 113835637 B CN113835637 B CN 113835637B
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- local server
- written
- writing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 53
- 230000006870 function Effects 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 13
- 238000010586 diagram Methods 0.000 description 11
- 238000005516 engineering process Methods 0.000 description 9
- 230000006872 improvement Effects 0.000 description 8
- 230000005540 biological transmission Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 238000004519 manufacturing process Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 229920001296 polysiloxane Polymers 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 239000010979 ruby Substances 0.000 description 1
- 229910001750 ruby Inorganic materials 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data writing method, which comprises the following steps: if the number m of the data blocks of the file to be read is smaller than or equal to the number n of the storage spaces of the storage strips, writing the data of at least two data blocks in the file to be read into a local server; if the number m of the data blocks of the file to be read is greater than the number n of the storage spaces of the storage stripes, writing the data of the first n data blocks in the file to be read into a local server, wherein the storage stripes are positioned on different servers. According to the embodiment of the application, more data blocks of the data required by the local server are written into the local server, so that the performance of the server in reading the data is improved.
Description
The application relates to a divisional application of a Chinese patent application CN111399780A, the application date of the original application is as follows: 3 months and 19 days 2020; the application number is: 202010198337.3; the application is named as follows: a data writing method, device and equipment.
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method, an apparatus, and a device for writing data.
Background
A memory stripe refers to a collection used to store data. The collection may write consecutive data to multiple servers while storing the data, and the data belonging to the same stripe of storage has an association. If a certain server fails, the data of the server is lost, and the lost data can be recovered through other servers in the same storage strip, so that the written data is safer and more reliable. For example, a piece of continuous data is written into a storage stripe, the storage stripe is distributed on 6 servers, if 2 servers fail, the data existing in the servers is lost, and the lost data can be recovered by other servers in the storage stripe.
When the existing method for writing data into the storage stripe is applied, continuous data is written into different servers, and the servers have poor performance when reading the data.
Disclosure of Invention
In view of the above, the embodiments of the present application provide a method, an apparatus, and a device for writing data, which are used for solving the problem of poor data reading performance of a server in the prior art.
The embodiment of the application adopts the following technical scheme:
The embodiment of the application provides a data writing method, which comprises the following steps:
if the number m of the data blocks of the file to be read is smaller than or equal to the number n of the storage spaces of the storage strips, writing the data of at least two data blocks in the file to be read into a local server;
if the number m of the data blocks of the file to be read is greater than the number n of the storage spaces of the storage stripes, writing the data of the first n data blocks in the file to be read into a local server, wherein the storage stripes are positioned on different servers.
The embodiment of the application also provides a data writing device, which comprises:
The writing unit is used for writing the data of at least two data blocks in the file to be read into the local server if the number m of the data blocks of the file to be read is smaller than or equal to the number n of the storage spaces of the storage strip; if the number m of the data blocks of the file to be read is greater than the number n of the storage spaces of the storage stripes, writing the data of the first n data blocks in the file to be read into a local server, wherein the storage stripes are positioned on different servers.
The embodiment of the application also provides a data writing device, which comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute the following means:
The writing unit is used for writing the data of at least two data blocks in the file to be read into the local server if the number m of the data blocks of the file to be read is smaller than or equal to the number n of the storage spaces of the storage strip; if the number m of the data blocks of the file to be read is greater than the number n of the storage spaces of the storage stripes, writing the data of the first n data blocks in the file to be read into a local server, wherein the storage stripes are positioned on different servers.
The above at least one technical scheme adopted by the embodiment of the application can achieve the following beneficial effects:
According to the embodiment of the application, more data blocks of the data required by the local server are written into the local server, so that the performance of the server in reading the data is improved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:
fig. 1 is a flowchart illustrating a method for writing data according to an embodiment of the disclosure;
Fig. 2 is a flow chart of a method for writing data according to a second embodiment of the disclosure;
FIG. 3 is a diagram illustrating a data writing manner of a memory stripe according to the second embodiment of the present disclosure;
FIG. 4 is a diagram illustrating a data writing manner of a memory stripe according to a second embodiment of the present application;
fig. 5 is a schematic structural diagram of a data writing device according to the third embodiment of the present disclosure.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments of the present application and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
The following describes in detail the technical solutions provided by the embodiments of the present application with reference to the accompanying drawings.
Fig. 1 is a flow chart of a data writing method according to an embodiment of the present disclosure, where the flow chart includes:
Step S101, if the number m of the data blocks of the file to be read is smaller than or equal to the number n of the storage spaces of the storage strips, writing the data of at least two data blocks in the file to be read into a local server; if the number m of the data blocks of the file to be read is greater than the number n of the storage spaces of the storage stripes, writing the data of the first n data blocks in the file to be read into a local server, wherein the storage stripes are positioned on different servers.
Fig. 2 is a schematic flow chart of a data writing method according to the second embodiment of the present disclosure, where the schematic flow chart includes:
Step S201, a storage stripe corresponding to the attribute information is generated according to the service requirement, and the storage stripe is divided into a plurality of storage blocks.
In step S201 of the embodiment of the present disclosure, the attribute information includes a type of the storage stripe or a storage space of the storage stripe, and if the service requirement is a specific type of storage stripe, the storage stripe of the attribute information may be a storage stripe of a corresponding type; if the service type is a storage stripe of a specific storage space, the storage stripe of the attribute information may correspond to the storage stripe of the storage space, for example, the service requirement is a storage stripe of 3M production storage space, or the service requirement is a storage stripe of 4+2 type production.
In step S201 of the embodiment of the present specification, the storage blocks are independent storage units, each for storing data. The system is initialized according to the service requirements, and one or more data strips of attribute information can be produced. Taking erasure codes as an example, different 4+2, 8+3 and other strips can be generated in the system according to different data reliability requirements; depending on the different data storage space requirements, a storage stripe of 3M or other storage space may be generated in the system. For example, the 4+2 type storage stripe is distributed to 6 servers, the storage stripe is divided into 6 storage blocks, and if 2 servers fail, the storage blocks distributed to the other 4 servers can be recovered to be written into the failed servers; if the storage space of the storage stripe is 3M and the storage stripe is required to be applied to a 4+2 type storage stripe, the storage space of each storage block can be determined to be 500kb, the dividing position of the storage stripe is determined, and then the storage stripe is divided into 6 storage blocks.
Step S202, if the number m of the data blocks of the file to be read is smaller than or equal to the number n of the storage spaces of the storage strips, writing the data of at least two data blocks in the file to be read into a local server; if the number m of the data blocks of the file to be read is greater than the number n of the storage spaces of the storage stripes, writing the data of the first n data blocks in the file to be read into a local server, wherein the storage stripes are positioned on different servers.
In step S202 of the embodiment of the present specification, it is necessary to place the memory stripe on a different server so that data can be restored by other memory blocks in the same memory stripe when the server fails.
For step S202, it is preferable to write all the data of the file to be read to the local server.
It should be noted that, according to the task requirement, the server to read the file may be designated as a local server, for example, when the server a needs to read the file a, the server a may be designated as a local server.
In the prior art, when writing data into a storage stripe, one storage stripe is written first and then other storage stripes are written, because the storage stripes are placed in different servers, the local servers can only store a part of data required by the local servers, and because the local servers need to read the data, the local servers need to call across the servers during reading, thereby bringing great inconvenience to the reading work of the local servers and possibly causing problems such as network delay, and aiming at the problems, detailed description is made:
In this embodiment, a 4+2 type storage stripe is taken as an example, and a technical solution in the prior art is described, where in order to make the reliability of data better, the storage stripe is distributed on 4 servers, and even if any 2 servers fail, the data will not be lost, and recovery can be performed through the remaining 4 servers. However, in the prior art, when writing continuous data into a storage stripe, integral writing is adopted, that is, when writing continuous data into one storage stripe, writing into other storage stripes until the storage space of the storage stripe is full; or if the memory space of the memory stripe is large enough, writing all the continuous data into one memory stripe.
Because of the continuity of data, the server needs to continuously read during running, but in the prior art, because continuous data is written to different servers, when a local server needs to read the data distributed on other servers later, a network or other transmission means is needed, which may reduce the performance of the whole system, increase the delay of the whole system, and also may cause the consumption of the network and the CPU. It should be noted that, the continuous file read by the local server in this embodiment is a file required by the server.
The present embodiment will further explain the technical solution of the prior art, referring to fig. 3, four consecutive data segments are respectively data segment one: s1d1, S2d1, S3d1, S4d1; and a second data segment: s1d2, S2d2, S3d2, S4d2; and data segment III: s1d3, S2d3, S3d3, S4d3; and a data segment IV: s1d4, S2d4, S3d4, S4d4. The six servers are Client1, client2, client3, client4, client5, and Client6, respectively. The four sections of memory stripes are respectively a Stripe1, a Stripe2, a Stripe3 and a Stripe4. When writing four continuous data into the server, writing S1d1, S2d1, S3d1 and S4d1 of the first data segment into the strip 1; writing S1d2, S2d2, S3d2, S4d2 of the second data segment into the strip 2; writing S1d3, S2d3, S3d3, S4d3 of data segment three to Stripe3; s1d4, S2d4, S3d4, S4d4 of data segment four are written to Stripe4. It can be seen that, in the prior art, when processing continuous data segments, in order to ensure the reliability and safety of data, the continuous data segments are distributed on different servers, if Client1 needs to read data segment one: s1d1, S2d1, S3d1, S4d1, client1 needs to read S2d1, S3d1, S4d1 after S1d1 is read locally, and S2d1, S3d1, S4d1 are distributed in other servers, so that a network or other transmission means is needed during reading, which may reduce the performance of the whole system, increase the delay of the whole system, and may cause the consumption of the network and the CPU.
In this embodiment, a 4+2 type storage stripe is taken as an example, and in order to make the reliability of data better, the storage stripe may be distributed on 4 servers, and even if any 2 servers fail, the data will not be lost, and the data can be recovered by the other 4 servers. The application divides a storage stripe into a plurality of storage blocks for storing data, and writes the data into different storage stripes located at a local server. Specifically, the data is written into the local server, if the storage space of the local server is inconvenient for storing the data, the rest data can be written into the adjacent server. In this case, since the continuous data is written into the local server, the local server does not need to use a network or other transmission means to read the required data when reading the continuous data, thereby improving the performance of the system. It should be noted that, the continuous file read by the local server in this embodiment is the data required by the server.
The embodiment will further describe the technical solution of the present application, referring to fig. 4, four consecutive data segments are respectively data segment one: s1d1, S2d1, S3d1, S4d1; and a second data segment: s1d2, S2d2, S3d2, S4d2; and data segment III: s1d3, S2d3, S3d3, S4d3; and a data segment IV: s1d4, S2d4, S3d4, S4d4. The six servers are Client1, client2, client3, client4, client5, and Client6, respectively. The four sections of memory stripes are respectively a Stripe1, a Stripe2, a Stripe3 and a Stripe4. When writing four continuous data into the server, writing S1d1, S2d1, S3d1 and S4d1 of the first data segment into the Client1; writing S1d2, S2d2, S3d2 and S4d2 of the second data segment into the Client2; writing S1d3, S2d3, S3d3, S4d3 of the data segment three to Client3; s1d4, S2d4, S3d4, S4d4 of data segment four are written to Client4. Wherein, data segment one is the data that Client1 needs, data segment two is the data that Client2 needs, data segment three is the data that Client3 needs, and data segment four is the data that Client4 needs. It can be seen that, in the present application, when writing the continuous data segments S1d1, S2d1, S3d1, S4d1, the continuous data segments S1d1, S2d1, S3d1, S4d1 are written in the Client1 (the Client1 needs to read the continuous data segments S1d1, S2d1, S3d1, S4d1 later), and meanwhile, since the continuous data segments S1d1, S2d1, S3d1, S4d1 are in four memory stripes of Stripe1, stripe2, stripe3, stripe4, if the Client1 fails, the present application can also be recovered by other servers. At this time, client1 can read S2d1, S3d1, S4d1 locally, and does not need to borrow a network or other transmission means, thereby improving the performance of the system.
In addition, in the prior art, a disk array (RAID) technology is adopted, and the technology stores continuous data on a single server, so that the local server is more convenient to read the continuous data, but cannot recover through other servers when the server fails. In the prior art, a multi-copy technology is adopted, the technology is used for avoiding the problem that a single server fails, so that data cannot be recovered, continuous data required by reading can be realized at a local server, but the greatest defect of multiple copies is cost, for example, two identical servers are required to be set up for two copies, and data stored by the two servers are synchronized in real time, and if one server fails, the other server can be applied; three copies require the establishment of three identical servers. It can be seen that the multiple copy technique requires more storage space to be built than the storage stripe of the present application, resulting in high construction costs.
The application only needs to distribute the data which should be distributed on a plurality of servers, more data are written into the local server, and the calling times of the local server data are reduced.
Further, before the step of performing the step of writing the data to the storage block of the corresponding server according to the first service requirement, the step of performing is further performed: and marking the storage blocks. The memory block may include a data block storing data and a check block storing a check code for checking the data stored in the memory stripe. Marking the storage block, which specifically comprises: marking the data block as allocatable and readable and writable; the check block is marked as allocable. For example, referring to fig. 4, S1p1, S2p1, S3p1, S4p1 are parity chunks distributed in Client5, S1p2, S2p2, S3p2, S4p2 bits are parity chunks distributed in Client6, S1d1, S2d1, S3d1, S4d1, etc. are data chunks, S1p1, S2p1, S3p1, S4p1, etc. are parity chunks according to the labeling.
Further, after the step of dividing the memory stripe into a plurality of memory blocks is performed, the method further comprises:
And marking the corresponding functions of the storage blocks in the storage strip according to the types of the predefined storage blocks. The type of the storage block comprises a data block for storing data and a check block for storing a check code, wherein the check code is used for checking the data in the storage strip.
The function corresponding to the storage block mark in the storage stripe specifically comprises the following steps:
marking the data block as allocatable and readable and writable;
the check block is marked as allocable.
When the number m of the data blocks of the file to be read is smaller than or equal to the number n of the storage spaces of the storage strips after the execution, writing the data of at least two data blocks in the file to be read into a local server; if the number m of the data blocks of the file to be read is greater than the number n of the storage spaces of the storage strip, when writing the data of the first n data blocks in the file to be read, writing the data of at least two data blocks into the local server, wherein the method further comprises the following steps:
and updating the check code in the check block according to all the written data. For example, referring to fig. 4, after the data block S1d1 is written to the Client1, check codes are generated in the S1p1 of the Client5 and the S1p2 of the Client6, and when the S1d2 is written to the Client2, since the S1d1 and the S1d2 are in the same storage stripe, the check codes in the S1p1 of the Client5 and the S1p2 of the Client6 are recalculated, and the calculation result is updated to the check codes in the S1p1 of the Client5 and the S1p2 of the Client 6.
Further, if the number m of the data blocks of the file to be read is smaller than or equal to the number n of the storage spaces of the storage strips, writing the data of at least two data blocks in the file to be read into a local server; if the number m of the data blocks of the file to be read is greater than the number n of the storage spaces of the storage strip, when the data of the first n data blocks in the file to be read is written, the data of at least two data blocks are written to the local server, and the method further comprises:
judging whether a pre-input instruction is data which is convenient for a local server to read the file to be read or not;
If the pre-input instruction is judged to be convenient for the local server to read the data of the file to be read, executing the process that if the number m of the data blocks of the file to be read is smaller than or equal to the number n of the storage spaces of the storage strip, and writing the data of at least two data blocks in the file to be read into the local server; if the number m of the data blocks of the file to be read is greater than the number n of the storage spaces of the storage strip, writing the data of at least two data blocks into a local server when writing the data of the first n data blocks in the file to be read;
If the fact that the pre-input instruction is not convenient for the local server to read the data of the file to be read is judged, the data of the file to be read is continuously written into different servers according to the position of the storage strip in the server.
In addition, the scheme can be applied to a distributed storage system, a distributed memory system and a storage system requiring high-reliability data. The storage blocks in the application can be stored in a hard disk, a memory or other storage devices.
According to the embodiment of the application, more data blocks of the data required by the local server are written into the local server, so that the performance of the server in reading the data is improved.
Fig. 5 is a schematic structural diagram of a data writing device according to the third embodiment of the present disclosure, where the schematic structural diagram includes: writing unit 1, generating unit 2, marking unit 3 and updating unit 4, judging unit 5 and executing unit 6.
The writing unit 1 is configured to write data of at least two data blocks in a file to be read into a local server if the number m of the data blocks of the file to be read is less than or equal to the number n of the storage spaces of the storage stripe; if the number m of the data blocks of the file to be read is greater than the number n of the storage spaces of the storage stripes, writing the data of the first n data blocks in the file to be read into a local server, wherein the storage stripes are positioned on different servers.
The generating unit 2 is configured to generate a storage stripe corresponding to attribute information according to a service requirement, and divide the storage stripe into a plurality of storage blocks, where the attribute information includes a type of the storage stripe and a storage space of the storage stripe.
The marking unit 3 is configured to mark the corresponding function for the memory block in the memory stripe according to a predefined memory block type.
Further, the type of the storage block comprises a data block for storing data and a check block for storing a check code, wherein the check code is used for checking the data in the storage stripe.
Further, the marking unit 3 is specifically configured to:
marking the data block as allocatable and readable and writable;
the check block is marked as allocable.
The updating unit 4 is configured to update the check code in the check block according to all the written data.
The judging unit 5 is used for judging whether a pre-input instruction is data which is convenient for the local server to read the file to be read;
the executing unit 6 is configured to execute the step of writing the data of at least two data blocks in the file to be read to the local server if the pre-input instruction is determined to be convenient for the local server to read the data of the file to be read, and if the number m of the data blocks of the file to be read is less than or equal to the number n of the storage spaces of the storage stripe; if the number m of the data blocks of the file to be read is greater than the number n of the storage spaces of the storage strip, writing the data of at least two data blocks into a local server when writing the data of the first n data blocks in the file to be read; if the fact that the pre-input instruction is not convenient for the local server to read the data of the file to be read is judged, the data of the file to be read is continuously written into different servers according to the position of the storage strip in the server.
The embodiment of the application also provides a data writing device, which comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute the following means:
The writing unit is used for writing the data of at least two data blocks in the file to be read into the local server if the number m of the data blocks of the file to be read is smaller than or equal to the number n of the storage spaces of the storage strip; if the number m of the data blocks of the file to be read is greater than the number n of the storage spaces of the storage stripes, writing the data of the first n data blocks in the file to be read into a local server, wherein the storage stripes are positioned on different servers.
According to the embodiment of the application, more data blocks of the data required by the local server are written into the local server, so that the performance of the server in reading the data is improved.
In the 90 s of the 20 th century, improvements to one technology could clearly be distinguished as improvements in hardware (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software (improvements to the process flow). However, with the development of technology, many improvements of the current method flows can be regarded as direct improvements of hardware circuit structures. Designers almost always obtain corresponding hardware circuit structures by programming improved method flows into hardware circuits. Therefore, an improvement of a method flow cannot be said to be realized by a hardware entity module. For example, a programmable logic device (Programmable Logic Device, PLD) (e.g., field programmable gate array (Field Programmable GATEARRAY, FPGA)) is an integrated circuit whose logic functions are determined by user programming of the device. A designer programs to "integrate" a digital system onto a PLD without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, such programming is mostly implemented with "logic compiler (logic compiler)" software, which is similar to the software compiler used in program development and writing, and the original code before being compiled is also written in a specific programming language, which is called hardware description language (Hardware Description Language, HDL), but HDL is not just one, but a plurality of kinds, such as ABEL(Advanced Boolean Expression Language)、AHDL(Altera Hardware Description Language)、Confluence、CUPL(Cornell UniversityProgramming Language)、HDCal、JHDL(Java Hardware Description Language)、Lava、Lola、MyHDL、PALASM、RHDL(Ruby Hardware Description Language), and VHDL (Very-High-SPEED INTEGRATED Circuit Hardware Description Language) and Verilog are currently most commonly used. It will also be apparent to those skilled in the art that a hardware circuit implementing the logic method flow can be readily obtained by merely slightly programming the method flow into an integrated circuit using several of the hardware description languages described above.
The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, application SPECIFIC INTEGRATED Circuits (ASICs), programmable logic controllers, and embedded microcontrollers, examples of controllers include, but are not limited to, the following microcontrollers: ARC625D, atmelAT91SAM, microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic of the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller in a pure computer readable program code, it is well possible to implement the same functionality by logically programming the method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Such a controller may thus be regarded as a kind of hardware component, and means for performing various functions included therein may also be regarded as structures within the hardware component. Or even means for achieving the various functions may be regarded as either software modules implementing the methods or structures within hardware components.
The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functions of each element may be implemented in the same piece or pieces of software and/or hardware when implementing the present application.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and variations of the present application will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the application are to be included in the scope of the claims of the present application.
Claims (13)
1. A method of writing data, the method comprising:
generating storage strips, wherein each storage strip is positioned on a different server;
Writing the data to be written into the server;
Wherein writing the data to be written to the server includes: writing data required by a local server to be written into a storage space formed by different storage strips on the local server; if the storage space of the storage strip on the local server is inconvenient for storing all the data required by the local server to be written, writing the rest data required by the local server to be written into an adjacent server after the storage space of the storage strip on the local server is fully written.
2. The method of claim 1, wherein each of the storage stripes has storage space located at the local server.
3. The method of claim 1, writing data required by a local server to be written to a storage space on the local server that is made up of different storage stripes comprising:
and writing the continuous data to be written into the storage space of different storage strips on the local server.
4. The method of claim 1, wherein the data to be written is a block of data of a file to be read; writing data required by a local server to be written into a storage space formed by different storage strips on the local server comprises:
If the number m of the data blocks of the file to be read is smaller than or equal to the number n of the storage spaces of the storage strips, writing the data of at least two data blocks in the file to be read into the storage spaces of the storage strips on the local server;
If the number m of the data blocks of the file to be read is greater than the number n of the storage spaces of the storage stripes, when the data of the first n data blocks in the file to be read is written, the data of at least two data blocks are written into the storage spaces of the storage stripes on the local server.
5. The method of claim 4, wherein before writing the data required by the local server to be written to the storage space on the local server, the method further comprises:
Judging whether a pre-input instruction is convenient for a local server to read the file to be read or not;
Writing data required by a local server to be written into a storage space formed by different storage strips on the local server comprises:
If the pre-input instruction is convenient for the local server to read the file to be read, the data to be written is written into the storage space of the storage strip on the local server.
6. The method of claim 1, the method further comprising:
dividing each memory stripe into a plurality of memory blocks;
And marking the corresponding functions of the storage blocks in the storage strip according to the types of the predefined storage blocks.
7. The method of claim 6, wherein the type of the memory block includes a data block storing data and a check block storing a check code; the check code is used for checking data in the storage strip.
8. The method of claim 7, marking the memory blocks in the memory stripe with corresponding functions, comprising:
Marking the data block as allocatable and readable and writable;
the check block is marked as allocable.
9. The method of claim 7, the method further comprising:
and updating the check code in the check block according to all the written data.
10. The method of claim 1, generating a memory stripe comprising: and generating a storage strip corresponding to the attribute information, wherein the attribute information comprises the type of the storage strip and the storage space of the storage strip.
11. The method of claim 10, generating a memory stripe corresponding to attribute information comprises: generating a storage strip corresponding to the attribute information according to the service requirement; if the service requirement is a storage stripe of a specific type, the attribute information is of the specific type; and if the service requirement is a storage stripe of a specific storage space, the attribute information is the specific storage space.
12. A data writing apparatus, the apparatus comprising:
The generation unit is used for generating storage strips, and each storage strip is located in a different server;
a writing unit, configured to write data to be written into the server;
Wherein writing the data to be written to the server includes: writing data required by a local server to be written into a storage space formed by different storage strips on the local server; if the storage space of the storage strip on the local server is inconvenient for storing all the data required by the local server to be written, writing the rest data required by the local server to be written into an adjacent server after the storage space of the storage strip on the local server is fully written.
13. A data writing device, the device comprising a memory for storing computer program instructions and a processor for executing the program instructions; wherein the computer program instructions, when executed by the processor, trigger the apparatus to perform the method of any of claims 1 to 11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111124319.1A CN113835637B (en) | 2020-03-19 | 2020-03-19 | Data writing method, device and equipment |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111124319.1A CN113835637B (en) | 2020-03-19 | 2020-03-19 | Data writing method, device and equipment |
| CN202010198337.3A CN111399780B (en) | 2020-03-19 | 2020-03-19 | Data writing method, device and equipment |
Related Parent Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010198337.3A Division CN111399780B (en) | 2020-03-19 | 2020-03-19 | Data writing method, device and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113835637A CN113835637A (en) | 2021-12-24 |
| CN113835637B true CN113835637B (en) | 2024-07-16 |
Family
ID=71432684
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010198337.3A Active CN111399780B (en) | 2020-03-19 | 2020-03-19 | Data writing method, device and equipment |
| CN202111124319.1A Active CN113835637B (en) | 2020-03-19 | 2020-03-19 | Data writing method, device and equipment |
Family Applications Before (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010198337.3A Active CN111399780B (en) | 2020-03-19 | 2020-03-19 | Data writing method, device and equipment |
Country Status (2)
| Country | Link |
|---|---|
| CN (2) | CN111399780B (en) |
| WO (1) | WO2021184901A1 (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111399780B (en) * | 2020-03-19 | 2021-08-24 | 蚂蚁金服（杭州）网络技术有限公司 | Data writing method, device and equipment |
| CN117873789B (en) * | 2024-03-13 | 2024-05-10 | 之江实验室 | Checkpoint writing method and device based on segmentation quantization |
| CN119002830B (en) * | 2024-10-24 | 2025-01-21 | 北京中科睿信科技有限公司 | An equidistant gate superposition method, device and medium based on FPGA |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102566942A (en) * | 2011-12-28 | 2012-07-11 | 华为技术有限公司 | File striping writing method, device and system |
Family Cites Families (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI254854B (en) * | 2004-11-19 | 2006-05-11 | Via Tech Inc | Method and related apparatus for data migration of disk arrays |
| GB0610335D0 (en) * | 2006-05-24 | 2006-07-05 | Oxford Semiconductor Ltd | Redundant storage of data on an array of storage devices |
| US8117388B2 (en) * | 2009-04-30 | 2012-02-14 | Netapp, Inc. | Data distribution through capacity leveling in a striped file system |
| JP2013196276A (en) * | 2012-03-19 | 2013-09-30 | Fujitsu Ltd | Information processor, program and data arrangement method |
| CN103577339B (en) * | 2012-07-27 | 2018-01-30 | 深圳市腾讯计算机系统有限公司 | A kind of date storage method and system |
| CN104881242A (en) * | 2014-02-28 | 2015-09-02 | 中兴通讯股份有限公司 | Data writing method and data writing device |
| WO2016051512A1 (en) * | 2014-09-30 | 2016-04-07 | 株式会社日立製作所 | Distributed storage system |
| CN105242879B (en) * | 2015-09-30 | 2018-11-06 | 华为技术有限公司 | A kind of date storage method and protocol server |
| CN107885612B (en) * | 2016-09-30 | 2020-02-21 | 华为技术有限公司 | Data processing method, system and device |
| CN108008909B (en) * | 2016-10-31 | 2019-11-22 | 杭州海康威视数字技术股份有限公司 | A kind of date storage method, apparatus and system |
| JP2018116526A (en) * | 2017-01-19 | 2018-07-26 | 東芝メモリ株式会社 | Storage control apparatus, storage control method and program |
| WO2019080015A1 (en) * | 2017-10-25 | 2019-05-02 | 华为技术有限公司 | Data reading and writing method and device, and storage server |
| CN111095217B (en) * | 2017-11-13 | 2024-02-06 | 清华大学 | Data storage system based on RAID mechanism with globally shared resources |
| CN110058961B (en) * | 2018-01-18 | 2023-05-05 | 伊姆西Ip控股有限责任公司 | Method and apparatus for managing storage system |
| JP2019159416A (en) * | 2018-03-07 | 2019-09-19 | Ｎｅｃソリューションイノベータ株式会社 | Data management device, file system, data management method, and program |
| CN109814807B (en) * | 2018-12-28 | 2022-05-06 | 曙光信息产业(北京)有限公司 | Data storage method and device |
| CN110347340A (en) * | 2019-07-05 | 2019-10-18 | 北京谷数科技有限公司 | A kind of method and apparatus improving storage system RAID performance |
| CN111399780B (en) * | 2020-03-19 | 2021-08-24 | 蚂蚁金服（杭州）网络技术有限公司 | Data writing method, device and equipment |
-
2020
- 2020-03-19 CN CN202010198337.3A patent/CN111399780B/en active Active
- 2020-03-19 CN CN202111124319.1A patent/CN113835637B/en active Active
- 2020-12-29 WO PCT/CN2020/140632 patent/WO2021184901A1/en active Application Filing
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102566942A (en) * | 2011-12-28 | 2012-07-11 | 华为技术有限公司 | File striping writing method, device and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111399780B (en) | 2021-08-24 |
| CN113835637A (en) | 2021-12-24 |
| WO2021184901A1 (en) | 2021-09-23 |
| CN111399780A (en) | 2020-07-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110008224B (en) | Database transaction processing method and device | |
| CN107239324B (en) | Service flow processing method, device and system | |
| CN107220148B (en) | Reconstruction method and device for redundant array of independent disks | |
| CN113835637B (en) | Data writing method, device and equipment | |
| CN108459898B (en) | Resource recovery method and device | |
| CN107577697B (en) | Data processing method, device and equipment | |
| CN108628688B (en) | Message processing method, device and equipment | |
| US10592337B1 (en) | Systems and methods for distributing information across distributed storage devices | |
| CN109033127B (en) | Synchronous data verification method, device and equipment | |
| CN111459724B (en) | Node switching method, device, equipment and computer readable storage medium | |
| CN110807013B (en) | Data Migration Method and Device for Distributed Data Storage Cluster | |
| CN109284066B (en) | Data processing method, device, equipment and system | |
| CN115391337A (en) | Database partitioning method and device, storage medium and electronic equipment | |
| CN110263050B (en) | Data processing method, device, equipment and storage medium | |
| CN108647112B (en) | Data backup method and device and distributed transaction processing system | |
| CN109388764A (en) | A kind of update method of local cache, device, equipment and system | |
| CN116541050A (en) | Method, device, equipment and medium for upgrading embedded software | |
| CN110119381A (en) | A kind of index updating method, device, equipment and medium | |
| CN110633321B (en) | Data synchronization method, device and equipment | |
| CN110244964B (en) | Operation and maintenance method, device and equipment based on operation and maintenance application | |
| CN112749152A (en) | Data migration method and device of intelligent contract and storage medium | |
| CN110502551A (en) | Data read-write method, system and infrastructure component | |
| CN107239270B (en) | Code processing method and device | |
| CN113485713B (en) | Method and device for quickly compiling program, electronic equipment and storage medium | |
| CN107391223B (en) | File processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |