CN111046057B - Data processing method, device, computer equipment and medium of server cluster - Google Patents
Data processing method, device, computer equipment and medium of server cluster Download PDFInfo
- Publication number
- CN111046057B CN111046057B CN201911373509.XA CN201911373509A CN111046057B CN 111046057 B CN111046057 B CN 111046057B CN 201911373509 A CN201911373509 A CN 201911373509A CN 111046057 B CN111046057 B CN 111046057B
- Authority
- CN
- China
- Prior art keywords
- self
- sequence
- identification data
- data
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2272—Management thereof
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Computer And Data Communications (AREA)
Abstract
The disclosure provides a data processing method of a server cluster, which comprises the following steps: an identification request about specified data from a first service deployed to a server cluster is received, and identification data for the identification request is generated. Wherein the identification data comprises: the server cluster comprises a time identifier, a first self-increasing sequence and a second self-increasing sequence, wherein the first self-increasing sequence is generated based on deployment information of a first service in the server cluster. The generated identification data is then sent to the first service to store the specified data in the distributed database by the first service based on the identification data. The present disclosure also provides a data processing apparatus of a server cluster, a computer device, and a computer readable storage medium.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data processing method, apparatus, computer device, and medium for a server cluster.
Background
In the prior art, an identification data scheme of the distributed system is generated, wherein the identification data comprises a time stamp, a machine code and a serial number. For services deployed in the same machine/instance, when time adjustment, clock callback and other conditions are encountered, the same timestamp may be generated, so that the generated identification data has a repeated risk, and the data processing requirement of the distributed system cannot be met.
Disclosure of Invention
In view of this, the present disclosure provides a data processing method, apparatus, computer device and medium for a server cluster.
One aspect of the present disclosure provides a data processing method of a server cluster, including: an identification request about specified data from a first service deployed to a server cluster is received, and identification data for the identification request is generated. Wherein the identification data comprises: the server cluster comprises a time identifier, a first self-increasing sequence and a second self-increasing sequence, wherein the first self-increasing sequence is generated based on deployment information of a first service in the server cluster. The generated identification data is then sent to the first service to store the specified data in the distributed database by the first service based on the identification data.
According to an embodiment of the present disclosure, the generating the identification data for the identification request includes: in one aspect, a time stamp is generated based on time information of a received stamp request. On the other hand, deployment information of the first service in the server cluster is acquired, and then a first self-increasing sequence is generated based on the deployment information. The deployment information comprises: deployment path and machine IP address. Next, a second self-increasing sequence is generated based on the time stamp, the first self-increasing sequence, and the number of stamp data generated. The time stamp, the first self-increasing sequence, and the second self-increasing sequence are combined to obtain the stamp data.
According to an embodiment of the present disclosure, generating the first self-increasing sequence based on the deployment information includes: a primary key for the first service is generated based on a deployment path of the first service in the server cluster and the machine IP address. Then, it is determined whether there is a sequence value for the primary key in the first mapping table. If so, the sequence value is taken as a first self-increasing sequence. If not, obtaining the used maximum first self-increment value, and generating a first self-increment sequence based on the used maximum first self-increment value. Wherein the first mapping table includes: mapping relation from primary key to sequence value.
According to an embodiment of the present disclosure, generating the first self-increment sequence based on the used maximum first self-increment value includes: and adding 1 to the maximum first self-added value to obtain updated maximum first self-added value. Then, it is determined whether there is a primary key in the second mapping table that is the largest first self-added value for the update. If so, the above operations are repeatedly performed. If not, taking the updated maximum first self-increment value as a first self-increment sequence, and recording the mapping relation between the primary key and the first self-increment sequence in a first mapping table and a second mapping table. Wherein the second mapping table includes: mapping relation from sequence value to primary key.
According to an embodiment of the present disclosure, generating the second self-increasing sequence based on the time stamp, the first self-increasing sequence, and the number of generated stamp data includes: the used maximum second self-increment value for the time identifier and the first self-increment sequence is obtained, and then 1 is added to the maximum second self-increment value to obtain a second self-increment sequence.
According to an embodiment of the present disclosure, the above method further includes: after the identification data is sent to the first service, an exception message regarding the identification data from the first service is received. And deleting the mapping relation between the primary key and the first self-increasing sequence aiming at the identification data from the first mapping table and the second mapping table. Then, a used maximum first self-increment value is obtained, and an updated first self-increment sequence is generated based on the used maximum first self-increment value.
According to an embodiment of the present disclosure, generating the updated first self-increment sequence based on the used maximum first self-increment value includes: and adding 1 to the maximum first self-added value to obtain updated maximum first self-added value. It is determined whether there is a primary key in the second mapping table that is the largest first self-added value for the update. If so, the above operations are repeatedly performed. If not, taking the updated maximum first self-increment value as an updated first self-increment sequence, and recording the mapping relationship between the primary key and the updated first self-increment sequence in a first mapping table and a second mapping table. Wherein the second mapping table includes: mapping relation from sequence value to primary key.
According to an embodiment of the present disclosure, the identification data further comprises a sign bit identification. The combining the time stamp, the first self-increasing sequence and the second self-increasing sequence to obtain the stamp data includes: and sequentially splicing the symbol bit identifier, the time identifier, the first self-increasing sequence and the second self-increasing sequence to obtain the identification data.
According to an embodiment of the present disclosure, the above method further includes: for any historical unit time, an identification data capacity for the any historical unit time is determined. Unused identification data for any of the historical unit times is determined based on the identification data capacity and the identification data actually generated for the any of the historical unit times. The unused identification data is then cached.
According to an embodiment of the present disclosure, generating the identification data for the identification request includes: and selecting one unused identification data from the cached unused identification data as the identification data.
Another aspect of the present disclosure provides a data processing apparatus of a server cluster, including: the device comprises a receiving module, a generating module and a sending module. The receiving module is used for receiving an identification request about specified data from a first service deployed on the server cluster. The generation module is used for generating the identification data for the identification request. Wherein the identification data comprises: the server cluster comprises a time identifier, a first self-increasing sequence and a second self-increasing sequence, wherein the first self-increasing sequence is generated based on deployment information of a first service in the server cluster. The sending module is used for sending the identification data to the first service so that the first service can store the specified data into the distributed database based on the identification data.
Another aspect of the present disclosure provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method as described above when executing the program.
Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are configured to implement a method as described above.
Another aspect of the present disclosure provides a computer program comprising computer executable instructions which when executed are for implementing a method as described above.
According to the embodiment of the disclosure, when any service in the server cluster performs data updating on the distributed database, the identification data is acquired from the shared storage space, and then the data updating operation is performed on the distributed database based on the acquired identification data, so that each data updating operation received by the distributed database corresponds to one identification data, the data stored in the distributed database can be globally and uniquely identified and indexed, and the data in the distributed database can be conveniently searched, used, managed and processed through the identification parameters corresponding to each data updating operation. The identification data corresponding to each data comprises a time identifier, a first self-increasing sequence and a second self-increasing sequence, and the time identifier, the first self-increasing sequence and the second self-increasing sequence together ensure global uniqueness of the identification data corresponding to any data in the identification data generated by the server cluster. Even if the same time mark is generated due to time adjustment and clock callback, the double variable of the first self-increasing sequence and the second self-increasing sequence has enough margin to ensure global uniqueness of the mark data, so that the data configuration has orderly and unique mark data, and the follow-up business processing is efficiently, accurately, quickly and flexibly carried out.
Drawings
The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments thereof with reference to the accompanying drawings in which:
FIG. 1 schematically illustrates an exemplary system architecture of a data processing method and apparatus for an application server cluster in accordance with an embodiment of the present disclosure;
FIG. 2 schematically illustrates an example data structure of identification data generated by one processing means;
FIG. 3 schematically illustrates a flow chart of a data processing method of a server cluster according to an embodiment of the disclosure;
FIG. 4A schematically illustrates an example data structure of identification data according to an embodiment of the disclosure;
FIG. 4B schematically illustrates an example flow chart of a data processing process for a server cluster in accordance with an embodiment of the disclosure;
FIG. 5 schematically illustrates a block diagram of a data processing apparatus of a server cluster, according to an embodiment of the disclosure; and
Fig. 6 schematically illustrates a block diagram of a computer device according to an embodiment of the disclosure.
Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.
Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a convention should be interpreted in accordance with the meaning of one of skill in the art having generally understood the convention (e.g., "a system having at least one of A, B and C" would include, but not be limited to, systems having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a formulation similar to at least one of "A, B or C, etc." is used, in general such a formulation should be interpreted in accordance with the ordinary understanding of one skilled in the art (e.g. "a system with at least one of A, B or C" would include but not be limited to systems with a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).
The embodiment of the disclosure provides a data processing method and device for a server cluster. The method may include a receiving process, a generating process, and a transmitting process. In a receiving process, an identification request for specified data from a first service deployed on a server cluster is received. Then, a generation process is performed to generate identification data for the identification request. Wherein the identification data comprises: the server cluster comprises a time identifier, a first self-increasing sequence and a second self-increasing sequence, wherein the first self-increasing sequence is generated based on deployment information of a first service in the server cluster. Then, a transmission process is performed to transmit the generated identification data to the first service to store the specified data in the distributed database based on the identification data by the first service.
FIG. 1 schematically illustrates an exemplary system architecture 100 of a data processing method and apparatus in which a server cluster may be applied, according to an embodiment of the present disclosure. It should be noted that fig. 1 is only an example of a system architecture to which embodiments of the present disclosure may be applied to assist those skilled in the art in understanding the technical content of the present disclosure, but does not mean that embodiments of the present disclosure may not be used in other devices, systems, environments, or scenarios.
As shown in fig. 1, the system architecture 100 according to this embodiment illustrates a distributed Database system architecture, where the distributed Database system (Distributed Database MANAGEMENT SYSTEM, DDBMS) includes servers 110, 120, and 130, where each server may have one or more services deployed, each server having its own Database management system (Database MANAGEMENT SYSTEM, DMS), corresponding to the different databases 111, 121, and 131, respectively, which is a large software that manipulates and manages the databases for creating, using, and maintaining the databases, which performs unified management and control of the databases to ensure the security and integrity of the databases, and where the Database manager performs maintenance work of the databases through the DBMS, as well as accessing the data in the databases through the DBMS. It enables multiple applications and users to build, modify and query databases in different ways at the same time or at different times. The three servers 110, 120 and 130 are connected through a network 140, each server has an own client (not shown in the figure), and a user can perform certain operations on databases in the server corresponding to the client through the client, and also can perform certain operations on databases in other servers in the distributed database system through the client.
The data processing method and device for the server cluster provided by the embodiment of the disclosure can be applied to the distributed database system shown in fig. 1, and an improved data updating operation is performed on the distributed database system shown in fig. 1, so that data stored in the distributed database has globally unique identification and index.
It should be understood that the number of servers and databases in fig. 1 is merely illustrative. There may be any number of servers and databases as desired for an implementation.
In one approach, identification data suitable for use in a distributed system is generated using, for example, a snowflake (snowflake) algorithm. Fig. 2 schematically shows an example data structure of identification data generated in a processing manner. As shown in fig. 2, from left to right, bit 1 is a sign bit. The next 41 bits are a time stamp for saving the millisecond time. Then 10 bits of machine code, each node in the server cluster has a fixed machine code, in this example a 10 bit length machine code supporting deployment of up to 1024 nodes. The last 12 bits are the self-increment sequence number, characterizing the count in milliseconds. Theoretically 409.6 ten thousand identification data per millisecond can be produced per node.
In this processing manner, if conditions such as time adjustment and clock callback are encountered, the same timestamp may be generated, and if the 10-bit machine codes are the same (for example, the service is deployed on the same machine/instance), if the self-increasing serial numbers are the same, the generated identification data has a repeated risk, and cannot meet the data processing requirement of the distributed system.
According to the embodiment of the disclosure, a data processing method of a server cluster capable of coping with special situations such as time adjustment is provided, and the method is described below by way of illustration. It should be noted that the sequence numbers of the respective operations in the following methods are merely representative of the operations for the purpose of description, and should not be construed as representing the order of execution of the respective operations. The method need not be performed in the exact order shown unless explicitly stated.
Fig. 3 schematically illustrates a flow chart of a data processing method of a server cluster according to an embodiment of the disclosure. The method can be applied to a shared storage space of a server cluster, and when one or more services in the server cluster need to execute data updating operation on a distributed database, an identification request can be sent to the shared storage space, and the shared storage space responds to the identification request to generate and feed back globally unique identification data.
As shown in fig. 3, the method may include the following operations S301 to S303.
In operation S301, an identification request for specified data from a first service deployed on a server cluster is received.
The first service may be any service deployed in a server cluster. The specific data may be service data, for example, the service data is data related to a specific service item, an important ring in the service data processing process is to generate identification data corresponding to the service data, further, the service data is put in storage by taking the identification data as an index, and service processing such as backtracking, problem positioning, statistical analysis and the like can be performed on the service data on the basis of the identification data, so as to assist smooth performance of the corresponding service item. In this operation S301, when any service needs to acquire identification data regarding specified data, an identification request needs to be transmitted to the shared storage space.
Then, in operation S302, identification data for the identification request is generated.
The identification data generated in the present operation S302 includes a time identification, a first self-increasing sequence, and a second self-increasing sequence. Wherein the time identifier is used for characterizing the time when the identification data is generated or for characterizing the time when the identification request is received. The first self-increasing sequence is generated based on deployment information of the first service in the server cluster and is used for representing deployment characteristics of the first service. Because the self-increasing sequence mode is adopted, the same service can correspond to different first self-increasing sequences under different conditions, and compared with a fixed machine code, the variation margin of the identification data is increased. The second self-increasing sequence is used to characterize the count of identification data for the same service for the same time period.
Next, the generated identification data is transmitted to the first service to store the designated data in the distributed database based on the identification data by the first service in operation S303.
According to the data processing method of the server cluster, as can be understood by those skilled in the art, when any service in the server cluster performs data update on the distributed database, firstly, identification data is obtained from the shared storage space, and then data update operation is performed on the distributed database based on the obtained identification data, so that each data update operation received by the distributed database corresponds to one identification data, and data stored in the distributed database can be globally and uniquely identified and indexed, and the data in the distributed database can be conveniently searched, used, managed and processed through the identification parameters corresponding to each data update operation. The identification data corresponding to each data comprises a time identifier, a first self-increasing sequence and a second self-increasing sequence, and the time identifier, the first self-increasing sequence and the second self-increasing sequence together ensure global uniqueness of the identification data corresponding to any data in the identification data generated by the server cluster. Even if the same time mark is generated due to time adjustment and clock callback, the double variable of the first self-increasing sequence and the second self-increasing sequence has enough margin to ensure global uniqueness of the mark data, so that the data configuration has orderly and unique mark data, and the follow-up business processing is efficiently, accurately, quickly and flexibly carried out.
According to an embodiment of the present disclosure, the above-described process of generating the identification data for the identification request may include: generating a time identifier, a first self-increasing sequence and a second self-increasing sequence respectively. The time stamp, the first self-increasing sequence, and the second self-increasing sequence are then combined to obtain the stamp data.
The process of generating the time stamp may include, for example: a time stamp is generated based on the time information of the received stamp request. The process of generating the first self-increasing sequence may comprise, for example: and acquiring deployment information of the first service in the server cluster, and then generating a first self-increasing sequence based on the deployment information. Illustratively, the deployment information includes: deployment path and machine IP (internet Protocol) address. The deployment path may be a deployment path of the first service in the server cluster, and the machine IP address may be an IP address of a machine on which the first service is deployed. The process of generating the second self-increasing sequence may comprise, for example: a second self-increasing sequence is generated based on the time stamp, the first self-increasing sequence, and the number of generated stamp data.
The process of generating the first self-increasing sequence is exemplarily described below.
For example, the shared memory space may be configured with a first mapping table and a second mapping table. The first mapping table includes a mapping relationship from the primary key to the sequence value, and the second mapping table includes a mapping relationship from the sequence value to the primary key.
Based on the deployment information, the process of generating the first self-increasing sequence may include: first, a primary key for a first service is generated based on a deployment path of the first service in a server cluster and a machine IP address. For example, the deployment path and the machine IP address may be spliced in a certain order to obtain a primary key for the first service, or the deployment path and/or the machine IP address may be operated by a predetermined algorithm (for example, a hashing algorithm, etc., which is not limited herein), and then the operation result is spliced to obtain the primary key for the first service.
Then, it is determined whether there is a sequence value for the primary key in the first mapping table. If so, the sequence value is used as the first self-increasing sequence to continue to be used, indicating that the first self-increasing sequence for the first service already exists. If not, the first self-increasing sequence aiming at the first service is not existed currently, the maximum first self-increasing value of the shared storage space which is used currently is obtained, and the first self-increasing sequence is generated based on the used maximum first self-increasing value.
For example, the process of generating the first self-increment sequence based on the maximum first self-increment value that has been used may be performed as follows: and adding 1 to the maximum first self-added value to obtain updated maximum first self-added value. Then, it is determined whether there is a primary key in the second mapping table that is the largest first self-added value for the update. If so, the operation is repeatedly executed until the maximum first self-added value of the update which is not used currently is found. If not, the updated maximum first self-increment value is not currently used by any service, the updated maximum first self-increment value is used as a first self-increment sequence, and the mapping relation between the primary key and the first self-increment sequence is recorded in a first mapping table and a second mapping table.
It will be appreciated that the above-described process of checking in the second mapping table may avoid repeated use of the same self-increment value. For example, the self-increment value of the shared memory space is circularly valued in the interval of 1-1024, and when the current maximum first self-increment value is 1024, 1 is added to the maximum first sub-increment value, so as to obtain the updated maximum first self-increment value of 1. The updated maximum first self-increment is most likely being used, and therefore a lookup confirmation in the second mapping table is required until the updated maximum first self-increment of the unrecorded mapping relationship is found as the first self-increment sequence.
After generating the time stamp and the first self-increasing sequence, according to an embodiment of the present disclosure, the generating the second self-increasing sequence based on the time stamp, the first self-increasing sequence, and the number of generated stamp data may include: the used maximum second self-increment value for the time identifier and the first self-increment sequence is obtained, and then 1 is added to the maximum second self-increment value to obtain a second self-increment sequence.
After generating and feeding back the identification data, the first service performs warehousing processing on the specified data based on the identification data, and needs to check whether historical identification data which is the same as the identification data exists in the distributed database or not so as to avoid damage to global uniqueness of the identification data in the distributed database caused by some abnormal factors. Upon determining that the same history identification data exists, an exception message regarding the identification data is sent to the shared memory space to cause the shared memory space to update the identification data. According to an embodiment of the present disclosure, the above method may further include: after the identification data is sent to the first service, an exception message regarding the identification data from the first service is received. And deleting the mapping relation between the primary key and the first self-increasing sequence aiming at the identification data from the first mapping table and the second mapping table. Then, the maximum first self-increment value of the shared memory space which is currently used is obtained, and an updated first self-increment sequence is generated based on the used maximum first self-increment value.
Illustratively, the generating the updated first self-increment sequence based on the used maximum first self-increment value may include: and adding 1 to the maximum first self-added value to obtain updated maximum first self-added value. It is determined whether there is a primary key in the second mapping table that is the largest first self-added value for the update. If so, the above operations are repeatedly performed. If not, taking the updated maximum first self-increment value as an updated first self-increment sequence, and recording the mapping relationship between the primary key and the updated first self-increment sequence in a first mapping table and a second mapping table. Wherein the second mapping table includes: mapping relation from sequence value to primary key. The principle of the process is the same as that of the process of generating the first self-increasing sequence, and the above detailed description is omitted here.
According to an embodiment of the present disclosure, the identification data further comprises a sign bit identification. The combining the time stamp, the first self-increasing sequence and the second self-increasing sequence to obtain the stamp data includes: and sequentially splicing the symbol bit identifier, the time identifier, the first self-increasing sequence and the second self-increasing sequence to obtain the identification data.
According to an embodiment of the present disclosure, the above method further includes: for any historical unit time, an identification data capacity for the any historical unit time is determined. Unused identification data for any of the historical unit times is determined based on the identification data capacity and the identification data actually generated for the any of the historical unit times. The unused identification data is then cached. On this basis, the generating of the identification data for the identification request includes, illustratively: and selecting one unused identification data from the cached unused identification data as the identification data.
The embodiments described above are exemplarily described with reference to fig. 4A and 4B in conjunction with specific examples.
Fig. 4A schematically illustrates an example data structure of identification data according to an embodiment of the present disclosure.
As shown in fig. 4A, the identification data may include, from left to right, a sign bit, a time identification, a first self-increasing sequence, and a second self-increasing sequence. In the example shown in fig. 4A, the sign bit takes 1 bit and the time stamp takes 41 bits, which can be used to preserve millisecond time. The shared storage space used for generating the annotation data is, for example, a Redis cluster, and the first self-increasing sequence is, for example, a Redis self-increasing sequence and occupies 10 bits. The second self-increasing sequence takes up 12 bits. In other examples, the sequence relationship of the parts in the identification data, the data length of the parts, and the like may be set according to actual needs, and are not limited herein.
In a server cluster, when any service needs to perform an update operation for a distributed database, an identification request may be sent to the Redis cluster. In response to the identification request, the Redis cluster may generate identification data as shown in FIG. 4A. In generating the identification data, according to the data processing method of the embodiment of the present disclosure, it is necessary to generate the first self-increasing sequence, unlike the related art in which a fixed machine code is generated for a service.
Fig. 4B schematically illustrates an example flow chart of a data processing procedure of a server cluster according to an embodiment of the disclosure.
In the example shown in fig. 4B, first, logic writing of a process of generating identification data is performed by implementing InitializingBean interfaces of a Spring framework. For example, when the Spring container is initialized at the start-up of the server, the afterPropertiesSet method implementing InitializingBean is called, and in this method, the generation logic writing of the identification data can be performed.
Then, the following operations S401 to S407 are performed.
In operation S401, a deployment path and a machine IP address of a service are acquired.
The deployment path of the service is illustratively obtained by a method System. GetProperty ("catenin. Base"), whose output address is the project deployment path. The machine IP address of the server where the service is located is acquired by the method inetaddress.
In operation S402, a KEY is generated according to the deployment path and the IP address of the currently acquired service.
In operation S403, it is queried whether there is a value corresponding to the KEY in the first mapping table of the Redis cluster, where the value is the sequence value above. If not, operation S404 is performed. If so, operation S407 is performed.
In operation S404, the maximum self-increment value X that has been currently used in the dis cluster is acquired, and an updated maximum self-increment value X' is obtained after adding 1 to X.
In operation S405, whether a primary key corresponding to X' exists is queried in the second mapping table of the dis cluster. If not, operation S406 is performed. If so, operation S404 is repeatedly performed.
In operation S406, taking X ' as the first self-increasing sequence, constructing a mapping relationship from KEY to X ' in the first mapping table, and constructing a mapping relationship from X ' to KEY in the second mapping table.
In operation S407, the value is continuously used as the first self-increasing sequence.
Thus, identification data, which may be referred to as distributed ID (identification), as shown in fig. 4A may be generated based on the first self-increasing sequence, the time stamp, and the second self-increasing sequence.
The self-increment in the description can be carried out by using a Redis self-increment sequence and carrying out increment cycle value in the interval of 1-1024.
It will be appreciated that in order to avoid operational conflicts, it should be ensured that the operations are performed within one atom during the process. For example, redis clusters may use setnx distributed locks and lua scripts to guarantee sequential execution of multiple servers. In addition, in the above procedure, the first mapping table and the second mapping table may be stored in various types of storage spaces such as Redis, mysql, zookeeper, which is not limited herein.
Further, based on the data processing method shown in fig. 4B, when the clock callback condition occurs, since 4096 pieces of identification data can be generated per unit time (for example, 1 ms) in the examples shown in fig. 4A to 4B, if the generated identification data is not repeated at the time of warehousing, the usage can be continued. If the conflict of the identification data occurs, the exception may be captured, the mapping keys (ip+ path, new self-increment) and (new self-increment, ip+ path) used at this time in the first mapping table and the second mapping table are deleted, and the above operation S404 is continuously performed, so as to change the first self-increment sequence in the identification data, and start to generate a new distributed ID.
In another embodiment of the present disclosure, for the case where clock callback is particularly large, the generated distributed IDs that are not used per unit time may be stored in the buffer during the early morning or during a period of relatively small traffic, and one day or a sufficient amount of distributed IDs may be buffered for the system to use in the special case by observing the daily use of the distributed IDs.
It can be appreciated that according to the data processing method of the server cluster in the embodiment of the disclosure, the situation that the distributed IDs are repeated can be rapidly processed.
Fig. 5 schematically illustrates a block diagram of a data processing apparatus of a server cluster according to an embodiment of the disclosure.
As shown in fig. 5, a data processing apparatus 500 of a server cluster may include: a receiving module 510, a generating module 520 and a transmitting module 530.
The receiving module 510 is configured to receive an identification request for specified data from a first service deployed on a server cluster.
The generation module 520 is configured to generate identification data for the identification request. Wherein the identification data comprises: the server cluster comprises a time identifier, a first self-increasing sequence and a second self-increasing sequence, wherein the first self-increasing sequence is generated based on deployment information of a first service in the server cluster.
The transmitting module 530 is configured to transmit the identification data to the first service, so that the first service stores the specified data in the distributed database based on the identification data.
It should be noted that, in the embodiment of the apparatus portion, the implementation manner, the solved technical problem, the realized function, and the achieved technical effect of each module/unit/subunit and the like are the same as or similar to the implementation manner, the solved technical problem, the realized function, and the achieved technical effect of each corresponding step in the embodiment of the method portion, and are not described herein again.
Any number of modules, sub-modules, units, sub-units, or at least some of the functionality of any number of the sub-units according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented as split into multiple modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system-on-chip, a system-on-substrate, a system-on-package, an Application Specific Integrated Circuit (ASIC), or in any other reasonable manner of hardware or firmware that integrates or encapsulates the circuit, or in any one of or a suitable combination of three of software, hardware, and firmware. Or one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be at least partially implemented as computer program modules, which, when executed, may perform the corresponding functions.
For example, any of the receiving module 510, the generating module 520, and the transmitting module 530 may be combined in one module to be implemented, or any of the modules may be split into a plurality of modules. Or at least some of the functionality of one or more of the modules may be combined with, and implemented in, at least some of the functionality of other modules. At least one of the receiving module 510, generating module 520, and transmitting module 530 may be implemented at least in part as hardware circuitry, such as a Field Programmable Gate Array (FPGA), programmable Logic Array (PLA), system-on-chip, system-on-substrate, system-on-package, application Specific Integrated Circuit (ASIC), or by hardware or firmware, such as any other reasonable way of integrating or packaging the circuitry, or in any one of or a suitable combination of three of software, hardware, and firmware. Or at least one of the receiving module 510, the generating module 520 and the transmitting module 530 may be at least partially implemented as computer program modules which, when executed, may perform the corresponding functions.
Fig. 6 schematically shows a block diagram of a computer device adapted to implement the above-described method according to an embodiment of the present disclosure. The computer device illustrated in fig. 6 is merely an example and should not be construed as limiting the functionality and scope of use of embodiments of the present disclosure.
As shown in fig. 6, a computer device 600 according to an embodiment of the present disclosure includes a processor 601 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. The processor 601 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or an associated chipset and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. Processor 601 may also include on-board memory for caching purposes. The processor 601 may comprise a single processing unit or a plurality of processing units for performing different actions of the method flows according to embodiments of the disclosure.
In the RAM 603, various programs and data required for the operation of the apparatus 600 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. The processor 601 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM 602 and/or the RAM 603. Note that the program may be stored in one or more memories other than the ROM 602 and the RAM 603. The processor 601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in the one or more memories.
According to an embodiment of the present disclosure, the device 600 may further include an input/output (I/O) interface 605, the input/output (I/O) interface 605 also being connected to the bus 604. The device 600 may also include one or more of the following components connected to the I/O interface 605: an input portion 606 including a keyboard, mouse, etc.; an output portion 607 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The drive 610 is also connected to the I/O interface 605 as needed. Removable media 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on drive 610 so that a computer program read therefrom is installed as needed into storage section 608.
According to embodiments of the present disclosure, the method flow according to embodiments of the present disclosure may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network through the communication portion 609, and/or installed from the removable medium 611. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by the processor 601. The systems, devices, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.
The present disclosure also provides a computer-readable storage medium that may be embodied in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the apparatus/device/system. The computer-readable storage medium carries one or more programs which, when executed, implement methods in accordance with embodiments of the present disclosure.
According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, the computer-readable storage medium may include ROM 602 and/or RAM 603 and/or one or more memories other than ROM 602 and RAM 603 described above.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Those skilled in the art will appreciate that the features recited in the various embodiments of the disclosure and/or in the claims may be combined in various combinations and/or combinations, even if such combinations or combinations are not explicitly recited in the disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or the claims may be variously combined and/or combined without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.
The embodiments of the present disclosure are described above. These examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be made by those skilled in the art without departing from the scope of the disclosure, and such alternatives and modifications are intended to fall within the scope of the disclosure.
Claims (12)
1. A data processing method for a server cluster, comprising:
Receiving an identification request about specified data from a first service deployed on the server cluster;
Generating identification data for the identification request, wherein the identification data comprises: a time stamp, a first self-increasing sequence, and a second self-increasing sequence, the time stamp being used to characterize one of: the first self-increasing sequence is generated based on deployment information of the first service in the server cluster, wherein the deployment information comprises: a deployment path and a machine IP address, the second self-increasing sequence being generated based on the time identification, the first self-increasing sequence, and the number of generated identification data, the first and second self-increasing sequences being doubly varied to cope with a time adjustment scenario and/or a clock callback scenario; and
Transmitting the identification data to the first service to store the specified data in a distributed database by the first service based on the identification data;
The method further comprises the steps of:
After the identification data is sent to the first service, receiving an exception message about the identification data from the first service;
Deleting a mapping relation between a primary key for the identification data and the first self-increasing sequence from a first mapping table and a second mapping table, wherein the first mapping table comprises: a mapping relationship from the primary key to the sequence value, the second mapping table comprising: mapping relation from the sequence value to the main key;
Obtaining a used maximum first self-increment value; and
Generating an updated first self-increment sequence based on the used maximum first self-increment value.
2. The method of claim 1, wherein the generating identification data for the identification request comprises:
generating the time identifier based on the time information of the identification request;
Acquiring the deployment information of the first service in the server cluster;
generating the first self-increasing sequence based on the deployment information;
Generating the second self-increasing sequence based on the time stamp, the first self-increasing sequence, and the number of stamp data that have been generated; and
And combining the time mark, the first self-increasing sequence and the second self-increasing sequence to obtain the mark data.
3. The method of claim 2, wherein the generating the first self-increasing sequence based on the deployment information comprises:
Generating a primary key for the first service based on the deployment path and the machine IP address;
Determining whether a sequence value for the primary key exists in a first mapping table;
if yes, taking the sequence value as the first self-increasing sequence;
if not, acquiring the used maximum first self-increment value; and
The first self-increasing sequence is generated based on the used maximum first self-increasing value.
4. The method of claim 3, wherein the generating the first self-increment sequence based on the used maximum first self-increment value comprises:
Adding 1 to the maximum first self-added value to obtain an updated maximum first self-added value;
Determining whether there is a primary key for the updated maximum first self-added value in the second mapping table;
if yes, repeating the above operation; and
And if not, taking the updated maximum first self-increment value as the first self-increment sequence, and recording the mapping relationship between the primary key and the first self-increment sequence in the first mapping table and the second mapping table.
5. The method of claim 2, wherein the generating the second self-increasing sequence based on the temporal identification, the first self-increasing sequence, and the number of identification data that have been generated comprises:
Obtaining a used maximum second self-increment value for the time identifier and the first self-increment sequence; and
And adding 1 to the maximum second self-increment value to obtain the second self-increment sequence.
6. The method of claim 1, wherein the generating an updated first self-increment sequence based on the used maximum first self-increment value comprises:
Adding 1 to the maximum first self-added value to obtain an updated maximum first self-added value;
Determining whether there is a primary key for the updated maximum first self-increment value in a second mapping table, the second mapping table comprising: mapping relation from sequence value to main key;
if yes, repeating the above operation; and
And if not, taking the updated maximum first self-increment value as the updated first self-increment sequence, and recording the mapping relationship between the primary key and the updated first self-increment sequence in the first mapping table and the second mapping table.
7. The method of claim 2, wherein the identification data further comprises a sign bit identification;
The combining the time stamp, the first self-increasing sequence, and the second self-increasing sequence to obtain the stamp data includes: and splicing the sign bit identifier, the time identifier, the first self-increasing sequence and the second self-increasing sequence in sequence to obtain the identifier data.
8. The method of claim 1, further comprising:
for any historical unit time, determining an identification data capacity for the any historical unit time;
Determining unused identification data for any historical unit time based on the identification data capacity and the identification data actually generated in the any historical unit time; and
And caching the unused identification data.
9. The method of claim 8, wherein the generating identification data for the identification request comprises:
Selecting one unused identification data from the cached unused identification data as the identification data.
10. A data processing apparatus of a server cluster, comprising:
The receiving module is used for receiving an identification request of a first service deployed on the server cluster on specified data;
A generating module, configured to generate identification data for the identification request, where the identification data includes: a time stamp, a first self-increasing sequence, and a second self-increasing sequence, the time stamp being used to characterize one of: the first self-increasing sequence is generated based on deployment information of the first service in the server cluster, wherein the deployment information comprises: a deployment path and a machine IP address, the second self-increasing sequence being generated based on the time identification, the first self-increasing sequence, and the number of generated identification data, the first and second self-increasing sequences being doubly varied to cope with a time adjustment scenario and/or a clock callback scenario; and
A transmitting module for transmitting the identification data to the first service to store the specified data in a distributed database based on the identification data by the first service;
The data processing apparatus is further configured to:
After the identification data is sent to the first service, receiving an exception message about the identification data from the first service; deleting a mapping relation between a primary key for the identification data and the first self-increasing sequence from a first mapping table and a second mapping table, wherein the first mapping table comprises: a mapping relationship from the primary key to the sequence value, the second mapping table comprising: mapping relation from the sequence value to the main key; obtaining a used maximum first self-increment value; generating an updated first self-increment sequence based on the used maximum first self-increment value.
11. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing when executing the program:
the method of any one of claims 1 to 9.
12. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform:
the method of any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911373509.XA CN111046057B (en) | 2019-12-26 | 2019-12-26 | Data processing method, device, computer equipment and medium of server cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911373509.XA CN111046057B (en) | 2019-12-26 | 2019-12-26 | Data processing method, device, computer equipment and medium of server cluster |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111046057A CN111046057A (en) | 2020-04-21 |
| CN111046057B true CN111046057B (en) | 2024-08-16 |
Family
ID=70240476
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911373509.XA Active CN111046057B (en) | 2019-12-26 | 2019-12-26 | Data processing method, device, computer equipment and medium of server cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111046057B (en) |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111708775B (en) * | 2020-05-21 | 2023-06-27 | 四川虹美智能科技有限公司 | Self-increment ID generation method, device and system |
| CN111694792A (en) * | 2020-05-29 | 2020-09-22 | 中国建设银行股份有限公司 | Identification generation method and device based on snowfly |
| CN111694845A (en) * | 2020-05-29 | 2020-09-22 | 中国建设银行股份有限公司 | Redis-based identifier generation method and device |
| CN111651459A (en) * | 2020-06-08 | 2020-09-11 | 北京首汽智行科技有限公司 | Service ID generation method based on Redis |
| CN113822015B (en) * | 2020-06-16 | 2025-01-14 | 北京沃东天骏信息技术有限公司 | Serial number generation method, device, electronic device and computer readable medium |
| CN111813559A (en) * | 2020-07-23 | 2020-10-23 | 济南大陆机电股份有限公司 | Industrial Internet distributed identification coding method and system |
| CN111953772B (en) * | 2020-08-11 | 2022-11-22 | 北京达佳互联信息技术有限公司 | Request processing method, device, server and storage medium |
| CN113220682B (en) * | 2021-05-08 | 2025-07-22 | 北京猿力未来科技有限公司 | Service data analysis method, data processing method, data analysis system and storage medium |
| CN113297327A (en) * | 2021-05-24 | 2021-08-24 | 建信金融科技有限责任公司 | System and method for generating distributed ID |
| CN113407491B (en) * | 2021-06-10 | 2024-10-29 | 北京猿力未来科技有限公司 | Data processing method and device |
| CN113360172B (en) * | 2021-06-30 | 2024-02-27 | 李建 | Application deployment method, device, computer equipment and storage medium |
| CN114153846A (en) * | 2021-11-29 | 2022-03-08 | 北京无忧创想信息技术有限公司 | A method and device for rapidly generating a unique ID of a distributed system |
| CN114546984A (en) * | 2022-01-18 | 2022-05-27 | 兴业消费金融股份公司 | Identification generation method, apparatus, computer equipment and storage medium |
| CN116401242B (en) * | 2023-02-21 | 2025-10-24 | 超聚变数字技术有限公司 | Unique identifier generation method and computing device |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109697133A (en) * | 2018-12-28 | 2019-04-30 | 广州华多网络科技有限公司 | ID generation method, apparatus and system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103020268B (en) * | 2012-12-26 | 2016-05-04 | 大唐软件技术股份有限公司 | Relevant database sequence number application process and system |
| CN108984639B (en) * | 2018-06-22 | 2021-12-24 | 联想(北京)有限公司 | Data processing method and device for server cluster |
| CN109344162A (en) * | 2018-09-14 | 2019-02-15 | 北京京东金融科技控股有限公司 | Business data processing method and device |
| CN109951541A (en) * | 2019-03-11 | 2019-06-28 | 中国银联股份有限公司 | A serial number generation method and server |
| CN110263565B (en) * | 2019-06-28 | 2025-12-30 | 北京百度网讯科技有限公司 | Methods and apparatus for invoking services |
-
2019
- 2019-12-26 CN CN201911373509.XA patent/CN111046057B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109697133A (en) * | 2018-12-28 | 2019-04-30 | 广州华多网络科技有限公司 | ID generation method, apparatus and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111046057A (en) | 2020-04-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111046057B (en) | Data processing method, device, computer equipment and medium of server cluster | |
| US9135454B2 (en) | Systems and methods for enabling searchable encryption | |
| US8069224B2 (en) | Method, equipment and system for resource acquisition | |
| CA3141329A1 (en) | Request link tracking method and service request processing method | |
| CN107733708B (en) | Equipment parameter configuration method and device, computer equipment and storage medium | |
| US9491238B2 (en) | Rapid client-side component processing based on component relationships | |
| CN114428820B (en) | Distributed data real-time synchronization method, system and data synchronization device | |
| US9684689B2 (en) | Distributed parallel processing system having jobs processed by nodes based on authentication using unique identification of data | |
| CN110083627B (en) | Data processing method, system, computer device and storage medium | |
| CN113934792B (en) | Processing method and device of distributed database, network equipment and storage medium | |
| CN107040576A (en) | Information-pushing method and device, communication system | |
| CN103716384A (en) | Method and device for realizing cloud storage data synchronization in cross-data-center manner | |
| CN113704790A (en) | Abnormal log information summarizing method and computer equipment | |
| US8892582B2 (en) | Method and system for identifying objects of service | |
| CN111031126A (en) | Cluster cache sharing method, system, equipment and storage medium | |
| CN111367869A (en) | Mirror image file processing method and device, storage medium and electronic equipment | |
| EP1704672A1 (en) | Automatic update system and method for using a meta mib | |
| WO2015139565A1 (en) | Heterogeneous logging system management configuration | |
| US8341368B2 (en) | Automatic reallocation of structured external storage structures | |
| US10185735B2 (en) | Distributed database system and a non-transitory computer readable medium | |
| CN113590643B (en) | Data synchronization method, device, equipment and storage medium based on dual-track database | |
| US12430051B2 (en) | Method, electronic device, and computer program product for expanding storage device cluster | |
| CN114265853A (en) | Data processing method, data processing device, storage medium and electronic equipment | |
| CN109254997A (en) | Method of data synchronization, system, computer equipment and readable storage medium storing program for executing | |
| CN104539449A (en) | A fault information processing method and related device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Daxing District, Beijing, 100176 Applicant after: Jingdong Technology Holding Co.,Ltd. Address before: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Daxing District, Beijing, 100176 Applicant before: JINGDONG DIGITAL TECHNOLOGY HOLDINGS Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |