CN116775669A - Data processing method, device, electronic equipment and computer storage medium - Google Patents

Abstract

Embodiments of the present application provide a data processing method, device, electronic equipment and computer storage medium. The method includes: intercepting multiple data operation requests received by the server within the current time period; the data operation requests are used to perform operation data Write operation or deletion operation; count the data inflow rate corresponding to multiple operation data completed in the current time period; the data inflow rate includes at least one of the data write rate and the data deletion rate; based on the data inflow rate to determine the lock release time corresponding to each target data operation request received in the next time period; in this way, the lock release time can be elastically scaled according to the data inflow rate, improving data processing efficiency in high concurrency scenarios.

Description

Data processing methods, devices, electronic equipment and computer storage media

Technical field

The present application relates to the field of computer technology, and in particular, to a data processing method, device, electronic equipment and computer storage medium.

Background technique

There are often problems in data storage: how to maintain the consistency of data deletion and writing; relational database management systems (MYSQL, My Structured Query Language), Structured Query Language Sever (SqlSever), Oracle and other relational databases It can be solved through key association operation; but when users have strict requirements on data reading and writing speed, they need to use a non-relational database. The first thing that comes to mind is the Redis database.

Redis database usually uses the lock mechanism to operate, that is, Redis lock, referred to as lock. When a thread operates on a certain object data, it acquires the lock of the object data. When another thread accesses the object data, if it cannot If the lock is reached, wait for execution. Although this operation method can ensure the consistency of deletion and writing of hierarchical associated data, there are problems such as thread waiting, resulting in low data processing efficiency in high concurrency scenarios.

Contents of the invention

Embodiments of the present application provide a data processing method, device, electronic device, and computer storage medium, which can improve data processing efficiency in high concurrency scenarios.

The technical solution of the embodiment of this application is implemented as follows:

The embodiment of the present application provides a data processing method, which method includes:

Intercept multiple data operation requests received by the server within the current time period; the data operation requests are used to write or delete operation data;

Count the data inflow rate corresponding to multiple operation data completed within the current time period; the data inflow rate includes at least one of a data writing rate and a data deletion rate;

Based on the data inflow rate, determine the release time of the lock corresponding to each target data operation request received within the next time period;

Based on the determined release time of the lock, an operation corresponding to each target data operation request is performed.

In some embodiments, in the case where the data inflow rate includes a data write rate and a data deletion rate, based on the data inflow rate, it is determined that each target data operation request received in the next time period corresponds to The lock release time includes:

Obtain the write release time and delete release time of the lock within the current time period;

Determine a first ratio between the data writing rate and the write release time of the lock in the current time period, and a second ratio between the data deletion rate and the deletion release time of the lock in the current time period;

Based on the first ratio and the second ratio, the release time of the lock corresponding to each target data operation request received within the next time period is determined.

In some embodiments, determining the release time of the lock corresponding to each target data operation request received within the next time period based on the first ratio and the second ratio includes:

Adjust the first ratio and the second ratio so that the first ratio and the second ratio are equal;

Based on the adjusted first ratio and the second ratio, the release time of the lock corresponding to each target data operation request received within the next time period is determined.

In some embodiments, performing an operation corresponding to each target data operation request based on the determined release time of the lock includes:

For each target data operation request, obtain the lock of the target operation data; the target data operation request is used to perform a write operation or delete operation on the target operation data;

Based on the determined release time of the lock, a locking operation is performed on the target operation data, and corresponding operations are performed on the target operation data according to the target data operation request.

In some embodiments, obtaining the lock of target operation data includes:

Obtain the lock of the target operation data from the Redis database; the lock of the target operation data is a lock in the form of key-value, the key value is the identification of the target operation data, and the value value is the target operation The release time of the data lock.

In some embodiments, the method further includes:

The execution process corresponding to each target data operation request is monitored until the monitoring time reaches the determined release time of the lock, and the lock of the target operation data is released.

In some embodiments, the method further includes:

When the listening duration reaches the determined release time of the lock and it is detected that the lock of the target operation data has not been released, a rollback operation is performed on the execution of the target operation data.

Embodiments of the present application provide a data processing device, which includes an interception module, a statistics module, a determination module and an execution module, wherein,

The interception module is used to intercept multiple data operation requests received by the server within the current time period; the data operation requests are used to write or delete operation data;

A statistics module, used to count the data inflow rate corresponding to multiple operation data completed within the current time period; the data inflow rate includes at least one of a data writing rate and a data deletion rate;

A determination module configured to determine, based on the data inflow rate, the release time of the lock corresponding to each target data operation request received within the next time period;

An execution module, configured to execute an operation corresponding to each target data operation request based on the determined release time of the lock.

Embodiments of the present application provide an electronic device. The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, one or more of the aforementioned technologies are implemented. The data processing method provided by the plan.

Embodiments of the present application provide a computer storage medium that stores a computer program; after being executed, the computer program can implement the data processing method provided by one or more of the foregoing technical solutions.

Embodiments of the present application provide a data processing method, device, electronic equipment and computer storage medium. The method includes: intercepting multiple data operation requests received by the server within the current time period; the data operation requests are used to perform operation data Write operation or deletion operation; count the data inflow rate corresponding to multiple operation data completed in the current time period; the data inflow rate includes at least one of the data write rate and the data deletion rate; based on the data inflow rate, determine the release time of the lock corresponding to each target data operation request received within the next time period; based on the determined release time of the lock, perform the operation corresponding to each target data operation request.

It can be seen that in the embodiment of the present application, each data operation request in the current time period is intercepted to count the data inflow rate in the time period, and then dynamically adjust the lock corresponding to each target data operation request in the next time period. The release time of the lock, that is, can achieve elastic expansion and contraction of the lock release time; for high concurrency scenarios, because the lock release time is dynamically adjusted according to the data inflow rate, that is, different data inflow rates correspond to different lock release times, Therefore, it is possible to ensure that data deletion and writing operations are completed as much as possible within the lock release time, ensuring the consistency of data deletion and writing operations while optimizing the thread waiting problem and improving data processing efficiency in high concurrency scenarios.

Description of drawings

Figure 1 is a schematic flow chart of a data processing method provided by an embodiment of the present application;

Figure 2 is a schematic flow chart of writing and deleting operation data provided by an embodiment of the present application;

Figure 3 is a schematic flow chart of another data processing method provided by an embodiment of the present application;

Figure 4 is a schematic structural diagram of a data processing device provided by an embodiment of the present application;

FIG. 5 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

The technical solution in this application will be clearly and completely described below in conjunction with the accompanying drawings in this application.

The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the embodiments provided here are only used to explain the present application and are not used to limit the present application. In addition, the embodiments provided below are partial embodiments for implementing the present application, rather than providing all the embodiments for implementing the present application. The technical solutions described in the present application can be implemented in any combination as long as there is no conflict.

It should be noted that in this application, the terms "comprising", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a method or device including a series of elements not only includes the explicitly stated elements, It also includes other elements that are not explicitly listed or that are intrinsic to the implementation of the method or apparatus. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of other related elements (such as steps in the method or devices) in the method or device including the element. A unit in the device, for example, a unit may be part of a processor, part of a program or software, etc.).

The term "and/or" in this article is just an association relationship that describes related objects, indicating that three relationships can exist. For example, C and/or D can mean: C exists alone, C and D exist simultaneously, and they exist alone. D these three situations. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of C, D, and E, which can mean including from C, Any one or more elements selected from the set composed of D and E.

For example, the data processing method provided by this application includes a series of steps, but the data processing method provided by this application is not limited to the steps described. Similarly, the data processing device provided by this application includes a series of modules, but this application The provided data processing device is not limited to include explicitly stated modules, but may also include modules that are required to obtain relevant information or perform processing based on the information.

In related technology, regarding the meaning of locks, simply speaking, each object data stored by the user in the Redis database has a lock. This lock can only be acquired and consumed by a single thread; when the lock has been acquired, other threads can only Suspended waiting. Normally, the lock release time is fixed, and in order to ensure the accuracy of each thread's task operation, the lock release time is set relatively long. In this way, when there is a lot of object data that needs to be processed, there will be threads waiting. problem, resulting in low data processing efficiency in high concurrency scenarios; in addition, when there is less object data to be processed in the first time period, and more object data is processed in the second time period, if in the second time period Continuing to use the lock release time set in the first time period will result in some object data not being completed after the lock release time is exceeded. Furthermore, there is a risk of data loss and it is difficult to meet the data processing in high concurrency scenarios. Require.

Among them, the implementation of Redis lock is related to the atomicity characteristics of Redis itself. Redis is single-threaded; Redis lock is equivalent to a label of an object data, which is usually represented by key-value. The key value is usually the universal value of the object. Unique identification code (universally unique identifier, uuid), and the value value usually indicates that the Redis lock of the object data has been acquired, and 0 indicates that the Redis lock of the object data can be acquired. The benefit of this Redis lock is that it can perfectly guarantee the atomicity of a single data operation. However, when the stored object data is multiple and has hierarchical or dependency relationships, this method is difficult to guarantee data deletion. consistency.

In view of the above problems, the following embodiments are proposed.

In some embodiments of the present application, the data processing method can be implemented using a processor in a data processing device. The processor can be an Application Specific Integrated Circuit (ASIC) or a Digital Signal Processor (Digital Signal Processor, DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU) ), at least one of a controller, a microcontroller, and a microprocessor.

Figure 1 is a schematic flow chart of a data processing method provided by an embodiment of the present application. As shown in Figure 1, the process may include:

Step 100: Intercept multiple data operation requests received by the server within the current time period.

For example, the user can initiate a data operation request on the client, the client sends the data operation request initiated by the user to the server, and the server can receive the data operation request sent by the client; here, for each data operation request received by the server A data operation request can be intercepted through an interceptor, thereby intercepting multiple data operation requests received by the server within the current time period.

Here, the current time period may be the first time period in which the interceptor performs the interception operation; the length of the current time period may be determined according to the actual scenario, and the embodiments of this application do not limit this; for example, in a high concurrency scenario, The length of the current period is usually less than 1 second.

In the embodiment of the present application, the data operation request is used to perform a write operation or a deletion operation on the operation data; wherein the operation data can be data with a hierarchical association relationship, and the operation data has corresponding identification information, and the identification information can be operation data uuid.

For example, writing or deleting operation data are operations on the database; that is, writing operation data into a certain database or deleting operation data in a certain database; here, for the database The type is not limited. For example, it can be a Redis database or other types of databases.

Here, taking the Redis database as an example, the meaning of operation data with hierarchical relationships is explained in conjunction with Figure 2. Assume that it is currently necessary to write an operation data b into the Redis database. The writing of this data is divided into two parts: the first part, Write an operation data b to the Redis database (corresponding to step (2) in the figure); in the second part, put the identification of the operation data b (such as uuid) into the object data a (corresponding to step (1) in the figure, object data a is a list that can contain multiple uuid similar to operation data b). Similarly, assuming that an operation data b needs to be deleted from the Redis database, the deletion of the data is also divided into two parts: the first part, deleting an operation data b from the Redis database (corresponding to step (4) in the figure); the second part, Delete the uuid of operation data b from object data a (corresponding to step (3) in the figure). At this time, the operation data b and the object data a have a hierarchical association relationship, that is, the operation data b is data with a hierarchical association relationship.

It should be noted that multiple data operation requests correspond to multiple operation data, and the operation data corresponding to different data operation requests may be the same or different. For example, when a certain data operation request is used to write operation data b, another data operation request may be used to delete operation data b or may be used to delete operation data c.

Step 101: Count the data inflow rate corresponding to multiple operation data completed within the current time period.

In the embodiment of this application, after intercepting multiple data operation requests for multiple operation data in the current time period according to the above steps, the data inflow rate corresponding to the multiple operation data completed in the time period is further calculated; here, the data inflow rate The rate may include at least one of a data writing rate and a data deletion rate.

For example, the intercepted multiple data operation requests may include the following three situations: 1) The multiple data operation requests are requests to write multiple operation data; 2) The multiple data operation requests are all requests to write multiple operation data; It is a request to delete multiple operation data; 3) Multiple data operation requests include both requests to write several operation data and requests to delete several operation data.

It can be understood that when counting the multiple operation data corresponding to the above-mentioned first situation, the obtained data inflow rate only includes the data writing rate; when counting the multiple operation data corresponding to the above-mentioned second situation, the obtained The data inflow rate only includes the data deletion rate; when counting the multiple operation data corresponding to the third case above, the obtained data inflow rate includes the data write rate and the data deletion rate.

Step 102: Based on the data inflow rate, determine the release time of the lock corresponding to each target data operation request received in the next time period.

Here, the next time period represents a certain time period after the current time period, and the time period is the same as the current time period.

For example, the target data operation request is used to write or delete the target operation data, and the target operation data is data with a hierarchical association relationship; it should be noted that the target data operation request and the above data operation request are both services. The request received by the server has the same effect. The only difference is that the time period received by the server is different. The target operation data is similar to the above operation data and will not be described again here.

In the embodiment of the present application, the release time of the lock is also called the lock time of the lock, which may include at least one of the write release time and the delete release time of the lock; it can be understood that the content included in the lock release time is based on the data The content included in the inflow rate is determined accordingly.

For example, if the data inflow rate only includes the data write rate, based on the data write rate, the write release of the lock corresponding to each target data operation request received for the write operation in the next time period can be determined time. If the data inflow rate only includes the data deletion rate, based on the data deletion rate, the deletion release time of the lock corresponding to each target data operation request received for the deletion operation in the next time period can be determined. If the data inflow rate includes the data writing rate and the data deletion rate, based on the data writing rate and the data deletion rate, the lock corresponding to each target data operation request for the write operation received in the next time period can be determined. The write release time, and the delete release time of the lock corresponding to each target data operation request for the delete operation. Because when the data inflow rate includes the data writing rate and the data deletion rate, the write release time and delete release time of the lock corresponding to the next time period can be determined at the same time; below, taking this situation as an example, the write release time of the lock Describe how to determine the entry release time and delete release time.

In one embodiment, when the data inflow rate includes a data writing rate and a data deletion rate, the release time of the lock corresponding to each target data operation request received in the next time period is determined based on the data inflow rate. , may include: obtaining the write release time and deletion release time of the lock in the current time period; determining the first ratio of the data write rate and the write release time of the lock in the current time period, and the data deletion rate and the current time period The second ratio of the lock deletion release time; based on the first ratio and the second ratio, determine the lock release time corresponding to each target data operation request received in the next time period.

Here, the write release time and delete release time of the lock in the current time period can be preset according to the actual situation, and their values can be the same or different; the embodiment of the present application does not limit this. For example, it can be Set both values to 5 minutes, or set one to 3 minutes and the other to 5 minutes.

For example, after obtaining the write release time t _write and the delete release time t _delete of the lock in the current time period, further determine the data write rate v _write and the write release time t _delete of the lock in the current time period. a ratio And the second ratio of the data deletion rate v _delete and the lock deletion and release time t _delete in the current time period Then, based on these two ratios, the lock release time corresponding to each target data operation request received in the next time period is determined.

In some embodiments, determining the release time of the lock corresponding to each target data operation request received within the next time period based on the first ratio and the second ratio may include: adjusting the first ratio and the second ratio. , making the first ratio and the second ratio equal; based on the adjusted first ratio and the second ratio, determine the release time of the lock corresponding to each target data operation request received in the next time period.

In the embodiment of the present application, by adjusting the first ratio and the second ratio, the first ratio and the second ratio are equal, that is, the relationship shown in formula (1) is satisfied:

Here, the unit of data writing rate v _write and data deletion rate v _delete is kilobytes per second (KB/s); the unit of lock writing release time t _write and deletion release time t _delete is seconds (s).

For example, during the adjustment process of formula (1), the following conditions need to be met: the values of the first ratio and the second ratio are both greater than 1, and the adjusted lock write release time and lock delete release time are both Greater than the set value; here, the set value can be set according to the actual situation, for example, it can be set to 30s.

For example, according to formula (1), if the data write rate v _write in the current time period increases, the write release time t _write of the lock increases proportionally, that is, the adjusted write write rate of the lock can be obtained Enter the release time; similarly, if the data deletion rate _vdelete in the current time period increases, the lock deletion and release time _tdelete increases proportionally, that is, the adjusted lock deletion and release time can be obtained; after that, The adjusted write release time of the lock is determined as the write release time of the lock corresponding to each target data operation request received for the write operation in the next time period. At the same time, the adjusted lock deletion time is released. The time is determined as the deletion release time of the lock corresponding to each target data operation request received for deletion operation in the next time period.

It should be noted that after intercepting multiple target data operation requests in the next time period, continue to count the data inflow rate corresponding to the multiple target operation data to determine the lock value corresponding to the data operation request in the next time period. Release time, and so on, until the server no longer receives data operation requests from the client.

It can be seen that the embodiment of the present application counts the data inflow rate in a certain time period to control the dynamic expansion and contraction of the lock release time corresponding to each target data operation request in the next time period of the time period; specifically, when a certain time period When the data inflow speed in a period of time is relatively high, the lock release time in the period is adjusted so that the lock release time in the next period of time is relatively extended to ensure that each operation data can be processed within the lock release time. Complete writing or deleting operations within 1 hour, thereby greatly reducing the risk of data loss and thus meeting the data processing requirements in high concurrency scenarios. On the contrary, when the data inflow speed in a certain period of time is small, by adjusting the lock release time in this period of time, the lock release time in the next period of time is relatively shortened. In this way, not only can each operation be ensured The data is written or deleted as much as possible within the lock release time. It can also solve the thread waiting problem and improve the data processing efficiency in high concurrency scenarios.

Step 103: Based on the determined release time of the lock, perform an operation corresponding to each target data operation request.

In some embodiments, based on the determined release time of the lock, performing operations corresponding to each target data operation request may include: for each target data operation request, acquiring a lock on the target operation data; based on the determined lock Within the release time, the target operation data is locked, and corresponding operations are performed on the target operation data according to each target data operation request.

For example, obtaining the lock of the target operation data may include: obtaining the lock of the target operation data from the Redis database; it should be noted that the locks involved in the embodiments of this application are all locks in the form of key-value; wherein, For a lock on operation data, the key value is the identification of the operation data, and the value is the release time of the lock on the operation data; for a lock on target operation data, the key value is the identification of the target operation data, and the value is the release time of the lock on the operation data. Lock release time.

For example, after acquiring the lock of the target operation data, the target operation data can be locked according to the lock release time determined in the above steps, and the target operation data can be locked according to the target data operation request within the lock release time. Perform the corresponding write or delete operation.

In some embodiments, the above method further includes: monitoring the execution process corresponding to each target data operation request until the monitoring time reaches the determined lock release time, and then releasing the lock on the target operation data.

For example, the execution process corresponding to each target data operation request can be monitored through a listening function or a listening program. When the listening time reaches the determined lock release time, the lock on the target operation data will be released; Generally speaking, if there are no exceptions or interruptions in the execution of the target operation data, the lock will be actively released.

Here, after the lock on the target operation data is released, the released lock will be returned to the Redis database.

In some embodiments, the above method may further include: performing a rollback operation on the execution of the target operation data when the listening duration reaches the determined lock release time and it is detected that the lock on the target operation data has not been released.

For example, if the listening time reaches the determined lock release time, but the lock of the target operation data is not released, it means that there may be some abnormality in the execution process of the target operation data. At this time, you can pass the target operation data The execution is rolled back, that is, the execution of the target operation data is rolled back to the state before execution and then the relevant operations are performed again, which can ensure the accuracy of the execution process. Here, the rollback operation can be implemented by adding transaction operations to the data execution process, or by other methods.

It should be noted that if after the rollback operation, it is still detected that the lock of the target operation data has not been released, at this time, the exception needs to be captured and a log printed to remind the user to handle the exception in a timely manner.

Embodiments of the present application provide a data processing method, which method includes: intercepting multiple data operation requests received by the server within the current time period; the data operation requests are used to write or delete operation data; and counting the current time period. The data inflow rate corresponding to multiple operation data completed within the period; the data inflow rate includes at least one of the data writing rate and the data deletion rate; based on the data inflow rate, determine each target data operation request received in the next time period The release time of the corresponding lock; based on the determined release time of the lock, perform the operation corresponding to each target data operation request. It can be seen that in the embodiment of the present application, each data operation request in the current time period is intercepted to count the data inflow rate in the time period, and then dynamically adjust the lock corresponding to each target data operation request in the next time period. The release time of the lock, that is, can achieve elastic expansion and contraction of the lock release time; for high concurrency scenarios, because the lock release time is dynamically adjusted according to the data inflow rate, that is, different data inflow rates correspond to different lock release times, Therefore, it is possible to ensure that data deletion and writing operations are completed as much as possible within the lock release time, ensuring the consistency of data deletion and writing operations while optimizing the thread waiting problem and improving data processing efficiency in high concurrency scenarios.

In order to better reflect the purpose of this application, the embodiments of this application are further described based on the above embodiments.

In high-concurrency scenarios, a large number of access requests (corresponding to the above-mentioned data operation requests or target data operation requests) are usually received, and data writing operations and deletion operations need to be continuously performed on the database; here, take the above Figure 2 as To illustrate, there may be multiple execution orders for steps (1)(2)(3)(4) shown in Figure 2. Ideally, when a write operation and a delete operation for the operation data are received at the same time, When (assuming that the execution of step (1) takes precedence over step (2), and the execution of step (3) takes precedence over step (4)), the optimal step execution order is (1)(2)(3)(4) or (3)(4)(1)(2), so that when the user performs a read operation during the execution of any step, he will not see dirty data; however, in actual scenarios, such step execution sequence (1)( 3)(4)(2) or (3)(1)(2)(4), etc.; it can be seen that in these cases, the operation data that needs to be written is deleted, or the operation data that needs to be deleted Abnormal situations such as not deleting data, and these situations are difficult to troubleshoot and modify online, often causing accidents on the live network.

In order to overcome the above problems, the embodiment of the present application uses an interceptor to obtain multiple data operation requests received by the server within the current time period, and optimizes the Redis lock mechanism in related technologies based on these intercepted data operation requests.

For example, the optimization process of the Redis lock mechanism is as follows. Key-value is also used to maintain locks on object data (corresponding to the above operation data or target operation data). The difference from related technologies is that the value value is not stored as 0 and 1, but Is the release time of the object data lock. A major hidden danger in ordinary Redis locks is that the lock may not be released, and then the object data can never be obtained. Here, the reason why the Redis lock is optimized is to prevent the lock from being actively released.

For example, the situation in which the lock is not actively released may include the following two situations: one is that during the execution of intermediate steps (1)(2)(3)(4), an exception occurs but is not caught, causing the lock to be released. The program is not executed; the other is that the execution of intermediate steps (1)(2)(3)(4) takes too long, causing a large number of tasks to accumulate and overflow the memory. Therefore, time is used here to realize the automatic release of the object data lock, also known as passive release. The process of releasing the lock will be described below with reference to Figure 3.

Figure 3 is a schematic flow chart of another data processing method provided by the embodiment of the present application. As shown in Figure 3, the figure includes an interceptor and a Redis database. The process may include: whenever the server receives a deletion operation of object data or write operation, the lock of the object data will be automatically read from the Redis database. If the lock of the object data is obtained, the object data will be locked, that is, the object data will be locked, and the object data will be locked. The value of the lock is set to System.currentTimeMillis()+t, where System.currentTimeMillis() represents the current time and t represents the lock release time; when the lock release time t is set to 5 minutes, it means that the lock will be released in 5 minutes. Release automatically after minutes. If the lock is not obtained after 5 minutes, wait for t time and request the lock again. When steps (1)(2)(3)(4) are successfully executed, the system will release the lock, that is, actively releasing the current lock. Generally speaking, if there are no exceptions or interruptions during program execution, the lock will be automatically released. Release, the lock will return to the Redis database after it is released.

For example, for the above two situations where the lock cannot be actively released, the following targeted processing can be performed. For the first case, you can catch the exception and print the log; for the second case, monitor the duration of the intermediate execution steps (1) (2) (3) (4). If the lock release time t is exceeded, an exception will be thrown and captured. Here, steps (1) (2) and steps (3) (4) can also be added to the transaction operation at the same time. When an exception occurs, operations such as writing and deleting the object data are rolled back.

It can be seen that the data processing method provided by the embodiment of the present application optimizes parameter settings, realizes dynamic adjustment of the lock release time, saves the cost of manual intervention, and reduces redundant work of parameter adjustment. In addition, the storage process of Redis-level associated data based on dynamic locks can provide a universal storage solution for more business scenarios and has certain usage prospects.

Figure 4 is a schematic structural diagram of a data processing device provided by an embodiment of the present application. As shown in Figure 4, the device includes: an interception module 400, a statistics module 401, a determination module 402 and an execution module 403, wherein:

The interception module 400 is used to intercept multiple data operation requests received by the server within the current time period; the data operation requests are used to write or delete operation data;

The statistics module 401 is used to count the data inflow rate corresponding to multiple operation data completed within the current time period; the data inflow rate includes at least one of a data writing rate and a data deletion rate;

Determining module 402, configured to determine the release time of the lock corresponding to each target data operation request received in the next time period based on the data inflow rate;

The execution module 403 is configured to execute an operation corresponding to each target data operation request based on the determined release time of the lock.

In some embodiments, when the data inflow rate includes a data writing rate and a data deletion rate, the determining module 402 is configured to determine, based on the data inflow rate, each data rate received in the next time period. The release time of the lock corresponding to each target data operation request, including:

Obtain the write release time and delete release time of the lock within the current time period;

Determine a first ratio between the data writing rate and the write release time of the lock in the current time period, and a second ratio between the data deletion rate and the deletion release time of the lock in the current time period;

Based on the first ratio and the second ratio, the release time of the lock corresponding to each target data operation request received within the next time period is determined.

In some embodiments, the determining module 402 is configured to determine the release time of the lock corresponding to each target data operation request received in the next time period based on the first ratio and the second ratio. ,include:

Adjust the first ratio and the second ratio so that the first ratio and the second ratio are equal;

Based on the adjusted first ratio and the second ratio, the release time of the lock corresponding to each target data operation request received within the next time period is determined.

In some embodiments, the execution module 403 is configured to execute operations corresponding to each target data operation request based on the determined release time of the lock, including:

For each target data operation request, obtain the lock of the target operation data; the target data operation request is used to perform a write operation or delete operation on the target operation data;

Based on the determined release time of the lock, a locking operation is performed on the target operation data, and corresponding operations are performed on the target operation data according to the target data operation request.

In some embodiments, the execution module 403 is used to obtain the lock of target operation data, including:

Obtain the lock of the target operation data from the Redis database; the lock of the target operation data is a lock in the form of key-value, the key value is the identification of the target operation data, and the value value is the target operation The release time of the data lock.

In some embodiments, the device further includes a listening module, and the listening module is used to:

The execution process corresponding to each target data operation request is monitored until the monitoring time reaches the determined release time of the lock, and the lock of the target operation data is released.

In some embodiments, the device further includes a rollback module, and the rollback module is used to:

When the listening duration reaches the determined release time of the lock and it is detected that the lock of the target operation data has not been released, a rollback operation is performed on the execution of the target operation data.

In practical applications, the above-mentioned interception module 400, statistics module 401, determination module 402, execution module 403, monitoring module and rollback module can all be implemented by a processor located in an electronic device. The processor can be an ASIC, DSP, or DSPD. , at least one of PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

In addition, each functional module in this embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or software function modules.

If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially The part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions to make a computer device (which can be a personal computer , server, or network equipment, etc.) or processor (processor) executes all or part of the steps of the method of this embodiment. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes.

Specifically, the computer program instructions corresponding to a data processing method in this embodiment can be stored on storage media such as optical disks, hard disks, and U disks. When the computer program instructions corresponding to a data processing method in the storage medium When read or executed by an electronic device, any data processing method in the aforementioned embodiments is implemented.

Based on the same technical concept of the previous embodiment, see Figure 5, which shows an electronic device 500 provided by an embodiment of the present application, which may include: a memory 501 and a processor 502; wherein,

Memory 501, used to store computer programs and data;

The processor 502 is used to execute the computer program stored in the memory to implement any of the data processing methods in the aforementioned embodiments.

In practical applications, the above-mentioned memory 501 can be a volatile memory (volatile memory), such as RAM; or a non-volatile memory (non-volatile memory), such as ROM, flash memory (flash memory), hard disk (Hard Disk). Drive, HDD) or solid-state drive (Solid-State Drive, SSD); or a combination of the above types of memories, and provides instructions and data to the processor 502.

The above-mentioned processor 502 may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. It can be understood that for different data processing systems, the electronic device used to implement the above processor function can also be other, which is not specifically limited in the embodiment of the present application.

In some embodiments, the functions or modules provided by the device provided by the embodiments of the present application can be used to execute the methods described in the above method embodiments. For specific implementation, refer to the description of the above method embodiments. For the sake of brevity, here No longer.

The above description of various embodiments tends to emphasize the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described again here.

The methods disclosed in each method embodiment provided in this application can be combined arbitrarily to obtain a new method embodiment if there is no conflict.

The features disclosed in each product embodiment provided in this application can be arbitrarily combined to obtain a new product embodiment if there is no conflict.

The features disclosed in each method or device embodiment provided in this application can be combined arbitrarily without conflict to obtain a new method embodiment or device embodiment.

Those skilled in the art will understand that embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, magnetic disk storage and optical storage, etc.) embodying computer-usable program code therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a A device for realizing the functions specified in one process or multiple processes of the flowchart and/or one block or multiple blocks of the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device, causing a series of operating steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby executing on the computer or other programmable device. Instructions provide steps for implementing the functions specified in a process or processes of a flowchart diagram and/or a block or blocks of a block diagram.

The above are only preferred embodiments of the present application and are not used to limit the protection scope of the present application.

Claims (10)

1. A method of data processing, the method comprising:

intercepting a plurality of data operation requests received by a server in a current time period; the data operation request is used for performing writing operation or deleting operation on operation data;

counting the data inflow rates corresponding to the operation data completed in the current time period; the data inflow rate includes at least one of a data writing rate and a data deleting rate;

determining the release time of the lock corresponding to each target data operation request received in the next time period based on the data inflow rate;

and executing the operation corresponding to each target data operation request based on the determined release time of the lock.

2. The method of claim 1, wherein, in the case where the data inflow rate includes a data write rate and a data delete rate, the determining, based on the data inflow rate, a release time of a lock corresponding to each target data operation request received in a next period of time includes:

acquiring write release time and delete release time of the lock in the current time period;

Determining a first ratio of the data writing rate to the writing release time of the lock in the current time period and a second ratio of the data deleting rate to the deleting release time of the lock in the current time period;

and determining the release time of the lock corresponding to each target data operation request received in the next time period based on the first ratio and the second ratio.

3. The method of claim 2, the determining, based on the first ratio and the second ratio, a release time of a lock corresponding to each target data operation request received in the next time period, comprising:

adjusting the first ratio and the second ratio so that the first ratio and the second ratio are equal;

and determining the release time of the lock corresponding to each target data operation request received in the next time period based on the adjusted first ratio and second ratio.

4. The method of claim 1, wherein the performing an operation corresponding to each of the target data operation requests based on the determined release time of the lock comprises:

for each target data operation request, acquiring a lock of target operation data; the target data operation request is used for performing writing operation or deleting operation on the target operation data;

And locking the target operation data based on the determined release time of the lock, and executing corresponding operation on the target operation data according to the target data operation request.

5. The method of claim 4, wherein the acquiring the lock of the target operational data comprises:

acquiring a lock of the target operation data from a Redis database; the lock of the target operation data is in a key-value form, the key value is the identification of the target operation data, and the value is the release time of the lock of the target operation data.

6. The method according to claim 4, wherein the method further comprises:

monitoring the execution process corresponding to each target data operation request until the monitoring time length reaches the determined lock release time, and releasing the locks of the target operation data.

7. The method of claim 6, wherein the method further comprises:

and executing rollback operation on the execution of the target operation data under the condition that the monitoring time length reaches the determined release time of the lock and the lock of the target operation data is detected not to be released.

8. A data processing apparatus, the apparatus comprising:

the intercepting module is used for intercepting a plurality of data operation requests received by the server side in the current time period; the data operation request is used for performing writing operation or deleting operation on operation data;

the statistics module is used for counting the data inflow rates corresponding to the operation data completed in the current time period; the data inflow rate includes at least one of a data writing rate and a data deleting rate;

the determining module is used for determining the release time of the lock corresponding to each target data operation request received in the next time period based on the data inflow rate;

and the execution module is used for executing the operation corresponding to each target data operation request based on the determined release time of the lock.

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method of any one of claims 1 to 7 when the program is executed.

10. A computer storage medium having stored thereon a computer program, which when executed by a processor implements the method of any of claims 1 to 7.