CN111897490B - Method and device for deleting data - Google Patents

Abstract

The embodiment of the specification provides a method and a device for deleting data, wherein the method for deleting data comprises the following steps: recording a main key of first designated data to be deleted by a first deleting task and/or a main key of first associated data associated with the main key of the first designated data in a deleting transaction log; extracting a primary key of the first specified data and/or a primary key of the first associated data from the deletion transaction log under the condition that the first deletion task is abnormally exited; deleting the first specified data according to the extracted primary key of the first specified data, and/or deleting the first associated data according to the extracted primary key of the first associated data.

Description

Method and device for deleting data

Technical Field

The embodiments of this specification relate to the field of data processing, and more particularly to a method for deleting data. One or more embodiments of this specification also relate to a device for deleting data, a computing device, and a computer-readable storage medium.

Background technique

With the rapid development of Internet technology, mankind has entered the era of industrial Internet where everything is connected. The extensive interconnection between intelligent machines and humans, and between machines, has generated massive amounts of data, such as spatiotemporal data with time and space information.

In order to support the storage, analysis and calculation of big data, a big data storage and computing architecture that is suitable for the application scenario can be adopted, such as the spatiotemporal big data architecture based on Geomesa, Spark, and Hbase. However, when deleting data in these big data storage and computing architectures, if the deletion task exits abnormally due to some errors, data may have been deleted in some data tables, and these data are logically marked as deleted by the database, but have not been deleted in other data tables. These data become abnormal data, posing a threat to data security.

Summary of the invention

In view of this, the present specification provides a method for deleting data. One or more embodiments of the present specification also relate to a device for deleting data, a computing device, and a computer-readable storage medium to solve the technical defects existing in the prior art.

According to a first aspect of an embodiment of the present specification, a method for deleting data is provided, comprising: recording in a deletion transaction log a primary key of first designated data to be deleted by a first deletion task and/or a primary key of first associated data associated with the primary key of the first designated data; in the event that the first deletion task exits abnormally, extracting from the deletion transaction log the primary key of the first designated data and/or the primary key of the first associated data; deleting the first designated data according to the extracted primary key of the first designated data, and/or deleting the first associated data according to the extracted primary key of the first associated data.

Optionally, the method further includes: finding out the associated data table of the second specified data table; generating a primary key of the second associated data associated with the primary key of the second specified data table based on the primary key of the second specified data table; finding out data that is not within the range of the primary key of the second associated data from the associated data table of the second specified data table; and deleting data that is not within the range of the primary key of the second associated data from the associated data table of the second specified data table.

Optionally, the method further includes: performing a deletion legitimacy check on the second designated data table to be deleted by the second deletion task; if the deletion legitimacy check passes, entering the step of finding out the associated data table of the second designated data table.

Optionally, deleting the data in the associated data table of the second designated data table that is not within the primary key range of the second associated data includes: concurrently deleting the data in the associated data table of the second designated data table that is not within the primary key range of the second associated data in batches.

Optionally, the first deletion task is used to first delete the data corresponding to the primary key of the first associated data, and then delete the data corresponding to the primary key of the first specified data; deleting the first specified data and the first associated data based on the extracted primary key of the first specified data and the primary key of the first associated data includes: first deleting the first associated data based on the primary key of the first associated data; and then deleting the first specified data based on the extracted primary key of the first specified data.

Optionally, the first deletion task includes multiple deletion tasks executed concurrently.

Optionally, the method further includes: based on a producer-consumer model, concurrently executing a plurality of second deletion tasks, wherein the plurality of second deletion tasks are used to delete a second designated data table.

Optionally, the method is applied to a big data architecture based on Spark as the computing layer; the producer-consumer model-based, concurrent execution of multiple second deletion tasks includes: obtaining the primary key of the second specified data table from the database to the Spark master node based on the Spark interface; the Spark master node, as a producer, sends the primary key of the second specified data table to multiple consumer queues in batches; the multiple consumer queues generate the primary key of the second associated data according to the received primary key; the multiple consumer queues generate deletion requests corresponding to the second deletion task for the received primary key and the primary key of the second associated data respectively; the multiple consumer queues send the deletion requests to the database concurrently respectively. The method also includes: the multiple consumer queues write the received primary key and the corresponding generated primary key of the second associated data into the deletion transaction log respectively.

Optionally, the deletion transaction log is used to perform log recording in a memory file mapping manner.

According to a second aspect of the embodiments of this specification, there is provided a device for deleting data, comprising: a deletion log recording module, configured to record in a deletion transaction log the primary key of the first designated data to be deleted by the first deletion task and/or the primary key of the first associated data associated with the primary key of the first designated data. A primary key extraction module, configured to extract the primary key of the first designated data and/or the primary key of the first associated data from the deletion transaction log when the first deletion task exits abnormally. A deletion recovery module, configured to delete the first designated data according to the extracted primary key of the first designated data, and/or to delete the first associated data according to the extracted primary key of the first associated data.

According to a third aspect of an embodiment of the present specification, a computing device is provided, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions: recording in a deletion transaction log the primary key of a first designated data to be deleted by a first deletion task and/or the primary key of a first associated data associated with the primary key of the first designated data; in the event that the first deletion task exits abnormally, extracting from the deletion transaction log the primary key of the first designated data and/or the primary key of the first associated data; deleting the first designated data according to the extracted primary key of the first designated data, and/or deleting the first associated data according to the extracted primary key of the first associated data.

According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer instructions, and when the instructions are executed by a processor, the steps of the method for deleting data described in any embodiment of this specification are implemented.

An embodiment of the present specification implements a method for deleting data. Since the method records the primary key of the first specified data to be deleted by the first deletion task and/or the primary key of the first associated data associated with the primary key of the first specified data in a deletion transaction log, when the first deletion task exits abnormally, the primary key of the first specified data and/or the primary key of the first associated data are extracted from the deletion transaction log, and the first specified data is deleted according to the extracted primary key of the first specified data, and/or the first associated data is deleted according to the extracted primary key of the first associated data. Therefore, when the deletion task exits abnormally and abnormal data is generated, the abnormal data can be cleaned up according to the recorded deletion transaction log, thereby ensuring successful deletion and improving data security.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a method for deleting data provided by an embodiment of the present specification;

FIG2 is a flow chart of a method for deleting data provided by another embodiment of the present specification;

FIG3 is a flow chart of a method for deleting data provided by another embodiment of the present specification;

FIG4 is a schematic diagram of a spatiotemporal big data architecture provided by an embodiment of this specification;

FIG5 is a flow chart of a method for deleting data provided by yet another embodiment of the present specification;

FIG6 is a structural block diagram of a device for deleting data provided by an embodiment of the present specification;

FIG7 is a structural block diagram of a device for deleting data provided by another embodiment of the present specification;

FIG8 is a structural block diagram of a computing device provided by an embodiment of the present specification.

Detailed ways

Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and/or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

In this specification, a method for deleting data is provided. This specification also relates to an apparatus for deleting data, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

FIG. 1 shows a flow chart of a method for deleting data according to an embodiment of the present specification, including steps 110 to 130 .

Step 110: Recording the primary key of the first designated data to be deleted by the first deletion task and/or the primary key of the first associated data associated with the primary key of the first designated data in the deletion transaction log.

The primary key is an identifier that can uniquely identify a record in a data table. For example, it can be the column value of each row of data.

The embodiments of this specification do not limit the deleted data and the deletion conditions. For example, the first specified data may be relational data, key-value data or other structured data.

In order to ensure that the deletion transaction log can be effectively recorded, for example, the log can be recorded in memory file mapping mode. The memory file mapping technology entrusts the operation of persisting the log to the disk to the operating system. If the program exits abnormally during the process of sending a command to the data warehouse to delete data, the operating system can write the log currently in the memory to the disk, and the data security will not be affected by only writing the data to the memory without actually persisting it to the disk.

Optionally, for example, when the deletion task is successful, the record of the deletion task may be cleared from the deletion transaction log to prevent waste of storage space.

In order to improve deletion efficiency, for example, the first deletion task may include multiple deletion tasks executed concurrently.

In order to support the deletion of abnormal data in the database after the deletion task exits abnormally, the method provided in the embodiment of this specification can record the transaction log for the deletion task. For each deletion task, a special log storage path can be established for this deletion task under the deletion transaction log path with time as the path name, which is used to record the data table and related data table specified for deletion of the deletion task.

Step 120: When the first deletion task exits abnormally, extract the primary key of the first designated data and/or the primary key of the first associated data from the deletion transaction log.

Step 130: deleting the first designated data according to the extracted primary key of the first designated data, and/or deleting the first associated data according to the extracted primary key of the first associated data.

Since the method records the primary key of the first designated data to be deleted by the first deletion task and/or the primary key of the first associated data associated with the primary key of the first designated data in the deletion transaction log, when the first deletion task exits abnormally, the primary key of the first designated data and/or the primary key of the first associated data are extracted from the deletion transaction log, and the first designated data is deleted according to the extracted primary key of the first designated data, and/or the first associated data is deleted according to the extracted primary key of the first associated data. Therefore, when the deletion task exits abnormally and abnormal data is generated, the abnormal data can be cleaned up according to the recorded deletion transaction log, thereby ensuring successful deletion and improving data security.

For example, after a deletion task exits abnormally, since the data in the original database is deleted, but some related data is not deleted, the indexes of each table cannot correspond to these undeleted related data, causing data security issues. According to the method provided in the embodiments of this specification, the remaining related data can be deleted based on the primary key of the related data recorded in the deletion transaction log, thereby ensuring successful deletion and improving data security.

In one or more embodiments of the present specification, before executing a deletion task, in order to further clear the abnormal data remaining in the data table related to the deletion task, the legality of the deletion of the data table by the deletion task can be first detected, and after the verification is passed, the abnormal data already existing in the specified data table will be detected and deleted. Specifically, FIG2 shows a flow chart of a method for deleting data provided according to another embodiment of the present specification. As shown in FIG2, the method also includes steps 101 to 105.

Step 101: Perform a deletion legitimacy check on the second designated data table to be deleted by the second deletion task.

It should be noted that the second deletion task and the first deletion task may be the same task or different tasks. The second designated data table and the first designated data may refer to the same data or different data.

For example, the deletion legitimacy check may include any one or more of: checking whether the database where the data table is located exists, checking whether the data table exists, checking whether the deletion caller has deletion authority, checking whether the data involved in the deletion filter conditions exists in the data table, etc.

Step 102: If the deletion legality check passes, find out the associated data table of the second designated data table.

For example, the name of the associated data table of the second specified data table may be generated according to the table name rule of the associated table provided by the big data architecture, and then a request may be sent to the database to search for the associated data table corresponding to the table name.

Step 103: Generate a primary key of second associated data associated with the primary key of the second designated data table according to the primary key of the second designated data table.

For example, the primary key of the second associated data associated with the primary key of the second specified data table may be generated according to the generation rule of the primary key of the associated table provided by the big data architecture.

Step 104: Searching for data that is not within the primary key range of the second associated data from the associated data table of the second designated data table.

Step 105: Delete the data in the associated data table of the second designated data table that is not within the primary key range of the second associated data.

For example, to further improve deletion efficiency, data may be deleted in batches and concurrently. Specifically, for example, data in the associated data table of the second designated data table that is not within the primary key range of the second associated data may be deleted in batches and concurrently.

In this embodiment, since the legality of the second deletion task deleting the second designated data table is first detected, it can be determined that the residual abnormal data can be further removed according to the relevant information of the second deletion task when the detection is passed. Therefore, when the detection is passed, the associated data table associated with the second designated data table is found out according to the second designated data table. It can be understood that if a deletion task exits abnormally, there may be residual abnormal data that has not been successfully deleted in the associated data table of the second designated data table. In order to be able to remove these abnormal data, this embodiment generates the primary key of the second associated data associated with the primary key according to the primary key of the second designated data table, that is, the primary key of the associated data that should normally exist. It can be understood that the data that is not within the primary key range of the associated data that should normally exist is the residual abnormal data. Therefore, this embodiment finds out the data that is not within the primary key range of the second associated data from the associated data table of the second designated data table, and deletes it, thereby achieving the purpose of monitoring and deleting the residual abnormal data in the database table.

Considering that when deleting data, the primary key of the associated data table can be obtained by concatenating and converting the primary key of the specified data table, the data in the associated data table is deleted first, and then the data in the specified data table is deleted. In this way, even if the deletion of the associated data table fails, it can still be deleted again through the primary key of the specified data table. Therefore, in one or more embodiments of the present specification, the first deletion task is used to first delete the data corresponding to the primary key of the first associated data, and then delete the data corresponding to the primary key of the first specified data. Deleting the first specified data and the first associated data according to the extracted primary key of the first specified data and the primary key of the first associated data may include: first deleting the first associated data according to the primary key of the first associated data; and then deleting the first specified data according to the extracted primary key of the first specified data.

In one or more embodiments of the present specification, in order to improve deletion efficiency, the method may also concurrently execute multiple second deletion tasks based on a producer-consumer model, where the multiple second deletion tasks are used to delete the second designated data table.

For example, the method can be applied to a big data architecture based on Spark as a computing layer. FIG3 shows a flowchart of a method for deleting data according to another embodiment of the present specification. As shown in FIG3 , the method can also include steps 140 to 145 .

Step 140: Based on the Spark interface, obtain the primary key of the second specified data table from the database to the Spark master node.

Step 141: The Spark master node, as a producer, sends the primary key of the second designated data table to multiple consumer queues in batches.

Step 142: The multiple consumer queues respectively generate primary keys for the second associated data according to the received primary keys.

Step 143: The multiple consumer queues respectively write the received primary key and the correspondingly generated primary key of the second associated data into the deletion transaction log.

Step 144: The multiple consumer queues generate deletion requests corresponding to the second deletion task respectively for the received primary key and the primary key of the second associated data.

Step 145: The multiple consumer queues send the deletion request to the database concurrently.

It can be seen that in this embodiment, since the Spark master node is used as the producer, the primary keys of the data table to be deleted are sent to multiple consumer queues in batches, so that the multiple consumer queues send deletion requests to the database concurrently, so that the deletion tasks corresponding to the multiple deletion requests are independent of each other and executed concurrently, which greatly exerts the processing capacity of the deletion tasks and improves the deletion efficiency.

Below, an implementation method that combines multiple embodiments of this specification is described in detail. For example, the method for deleting data described in the embodiments of this specification can be applied to the spatiotemporal big data architecture based on Geomesa, Spark, and Hbase as shown in Figure 4. The client access method of this big data architecture is flexible, for example, JDBC access, Beeline access, etc. can be accessed. Among them, Spark and Geomesa are used as the computing layer, and Hbase is used as the storage layer. Spark is a fast and general computing engine designed for large-scale data processing that is widely used. Hbase is a distributed column-oriented open source database. Geomesa: An open source toolkit for spatiotemporal data processing. The underlying storage type is not limited, for example: EXT4 file system, NTFS file system and other file systems.

Based on the spatiotemporal big data architecture shown in FIG. 4 , FIG. 5 shows a flowchart of a method for deleting data provided according to an embodiment of the present specification. As shown in FIG. 5 , the method includes steps 502 to 528 .

Step 502: Perform deletion legitimacy check on the second designated data table.

For example, in this embodiment, the existence of the database of the second specified data table can be queried by calling the Hbase interface; the existence of the second specified data table can be queried by calling the Hbase interface; the existence of the second specified data table can be determined by calling the Hbase interface whether the deletion caller has the deletion authority for the second specified data table; and the existence of the columns involved in the filtering conditions of the second deletion task can be determined in the specified database table. More specifically, for example, the method provided in the embodiment of this specification can be applied to the spatiotemporal data warehouse based on Figure 4. The deletion statement can use the SQL language commonly used in the database field to describe the filtering conditions. The filtering conditions can be specified by column constraints or composed in the form of composite functions. For example: "where id>0andname＝'hello'", "where function(id,name)>0and user_defined_function(id,name,id)<1", according to the deletion statement, it can be detected whether there are columns corresponding to the two column names "id" and "name" in the database table.

Step 504: If the deletion legality check passes, search the Hbase database for the associated data table of the second designated data table.

For example, according to the rules of the table names of associated data tables, the Geomesa interface can be called to determine whether there are associated data tables associated with the specified data table, such as spatiotemporal index data tables, ordinary Btree index data tables, etc., according to the type of the specified data table, and send a request to the Hbase database to confirm whether the associated data table exists in Hbase. Specifically, as Geomesa is an open source spatiotemporal computing project, the rules for generating the table names of associated data tables (such as index tables) given the original data table are given in the Geomesa source code. For example, for the data table "gdeltable", according to the rules of the table names of associated data tables, the associated data tables with the following table names will be generated in the database:

gdeltable_gdelt_id (id table)

gdeltable_gdelt_z2_v2 (z2 index table)

gdeltable_gdelt_z3_v2 (z3 index table)

Step 506: Extract the primary key of the second specified data table.

For example, after the associated data table is searched, the Hbase interface can be called through Spark to pull all the data of the specified data table to the Spark master node. The Spark master node can extract the primary key of the column in the specified data table and write it into the deletion transaction log. For example, the deletion transaction log can be persisted to one of the local file system, the distributed file system, or the Hbase temporary table. It is understandable that, since there is a possibility of exiting due to deletion failure in the process of deleting abnormal data from step 506 to step 510, in the case of failure, the deletion can be restored according to the persistent deletion transaction log to improve performance. In another embodiment, the primary key of step 506 may not be written into the deletion transaction log, but by repeatedly executing steps 506 to step 510, the deletion of residual abnormal data can also be achieved.

Step 508: Generate a primary key of the second associated data of the second specified data table.

For example, after the primary key of the specified data table is extracted, the Geomesa interface can be called through Spark to generate the corresponding primary key column of the associated database table locally based on the primary key column of the specified data table. Specifically, for example, the source code of Geomesa provides the generation rules for calculating the associated column given the original primary key. According to the generation rules, the primary key of the associated database table can be generated based on the primary key of the specified data table. For example, for a piece of data containing spatial data points, the original primary key is "FeatureID", and the generation rule of the primary key of the associated "spatial Z2 index data table" is defined in geomea: hash key + Z2 index value + FeatureID.

Step 510: In the associated data table of the second designated data table, data that is not within the primary key range of the second associated data is treated as abnormal data and deleted in parallel.

For example, multiple threads can be started, and the Hbase interface can be called to concurrently delete the data in the Hbase associated database table found in step 504 whose primary key values are not within the primary key range generated in step 508. For example, before deleting, the primary keys whose primary key values are not within the primary key range generated in step 508 can be written into the deletion transaction log, so that in the event of a deletion failure, the deletion can be restored according to the deletion transaction log. Of course, the primary keys to be deleted in step 510 can also be deleted without writing them into the deletion transaction log, but the deletion of the residual abnormal data can be achieved by repeatedly executing steps 506 to 510.

Step 512: Determine whether any abnormal deletion tasks remain by checking the deletion transaction log.

For example, the first deletion task may be a deletion task before the second deletion task, or, for example, the first deletion task may be a step. When the first deletion task exits abnormally, the deletion transaction log will record the primary key of the first specified data and the primary key of the first associated data specified to be deleted by the first deletion task.

For example, the method provided in the embodiments of this specification may provide in advance a default parameter configuration and storage path for determining the deletion of the transaction log, or the parameter configuration and storage path for the deletion of the transaction log may be determined by a user-defined environment variable. When detecting the deletion of the transaction log, the deletion transaction log path may be found based on the environment variable, configuration parameter, or default parameter. For example, if the storage path is configured in the startup parameters, the configuration parameter may be used; if there is no configuration parameter, the environment variable may be searched to see whether the storage path is set; if there is no environment variable, the default path may be used.

In one embodiment, it is possible to find out whether there is a deletion transaction log in the deletion transaction path. If there is no deletion transaction log, it means that there is no deletion transaction that is abnormally exited, and then the process proceeds to step 520 to perform a normal deletion process; if there is a deletion transaction log, then the process proceeds to steps 514-516 to delete the abnormal data in the database according to the deletion transaction log.

Step 514: Parallel deletion of the data corresponding to the primary key of the first associated data recorded in the deletion transaction log.

For example, when there is a deletion transaction log, the log part of the associated data in the deletion transaction log can be read, and the Hbase Delete interface can be called through multiple threads to perform batch concurrent deletion of the data of the associated data.

Step 516: Parallelize and delete the data corresponding to the primary key of the first designated data recorded in the deletion transaction log.

After deleting the remaining abnormally associated data, you can continue to read the original data in the deletion transaction log, extract the primary key of the original data to be deleted, and perform batch concurrent deletion of the original data.

After completing the above steps of deleting the residual abnormal data of the second designated data table and recovering the abnormal deletion based on the deletion transaction log, the process proceeds to step 520 to perform a normal deletion process for the second designated data table.

Step 520: The producer queue obtains the primary key of the second designated data table that meets the filtering condition from the Hbase database based on the Spark interface.

For example, Spark can call the Hbase interface to pull the primary key of the second specified data table that meets the filtering conditions to the Spark master node. As a producer, the Spark master node will send the pulled local data in batches to multiple consumer queues in turn.

Step 522: Multiple consumer queues consume data and call the Geomesa interface to calculate the primary key of the associated column, that is, the primary key of the second associated data.

For example, multiple threads are used as multiple consumers, each consumer will continuously pull data from the corresponding data queue for consumption, each consumer can extract the primary key of the first specified data to be deleted from the pulled data, and call the Geomesa interface to generate the primary key of the corresponding second associated data.

Step 524: The plurality of consumer queues write the primary key of the second designated data and the primary key of the second associated data into the memory file mapping area respectively.

After obtaining the primary key of the second specified data and the primary key of the second associated data, in order to support fault recovery after abnormal deletion, the primary key of the data that needs to be recorded in the deletion transaction log can be written in this step. Thus, when the second deletion task fails, the corresponding data can be deleted based on the primary key recorded in the deletion transaction log, that is, the primary key of the data that needs to be deleted in the Hbase warehouse. It can be understood that in data storage such as Hbase, a primary key will be stored for each data, and data can be deleted in the specified database table based on the primary key. Each consumer writes the primary key to the deletion transaction log through the memory file mapping method. Before writing, the current position point can be recorded first, and then the primary key of the data to be deleted, that is, the primary key of the second specified data and the primary key of the second associated data are written into the memory file mapping area through several commonly used interfaces for writing data to the memory mapping area in the memory file mapping technology, such as the standard interface of Oracle JDK.

Step 526: Multiple consumer queues concurrently send deletion requests to Hbase.

For example, after the primary key of the second specified data and the primary key of the second associated data are written into the deletion transaction log, the current consumer can splice the primary key of the second associated data of the current batch into an Hbase Delete request, delete the associated data, and then splice and send a request to delete the primary key of the second specified data. In this step, the deletion tasks of multiple consumers are independent and executed in parallel to maximize the concurrent processing capabilities of Hbase.

Step 528: After the deletion is successful, clear the deletion transaction log in the memory file mapping area.

For example, since the current position point can be recorded before writing the primary key into the deletion transaction log, after each data consumer completes the data deletion through the Hbase delete interface, the corresponding memory file mapping area can be overwritten with a null character according to the recorded current position point, so that the write mark point of the deletion transaction log is reset to the initial position recorded in step 524. In this embodiment, the deletion transaction log is recorded first, and then a command to delete the data is sent to the Hbase data warehouse. Through the null character overwriting and the mark point rollback, the log records of this batch can be cleared after the deletion task is normally completed, thereby preventing storage space waste.

For each data consumer in the deletion process, when the data producer has finished pulling data, the consumption queue corresponding to the current consumer is empty, and when the consumption has been completed, the consumption task of the current consumer can be considered to be completed. When the consumption tasks of all consumers are completed, the overall data deletion process is completed.

It can be seen that in this embodiment, since the deletion process adopts a parallel process based on the producer-consumer model, the producer pulls data in batches and hands it over to multiple independent and parallel consumers for logging and generating Hbase Delete requests, thereby improving the efficiency of concurrent deletion and maximizing the concurrency capability of the storage layer Hbase.

Corresponding to the above method embodiment, this specification also provides an embodiment of a device for deleting data, and Figure 6 shows a schematic diagram of the structure of a device for deleting data provided by an embodiment of this specification. As shown in Figure 6, the device includes: a deletion log recording module 602, a primary key extraction module 604 and a deletion recovery module 606.

The deletion log recording module 602 may be configured to record in the deletion transaction log the primary key of the first designated data to be deleted by the first deletion task and/or the primary key of the first associated data associated with the primary key of the first designated data.

The primary key extraction module 604 may be configured to extract the primary key of the first designated data and/or the primary key of the first associated data from the deletion transaction log when the first deletion task exits abnormally.

The deletion recovery module 606 may be configured to delete the first designated data according to the extracted primary key of the first designated data, and/or to delete the first associated data according to the extracted primary key of the first associated data.

Since the device records the primary key of the first designated data to be deleted by the first deletion task and/or the primary key of the first associated data associated with the primary key of the first designated data in the deletion transaction log, when the first deletion task exits abnormally, the primary key of the first designated data and/or the primary key of the first associated data are extracted from the deletion transaction log, and the first designated data is deleted according to the extracted primary key of the first designated data, and/or the first associated data is deleted according to the extracted primary key of the first associated data. Therefore, when the deletion task exits abnormally and abnormal data is generated, the abnormal data can be cleaned up according to the recorded deletion transaction log, thereby ensuring successful deletion and improving data security.

Fig. 7 shows a schematic diagram of a structure of a device for deleting data provided by another embodiment of the present specification. As shown in Fig. 7, the device may also include: a legal detection module 608, an association table search module 610, an association primary key generation module 612, an abnormal data screening module 614 and an abnormal data deletion module 616.

The legality detection module 608 may be configured to perform a deletion legality detection on the second designated data table designated for deletion by the second deletion task.

The association table search module 610 may be configured to search for an association data table of the second designated data table if the deletion legality check passes.

The associated primary key generating module 612 may be configured to generate a primary key of the second associated data associated with the primary key of the second designated data table according to the primary key of the second designated data table.

The abnormal data screening module 614 may be configured to search for data that is not within the primary key range of the second associated data from the associated data table of the second designated data table.

The abnormal data deletion module 616 may be configured to delete data in the associated data table of the second designated data table that is not within the primary key range of the second associated data.

In this embodiment, since the legality of the second deletion task deleting the second designated data table is first detected, it can be determined that the residual abnormal data can be further removed according to the relevant information of the second deletion task when the detection is passed. Therefore, when the detection is passed, the associated data table associated with the second designated data table is found out according to the second designated data table. It can be understood that if a deletion task exits abnormally, there may be residual abnormal data that has not been successfully deleted in the associated data table of the second designated data table. In order to be able to remove these abnormal data, this embodiment generates the primary key of the second associated data associated with the primary key according to the primary key of the second designated data table, that is, the primary key of the associated data that should normally exist. It can be understood that the data that is not within the primary key range of the associated data that should normally exist is the residual abnormal data. Therefore, this embodiment finds out the data that is not within the primary key range of the second associated data from the associated data table of the second designated data table, and deletes it, thereby achieving the purpose of monitoring and deleting the residual abnormal data in the database table.

To improve deletion efficiency, optionally, the abnormal data deletion module 616 may be configured to concurrently delete in batches the data in the associated data table of the second designated data table that is not within the primary key range of the second associated data.

Considering that when deleting data, the primary key of the associated data table can be obtained by concatenating and converting the primary key of the specified data table, the data in the associated data table is deleted first, and then the data in the specified data table is deleted. In this way, even if the deletion of the associated data table fails, it can be deleted again by the primary key of the specified data table. Therefore, in one or more embodiments of this specification, the first deletion task is used to first delete the data corresponding to the primary key of the first associated data, and then delete the data corresponding to the primary key of the first specified data. The deletion recovery module 606 can be configured to first delete the first associated data according to the primary key of the first associated data; and then delete the first specified data according to the extracted primary key of the first specified data.

In one or more embodiments of the present specification, in order to improve deletion efficiency, as shown in Figure 7, the device may also include: a task execution module 618, which can be configured to concurrently execute multiple second deletion tasks based on the producer consumer model, and the multiple second deletion tasks are used to delete the second specified data table.

Optionally, the device can be configured in a big data architecture based on Spark as a computing layer. As shown in FIG7 , the task execution module 618 may include: a designated data acquisition submodule 6180, a designated data primary key distribution submodule 6182, an associated data primary key generation submodule 6184, a deletion request generation submodule 6186, and a deletion request sending submodule 6188. The device may also include: a primary key writing submodule 620, which may be configured to write the received primary key and the corresponding generated primary key of the second associated data into the deletion transaction log in the multiple consumer queues.

Optionally, the deletion transaction log is used to perform log recording in a memory file mapping manner.

The designated data acquisition submodule 6180 may be configured to acquire the primary key of the second designated data table from the database to the Spark master node based on the Spark interface.

The designated data primary key distribution submodule 6182 may be configured to send the primary key of the second designated data table to multiple consumer queues in batches as a producer on the Spark master node.

The associated data primary key generation submodule 6184 may be configured to generate the primary key of the second associated data in the multiple consumer queues respectively according to the received primary keys.

The deletion request generation submodule 6186 may be configured to generate a deletion request corresponding to the second deletion task in the multiple consumer queues respectively for the received primary key and the primary key of the second associated data.

The deletion request sending submodule 6188 may be configured to send the deletion request to the database concurrently in the multiple consumer queues respectively.

In this embodiment, since the Spark master node is used as the producer, the primary keys of the data table to be deleted are sent to multiple consumer queues in batches, so that the multiple consumer queues send deletion requests to the database concurrently, so that the deletion tasks corresponding to the multiple deletion requests are independent of each other and executed concurrently, which greatly exerts the processing capacity of the deletion tasks and improves the deletion efficiency.

Optionally, as shown in FIG. 7 , the apparatus may further include: a log clearing module 630 , which may be configured to clear the record of the deletion task in the deletion transaction log when the deletion task is successful.

The above is a schematic scheme of a device for deleting data in this embodiment. It should be noted that the technical scheme of the device for deleting data and the technical scheme of the method for deleting data belong to the same concept, and the details of the technical scheme of the device for deleting data that are not described in detail can all be referred to the description of the technical scheme of the method for deleting data.

Fig. 8 shows a block diagram of a computing device 800 according to an embodiment of the present specification. The components of the computing device 800 include but are not limited to a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

The computing device 800 also includes an access device 840 that enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 840 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

In one embodiment of the present specification, the above components of the computing device 800 and other components not shown in FIG8 may also be connected to each other, for example, through a bus. It should be understood that the computing device structure block diagram shown in FIG8 is only for illustrative purposes and is not intended to limit the scope of the present specification. Those skilled in the art may add or replace other components as needed.

The computing device 800 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 800 may also be a mobile or stationary server.

The processor 820 is used to execute the following computer executable instructions:

Recording in the deletion transaction log the primary key of the first designated data to be deleted by the first deletion task and/or the primary key of the first associated data associated with the primary key of the first designated data;

In the case where the first deletion task exits abnormally, extracting the primary key of the first designated data and/or the primary key of the first associated data from the deletion transaction log;

The first designated data is deleted according to the extracted primary key of the first designated data, and/or the first associated data is deleted according to the extracted primary key of the first associated data.

The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above method for deleting data belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above method for deleting data.

An embodiment of the present specification further provides a computer-readable storage medium storing computer instructions, which are used when executed by a processor to:

Recording in the deletion transaction log the primary key of the first designated data to be deleted by the first deletion task and/or the primary key of the first associated data associated with the primary key of the first designated data;

In the case where the first deletion task exits abnormally, extracting the primary key of the first designated data and/or the primary key of the first associated data from the deletion transaction log;

The first designated data is deleted according to the extracted primary key of the first designated data, and/or the first associated data is deleted according to the extracted primary key of the first associated data.

The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned method for deleting data belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned method for deleting data.

The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the described action sequence, because according to the embodiments of this specification, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims (10)

1. A method of deleting data, comprising:

recording a main key of first designated data to be deleted by a first deleting task and/or a main key of first associated data associated with the main key of the first designated data in a deleting transaction log;

extracting a primary key of the first specified data and/or a primary key of the first associated data from the deletion transaction log under the condition that the first deletion task is abnormally exited;

Deleting the first specified data according to the extracted primary key of the first specified data and/or deleting the first associated data according to the extracted primary key of the first associated data;

the first deleting task is used for deleting the data corresponding to the primary key of the first associated data first and then deleting the data corresponding to the primary key of the first designated data;

the deleting the first specified data and the first associated data according to the extracted primary key of the first specified data and the primary key of the first associated data includes:

firstly deleting the first associated data according to the primary key of the first associated data;

then deleting the first appointed data according to the extracted main key of the first appointed data;

The method further comprises the steps of:

Searching an associated data table of the second designated data table;

Generating a main key of second associated data associated with the main key of the second designated data table according to the main key of the second designated data table;

Searching data which is not in the primary key range of the second associated data from the associated data table of the second designated data table;

and deleting data which are not in the primary key range of the second associated data in the associated data table of the second designated data table.

2. The method of claim 1, further comprising:

Performing deletion legitimacy detection on a second designated data table designated to be deleted by a second deletion task;

And if the deletion validity detection is passed, entering the step of searching out the associated data table of the second designated data table.

3. The method of claim 1, the deleting data in the association data table of the second designated data table that is not within a primary key range of the second association data comprising:

and deleting the data which is not in the primary key range of the second associated data in the associated data table of the second designated data table in batches and concurrently.

4. The method of claim 1, the first deletion task comprising a plurality of deletion tasks that are concurrently performed.

5. The method of claim 1, further comprising:

based on the producer consumer model, a plurality of second deletion tasks are concurrently performed, the plurality of second deletion tasks for deleting the second specified data table.

6. The method of claim 5 applied to a big data architecture based on Spark as a compute layer;

the concurrently performing a plurality of second deletion tasks based on the producer consumer model includes:

acquiring a primary key of the second specified data table from a database to a Spark primary node based on a Spark interface;

the Spark master node is used as a producer to distribute the primary keys of the second designated data table to a plurality of consumer queues in batches;

the plurality of consumer queues respectively generate a main key of second associated data according to the received main key;

the plurality of consumer queues respectively generate a deletion request corresponding to the second deletion task for the received primary key and the primary key of the second associated data;

the plurality of consumer queues send the deletion request to a database respectively and concurrently;

The method further comprises the steps of:

And the plurality of consumer queues respectively write the received primary key and the primary key of the second associated data correspondingly generated into the deletion transaction log.

7. The method of claim 1 or 6, wherein the deletion transaction log is used for logging in a memory file mapping manner.

8. An apparatus for deleting data, comprising:

the deletion log recording module is configured to record a main key of first specified data to be deleted by a first deletion task and/or a main key of first associated data associated with the main key of the first specified data in a deletion transaction log;

The primary key extraction module is configured to extract a primary key of the first specified data and/or a primary key of the first associated data from the deletion transaction log under the condition that the first deletion task is abnormally exited;

a deletion restoration module configured to delete the first specified data according to the extracted primary key of the first specified data and/or delete the first associated data according to the extracted primary key of the first associated data;

the first deleting task is used for deleting the data corresponding to the primary key of the first associated data first and then deleting the data corresponding to the primary key of the first designated data;

the deleting the first specified data and the first associated data according to the extracted primary key of the first specified data and the primary key of the first associated data includes:

firstly deleting the first associated data according to the primary key of the first associated data;

then deleting the first appointed data according to the extracted main key of the first appointed data;

the apparatus further comprises:

The association table searching module is configured to search out an association data table of the second designated data table;

an associated primary key generation module configured to generate primary keys of second associated data associated with primary keys of the second specified data table in accordance with the primary keys of the second specified data table;

The abnormal data screening module is configured to search out data which is not in the range of the primary key of the second associated data from the associated data table of the second designated data table;

And the abnormal data deleting module is configured to delete data which are not in the primary key range of the second associated data in the associated data table of the second designated data table.

9. A computing device, comprising:

A memory and a processor;

The memory is for storing computer-executable instructions, and the processor is for executing the computer-executable instructions:

recording a main key of first designated data to be deleted by a first deleting task and/or a main key of first associated data associated with the main key of the first designated data in a deleting transaction log;

extracting a primary key of the first specified data and/or a primary key of the first associated data from the deletion transaction log under the condition that the first deletion task is abnormally exited;

Deleting the first specified data according to the extracted primary key of the first specified data and/or deleting the first associated data according to the extracted primary key of the first associated data;

the first deleting task is used for deleting the data corresponding to the primary key of the first associated data first and then deleting the data corresponding to the primary key of the first designated data;

the deleting the first specified data and the first associated data according to the extracted primary key of the first specified data and the primary key of the first associated data includes:

firstly deleting the first associated data according to the primary key of the first associated data;

then deleting the first appointed data according to the extracted main key of the first appointed data;

the computer-executable instructions further comprise:

Searching an associated data table of the second designated data table;

Generating a main key of second associated data associated with the main key of the second designated data table according to the main key of the second designated data table;

Searching data which is not in the primary key range of the second associated data from the associated data table of the second designated data table;

and deleting data which are not in the primary key range of the second associated data in the associated data table of the second designated data table.

10. A computer readable storage medium storing computer instructions which, when executed by a processor, implement the steps of the method of deleting data of any one of claims 1 to 7.