CN117271570A - Data storage methods, devices, media and computing equipment - Google Patents
Data storage methods, devices, media and computing equipment Download PDFInfo
- Publication number
- CN117271570A CN117271570A CN202311205621.9A CN202311205621A CN117271570A CN 117271570 A CN117271570 A CN 117271570A CN 202311205621 A CN202311205621 A CN 202311205621A CN 117271570 A CN117271570 A CN 117271570A
- Authority
- CN
- China
- Prior art keywords
- data
- service
- data table
- query
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Embodiments of the present disclosure provide a data storage method, apparatus, medium, and computing device, the method comprising: acquiring each enumeration value of a service, wherein the enumeration values comprise a data query mode, a query response time length and a data volume of a data table where query data are located; determining a target storage engine matched with each enumeration value, and acquiring a service grade corresponding to a service; and determining a target database corresponding to the service level in the target storage engine, and creating a data table in the target database. In the method, the target storage engine for storing the data is determined through the data query mode, the query response time length and the data quantity of the data table where the query data is located, so that the data table is created in the target database corresponding to the service level of the service in the target storage engine to store the data, the selection of the storage engine and the database in the storage engine is not dependent on manual operation, the selection time length of the storage engine and the database is shortened, and the storage efficiency of the data is improved.
Description
Technical Field
Embodiments of the present disclosure relate to the field of big data technology, and more particularly, to a data storage method, apparatus, medium, and computing device.
Background
This section is intended to provide a background or context for embodiments of the present disclosure. The description herein is not admitted to be prior art by inclusion in this section.
The data needs to be stored in a database before a data query is made. When data is stored, a storage engine is required to be selected, and the data to be stored is stored in a database in the storage engine.
In the exemplary technology, the selection of the storage engine and the selection of the database in the storage engine are manually performed by a technician, that is, the selection of the storage engine and the database depends on experience of the technician, if the experience of the technician is insufficient, it takes a long time to select the storage engine and the database, resulting in a long time required for data storage and a low data storage efficiency.
Disclosure of Invention
The disclosure provides a data storage method, a data storage device, a data storage medium and a computing device, which are used for solving the problem of low data storage efficiency.
In a first aspect of embodiments of the present disclosure, there is provided a data storage method, including: acquiring each enumeration value of a service, wherein the enumeration values comprise a data query mode, a query response time length and a data volume of a data table where query data are located; determining target storage engines matched with the enumeration values, and acquiring service grades corresponding to the businesses; and determining a target database corresponding to the service level in the target storage engine, and creating a data table in the target database, wherein the data table is used for storing data generated by the service.
In an embodiment of the disclosure, the determining the target storage engine that each of the enumerated values matches includes: constructing a plurality of group parameters according to each enumeration value; among the storage engines, a storage engine matching the multi-group parameter is determined as the target storage engine.
In another embodiment of the present disclosure, the determining, in the target storage engine, a target database corresponding to the service level includes: determining a plurality of databases to be determined corresponding to the service level in the target storage engine; acquiring a load balancing strategy corresponding to the target storage engine; and determining the databases to be determined matched with the load balancing strategy in each database to be determined as the target database.
In another embodiment of the present disclosure, after the target database creates the data table, the method further includes: analyzing service nodes generated by data stored in the data table based on the query statement of the service; constructing a data synchronization task corresponding to the data output task of the service node; and sending the data synchronization task to a data scheduling device, wherein the data synchronization task is used for synchronizing the data produced by the data production task to the data table.
In another embodiment of the present disclosure, after the sending the data synchronization task to the data scheduling device, the method further includes: acquiring first data synchronized by the data synchronization task, and acquiring a data synchronization mode of the data table; and storing the first data to the data table based on the data synchronization mode.
In another embodiment of the present disclosure, the storing the first data to the data table based on the data synchronization manner includes: copying the data table to obtain a temporary table in response to the data synchronization mode being full synchronization; storing the first data to the temporary table to obtain a synchronous temporary table; and changing the identification of the synchronous temporary table into the identification of the data table, and deleting the data table.
In another embodiment of the present disclosure, the storing the first data to the data table based on the data synchronization manner includes: responding to the data synchronization mode as increment synchronization, comparing the first data with the second data of the data table to determine sub-data in the first data, wherein the sub-data is not stored in the data table; and storing the sub data into the data table.
In another embodiment of the present disclosure, further comprising: and responding to the deletion of the data table or the pause of the query service corresponding to the data table, and sending a deletion request to the data scheduling device, wherein the deletion request is used for requesting the data scheduling device to delete the data synchronization task.
In another embodiment of the present disclosure, after the target database creates the data table, the method further includes: acquiring the query frequency of each data in the data table; and deleting the data corresponding to the query frequency lower than the preset frequency.
In a second aspect of embodiments of the present disclosure, there is also provided a data storage device comprising: the first acquisition module is used for acquiring each enumeration value of the service, wherein the enumeration values comprise a data query mode, a query response time length and a data volume of a data table where query data are located; the first determining module is used for determining the target storage engine matched with each enumeration value and acquiring the service grade corresponding to the service; and the second determining module is used for determining a target database corresponding to the service level in the target storage engine and creating a data table in the target database, wherein the data table is used for storing data generated by the service.
In an embodiment of the disclosure, the first determining module includes: the construction unit is used for constructing a plurality of group parameters according to each enumeration value; and the first determining unit is used for determining the storage engine matched with the multi-group parameter in each storage engine as the target storage engine.
In another embodiment of the present disclosure, the second determining module includes: the second determining unit is used for determining a plurality of databases to be determined corresponding to the service level in the target storage engine; the acquisition unit is used for acquiring a load balancing strategy corresponding to the target storage engine; and the third determining unit is used for determining the databases to be determined matched with the load balancing strategy in the databases to be determined as the target databases.
In another embodiment of the present disclosure, the data storage device further includes:
the analysis module is used for analyzing service nodes generated by data stored in the data table based on the query statement of the service;
the construction module is used for constructing a data synchronization task corresponding to the data output task of the service node; and the first sending module is used for sending the data synchronization task to a data scheduling device, and the data synchronization task is used for synchronizing the data produced by the data production task to the data table.
In another embodiment of the present disclosure, the data storage device further includes: the second acquisition module is used for acquiring the first data synchronized by the data synchronization task and acquiring a data synchronization mode of the data table; and the storage module is used for storing the first data to the data table based on the data synchronization mode.
In another embodiment of the present disclosure, the memory module includes: the copying unit is used for copying the data table to obtain a temporary table in response to the data synchronization mode being full synchronization; the first storage unit is used for storing the first data into the temporary table to obtain a synchronous temporary table; and the processing unit is used for changing the identification of the synchronous temporary table into the identification of the data table and deleting the data table.
In another embodiment of the present disclosure, the memory module includes: the comparison unit is used for responding to the data synchronization mode to be incremental synchronization and comparing the first data with the second data of the data table so as to determine sub-data in the first data, wherein the sub-data is the incremental data which is not stored in the data table; and the second storage unit is used for storing the sub data into the data table.
In another embodiment of the present disclosure, the data storage device further includes: and the second sending module is used for responding to the deletion of the data table or the pause of the query service corresponding to the data table, sending a deletion request to the data scheduling device, wherein the deletion request is used for requesting the data scheduling device to delete the data synchronization task.
In another embodiment of the present disclosure, the data storage device further includes: the third acquisition module is used for acquiring the query frequency of each data in the data table; and the deleting module is used for deleting the data corresponding to the query frequency lower than the preset frequency.
In a third aspect of embodiments of the present disclosure, there is provided a medium comprising: computer-executable instructions for implementing the method as described above when executed by a processor.
In a fourth aspect of embodiments of the present disclosure, there is provided a computing device comprising:
a memory and a processor;
the memory stores computer-executable instructions;
the processor executes the computer-executable instructions stored by the memory, causing the processor to perform the method as described above.
In the embodiment of the disclosure, the target storage engine for storing the data is determined by the data query mode, the query response time length and the data amount of the data table where the query data is located, so that the data table is created in the target database corresponding to the service level of the service in the target storage engine to store the data, the selection of the storage engine and the database in the storage engine is not dependent on manual operation, the selection time length of the storage engine and the database is shortened, and the storage efficiency of the data is improved.
Drawings
The above, as well as additional purposes, features, and advantages of exemplary embodiments of the present disclosure will become readily apparent from the following detailed description when read in conjunction with the accompanying drawings. Several embodiments of the present disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which:
FIG. 1 schematically illustrates a data storage method application scenario diagram according to an embodiment of the present disclosure;
FIG. 2 schematically illustrates a flow diagram according to an embodiment of the present disclosure;
FIG. 3 schematically illustrates a flow diagram according to another embodiment of the present disclosure;
FIG. 4 schematically illustrates a flow diagram according to yet another embodiment of the present disclosure;
FIG. 5 schematically illustrates a flow diagram according to yet another embodiment of the present disclosure;
FIG. 6 schematically illustrates a schematic diagram of a program product provided in accordance with an embodiment of the present disclosure;
FIG. 7 schematically illustrates a schematic structure of a data storage device provided in accordance with an embodiment of the present disclosure;
fig. 8 schematically illustrates a structural schematic diagram of a computing device provided in accordance with an embodiment of the present disclosure.
In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.
Detailed Description
The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are presented merely to enable one skilled in the art to better understand and practice the present disclosure and are not intended to limit the scope of the present disclosure in any way. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
Those skilled in the art will appreciate that embodiments of the present disclosure may be implemented as a system, apparatus, device, method, or computer program product. Accordingly, the present disclosure may be embodied in the following forms, namely: complete hardware, complete software (including firmware, resident software, micro-code, etc.), or a combination of hardware and software.
According to an embodiment of the disclosure, a data storage method, a data storage device, a data storage medium and a computing device are provided.
Furthermore, any number of elements in the figures is for illustration and not limitation, and any naming is used for distinction only and not for any limiting sense.
In addition, the data related to the disclosure may be data authorized by the user or fully authorized by each party, and the collection, transmission, use and the like of the data all conform to the requirements of national related laws and regulations, and the embodiments of the disclosure may be mutually combined.
The principles and spirit of the present disclosure are explained in detail below with reference to several representative embodiments thereof.
Summary of The Invention
The data needs to be stored in a database before a data query is made. When data is stored, a storage engine is required to be selected, and the data to be stored is stored in a database in the storage engine.
The inventor of the present disclosure found that the selection of the storage engine and the selection of the database in the storage engine are manually performed by a technician, that is, the selection of the storage engine and the database depends on experience of the technician, if the experience of the technician is insufficient, it takes a long time to select the storage engine and the database, resulting in a long time required for data storage and a low data storage efficiency.
The inventor of the present disclosure therefore thinks that a target storage engine for storing data is determined by a data query manner, a query response time length, and a data amount of a data table where query data is located, so that a data table is created in a target database corresponding to a service level of a service in the target storage engine to store data, and selection of the storage engine and the database in the storage engine is not dependent on manual work, so that the selection time length of the storage engine and the database is shortened, and the storage efficiency of the data is improved.
Application scene overview
Referring first to fig. 1, fig. 1 is an application scenario schematic diagram of a data storage method according to an embodiment of the disclosure. The data storage device 100 may be a terminal device, a data processing system, or a cloud server with data storage capabilities. The data storage device 100 is associated with a service, and obtains each enumeration value of the service, where the enumeration values include a data query mode, a query response duration, a data amount of a data table where query data is located, and the like. The data storage device 100 includes different types of storage engines, such as storage engine a, storage engine B, and storage engine C, and the data storage device 100 determines a target storage engine among the plurality of storage engines based on each enumeration value. Illustratively, storage engine A is the target storage engine. The data storage device 100 determines a target database among the databases of the target storage engine, the target database being determined by the service class corresponding to the service. Illustratively, the databases include database a, database b, and database c, which is the target database. The data storage device 100 creates a data table 200 in the target database for storing data generated by the business.
Exemplary method
A data storage method according to an exemplary embodiment of the present disclosure is described below with reference to fig. 2 to 5 in conjunction with the application scenario of fig. 1. It should be noted that the above application scenario is only shown for the convenience of understanding the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in any way in this respect. Rather, embodiments of the present disclosure may be applied to any scenario where applicable.
Referring to fig. 2, fig. 2 is a schematic flow chart schematically illustrating an embodiment of a data storage method according to an embodiment of the disclosure, where the data storage method includes:
step S201, each enumeration value of the service is obtained, wherein the enumeration value comprises a data query mode, a query response time length and a data volume of a data table where query data are located.
In this embodiment, the execution body is a data storage device, and for convenience of description, the device is referred to as a data storage device hereinafter. The device is a terminal device, a cloud server or a data processing system with data storage capability.
The service has a corresponding query scene, which is characterized by the combination of the enumerated values of the service. The device acquires each enumeration value corresponding to the service. Illustratively, the service is associated with a plurality of enumeration values, and the device obtains the plurality of enumeration values associated with the service through the identification of the service. The enumeration value comprises a data query mode, a query response time length and a data volume of a data table where query data are located. Exemplary data query means include single-table queries and multi-table associative queries; the query response time length refers to a response time length section for responding to the query request, and for example, the query response time length includes one hundred milliseconds or less, one hundred milliseconds to five hundred milliseconds, five hundred milliseconds to one thousand milliseconds, one to five seconds, and five to thirty seconds; the data volume refers to a data volume section of the data table, including less than one hundred thousand, one hundred thousand to one million, one million to five million, five million to one million, and more than one million, and the unit of the data volume may be KB. The above-mentioned query response time period and data amount are merely examples, and are not limited to the above-mentioned intervals, and may be set as desired.
Step S202, determining a target storage engine matched with each enumeration value, and acquiring a service grade corresponding to the service.
Different types of storage engines are provided in the device, such as mysql (relational database management system), doris (analytical database) and hbase (distributed open source database).
Different storage engines have different capabilities, while the enumerated values are actually the capabilities that the service requires for the storage engines. In this regard, the device determines a target storage engine based on the respective enumeration values.
In an example, the device determines a target storage engine to which each enumerated value matches, e.g., the storage engine has a tag, e.g., to support multi-table associative queries, data amounts of more than ten millions of data in a data table, and query response durations of less than one hundred milliseconds. The device determines whether each enumerated value has a matching tag in the storage engine, and if each enumerated value has a matching tag in the storage engine, the storage engine may be determined to be the target storage engine to which each enumerated value matches. Illustratively, the enumerated value is a single table query and the tag of the storage engine is a support for the single table query, then the enumerated value is matched with the tag.
In another example, the apparatus builds a tuple parameter from each enumerated value and determines, among the respective storage engines, a storage engine matching the tuple parameter as the target storage engine. Illustratively, each enumerated value constitutes a triplet parameter, such as (single table lookup, one hundred milliseconds or less, one hundred thousand or less). The storage engine is associated with corresponding multi-group parameters, the device compares the constructed multi-group parameters with the multi-group parameters associated with the storage engine, and if the constructed multi-group parameters are matched with the multi-group parameters, the storage engine is taken as a target storage engine. For example, if the constructed tuple parameters are the same as the tuple parameters associated with the storage engine, then the two tuple parameters are matched. It should be noted that, after the multi-element parameter is constructed, if the constructed multi-element parameter does not have a matched storage engine, the multi-element parameter formed by each enumerated value does not conform to the specification, and prompt information needs to be output to change the enumerated value of the service.
In step S203, a target database corresponding to the service level is determined in the target storage engine, and a data table is created in the target database, where the data table is used to store data generated by the service.
Multiple databases are configured in the storage engine, and each database is for use by a different tenant in the business, and different tenants may correspond to multiple parallel databases. After the target storage engine is determined, a suitable database needs to be found among the multiple databases for storing data of the service.
The services have different levels, which refers to the levels defined by the services of the services in the application management system, which are defined as service levels. The service level is associated with the identification of the service, and the device acquires the service level corresponding to the service through the identification of the service. Each class of service corresponds to a different database. Illustratively, the service level is level 1, and then corresponds to databases 1 to 30; the service level is 2, and the database 31 to the database 50 are corresponding; the service class is 3, and the database 51 to the database 60 are corresponding. The apparatus determines a target database based on the service level.
In an example, one database is optionally selected from the respective databases corresponding to the service level as the target database.
In another example, a plurality of databases corresponding to the service level are determined in the target storage engine, the databases being defined as databases to be determined; the device acquires a load balancing strategy corresponding to the target storage engine, and determines the database to be determined matched with the load balancing strategy in each database to be determined as a target database. The load balancing strategy is to keep the difference of the data amount in each database small, and the device determines the database with the smallest data amount in each database to be determined as the target database. In addition, the load balancing policy may also be a polling of the databases, that is, each database has a sequence, the device obtains a first sequence number of a database that creates the data table, and uses a database corresponding to a second sequence number located below the first sequence number as the target database.
After determining the target database, the apparatus creates a data table in the target database, the data table being used to store data generated by the service. The device obtains the structure of the source table in the database, creates the data table based on the structure, for example, the source table is a HIVE type data table, and after obtaining the structure of the HIVE source table, the device converts part of fields included in the structure, for example, converts the fields of the characterization type, for example, converts the fields of the characterization MYSQL and DORIS into string- > varchar.
In this embodiment, the target storage engine for storing data is determined according to the data query mode, the query response time length and the data amount of the data table where the query data is located, so that the data table is created in the target database corresponding to the service level of the service in the target storage engine to store the data, the selection of the storage engine and the database in the storage engine is not performed manually, the selection time length of the storage engine and the database is shortened, and the storage efficiency of the data is improved.
Referring to fig. 3, fig. 3 is a schematic flow chart schematically illustrating another embodiment of a data storage method according to an embodiment of the disclosure, based on the foregoing embodiment, after step S203, further includes:
step S301, based on the query statement of the service, analyzing the service node generated by the data stored in the data table.
In this embodiment, the offline task of the database includes a task and a node, where the task is generally composed of a plurality of nodes with operation dependencies, and the operation dependencies refer to operation results of an operation dependency of a certain node and another node. The data stored in the database are generated by the nodes, the nodes are defined as service nodes, and the nodes correspond to data output tasks, namely, the data generated by the data output tasks need to be stored in the created data table in the database. For this, the device needs to determine the blood-edge relationship between the service node and the data table.
In this regard, the device obtains a query statement of the service, for example, an sql (Structured Query Language, database language) statement, so that a relationship between a source table and a service node in the database can be obtained, and the data table is created based on the source table, so that a blood-edge relationship from the service node to the data table can be found, that is, the service node generated by data stored in the data table can be determined by analyzing the query statement.
Step S302, a data synchronization task corresponding to the data output task of the service node is constructed.
The service node is provided with a data output task, the device pulls metadata of the task, a task relation corresponding to the service node can be determined based on the metadata, the data output task corresponding to the service node can be determined from the task relation, the device can construct a blood-margin relation between the data output task and the data table, and data output by the data output task is stored in the data table based on the blood-margin relation.
In an exemplary technique, after the data output task is completed, data needs to be transmitted to the data table through a data link and a data transmission period set by the service node, where the data transmission link is longer and the data transmission period is longer, for example, the data transmission period is one day in a compartment. In order to timely schedule the produced data, the device constructs a data synchronization task corresponding to the data production task.
In step S303, the data synchronization task is sent to the data scheduling device, where the data synchronization task is used to synchronize the data generated by the data generating task to the data table.
The device sends the data synchronous task to the data scheduling device, and the data scheduling device is entity equipment of a task scheduling platform and is used for monitoring whether the data output task is finished or not and acquiring the data output by the data output task. After the data output task is completed, the data synchronization task can timely synchronize data generated by the data output task into the data table, so that the data is timely stored, the overlong storage period of the data is avoided, and the timely scheduling of the data is realized.
In addition, in order to place modification of the scheduling period or time of the upstream task, the device sets a monitoring thread for periodically monitoring the scheduling configuration of the data output task, if the scheduling configuration of the data output task changes, the device synchronously updates the data synchronization task based on the changed scheduling configuration and sends the updated data synchronization task to the data scheduling device, so that the updated data synchronization task can acquire data generated by the data output task with the changed scheduling configuration.
In this embodiment, the device analyzes the service node generated by the data stored in the data table based on the query statement of the service, and constructs a data synchronization task corresponding to the data output task of the service node, so that the data generated by the data output task is quickly obtained based on the data synchronization task, so as to store the data in the data table, and schedule the data in time.
Referring to fig. 4, fig. 4 is a schematic flow chart schematically illustrating a further embodiment of a data storage method according to an embodiment of the disclosure, and after step 303, based on the embodiment illustrated in fig. 3, further includes:
step S401, first data synchronized by a data synchronization task is obtained, and a data synchronization mode of a data table is obtained.
In this embodiment, in order to ensure consistency of data and shielding synchronization details of services, a complete consistency snapshot is always provided, and when a data synchronization task is scheduled to run, a front processing and a rear processing are required to be performed on a data synchronization process based on different data synchronization modes, that is, data synchronized by the data synchronization task is stored according to the data synchronization modes.
In this regard, the device acquires data synchronized by the data synchronization task, the data being defined as first data, and the device acquires the data synchronization pattern of the data table. The data synchronization mode comprises full synchronization and increment synchronization. The data synchronization pattern is associated with the storage engine and the device obtains the data synchronization pattern based on the target storage engine.
Step S402, the first data is stored in the data table based on the data synchronization mode.
After determining the data synchronization pattern, the device stores the first data to the data table based on the data synchronization pattern.
In one example, the data synchronization mode is full synchronization, and the temporary table mode is adopted for data storage. The device copies the data table as a temporary table, and uses the temporary table as the data table in the data synchronization process, namely, the device stores the first data into the temporary table to obtain a synchronous temporary table. The device changes the identification of the synchronous temporary table into the identification of the data table, and the data table is deleted to finish the data synchronization, the data table is deleted, and the query service is not perceived, i.e. the query task is not influenced.
In another example, when the data synchronization mode is incremental synchronization, the device compares the first data with the second data of the data table to determine the sub-data in the first data, where the sub-data is incremental data, that is, the sub-data is not stored in the data table, and the device stores the sub-data to the data volume, the data storage may be completed.
In this embodiment, the device acquires the first data synchronized by the data synchronization task and the data synchronization manner of the data table, so that the first data is stored in the data table according to the data synchronization manner.
Referring to fig. 5, fig. 5 is a schematic flow chart schematically illustrating still another embodiment of a data storage method according to an embodiment of the present disclosure, after step 303, based on the embodiment shown in fig. 3 or fig. 4, further includes:
in step S501, in response to the deletion of the data table or the suspension of the query service corresponding to the data table, a deletion request is sent to the data scheduling device, where the deletion request is used to request the data scheduling device to delete the data synchronization task.
In this embodiment, after the data synchronization is combined with the data query, the device has a normative constraint and a lifecycle of data service consistency, and can perform automated operation and maintenance on the data synchronization.
In one example, the apparatus performs inspection of the data synchronization tasks, such as checking the number of single-table databases, etc., for monitoring alarms, in a regular task manner.
In another example, for event driven, the lifecycle of the data synchronization task is managed based on the lifecycle of the data. For example, when the deletion of the data table or the suspension of the query service corresponding to the data table is detected, the scheduling of the data synchronization task may be canceled, that is, the data synchronization task is deleted, and the device sends a deletion request to the data scheduling device, where the deletion request is used to request the data scheduling device to delete the data synchronization task.
In yet another example, the apparatus may perform recognition of the cold model, that is, recognize data that is not frequently queried, perform deletion, and make room for the database. In contrast, after the target database creates the data table, the device acquires the query frequency of each data in the data table, and deletes the data corresponding to the query frequency lower than the preset frequency. In addition, the alarm information can be generated based on the data corresponding to the query frequency lower than the preset frequency, and the alarm information is output, so that related personnel find the data based on the alarm information to determine whether the data can be deleted.
In this embodiment, the device manages the data synchronization task to end the storage of the data in the data table, so as to avoid the suspension of the query service, and the data table corresponding to the query service still stores the data, thereby avoiding the waste of storage resources in the database.
Exemplary Medium
Having described the method of the exemplary embodiments of the present disclosure, next, a storage medium of the exemplary embodiments of the present disclosure will be described with reference to fig. 6.
Referring to fig. 6, a storage medium 60 has stored therein a program product for implementing the methods described above according to embodiments of the present disclosure, which may employ a portable compact disc read only memory (CD-ROM) and include computer-executable instructions for causing a computing device to perform the data storage methods provided by the present disclosure. However, the program product of the present disclosure is not limited thereto.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave in which the computer-executable instructions are carried. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. The readable signal medium may also be any readable medium other than a readable storage medium.
Computer-executable instructions for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-executable instructions may be executed entirely on the user computing device, partly on the user device, partly on the remote computing device, or entirely on the remote computing device or server. In the context of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN).
Exemplary apparatus
Having described the medium of the exemplary embodiment of the present disclosure, next, a description will be given of a terminal device of the exemplary embodiment of the disclosure with reference to fig. 7, where the terminal device is configured to implement the method in any of the foregoing data storage method embodiments, and implementation principles and technical effects are similar.
Referring to fig. 7, fig. 7 schematically illustrates a schematic structure of a data storage device provided according to an embodiment of the present disclosure.
As shown in fig. 7, the data storage device includes: a first obtaining module 710, configured to obtain each enumeration value of a service, where the enumeration value includes a data query mode, a query response duration, and a data amount of a data table where query data is located; a first determining module 720, configured to determine a target storage engine with which each enumerated value is matched, and obtain a service class corresponding to the service; and a second determining module 730, configured to determine a target database corresponding to the service level in the target storage engine, and create a data table in the target database, where the data table is used to store data generated by the service.
In one embodiment, the first determining module 720 includes: the construction unit is used for constructing a plurality of group parameters according to each enumeration value; and the first determining unit is used for determining the storage engine matched with the multi-group parameters in the storage engines as a target storage engine.
In one embodiment, the second determining module 730 includes: the second determining unit is used for determining a plurality of databases to be determined corresponding to the service level in the target storage engine; the acquisition unit is used for acquiring a load balancing strategy corresponding to the target storage engine; and the third determining unit is used for determining the databases to be determined matched with the load balancing strategy in the databases to be determined as target databases.
In one embodiment, the data storage device further comprises: the analysis module is used for analyzing service nodes generated by data stored in the data table based on query sentences of the service; the construction module is used for constructing a data synchronization task corresponding to the data output task of the service node; the first sending module is used for sending the data synchronization task to the data scheduling device, and the data synchronization task is used for synchronizing the data produced by the data production task to the data table.
In one embodiment, the data storage device further comprises: the second acquisition module is used for acquiring the first data synchronized by the data synchronization task and acquiring a data synchronization mode of the data table; and the storage module is used for storing the first data to the data table based on the data synchronization mode.
In one embodiment, a memory module includes: the copying unit is used for copying the data table to obtain a temporary table in response to the data synchronization mode being full synchronization; the first storage unit is used for storing the first data into the temporary table to obtain a synchronous temporary table; and the processing unit is used for changing the identification of the synchronous temporary table into the identification of the data table and deleting the data table.
In one embodiment, a memory module includes: the comparison unit is used for responding to the data synchronization mode as increment synchronization and comparing the first data with the second data of the data table so as to determine sub-data in the first data, wherein the sub-data is the increment data which is not stored in the data table; and the second storage unit is used for storing the sub data into the data table.
In one embodiment, the data storage device further comprises: and the second sending module is used for responding to the deletion of the data table or the pause of the query service corresponding to the data table, sending a deletion request to the data scheduling device, and requesting the data scheduling device to delete the data synchronization task.
In one embodiment, the data storage device further comprises: the third acquisition module is used for acquiring the query frequency of each data in the data table; and the deleting module is used for deleting the data corresponding to the query frequency lower than the preset frequency.
Exemplary computing device
Having described the methods, media, and apparatus of exemplary embodiments of the present disclosure, a computing device of exemplary embodiments of the present disclosure is next described with reference to fig. 8.
The computing device 80 shown in fig. 8 is merely an example and should not be taken as limiting the functionality and scope of use of embodiments of the present disclosure. As shown in fig. 8, the computing device 80 is in the form of a general purpose computing device. Components of computing device 80 may include, but are not limited to: at least one processing unit 801, at least one memory unit 802, a bus 803 connecting the different system components, including the processing unit 801 and the memory unit 802. Wherein at least one memory unit 802 has stored therein computer-executable instructions; at least one processing unit 801 includes a processor that executes computer-executable instructions to implement the methods described above.
Bus 803 includes a data bus, a control bus, and an address bus.
The storage unit 802 may include readable media in the form of volatile memory, such as Random Access Memory (RAM) 8021 and/or cache memory 8022, and may further include readable media in the form of nonvolatile memory, such as Read Only Memory (ROM) 8023.
The storage unit 802 may also include a program/utility 8025 having a set (at least one) of program modules 8024, such program modules 8024 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
The computing device 80 may also communicate with one or more external devices 804 (e.g., keyboard, pointing device, etc.). Such communication may occur through an input/output (I/O) interface 805. Moreover, computing device 80 may also communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet, through network adapter 806. As shown in fig. 8, network adapter 806 communicates with other modules of computing device 80 over bus 803. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with computing device 80, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.
It should be noted that although in the above detailed description several units/modules or sub-units/modules of a terminal device/server are mentioned, such a division is only exemplary and not mandatory. Indeed, the features and functionality of two or more units/modules described above may be embodied in one unit/module in accordance with embodiments of the present disclosure. Conversely, the features and functions of one unit/module described above may be further divided into ones that are embodied by a plurality of units/modules.
Furthermore, although the operations of the methods of the present disclosure are depicted in the drawings in a particular order, this is not required to or suggested that these operations must be performed in this particular order or that all of the illustrated operations must be performed in order to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform.
While the spirit and principles of the present disclosure have been described with reference to several particular embodiments, it is to be understood that this disclosure is not limited to the particular embodiments disclosed nor does it imply that features in these aspects are not to be combined to benefit from this division, which is done for convenience of description only. The disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
Claims (10)
1. A method of data storage, comprising:
acquiring each enumeration value of a service, wherein the enumeration values comprise a data query mode, a query response time length and a data volume of a data table where query data are located;
determining target storage engines matched with the enumeration values, and acquiring service grades corresponding to the businesses;
and determining a target database corresponding to the service level in the target storage engine, and creating a data table in the target database, wherein the data table is used for storing data generated by the service.
2. The data storage method of claim 1, wherein the determining a target storage engine for which each of the enumerated values matches comprises:
constructing a plurality of group parameters according to each enumeration value;
among the storage engines, a storage engine matching the multi-group parameter is determined as the target storage engine.
3. The data storage method according to claim 1, wherein determining, in the target storage engine, a target database corresponding to the service class, comprises:
determining a plurality of databases to be determined corresponding to the service level in the target storage engine;
acquiring a load balancing strategy corresponding to the target storage engine;
and determining the databases to be determined matched with the load balancing strategy in each database to be determined as the target database.
4. The data storage method of claim 1, wherein after the target database creates a data table, further comprising:
analyzing service nodes generated by data stored in the data table based on the query statement of the service;
constructing a data synchronization task corresponding to the data output task of the service node;
and sending the data synchronization task to a data scheduling device, wherein the data synchronization task is used for synchronizing the data produced by the data production task to the data table.
5. The data storage method according to claim 4, further comprising, after the data synchronization task is sent to the data scheduling device:
acquiring first data synchronized by the data synchronization task, and acquiring a data synchronization mode of the data table;
and storing the first data to the data table based on the data synchronization mode.
6. The data storage method according to claim 5, wherein the storing the first data to the data table based on the data synchronization manner comprises:
copying the data table to obtain a temporary table in response to the data synchronization mode being full synchronization;
storing the first data to the temporary table to obtain a synchronous temporary table;
and changing the identification of the synchronous temporary table into the identification of the data table, and deleting the data table.
7. The data storage method according to claim 5, wherein the storing the first data to the data table based on the data synchronization manner comprises:
responding to the data synchronization mode as increment synchronization, comparing the first data with the second data of the data table to determine sub-data in the first data, wherein the sub-data is not stored in the data table;
and storing the sub data into the data table.
8. A data storage device, comprising:
the first acquisition module is used for acquiring each enumeration value of the service, wherein the enumeration values comprise a data query mode, a query response time length and a data volume of a data table where query data are located;
the first determining module is used for determining the target storage engine matched with each enumeration value and acquiring the service grade corresponding to the service;
and the second determining module is used for determining a target database corresponding to the service level in the target storage engine and creating a data table in the target database, wherein the data table is used for storing data generated by the service.
9. A medium, comprising: computer-executable instructions for implementing the method of any one of claims 1 to 7 when executed by a processor.
10. A computing device, comprising:
a memory and a processor;
the memory stores computer-executable instructions;
the processor executing computer-executable instructions stored in the memory, causing the processor to perform the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311205621.9A CN117271570A (en) | 2023-09-18 | 2023-09-18 | Data storage methods, devices, media and computing equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311205621.9A CN117271570A (en) | 2023-09-18 | 2023-09-18 | Data storage methods, devices, media and computing equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117271570A true CN117271570A (en) | 2023-12-22 |
Family
ID=89218951
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311205621.9A Pending CN117271570A (en) | 2023-09-18 | 2023-09-18 | Data storage methods, devices, media and computing equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117271570A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN120011360A (en) * | 2025-01-15 | 2025-05-16 | 临海市新睿电子科技股份有限公司 | A dynamic storage management method and management platform for Internet of Things device data |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080162414A1 (en) * | 2006-12-28 | 2008-07-03 | Kirk Steven A | Accelerating queries using delayed value projection of enumerated storage |
| CN112905595A (en) * | 2021-03-05 | 2021-06-04 | 腾讯科技（深圳）有限公司 | Data query method and device and computer readable storage medium |
| CN114579617A (en) * | 2022-03-14 | 2022-06-03 | 中国工商银行股份有限公司 | Data query method and device, computer equipment and storage medium |
| CN115544081A (en) * | 2022-05-24 | 2022-12-30 | 北京字跳网络技术有限公司 | Data query method, device, electronic equipment and storage medium |
| CN116303354A (en) * | 2023-03-10 | 2023-06-23 | 北京奇艺世纪科技有限公司 | A storage engine determination method, device, electronic equipment and storage medium |
-
2023
- 2023-09-18 CN CN202311205621.9A patent/CN117271570A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080162414A1 (en) * | 2006-12-28 | 2008-07-03 | Kirk Steven A | Accelerating queries using delayed value projection of enumerated storage |
| CN112905595A (en) * | 2021-03-05 | 2021-06-04 | 腾讯科技（深圳）有限公司 | Data query method and device and computer readable storage medium |
| CN114579617A (en) * | 2022-03-14 | 2022-06-03 | 中国工商银行股份有限公司 | Data query method and device, computer equipment and storage medium |
| CN115544081A (en) * | 2022-05-24 | 2022-12-30 | 北京字跳网络技术有限公司 | Data query method, device, electronic equipment and storage medium |
| CN116303354A (en) * | 2023-03-10 | 2023-06-23 | 北京奇艺世纪科技有限公司 | A storage engine determination method, device, electronic equipment and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN120011360A (en) * | 2025-01-15 | 2025-05-16 | 临海市新睿电子科技股份有限公司 | A dynamic storage management method and management platform for Internet of Things device data |
| CN120011360B (en) * | 2025-01-15 | 2025-10-24 | 临海市新睿电子科技股份有限公司 | A dynamic storage management method and management platform for Internet of Things device data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11615076B2 (en) | Monolith database to distributed database transformation | |
| US9971820B2 (en) | Distributed system with accelerator-created containers | |
| US20100088271A1 (en) | Hsm two-way orphan reconciliation for extremely large file systems | |
| CN103514223A (en) | Data synchronism method and system of database | |
| CN115374102A (en) | Data processing method and system | |
| US20150317331A1 (en) | Unified platform for application development | |
| US9898372B2 (en) | Backing up a computer application | |
| CN116028577B (en) | Data synchronization method and device, electronic device and storage medium | |
| CN117235101B (en) | Systems and methods for querying resource caches | |
| US8538963B2 (en) | Optimal persistence of a business process | |
| CN112817989B (en) | Data processing method, data processing device, storage medium and electronic equipment | |
| CN117271570A (en) | Data storage methods, devices, media and computing equipment | |
| CN115391457B (en) | Cross-database data synchronization method, device and storage medium | |
| CN107004036B (en) | Method and system for searching logs containing a large number of entries | |
| US7483870B1 (en) | Fractional data synchronization and consolidation in an enterprise information system | |
| CN118445306A (en) | Method, device, equipment and medium for offline analysis of pre-written log | |
| US11947490B2 (en) | Index generation and use with indeterminate ingestion patterns | |
| CN113010616B (en) | Data processing method and data processing system | |
| CN113157716A (en) | Data processing method, device, equipment and medium | |
| CN115617900A (en) | Data synchronization method, device, equipment and medium | |
| CN113761446B (en) | Network public opinion monitoring method, device, equipment, program product and storage medium | |
| Fjällid | A comparative study of databases for storing sensor data | |
| HK40079476A (en) | Data synchronization method, apparatus, device and medium | |
| CN118035178B (en) | Data investigation method, device, electronic equipment, computer readable storage medium and program product | |
| CN118626501A (en) | A method and device for updating table data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |