CN114020629B - Data processing method and device, management platform of test environment and electronic equipment - Google Patents

Abstract

The invention discloses a data processing method and device, a management platform of a test environment and electronic equipment. The processing method comprises the following steps: receiving a service processing request, wherein the service processing request carries an environment identifier of an environment to be processed, responding to the service processing request, inquiring environment information of the environment to be processed from an environment management module, generating a data task of the environment to be processed based on the environment information of the environment to be processed, controlling a task processing module to execute the data task to obtain an execution result, and updating version information of the environment to be processed based on the execution result to complete the service processing request. The invention solves the technical problem of low service processing efficiency caused by the fact that various types of test environments cannot be uniformly managed in the related technology.

Description

Data processing method and device, test environment management platform, and electronic equipment

Technical Field

The present invention relates to the field of data processing technology, and in particular to a data processing method and device thereof, a management platform for a test environment, and electronic equipment.

Background Art

With the widespread use of DevOps and agile software development, building, developing, testing, and releasing software have become faster and more frequent. Therefore, a management platform is needed that can effectively manage complex and changeable test environments so that these test environments can have both flexibility and reliability to support the needs of fast and frequent software iterations.

In the related art, there are several difficulties in establishing a management platform that can effectively manage complex and ever-changing test environments: (1) The test environment has a complex structure with multiple branches and versions. In DevOps activities, it is often necessary to quickly create new branches and versions, and it is also often necessary to frequently switch between different branches and versions. (2) With the development of microservices, the test database is split into multiple databases along with the services, and the sub-databases as a cluster also need to be quickly and conveniently migrated as a whole along with the test environment. Therefore, it brings great inconvenience to the management of the test environment and also makes the efficiency of handling related business low.

To address the above-mentioned problems, no effective solution has been proposed yet.

Summary of the invention

The embodiments of the present invention provide a data processing method and device thereof, a test environment management platform, and an electronic device, so as to at least solve the technical problem in the related art that various types of test environments cannot be managed uniformly, resulting in low efficiency in processing business.

According to one aspect of an embodiment of the present invention, there is provided a data processing method, which is applied to a version control module in a development environment management platform, wherein the development environment management platform further includes: an environment management module and a task processing module, and the data processing method includes: receiving a business processing request, wherein the business processing request carries an environment identifier of an environment to be processed; responding to the business processing request, querying the environment management module for environment information of the environment to be processed; generating a data task for the environment to be processed based on the environment information of the environment to be processed; controlling the task processing module to execute the data task, obtaining an execution result, and updating the version information of the environment to be processed based on the execution result to complete the business processing request.

Optionally, the environment type of the environment to be processed includes: test environment and production environment. If the environment to be processed is a test environment, the environment information of the environment to be processed includes: source data of the test environment and a list of data sources corresponding to the test environment. Each of the test environments corresponds to at least one environment version, and each environment version corresponds to a data source.

Optionally, the types of the business processing request include: creating a test environment version, switching a test environment version, and restoring production data.

Optionally, if the environment to be processed is a test environment and the type of the business processing request is to create a test environment version, then based on the environmental information of the environment to be processed, a step of generating a data task for the environment to be processed includes: based on the environmental information of the test environment, generating a version creation task for the test environment; controlling the task processing module to execute the data task and obtaining an execution result includes: after receiving the version creation task, controlling the task processing module to send a test data backup request to the database operation module in the development environment management platform according to the data source list carried in the version creation task; updating the version information of the environment to be processed based on the execution result and completing the business processing request includes: controlling the database operation module in the development environment management platform to respond to the test data backup request, execute a test data backup operation, and save the backup file to the data resource library; controlling the task processing module to register the created target version of the data resources, and update the version information of the test environment to complete the version creation task.

Optionally, if the environment to be processed is a test environment and the type of the business processing request is to switch the test environment version, then based on the environment information of the environment to be processed, the step of generating a data task for the environment to be processed includes: based on the environment information of the current test environment and the data resource information of the test environment to be switched, generating a version switching task corresponding to the test environment to be switched; the step of controlling the task processing module to execute the data task and obtaining an execution result includes: after receiving the version switching task, controlling the task processing module to extract the data resources of the switched version of the current test environment according to the data resource information, and sending a test data restoration request to the database operation module in the development environment management platform, wherein the test data restoration request carries a data source list and the data resource information; the step of updating the version information of the environment to be processed based on the execution result and completing the business processing request includes: controlling the database operation module in the development environment management platform to respond to the test data restoration request, accessing the data source to be switched according to the data source list and data resource information in the test data restoration request, executing the test data restoration operation, returning the data restoration result to the task processing module, and updating the version information of the test environment to be switched to complete the version switching task.

Optionally, the step of extracting the switched version of data resources of the current test environment according to the data resource information includes: sending a resource extraction request to the data resource management module in the development environment management platform according to the data resource information to extract the switched version of data resources of the current test environment.

Optionally, if the environment to be processed is a test environment, then after receiving the business processing request, the data processing method also includes: sending a data source import request to a data resource management module in the development environment management platform, wherein the data source import request is used to import the environmental data resources of the test environment into a data resource library.

Optionally, if the environment to be processed is a test environment and the type of the business processing request is production data restoration, the step of generating a data task for the environment to be processed based on the environmental information of the environment to be processed includes: creating a target production data version, and establishing an association between the target production data version and the environmental data resources in the data resource library; sending a production data restoration request to the task processing module based on the version information of the target production data version; updating the version information of the environment to be processed based on the execution result, and completing the step of the business processing request includes: controlling the database operation module to respond to the production data restoration request, accessing the data source according to the production data resources and the version information, performing the production data restoration on-site operation, and obtaining the execution result; returning the execution result to the task processing module, and updating the version information of the test environment to complete the production data restoration task.

Optionally, the step of controlling the task processing module to execute the data task and obtain the execution result includes: controlling the task processing module to send the data task to the task buffer pool after receiving the data task; and executing each of the data tasks according to the task queuing order of the task buffer pool.

According to another aspect of an embodiment of the present invention, a test environment management platform is also provided, including: an environment management module for managing environment information of a test environment or production; a version control module for executing any of the data processing methods described above; a database operation module for library operations of a data resource library corresponding to the test environment, wherein the library operations include at least one of the following: library connection, data backup, and data restoration of the data resource library; a task processing module for executing business processing requests and returning execution results; a data resource management module for managing test data under each environment version under the test environment and completing version switching tasks of the test environment.

Optionally, the task processing module includes: a task buffer pool, and the task buffer pool is used to schedule multiple data tasks to be processed.

According to another aspect of an embodiment of the present invention, there is also provided a data processing device, which is applied to a version control module in a development environment management platform. The development environment management platform also includes: an environment management module and a task processing module. The data processing device includes: a receiving unit, which is used to receive a business processing request, wherein the business processing request carries an environment identifier of the environment to be processed; a query unit, which is used to respond to the business processing request and query the environment management module for the environment information of the environment to be processed; a generation unit, which is used to generate a data task for the environment to be processed based on the environment information of the environment to be processed; and a control unit, which is used to control the task processing module to execute the data task, obtain an execution result, and update the version information of the environment to be processed based on the execution result to complete the business processing request.

Optionally, the environment type of the environment to be processed includes: test environment and production environment. If the environment to be processed is a test environment, the environment information of the environment to be processed includes: source data of the test environment and a list of data sources corresponding to the test environment. Each of the test environments corresponds to at least one environment version, and each environment version corresponds to a data source.

Optionally, the types of the business processing request include: creating a test environment version, switching a test environment version, and restoring production data.

Optionally, the data processing device includes: a first generating module, which is used to generate a data task for the environment to be processed based on the environment information of the environment to be processed if the environment to be processed is a test environment and the type of the business processing request is to create a test environment version, including: generating a version creation task for the test environment based on the environment information of the test environment; controlling the task processing module to execute the data task and obtain an execution result, including: a first sending module, which is used to control the task processing module to send a test data backup request to the database operation module in the development environment management platform according to the data source list carried in the version creation task after receiving the version creation task; updating the version information of the environment to be processed based on the execution result to complete the business processing request, including: a first saving module, which is used to control the database operation module in the development environment management platform to respond to the test data backup request, perform a test data backup operation, and save the backup file to the data resource library; a first updating module, which is used to control the task processing module to register the data resources of the target version created, and update the version information of the test environment to complete the version creation task.

Optionally, the data processing device includes: a second generation module, which is used to generate a version switching task corresponding to the test environment to be switched based on the environment information of the current test environment and the data resource information of the test environment to be switched if the environment to be processed is a test environment and the type of the business processing request is to switch the test environment version; a first extraction module, which is used to control the task processing module to extract the data resources of the switched version of the current test environment according to the data resource information after receiving the version switching task, and send a test data restoration request to the database operation module in the development environment management platform, wherein the test data restoration request carries a data source list and the data resource information; a second update module, which is used to control the database operation module in the development environment management platform to respond to the test data restoration request, access the data source to be switched according to the data source list and data resource information in the test data restoration request, perform a test data restoration operation, return the data restoration result to the task processing module, and update the version information of the test environment to be switched to complete the version switching task.

Optionally, the first extraction module includes: a first extraction submodule, configured to send a resource extraction request to a data resource management module in the development environment management platform according to the data resource information, so as to extract the switched version of the data resource of the current test environment.

Optionally, the data processing device also includes: a first import module, which is used to send a data source import request to the data resource management module in the development environment management platform after receiving a business processing request if the environment to be processed is a test environment, wherein the data source import request is used to import the environmental data resources of the test environment into the data resource library.

Optionally, the data processing device includes: a first creation module, which is used to create a target production data version if the environment to be processed is a test environment and the type of the business processing request is production data restoration, and to establish an association relationship between the target production data version and the environmental data resource in the data resource library; a second sending module, which is used to send a production data restoration request to the task processing module based on the version information of the target production data version; a third update module, which is used to control the database operation module to respond to the production data restoration request, access the data source according to the production data resources and the version information, perform the production data restoration on-site operation, and obtain the execution result; return the execution result to the task processing module, and update the version information of the test environment to complete the production data restoration task.

Optionally, the control unit includes: a third sending module, used to control the task processing module to send the data task to the task buffer pool after receiving the data task; and a first execution module, used to execute each of the data tasks according to the task queuing order of the task buffer pool.

According to another aspect of an embodiment of the present invention, there is further provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned data processing methods by executing the executable instructions.

According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the data processing methods described above.

In the present disclosure, a development environment management platform (or test data environment management platform) supporting multiple branches, multiple versions and multiple libraries is provided. By uniformly managing the test data corresponding to the environments of multiple branches, multiple versions and multiple databases under each application, it is suitable for functional scenarios such as the production and restoration of test data versions, and the version switching between different environments within the application. In addition, the present disclosure also has the function of restoring production data on site.

In the present disclosure, a business processing request is received, wherein the business processing request carries an environment identifier of an environment to be processed, a business processing request is responded to, the environment information of the environment to be processed is queried from the environment management module, a data task of the environment to be processed is generated based on the environment information of the environment to be processed, a task processing module is controlled to execute the data task, an execution result is obtained, and the version information of the environment to be processed is updated based on the execution result to complete the business processing request. In the present application, a development environment management platform is used to uniformly manage the environment information of the environment to be processed (the management content that can be realized includes but is not limited to: the production and restoration of test data versions, the version switching between different environments within the application, and the restoration of production data), which is suitable for processing businesses in multiple environments and improving the efficiency of processing businesses, thereby solving the technical problem that the related technology cannot uniformly manage various types of test environments, resulting in low efficiency in processing businesses.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

FIG1 is a flow chart of an optional data processing method according to an embodiment of the present invention;

FIG2 is a schematic diagram of an optional process for creating a test environment version task according to an embodiment of the present invention;

FIG3 is a schematic diagram of an optional task of processing switching test environment versions according to an embodiment of the present invention;

FIG4 is a schematic diagram of an optional processing production data restoration task according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of an optional data processing device according to an embodiment of the present invention.

DETAILED DESCRIPTION

In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

To facilitate those skilled in the art to understand the present invention, some terms or nouns involved in the embodiments of the present invention are explained below:

DevOps: A collective name for a set of processes, methods, and systems that facilitate communication, collaboration, and integration between development (application/software engineering), technical operations, and quality assurance (QA) departments.

Agile software development: a software development capability that can cope with rapidly changing requirements, focusing on close collaboration and face-to-face communication between programmer teams and business experts, frequent delivery of new software versions, compact and self-organizing teams, code writing and team organization methods that can adapt well to changing requirements, and also pay more attention to the role of the software development process.

The following embodiments of the present invention can be applied to management platforms/systems/applications of various complex and changeable test environments, or to scenarios of processing business under complex and changeable test environments. The present invention can uniformly manage the test data corresponding to multiple branches, multiple versions and environments involving multiple databases under each application by supporting a test data environment management method for multiple branches, multiple versions and multiple libraries. It is suitable for functional scenarios such as the production and restoration of test data versions, as well as version switching between different environments in the same application. At the same time, it also has the function of restoring production data on site.

Embodiment 1

According to an embodiment of the present invention, a data processing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

FIG. 1 is a flow chart of an optional data processing method according to an embodiment of the present invention. As shown in FIG. 1 , the method includes the following steps:

Step S102: receiving a business processing request, wherein the business processing request carries an environment identifier of an environment to be processed.

Step S104, responding to the business processing request, querying the environment management module for the environment information of the environment to be processed.

Step S106: Generate data tasks for the environment to be processed based on the environment information of the environment to be processed.

Step S108, controlling the task processing module to execute the data task, obtaining the execution result, and updating the version information of the environment to be processed based on the execution result to complete the business processing request.

Through the above steps, a business processing request can be received, wherein the business processing request carries the environment identifier of the environment to be processed, responds to the business processing request, queries the environment information of the environment to be processed from the environment management module, generates data tasks of the environment to be processed based on the environment information of the environment to be processed, controls the task processing module to execute the data tasks, obtains the execution result, and updates the version information of the environment to be processed based on the execution result to complete the business processing request. In an embodiment of the present invention, the development environment management platform is used to uniformly manage the environment information of the environment to be processed, which is suitable for processing businesses in multiple environments and improving the efficiency of processing businesses, thereby solving the technical problem that the related technology cannot uniformly manage various types of test environments, resulting in low efficiency in processing businesses.

The embodiment of the present invention is described in detail below in combination with the above steps.

The embodiment of the present invention is applied to a version control module in a development environment management platform, wherein the management platform of the development environment management platform (or test environment) includes: an environment management module for managing environment information of a test environment or production; a version control module for executing any one of the above data processing methods; a database operation module for library operations of a data resource library corresponding to the test environment, wherein the library operations include at least one of the following: library connection, data backup, and data restoration of the data resource library; a task processing module for executing business processing requests and returning execution results; a data resource management module for managing test data under each environment version under the test environment and completing version switching tasks of the test environment.

Optionally, the task processing module includes: a task buffer pool, which is used to schedule multiple data tasks to be processed.

Step S102: receiving a business processing request, wherein the business processing request carries an environment identifier of an environment to be processed.

In an embodiment of the present invention, a business processing request can be received through a version control module, wherein the version control module is a core processing module in a development environment management platform, and is responsible for performing services such as creating versions and switching versions for a test environment. The business processing request carries an environment identifier of the environment to be processed, and the environment identifier is used to identify corresponding environment information.

Optionally, the environment types of the environment to be processed include: test environment and production environment. If the environment to be processed is a test environment, the environment information of the environment to be processed includes: source data of the test environment and a list of data sources corresponding to the test environment. Each test environment corresponds to at least one environment version, and each environment version corresponds to a data source.

In an embodiment of the present invention, each test environment may have multiple environment versions, each environment version corresponds to a corresponding data source, and the environmental information in the production environment cannot be changed at will. The data source of the production environment can be restored/copied to the test environment for analysis and processing.

Optionally, the types of business processing requests include: creating a test environment version, switching a test environment version, and restoring production data.

In an embodiment of the present invention, there may be three business usage scenarios: creating a test environment version, switching a test environment version, and restoring production data on site (i.e., the types of business processing requests may include creating a test environment version, switching a test environment version, and restoring production data).

Step S104, responding to the business processing request, querying the environment management module for the environment information of the environment to be processed.

In an embodiment of the present invention, different environmental information can be queried according to different types of environments to be processed. If the environment to be processed is a test environment, the environment management module can be queried for the current test environment source data and the data source list corresponding to the test environment, or the environment management module can be queried for the current test environment source data and the data source list corresponding to the test environment, as well as the data resource information of the test environment to be switched; if the environment to be processed is a production environment, the environment management module is queried for the environment information corresponding to the production environment, wherein the environment management module manages relevant information of the test environment, including but not limited to: each branch/version corresponding to each environment, the data source corresponding to each branch/version, etc.

Step S106: Generate data tasks for the environment to be processed based on the environment information of the environment to be processed.

In an embodiment of the present invention, the version control module can generate a create version asynchronous task for the corresponding environment or a switch version asynchronous task for the corresponding environment (i.e., generate a data task for the environment to be processed) based on the test environment information or the data resource information of the test environment to be switched (i.e., based on the environment information of the environment to be processed), and can also generate an asynchronous task to restore the production environment based on the environment information corresponding to the production environment.

Step S108, controlling the task processing module to execute the data task, obtaining the execution result, and updating the version information of the environment to be processed based on the execution result to complete the business processing request.

In an embodiment of the present invention, after the version control module generates a data task (including a version creation asynchronous task, a version switching asynchronous task, or a production environment restoration asynchronous task, etc.), the data task can be handed over to the task processing module for execution, and the request result (i.e., the execution result) is returned to the front end. The version information of the environment to be processed is updated based on the execution result to complete the business processing request (including a version creation task, a version switching task, or a production data restoration on-site task, etc.), wherein the task processing module is used to receive the version creation task, version switching task, or production data restoration on-site task, etc. sent by the version control module, execute and return the execution result.

1) Process the task of creating a test environment version

In an embodiment of the present invention, if the environment to be processed is a test environment and the type of the business processing request is to create a test environment version, then based on the environment information of the environment to be processed, a step of generating a data task for the environment to be processed includes: based on the environment information of the test environment, generating a version creation task for the test environment; controlling the task processing module to execute the data task and obtaining the execution result includes: after receiving the version creation task, the task processing module sends a test data backup request to the database operation module in the development environment management platform according to the data source list carried in the version creation task; updating the version information of the environment to be processed based on the execution result and completing the business processing request includes: controlling the database operation module in the development environment management platform to respond to the test data backup request, execute the test data backup operation, and save the backup file to the data resource library; controlling the task processing module to register the data resources of the created target version, and update the version information of the test environment to complete the version creation task.

In an embodiment of the present invention, FIG. 2 is a schematic diagram of an optional processing task of creating a test environment version according to an embodiment of the present invention. As shown in FIG. 2, it includes: an environment management module, a version control module, a database operation module, a task processing module, and a data resource management module, and has four steps of querying environment information, inserting a create version asynchronous task, executing a test data backup operation, and registering the current version data resource. The specific steps are as follows:

(1) Query environmental information

After receiving the request to create a test environment version, the version control module queries the environment management module for environment information, including: the current test environment source data and a data source list corresponding to the test environment.

(2) Insert the create version asynchronous task

The version control module generates a version creation asynchronous task for the corresponding environment (i.e., generates a version creation task for the test environment) based on the obtained test environment information, and sends the version creation asynchronous task to the task processing module for execution, and returns the request result to the front end.

(3) Perform test data backup operations

After receiving the version creation task (i.e., version creation task), the task processing module queues it in the buffer and starts executing it. Then, according to the environment data source list information carried in the version creation task, the task processing module sends a request to execute the test data backup operation to the database operation module in the development environment management platform until the data operation success information is received. Among them, the database operation module is responsible for the database operations corresponding to the test environment, including but not limited to: test database connection, test data backup, restore and other functions.

(4) Register the current version of data resources

After executing the test data backup operation, the backup file is saved in the data resource library, the current version (ie, the created target version) data resource record is registered, and the test environment version information is updated to complete the version creation task.

2) Handle the task of switching the test environment version

In an embodiment of the present invention, if the environment to be processed is a test environment and the type of the business processing request is to switch the test environment version, then based on the environment information of the environment to be processed, a step of generating a data task for the environment to be processed includes: based on the environment information of the current test environment and the data resource information of the test environment to be switched, generating a version switching task corresponding to the test environment to be switched; controlling the task processing module to execute the data task and obtaining the execution result includes: after receiving the version switching task, the task processing module extracts the data resource of the switched version of the current test environment according to the data resource information, and sends a test data restoration request to the database operation module in the development environment management platform, wherein the test data restoration request carries a data source list and data resource information; updating the version information of the environment to be processed based on the execution result and completing the business processing request includes: controlling the database operation module in the development environment management platform to respond to the test data restoration request, accessing the data source to be switched according to the data source list and data resource information in the test data restoration request, executing the test data restoration operation, returning the data restoration result to the task processing module, and updating the version information of the test environment to be switched to complete the version switching task.

Another optional step of extracting the switched version of data resources of the current test environment based on the data resource information includes: sending a resource extraction request to the data resource management module in the development environment management platform based on the data resource information to extract the switched version of data resources of the current test environment.

In an embodiment of the present invention, FIG3 is a schematic diagram of an optional processing task of switching test environment versions according to an embodiment of the present invention. As shown in FIG3, it includes: an environment management module, a version control module, a database operation module, a task processing module, and a data resource management module, and has four steps of querying environment information, inserting a switching version asynchronous task, taking out the current version data resource, and performing a test data restoration operation. The specific steps are as follows:

(1) Query environmental information

After receiving the request to switch the test environment version, the version control module queries the environment management module for environment information, including: the current test environment source data and the data source list corresponding to the test environment, as well as the data resource information of the test environment to be switched.

(2) Insert the asynchronous task of switching versions

The version control module generates a version switching asynchronous task for the corresponding environment (i.e., generates a version switching task corresponding to the test environment to be switched) based on the obtained test environment information and data resource information (i.e., the environment information of the current test environment and the data resource information of the test environment to be switched), and hands the version switching asynchronous task to the task processing module for execution, and returns the request result to the front end.

(3) Get the current version of the data resource

The task processing module receives the version switching task, queues it in the buffer and starts executing it. Then, based on the data resource information, it sends a request to the data resource management module in the development environment management platform to retrieve the data resources of the switched version (i.e., the current version). After retrieving the data resources of the switched version of the current test environment, it sends a request to the database operation module to perform a test data restoration operation (i.e., a test data restoration request). The test data restoration request carries a data source list and data resource information. The data resource management module can manage the test data under each branch/version under the test environment in the form of files to assist in completing the version switching of the test environment.

(4) Perform test data restoration operations

After receiving the test data restoration request, the database operation module accesses the data source to be switched according to the data source list information and data resource information in the request, performs the test data restoration operation, returns the execution result to the task processing module, and updates the version information of the test environment to be switched to complete the version switching task.

3) Processing production data restoration tasks

In an embodiment of the present invention, if the environment to be processed is a test environment, then after receiving the business processing request, the data processing method also includes: sending a data source import request to a data resource management module in a development environment management platform, wherein the data source import request is used to import the environmental data resources of the test environment into a data resource library.

Another optional step, if the environment to be processed is a test environment and the type of business processing request is production data restoration, based on the environmental information of the environment to be processed, the steps of generating a data task for the environment to be processed include: creating a target production data version, and establishing an association between the target production data version and the environmental data resources in the data resource library; based on the version information of the target production data version, sending a production data restoration request to the task processing module; updating the version information of the environment to be processed based on the execution result, and completing the steps of the business processing request include: controlling the database operation module to respond to the production data restoration request, accessing the data source according to the production data resources and version information, performing on-site production data restoration operations, and obtaining execution results; returning the execution results to the task processing module, and updating the version information of the test environment to complete the production data restoration task.

In an embodiment of the present invention, FIG4 is a schematic diagram of an optional processing production data restoration task according to an embodiment of the present invention. As shown in FIG4, it includes: an environment management module, a version control module, a database operation module, a task processing module and a data resource management module, and has six steps of querying environment information, importing production data resources, creating a special version of production data, inserting an asynchronous task to restore the production environment, taking out production data resources, and performing on-site operations for production data restoration. The specific steps are as follows:

(1) Query environmental information

After receiving the production data restoration request, the version control module queries the environment management module for the environment information corresponding to the production environment.

(2) Importing production data resources

Since the data environment of the production environment cannot be directly operated, the production environment operation and maintenance personnel can prepare the data resources of the production environment in advance. They can send an import production data source request (i.e., data source import request) to the data resource management module through the version control module, and import the prepared data resources into the data resource library (i.e., import the environmental data resources of the production environment into the data resource library).

(3) Create a special version of production data

The version control module can create a special version of the production data (ie, create a target production data version), and bind the version to the previous data resource (ie, establish an association relationship between the target production data version and the environmental data resource in the data resource library).

(4) Insert an asynchronous task to restore the production environment

The version control module sends an asynchronous task for inserting and restoring the production environment (ie, a production data restoration task) to the task processing module based on the special version information of the production data (ie, the version information of the target production data version).

(5) Extracting production data resources

The task processing module receives the production environment restoration task, queues it in the buffer and starts executing it, then sends a request to the data resource management module to retrieve the production data resources. After obtaining the production data resources (i.e., reading the data resources stored in the data resource management module), it sends a request to execute the production data restoration on-site operation (i.e., the production data restoration request) to the database operation module based on the production data resources and the special version information of the production data.

(6) Perform on-site operations to restore production data

After receiving the production data restoration request, the database operation module accesses the data source according to the production data resources and the special version information of the production data (i.e., the version information of the production data resources and the target production data version), performs the production data restoration on-site operation, returns the execution result to the task processing module, and updates the test environment version information to complete the production data restoration on-site task.

In an embodiment of the present invention, another optional step of controlling the task processing module to execute the data task and obtain the execution result includes: after receiving the data task, the control task processing module sends the data task to the task buffer pool; and executes each data task according to the task queuing order of the task buffer pool.

In an embodiment of the present invention, there is a task buffer pool inside the task processing module, which can solve the multi-task scheduling situation. That is, after receiving the data task, the task processing module sends the data task to the task buffer pool, and executes each data task according to the task queuing order of the task buffer pool (for example, it can be sorted according to the time order of the tasks, or it can be sorted according to a pre-set priority, etc.).

The embodiment of the present invention, based on the demand for a flexible test environment, proposes a test data environment management method that supports multiple branches, multiple versions, and multiple libraries, which can uniformly manage the test data corresponding to multiple branches, multiple versions, and environments involving multiple databases under each application, and has the following beneficial effects:

(1) It can support complex test environment requirements. By setting branches and versions for the test environment, the test environment can be switched between multiple branches and versions. It not only supports the test environment of multiple database combinations, but also can complete the test environment switching of the entire database cluster.

(2) It has the function of restoring production data on site. If a production problem occurs, since it is not possible to test directly in the production environment, the corresponding test environment can be used as a simulated test scenario for testing;

(3) The task buffer pool design makes the platform/system stable and reliable, which can effectively avoid the load pressure caused by high concurrent requests.

Embodiment 2

A data processing device provided in this embodiment includes multiple implementation units, each implementation unit corresponds to each implementation step in the above-mentioned embodiment 1.

FIG5 is a schematic diagram of an optional data processing device according to an embodiment of the present invention. As shown in FIG5 , the processing device may include: a receiving unit 50, a query unit 52, a generating unit 54, and a control unit 56, wherein:

The receiving unit 50 is configured to receive a service processing request, wherein the service processing request carries an environment identifier of an environment to be processed.

The query unit 52 is used to respond to the business processing request and query the environment management module for the environment information of the environment to be processed.

The generating unit 54 is used to generate a data task for the environment to be processed based on the environment information of the environment to be processed.

The control unit 56 is used to control the task processing module to execute the data task, obtain the execution result, and update the version information of the environment to be processed based on the execution result to complete the business processing request.

The above-mentioned processing device can receive a business processing request through the receiving unit 50, wherein the business processing request carries the environment identifier of the environment to be processed, respond to the business processing request through the query unit 52, query the environment information of the environment to be processed from the environment management module, generate the data task of the environment to be processed based on the environment information of the environment to be processed through the generating unit 54, control the task processing module to execute the data task through the control unit 56, obtain the execution result, and update the version information of the environment to be processed based on the execution result to complete the business processing request. In the embodiment of the present invention, by uniformly managing the environment information of the processing environment, it is suitable for processing businesses in multiple environments, improving the efficiency of processing businesses, thereby solving the technical problem that the related technology cannot uniformly manage various types of test environments, resulting in low efficiency in processing businesses.

Optionally, the environment types of the environment to be processed include: test environment and production environment. If the environment to be processed is a test environment, the environment information of the environment to be processed includes: source data of the test environment and a list of data sources corresponding to the test environment. Each test environment corresponds to at least one environment version, and each environment version corresponds to a data source.

Optionally, the types of business processing requests include: creating a test environment version, switching a test environment version, and restoring production data.

Optionally, the data processing device includes: a first generating module, which is used to generate a data task for the environment to be processed based on the environmental information of the environment to be processed if the environment to be processed is a test environment and the type of the business processing request is to create a test environment version, including: generating a version creation task for the test environment based on the environmental information of the test environment; controlling the task processing module to execute the data task and obtain the execution result, including: a first sending module, which is used to control the task processing module to send a test data backup request to the database operation module in the development environment management platform according to the data source list carried in the version creation task after receiving the version creation task; updating the version information of the environment to be processed based on the execution result to complete the business processing request, including: a first saving module, which is used to control the database operation module in the development environment management platform to respond to the test data backup request, perform the test data backup operation, and save the backup file to the data resource library; a first updating module, which is used to control the task processing module to register the data resources of the target version created, and update the version information of the test environment to complete the version creation task.

Optionally, the data processing device includes: a second generation module, which is used to generate a version switching task corresponding to the test environment to be switched based on the environment information of the current test environment and the data resource information of the test environment to be switched if the environment to be processed is a test environment and the type of business processing request is to switch the test environment version; a first extraction module, which is used to control the task processing module to extract the data resources of the switched version of the current test environment according to the data resource information after receiving the version switching task, and send a test data restoration request to the database operation module in the development environment management platform, wherein the test data restoration request carries a data source list and data resource information; a second update module, which is used to control the database operation module in the development environment management platform to respond to the test data restoration request, access the data source to be switched according to the data source list and data resource information in the test data restoration request, perform the test data restoration operation, return the data restoration result to the task processing module, and update the version information of the test environment to be switched to complete the version switching task.

Optionally, the first extraction module includes: a first extraction submodule, configured to send a resource extraction request to a data resource management module in the development environment management platform according to the data resource information, so as to extract the switched version of the data resource of the current test environment.

Optionally, the data processing device also includes: a first import module, which is used to send a data source import request to the data resource management module in the development environment management platform after receiving the business processing request if the environment to be processed is a test environment, wherein the data source import request is used to import the environmental data resources of the test environment into the data resource library.

Optionally, the data processing device includes: a first creation module, which is used to create a target production data version and establish an association relationship between the target production data version and the environmental data resources in the data resource library if the environment to be processed is a test environment and the type of business processing request is production data restoration; a second sending module, which is used to send a production data restoration request to the task processing module based on the version information of the target production data version; a third update module, which is used to control the database operation module to respond to the production data restoration request, access the data source according to the production data resources and version information, perform the production data restoration on-site operation, and obtain the execution result; return the execution result to the task processing module, and update the version information of the test environment to complete the production data restoration task.

Optionally, the control unit includes: a third sending module, used to control the task processing module to send the data task to the task buffer pool after receiving the data task; and a first execution module, used to execute each data task according to the task queuing order of the task buffer pool.

The above-mentioned processing device may also include a processor and a memory. The above-mentioned receiving unit 50, query unit 52, generation unit 54, control unit 56, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters are adjusted to update the version information of the environment to be processed based on the execution result to complete the business processing request.

The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and/or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one storage chip.

The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having the following method steps: receiving a business processing request, wherein the business processing request carries an environment identifier of the environment to be processed, responding to the business processing request, querying the environment management module for the environment information of the environment to be processed, generating a data task for the environment to be processed based on the environment information of the environment to be processed, controlling the task processing module to execute the data task, obtaining an execution result, and updating the version information of the environment to be processed based on the execution result, thereby completing the business processing request.

According to another aspect of an embodiment of the present invention, a test environment management platform is also provided, including: an environment management module for managing environment information of a test environment or production; a version control module for executing any one of the above data processing methods; a database operation module for library operations of a data resource library corresponding to the test environment, wherein the library operations include at least one of the following: library connection, data backup, and data restoration of the data resource library; a task processing module for executing business processing requests and returning execution results; a data resource management module for managing test data under each environment version under the test environment and completing version switching tasks of the test environment.

Optionally, the task processing module includes: a task buffer pool, which is used to schedule multiple data tasks to be processed.

According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned data processing methods by executing the executable instructions.

According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned data processing methods.

The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims (13)

1. A data processing method, characterized in that it is applied to a version control module in a development environment management platform, wherein the development environment management platform further comprises: an environment management module and a task processing module, and the data processing method comprises: Receiving a business processing request through the version control module, wherein the business processing request carries an environment identifier of a to-be-processed environment, and the type of the business processing request at least includes: creating a test environment version; In response to the business processing request, query the environment management module for environment information of the environment to be processed; Based on the environment information of the environment to be processed, generating a data task for the environment to be processed, including: if the environment to be processed is a test environment and the type of the business processing request is to create a test environment version, based on the environment information of the test environment, generating a version creation task for the test environment; Controlling the task processing module to execute the data task and obtain the execution result includes: controlling the task processing module to send a test data backup request to the database operation module in the development environment management platform according to the data source list carried in the version creation task after receiving the version creation task; The version information of the environment to be processed is updated based on the execution result to complete the business processing request, including: controlling the database operation module in the development environment management platform to respond to the test data backup request, executing the test data backup operation, and saving the backup file to the data resource library; controlling the task processing module to register the data resources of the target version created, and updating the version information of the test environment to complete the version creation task. 2. The method according to claim 1 is characterized in that the environment type of the environment to be processed includes: a test environment and a production environment. If the environment to be processed is a test environment, the environment information of the environment to be processed includes: source data of the test environment and a list of data sources corresponding to the test environment. Each of the test environments corresponds to at least one environment version, and each environment version corresponds to a data source. 3. The method according to claim 1 is characterized in that the types of business processing requests also include: switching test environment versions and restoring production data. 4. The method according to claim 3, characterized in that if the environment to be processed is a test environment and the type of the business processing request is to switch the test environment version, The step of generating a data task of the environment to be processed based on the environment information of the environment to be processed includes: generating a version switching task corresponding to the test environment to be switched based on the environment information of the current test environment and the data resource information of the test environment to be switched; The step of controlling the task processing module to execute the data task and obtain the execution result includes: after receiving the version switching task, controlling the task processing module to extract the data resource of the switched version of the current test environment according to the data resource information, and sending a test data restoration request to the database operation module in the development environment management platform, wherein the test data restoration request carries a data source list and the data resource information; The step of updating the version information of the environment to be processed based on the execution result and completing the business processing request includes: controlling the database operation module in the development environment management platform to respond to the test data restoration request, accessing the data source to be switched according to the data source list and data resource information in the test data restoration request, executing the test data restoration operation, returning the data restoration result to the task processing module, and updating the version information of the test environment to be switched to complete the version switching task. 5. The method according to claim 4, characterized in that the step of extracting the switched version of the data resource of the current test environment according to the data resource information comprises: According to the data resource information, a resource extraction request is sent to the data resource management module in the development environment management platform to extract the switched version of the data resource of the current test environment. 6. The method according to claim 3, characterized in that if the environment to be processed is a test environment, then after receiving the business processing request, the data processing method further comprises: A data source import request is sent to a data resource management module in the development environment management platform, wherein the data source import request is used to import the environmental data resources of the test environment into a data resource library. 7. The method according to claim 6, characterized in that if the environment to be processed is a test environment and the type of the business processing request is production data restoration, The step of generating a data task for the environment to be processed based on the environment information of the environment to be processed includes: creating a target production data version, and establishing an association relationship between the target production data version and the environment data resource in the data resource library; sending a production data restoration request to a task processing module based on the version information of the target production data version; The steps of updating the version information of the environment to be processed based on the execution result and completing the business processing request include: controlling the database operation module to respond to the production data restoration request, accessing the data source according to the production data resource and the version information, performing the production data restoration on-site operation, and obtaining the execution result; returning the execution result to the task processing module, and updating the version information of the test environment to complete the production data restoration task. 8. The method according to claim 1, characterized in that the step of controlling the task processing module to execute the data task and obtaining the execution result comprises: Controlling the task processing module to send the data task to the task buffer pool after receiving the data task; Each of the data tasks is executed according to the task queuing order of the task buffer pool. 9. A test environment management platform, characterized by comprising: The environment management module is used to manage the test environment or production environment information; A version control module, used to execute the data processing method according to any one of claims 1 to 7; A database operation module, used for library operation of a data resource library corresponding to the test environment, wherein the library operation includes at least one of the following: library connection, data backup, and data restoration of the data resource library; The task processing module is used to execute business processing requests and return execution results; The data resource management module is used to manage the test data under each environment version in the test environment and complete the version switching task of the test environment. 10 . The management platform according to claim 9 , wherein the task processing module comprises: a task buffer pool, wherein the task buffer pool is used to schedule a plurality of data tasks to be processed. 11. A data processing device, characterized in that it is applied to a version control module in a development environment management platform, the development environment management platform further comprising: an environment management module and a task processing module, the data processing device comprising: A receiving unit, configured to receive a business processing request, wherein the business processing request carries an environment identifier of an environment to be processed, and the type of the business processing request at least includes: creating a test environment version; A query unit, configured to respond to the business processing request and query the environment management module for environment information of the environment to be processed; A generating unit, configured to generate a data task for the environment to be processed based on the environment information of the environment to be processed, including: if the environment to be processed is a test environment and the type of the business processing request is to create a test environment version, generating a version creation task for the test environment based on the environment information of the test environment; A control unit, used to control the task processing module to execute the data task, obtain the execution result, and update the version information of the environment to be processed based on the execution result to complete the business processing request, including: controlling the task processing module to send a test data backup request to the database operation module in the development environment management platform according to the data source list carried in the version creation task after receiving the version creation task; controlling the database operation module in the development environment management platform to respond to the test data backup request, perform the test data backup operation, and save the backup file to the data resource library; controlling the task processing module to register the data resources of the created target version, and update the version information of the test environment to complete the version creation task. 12. An electronic device, comprising: Processor; and A memory, configured to store executable instructions of the processor; The processor is configured to execute the data processing method according to any one of claims 1 to 8 by executing the executable instructions. 13. A computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the data processing method described in any one of claims 1 to 8.