CN114443294B - Big data service component deployment method, system, terminal and storage medium - Google Patents

Abstract

The invention relates to the technical field of big data, and particularly provides a big data service component deployment method, a system, a terminal and a storage medium, wherein the method comprises the following steps: editing a dependency tree diagram through a browser interface, wherein the dependency tree diagram indicates the dependency relationship among service components to be deployed, and the identity information and the deployment address of the service components to be deployed; converting the dependency relationship into a deployment sequence of each service component, sequentially calling service component installation files according to the deployment sequence, and executing corresponding installation files according to the identity information and the deployment address of the service component to be deployed; a distributed monitoring process is created that monitors the running information of the deployed service components. The invention enables the big data cluster to have the management capability of multi-service deployment, provides clear and visual service dependency relationship, and effectively avoids node conflict among multiple deployments of the same big data assembly in the implementation process.

Description

Big data service component deployment method, system, terminal and storage medium

Technical field

The present invention relates to the field of big data technology, and specifically to a big data service component deployment method, system, terminal and storage medium.

Background technique

There are interdependencies between big data service components. For example, the underlying storage of HBase relies on HDFS, and the high-availability deployment of HDFS relies on Zookeeper. For enterprise-level data centers, in complex application scenarios, it is often necessary to implement multiple deployments of the same service component. For example, deploy multiple HBase or HDFS clusters in the same physical cluster. These clusters with the same component support different services. Business and data are independent of each other and jointly carry a huge and complex big data cluster. However, during the implementation process, the dependencies between multiple services are prone to overlap, conflict and other problems. The system is chaotic and lacks a unified management plan. As the business expands, it brings huge difficulties to later maintenance and management.

Contents of the invention

Aiming at problems such as dependencies between multiple services in existing big data service component deployment methods that are prone to overlap, conflict, etc., the present invention provides a big data service component deployment method, system, terminal and storage medium to avoid huge business downtime. The structural chaos of multi-service deployment provides great convenience for cluster management.

In a first aspect, the present invention provides a big data service component deployment method, including:

Edit a dependency tree diagram through the browser interface, which indicates the dependency relationship between the service components to be deployed and the identity information and deployment address of the service components to be deployed;

Convert the dependency relationship into the deployment sequence of each service component, sequentially retrieve the service component installation files according to the deployment sequence, and execute the corresponding installation file according to the identity information and deployment address of the service component to be deployed;

Create a distributed monitoring process that monitors running information of deployed service components.

Further, edit the dependency tree diagram through the browser interface. The dependency tree diagram indicates the dependency relationship between the service components to be deployed and the identity information and deployment address of the service components to be deployed, including:

Pre-set various service component labels and connection indicators;

Receive user drag instructions, which include drag objects and target point coordinates, and copy the drag objects to the area where the target point coordinates are located according to the user drag instructions. The drag objects are service component labels or connection instructions. mark;

Call the default configuration template for the service component label dragged to the target point coordinates. The default configuration template includes parameter items to be configured. After inputting parameter values to the parameter items to be configured, the default configuration template is saved as the service component. The configuration file of the label, the parameter items to be configured include identity information and deployment address;

Generate dependencies between service components in a dependency tree diagram based on connection indication marks between adjacent service components;

Verify the rationality of the dependency tree diagram according to the set verification rules. The verification rules include an acyclic graph, the number of service components that a service component depends on does not exceed 1, and the deployment address of the same service component in different dependency tree diagrams. No crossing.

Further, the dependency relationship is converted into the deployment sequence of each service component, the service component installation files are sequentially retrieved according to the deployment sequence, and the corresponding installation file is executed according to the identity information and deployment address of the service component to be deployed, including:

Parse the dependency tree graph to obtain the dependencies between service components and the identity information and deployment address of the service components to be deployed, and distribute the installation files of each service component to the deployment addresses of each service component in sequence based on the dependency relationships;

The installation files for installing each service component are sequentially decompressed according to the dependency relationship, and each service component is started sequentially according to the dependency relationship.

Further, create a distributed monitoring process that monitors the running information of deployed service components, including:

Use the distributed monitoring process to collect the running information of each service component. The running information includes the process, the physical node where it is located, and the running status information;

The dependency tree diagram and the collected running status information are saved in the metadata database, corresponding data is retrieved from the metadata according to the query instructions received by the browser interface, and the corresponding data is returned to the display page of the browser interface.

Further, the method also includes:

Update the dependency tree diagram through the browser interface, and synchronize the updated dependency tree diagram to the source database. The update operations include service component configuration modification, addition and deletion of service components, service offline and dependency tree deletion;

Compare the dependency tree before modification, parse the modified nodes and modification content, and perform related operations on the deployed service components based on the dependency relationships. The operations include configuration modification, node addition and deletion, and service removal; if the dependency tree is deleted, the current dependencies are parsed The dependency relationship of the tree, and all service components of the current dependency tree are uninstalled in sequence based on the reverse dependency relationship.

In a second aspect, the present invention provides a big data service component deployment system, including:

A tree diagram building unit for editing a dependency tree diagram through a browser interface, where the dependency tree diagram indicates the dependency relationship between the service components to be deployed and the identity information and deployment address of the service components to be deployed;

A component deployment unit, used to convert the dependency relationship into a deployment sequence of each service component, sequentially retrieve the service component installation files according to the deployment sequence, and execute the corresponding installation file according to the identity information and deployment address of the service component to be deployed;

A component monitoring unit is used to create a distributed monitoring process, and the distributed monitoring process monitors the running information of deployed service components.

Further, the tree diagram building unit includes:

Label preparation module, used to pre-set various service component labels and connection indication marks;

The object drag module is used to receive user drag instructions, the user drag instructions include the drag object and the target point coordinates, and copy the drag object to the area where the target point coordinates are located according to the user drag instructions, the drag object Labels or connection indicators for service components;

The component configuration module is used to call a default configuration template for the service component label dragged to the target point coordinates. The default configuration template includes a parameter item to be configured. After inputting a parameter value to the parameter item to be configured, the default configuration template is Save the configuration file as the service component label, and the parameter items to be configured include identity information and deployment address;

The relationship building module is used to generate dependency relationships between service components in the dependency tree diagram based on the connection indicator marks between adjacent service components;

Reasonable verification module, used to verify the rationality of the dependency tree diagram according to the set verification rules. The verification rules include acyclic graphs, the number of service components that a service component depends on does not exceed 1, and different dependency tree diagrams. The deployment addresses of the same service component do not overlap.

Further, the component deployment unit includes:

The tree diagram parsing module is used to parse the dependency tree diagram to obtain the dependency relationships between service components and the identity information and deployment addresses of the service components to be deployed, and distribute the installation files of each service component to each service component in sequence based on the dependency relationships. deployment address;

The component installation module is used to decompress and install the installation files of each service component in sequence according to the dependency relationship, and to start each service component in sequence according to the dependency relationship.

In the third aspect, a terminal is provided, including:

processor, memory, where,

This memory is used to store computer programs,

The processor is used to call and run the computer program from the memory, so that the terminal executes the above terminal method.

In a fourth aspect, a computer storage medium is provided. The computer-readable storage medium stores instructions, which when run on a computer, cause the computer to perform the methods described in the above aspects.

The beneficial effect of the present invention is that the big data service component deployment method, system, terminal and storage medium provided by the present invention enable the big data cluster to have the management capability of multi-service deployment, provide clear and intuitive service dependencies, and effectively avoid During the implementation process, node conflicts between multiple deployments of the same big data component; simultaneous modification of the configuration, expansion, and decommissioning of a service no longer require node-by-node operations, but can be performed uniformly on the interface, supporting one-click deployment, Modification or uninstallation greatly reduces management costs.

In addition, the design principle of the invention is reliable, the structure is simple, and it has very broad application prospects.

Description of the drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those of ordinary skill in the art, It is said that other drawings can also be obtained based on these drawings without exerting creative work.

Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

Figure 2 is a schematic block diagram of a system according to an embodiment of the present invention.

Figure 3 is a schematic structural diagram of a terminal provided by an embodiment of the present invention.

Detailed ways

In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described The embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts should fall within the scope of protection of the present invention.

Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention. Among them, the execution subject in Figure 1 can be a big data service component deployment system.

As shown in Figure 1, the method includes:

Step 110: Edit the dependency tree diagram through the browser interface. The dependency tree diagram indicates the dependency relationship between the service components to be deployed and the identity information and deployment address of the service components to be deployed;

Step 120: Convert the dependency relationship into the deployment sequence of each service component, sequentially retrieve the service component installation files according to the deployment sequence, and execute the corresponding installation file according to the identity information and deployment address of the service component to be deployed;

Step 130: Create a distributed monitoring process that monitors the running information of deployed service components.

In order to facilitate understanding of the present invention, the big data service component deployment method provided by the present invention will be further described below based on the principle of the big data service component deployment method of the present invention and the process of deploying the big data service components in the embodiment.

Specifically, the big data service component deployment method includes:

S1. Edit a dependency tree diagram through a browser interface. The dependency tree diagram indicates the dependency relationship between the service components to be deployed and the identity information and deployment address of the service components to be deployed.

The web interface provides visual query, monitoring and operation functions. Before deploying services, users can edit the dependency tree in the web interface and submit the task after the editing is completed. The function of dependency tree diagram requires the implementation of the following modules:

The visual interface provides a full range of big data service tags and connection pointers, such as Zookeeper, HDFS, etc. Users can create a tree diagram by dragging and connecting in the panel. The tree diagram is a directed acyclic graph and is not allowed. End-to-end connection occurs;

Each node of the dependency tree corresponds to each big data component. Each component has an independent ID and name, which is used to distinguish different service deployments. For example, if the HBase service is introduced in both tree diagrams, their IDs and names are different. ;

Each node's service has a default template for the configuration file. After the task is submitted, the configuration file required for the service will be generated based on the template. Users need to write and modify the configuration according to actual production needs. At the same time, users need to fill in the installation node IP of each service and the client's deployment node;

The web interface will automatically verify the rationality of the tree diagram edited by the user. For example, the same HDFS is not allowed to rely on two Zookeepers at the same time, and the IPs deployed by the same service in different tree diagrams are not allowed to cross, etc., when the user completes the edit When submitted, the dependency tree diagram will be submitted to the service manager as a task, and the relevant information will be written to the metadata database;

For existing deployed and running dependencies, you can perform editing operations, such as configuration modifications, node additions and deletions, service offline or even deletion of the entire dependency tree, etc. After editing is completed, it is also submitted to the service manager as a task.

S2. Convert the dependency relationship into the deployment sequence of each service component, sequentially retrieve the service component installation files according to the deployment sequence, and execute the corresponding installation file according to the identity information and deployment address of the service component to be deployed.

The service manager maintains the installation compressed packages of all big data components to be deployed, receives task requests submitted by the web interface, parses the relevant parameters, and implements installation, deployment, and startup. The process is as follows:

The service manager parses the submitted dependency tree, obtains the dependencies between services and the IP nodes to be deployed for each service, then automatically generates a script language, and decompresses the corresponding service installation package to the specified directory of the corresponding physical machine. ;

After the decompression and installation of each service is completed, the configuration file is generated according to the edited configuration template provided by the user and overwritten in the corresponding installation directory;

According to the dependency relationships in the dependency tree, the manager starts each big data service in the correct order through remote commands;

If the submitted task is to modify and edit the existing dependency tree, the service manager will compare the dependency tree before the modification, parse the modified nodes and modification content, generate a script language and send it to the remote host, and compare the currently deployed dependencies in the correct dependency order. The service performs related operations, such as configuration modification, node addition and deletion, service removal, etc.; if the entire dependency tree is deleted, the service manager also parses the linear order of the current dependency tree and uninstalls all services in the current tree with one click.

S3. Create a distributed monitoring process that monitors the running information of deployed service components.

The monitoring control is a lightweight distributed service. When each service node corresponding to the dependency tree diagram is started, the monitoring control will collect information such as the processes, physical nodes where these components are located, and running status, and synchronize them to the metadata database. When any service node changes, the monitoring control will collect information in real time and synchronize the data.

The metadatabase stores the user-edited treemap information, task information, and the running status of each service node. The web module queries this information for visual display. If the user makes changes to the dependency treemap, it will be synchronized to the metadatabase in a timely manner. , and generate new tasks to submit to the service manager.

As shown in Figure 2, the system 200 includes:

The tree diagram building unit 210 is configured to edit a dependency tree diagram through a browser interface, where the dependency tree diagram indicates the dependency relationship between the service components to be deployed and the identity information and deployment address of the service components to be deployed;

The component deployment unit 220 is used to convert the dependency relationship into the deployment sequence of each service component, sequentially retrieve the service component installation files according to the deployment sequence, and execute the corresponding installation file according to the identity information and deployment address of the service component to be deployed. ;

The component monitoring unit 230 is used to create a distributed monitoring process that monitors the running information of deployed service components.

Optionally, as an embodiment of the present invention, the tree diagram building unit includes:

Label preparation module, used to pre-set various service component labels and connection indication marks;

The object drag module is used to receive user drag instructions, the user drag instructions include the drag object and the target point coordinates, and copy the drag object to the area where the target point coordinates are located according to the user drag instructions, the drag object Labels or connection indicators for service components;

The component configuration module is used to call a default configuration template for the service component label dragged to the target point coordinates. The default configuration template includes a parameter item to be configured. After inputting a parameter value to the parameter item to be configured, the default configuration template is Save the configuration file as the service component label, and the parameter items to be configured include identity information and deployment address;

The relationship building module is used to generate dependency relationships between service components in the dependency tree diagram based on the connection indicator marks between adjacent service components;

Reasonable verification module, used to verify the rationality of the dependency tree diagram according to the set verification rules. The verification rules include acyclic graphs, the number of service components that a service component depends on does not exceed 1, and different dependency tree diagrams. The deployment addresses of the same service component do not overlap.

Optionally, as an embodiment of the present invention, the component deployment unit includes:

The tree diagram parsing module is used to parse the dependency tree diagram to obtain the dependency relationships between service components and the identity information and deployment addresses of the service components to be deployed, and distribute the installation files of each service component to each service component in sequence based on the dependency relationships. deployment address;

The component installation module is used to decompress and install the installation files of each service component in sequence according to the dependency relationship, and to start each service component in sequence according to the dependency relationship.

FIG. 3 is a schematic structural diagram of a terminal 300 provided by an embodiment of the present invention. The terminal 300 can be used to execute the big data service component deployment method provided by an embodiment of the present invention.

The terminal 300 may include: a processor 310, a memory 320, and a communication unit 330. These components communicate through one or more buses. Those skilled in the art can understand that the structure of the server shown in the figure does not limit the invention. It can be a bus structure, a star structure, or More or fewer components may be included than shown, or certain components may be combined, or may be arranged differently.

Among them, the memory 320 can be used to store execution instructions of the processor 310. The memory 320 can be implemented by any type of volatile or non-volatile storage terminals or their combination, such as static random access memory (SRAM), electronic Erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk . When the execution instructions in the memory 320 are executed by the processor 310, the terminal 300 is enabled to perform some or all of the steps in the following method embodiments.

The processor 310 is the control center of the storage terminal, using various interfaces and lines to connect various parts of the entire electronic terminal, by running or executing software programs and/or modules stored in the memory 320, and calling data stored in the memory, To perform various functions of the electronic terminal and/or process data. The processor may be composed of an integrated circuit (IC for short), for example, it may be composed of a single packaged IC, or it may be composed of multiple packaged ICs connected with the same function or different functions. For example, the processor 310 may only include a central processing unit (Central Processing Unit, CPU for short). In the embodiment of the present invention, the CPU may be a single computing core or may include multiple computing cores.

The communication unit 330 is used to establish a communication channel so that the storage terminal can communicate with other terminals. Receive user data sent by other terminals or send user data to other terminals.

The present invention also provides a computer storage medium, wherein the computer storage medium can store a program, and when executed, the program can include some or all of the steps in the embodiments provided by the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

Therefore, the present invention enables the big data cluster to have the management capability of multi-service deployment, provides clear and intuitive service dependencies, and effectively avoids node conflicts between multiple deployments of the same big data component during the implementation process; simultaneously modifying certain The configuration, expansion and decommissioning of individual services no longer need to be performed node by node. They can be performed uniformly on the interface, supporting one-click deployment, modification or uninstallation, which greatly reduces management costs. The technical effects achieved by this embodiment can be found in The above description will not be repeated here.

Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solutions in the embodiments of the present invention can be embodied in the form of software products in essence or in part that contribute to the existing technology. The computer software products are stored in a storage medium such as a USB flash drive, mobile phone, etc. Hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program code, including a number of instructions to make a computer terminal (It can be a personal computer, a server, a second terminal, a network terminal, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention.

The same and similar parts among the various embodiments in this specification can be referred to each other. In particular, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For relevant details, please refer to the description in the method embodiment.

In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, indirect coupling or communication connection of the system or unit, which may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

Although the present invention has been described in detail with reference to the accompanying drawings in conjunction with preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, those of ordinary skill in the art can make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions should be within the scope/any scope of the present invention. Those skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present invention, and they should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.

Claims (6)

1. A method for deploying a big data service component, comprising:

editing a dependency tree diagram through a browser interface, wherein the dependency tree diagram indicates the dependency relationship among service components to be deployed, and the identity information and the deployment address of the service components to be deployed;

converting the dependency relationship into a deployment sequence of each service component, sequentially calling service component installation files according to the deployment sequence, and executing corresponding installation files according to the identity information and the deployment address of the service component to be deployed;

creating a distributed monitoring process, wherein the distributed monitoring process monitors running information of deployed service components;

editing a dependency tree diagram through a browser interface, wherein the dependency tree diagram indicates the dependency relationship among service components to be deployed, and identity information and deployment addresses of the service components to be deployed, and the method comprises the following steps:

presetting various service component labels and connection indication marks;

receiving a user dragging instruction, wherein the user dragging instruction comprises a dragging object and target point coordinates, and copying the dragging object to an area where the target point coordinates are located according to the user dragging instruction, wherein the dragging object is a service component label or a connection indication mark;

calling a default configuration template for a service component tag dragged to a target point coordinate, wherein the default configuration template comprises a parameter item to be configured, and after parameter values are input into the parameter item to be configured, the default configuration template is stored as a configuration file of the service component tag, and the parameter item to be configured comprises identity information and a deployment address;

generating a dependency relationship between service components in the dependency tree graph according to the connection indication marks between adjacent service components;

verifying the rationality of the dependency tree graphs according to a set verification rule, wherein the verification rule comprises an acyclic graph, the number of service components on which one service component depends is not more than 1, and the deployment addresses of the same service component of different dependency tree graphs are not crossed;

converting the dependency relationship into a deployment sequence of each service component, sequentially calling service component installation files according to the deployment sequence, and executing corresponding installation files according to the identity information and the deployment address of the service component to be deployed, wherein the method comprises the following steps:

analyzing the dependency tree graph to obtain the dependency relationship among the service components, the identity information and the deployment address of the service components to be deployed, and distributing the installation files of the service components to the deployment addresses of the service components in sequence according to the dependency relationship;

and sequentially decompressing and installing the installation files of the service components according to the dependency relationship, and sequentially starting the service components according to the dependency relationship.

2. The method of claim 1, wherein creating a distributed monitoring process that monitors operational information of deployed service components comprises:

collecting operation information of each service component by using a distributed monitoring process, wherein the operation information comprises the process, the physical node and the operation state information;

and storing the dependency tree graphs and the collected running state information into a metadata base, calling corresponding data from the metadata according to a query instruction received by the browser interface, and returning the corresponding data to a display page of the browser interface.

3. The method according to claim 1, wherein the method further comprises:

updating the dependency graph through a browser interface, and synchronously updating the updated dependency graph to a source database, wherein the updating operation comprises service component configuration modification, adding and deleting of service components, service offline and dependency tree deletion;

analyzing the modified nodes and the modified content by comparing the dependency tree before modification, and performing related operations on the deployed service components according to the dependency relationship, wherein the operations comprise configuration modification, node addition and deletion and service off-shelf; if the dependency tree is deleted, analyzing the dependency relationship of the current dependency tree, and unloading all service components of the current dependency tree in sequence according to the inverse dependency relationship.

4. A big data service component deployment system, comprising:

the tree diagram construction unit is used for editing a dependency tree diagram through a browser interface, wherein the dependency tree diagram indicates the dependency relationship among the service components to be deployed, the identity information of the service components to be deployed and the deployment address;

the component deployment unit is used for converting the dependency relationship into the deployment sequence of each service component, sequentially calling the service component installation files according to the deployment sequence, and executing the corresponding installation files according to the identity information and the deployment address of the service component to be deployed;

the component monitoring unit is used for creating a distributed monitoring process, and the distributed monitoring process monitors the running information of the deployed service components;

the tree diagram construction unit includes:

the label preparation module is used for presetting various service component labels and connection indication marks;

the object dragging module is used for receiving a user dragging instruction, wherein the user dragging instruction comprises a dragging object and target point coordinates, the dragging object is copied to an area where the target point coordinates are located according to the user dragging instruction, and the dragging object is a service component label or a connection indication mark;

the component configuration module is used for calling a default configuration template for the service component tag dragged to the target point coordinate, the default configuration template comprises a parameter item to be configured, the default configuration template is stored as a configuration file of the service component tag after parameter values are input into the parameter item to be configured, and the parameter item to be configured comprises identity information and a deployment address;

the relation construction module is used for generating the dependency relation between the service components in the dependency tree graph according to the connection indication marks between the adjacent service components;

the reasonable verification module is used for verifying the rationality of the dependency tree graph according to a set verification rule, wherein the verification rule comprises an acyclic graph, the number of service components on which one service component depends is not more than 1, and the deployment addresses of the same service component of different dependency tree graphs are not crossed;

the component deployment unit includes:

the tree diagram analysis module is used for analyzing the dependency tree diagram to obtain the dependency relationship among the service components, the identity information and the deployment address of the service components to be deployed, and distributing the installation files of the service components to the deployment addresses of the service components in sequence according to the dependency relationship;

and the component installation module is used for sequentially decompressing and installing the installation files of the service components according to the dependency relationship and sequentially starting the service components according to the dependency relationship.

5. A terminal, comprising:

a processor;

a memory for storing execution instructions of the processor;

wherein the processor is configured to perform the method of any of claims 1-3.

6. A computer readable storage medium storing a computer program, which when executed by a processor implements the method of any one of claims 1-3.