CN108259567B - A Discovery Method of Large Data Center Business Subsystem Based on Server Application Logic - Google Patents

Abstract

The invention relates to the technical field of server operation and maintenance management, in particular to a large-scale data center business subsystem discovery method based on server application logic. The method effectively overcomes the defects of the traditional operation and maintenance method by constructing the system architecture diagram of the data center, and does not need to spend a lot of manpower, material and financial resources to collect and count data; it does not require the operation and maintenance personnel to have sufficient experience and knowledge of the application system architecture of the data center. ; Can automatically build an application-based business logic network, effectively assisting the operation and maintenance personnel to master the server architecture and usage of the data center.

Description

A Discovery Method of Large Data Center Business Subsystem Based on Server Application Logic

technical field

The invention relates to the technical field of server operation and maintenance management, in particular to a large-scale data center business subsystem discovery method based on server application logic.

Background technique

In recent years, the rapid increase in the number of servers in large data centers has brought enormous pressure to their operation and maintenance departments. It is increasingly difficult for operation and maintenance managers to grasp the actual usage of servers in the data center, as well as the application logic relationships between servers involved in various business applications.

At present, in the face of data centers running complex business systems, the traditional operation and maintenance solution is to simply rely on manual statistics and some primary operation and maintenance tools to store the server usage of the data center in the CMDB (Configuration Management Database) for registration. and statistics, and then based on the statistical results, manually draw the server's business application logic network. There are many flaws in this approach. On the one hand, during the actual operation of the data center, the business system architecture changes rapidly, the business system is highly complex, and the server usage records in the CMDB are often incomplete or outdated, making it difficult for operation and maintenance personnel to accurately grasp the real architecture of the system , manual statistics also require coordination of various departments, which consumes a lot of manpower and material resources; on the other hand, based on manual statistics, the server business application logic network drawn by the personal experience of operation and maintenance personnel is not inheritable. Once personnel are transferred, new Due to the lack of understanding of the business system and the lack of experience and knowledge, it is difficult for the personnel to update the business application logic network.

SUMMARY OF THE INVENTION

Aiming at the shortcomings of the above-mentioned prior art, the present invention provides a large-scale data center service subsystem discovery method based on server application logic, so as to realize effective operation and maintenance management of large-scale data center servers.

The present invention provides the following solution, a method for discovering a large-scale data center business subsystem based on server application logic, the method comprising the following steps:

S1 parses and aggregates server data, TCP connection data, and introduces external CMDB business system information data from the original log data of large-scale data center servers, which includes the following steps:

S1.1 Analyze and aggregate the raw log data of large data center servers within a certain period of time, and extract server data from the raw log data, which at least includes server name, server IP list, out-degree and in-degree , average weight, number of users, number of processes, memory and CPU usage, connection distribution per hour, server classification, and a series of top10 process information for CPU, VSZ, I/O, IPCS, etc.;

S1.2 extracts the TCP connection data from the original log data. The TCP connection data contains at least the server names at both ends of the TCP connection, the type of the connection, the weight of the connection, the hourly distribution of the connection, the sending and receiving queues and other information;

S1.3 Obtain business system information data from the CMDB, which at least includes the business system name, the mapping relationship between the business system and the server, and the person in charge of the business system;

S2 aggregates the data information of the business system and the relationship data information between the business systems based on the data extracted by S1;

The data information of the business system specifically includes: the business system name, the person in charge of the business system, the number of servers included in the business system, the number of connections S _{edge_sum} , the number of external connections S _{outside_edge} , the number of internal connections S _{inside_edge} , the out degree S _{out_degree} and the in degree S _{in_degree} , average number of connections S _{avg_edge} , average weight S _{avg_weight} and other information;

The number of connections S _{edge_sum} is equal to the sum of the number of external connections S _{outside_edge} and the number of internal connections S _{inside_edge} , the formula is as follows:

S _{edge_sum} =S _{outside_edge} +S _{inside_edge}

The number of external connections S _{outside_edge} indicates that the server at one end of the TCP connection is inside the business system, and the server at the other end does not belong to the business system; the number of internal connections S _{inside_edge} indicates that the servers at both ends of the TCP connection are inside the business system;

Out degree S _{out_degree} represents the number of other business systems accessed by the business system; S _{in_degree} represents the number of other business systems that have accessed the business system;

The average number of connections S _{avg_edge} represents the ratio of the number of internal connections S _{inside_edge} of the business system to the number of servers connected within the business system; the average weight S _{avg_weight} represents the average value of the sum of the weights of the internal connections of the business system, the formula is as follows:

S _{avg_weight} =Sum(W _{in_degree} )/S _{inside_edge}

Wherein, the association relationship data information between business systems includes: associated server business system, associated client business system, associated weight S _weight and other information;

The association weight S _weight between the business systems is equal to the cumulative sum of the TCP connection weights between the two business systems;

S3 divides the internal servers of the business system into layers based on S1 and S2 data information;

S3.1 Find out the border server border _vm inside the business system, the border server border _vm is defined as a non-database type server that has been connected to the public network IP or has been connected to an external non-monitoring business system; the border server border _vm The level of Level is marked as 0;

S3.2 Starting from the layer 0 boundary server, according to the TCP connection data of the server, find the server that has been accessed by the layer 0 boundary server among the servers that have not been divided into layers in the business system as the next layer of the layer 0 boundary server. The Level is marked as 1, among which, if the server no longer accesses other servers but only serves as a server to provide services to other servers, it is called the root server root _vm , and the Level of the root server root _vm is marked as -1;

S3.3 repeats S3.2 with the server marked as Level+1 as the starting point, until all servers in the business system have completed the level division;

S4 is based on the internal hierarchical division result of the S3 business system, and then groups each layer of servers according to the TCP connection data extracted by S1; by calculating whether the similarity between any two servers in each layer of servers reaches the threshold, it can be divided into The same group, otherwise different groups;

The similarity _vm1& vm2 between the two servers vm1 and vm2 is defined as the sum of the client Jacobian coefficient client _coef plus the server Jacobian coefficient server _coef , the formula is as follows:

similarity _vm1&vm2 =client _coef +server _coef

The client Jacobian coefficient client _coef is equal to the ratio of the intersection of the server set server _vm1 accessed by server vm1 as a client and the server set server _vm2 accessed by server vm2 as a client to the union set of server _vm1 and server _vm2 . The formula is as follows:

client _coef =(server _vm1 ∩server _vm2 )/(server _vm1 ∪server _vm2 )

The server Jacobian coefficient server _coef is equal to the ratio of the intersection of the server set client _vm1 served by server vm1 as the server and the server set client _vm2 served by server vm2 as the server to the union of client _vm1 and client _vm2 . The formula is as follows:

server _coef =(client _vm1 ∩client _vm2 )/(client _vm1 ∪client _vm2 )

S5 establishes grouped data information and grouped association relationship data information based on the business subsystem results of the grouping discovery of S3 and S4;

S6 constructs a data center business system architecture diagram based on the grouping association relationship established by S5.

The present invention has the following technical effects: the method for discovering large-scale data center business subsystems based on server application logic proposed by the present invention effectively overcomes the defects of traditional operation and maintenance methods by constructing a data center system architecture diagram, and does not need to spend a lot of manpower, material resources and financial resources. Collect and count data; it is not necessary for the operation and maintenance personnel to have sufficient experience and knowledge of the application system architecture of the data center; it can automatically build an application-based business logic network, effectively assisting the operation and maintenance personnel to master the server architecture and usage of the data center.

Description of drawings

Fig. 1 is a flow chart of constructing a business system architecture diagram based on a large-scale data center business subsystem discovery method based on server application logic proposed by the present invention;

FIG. 2 is an architecture diagram of a business system constructed by a method for discovering business subsystems of large data centers based on server application logic finally obtained by the present invention.

Detailed ways

A method for discovering a large-scale data center business subsystem based on server application logic proposed by an embodiment of the present invention, the processing flow for constructing a business system architecture diagram includes the following steps:

S1 parses and aggregates server data, TCP connection data and external CMDB business system information data from the original log data of large data center servers, including:

S1.1 Parse and aggregate the raw log data of large-scale data center servers for a certain period of time. The above period of time is in days. In practical applications, multiple days of data can also be selected for fusion. The extracted server data includes the server name. , IP list of servers, out-degree and in-degree, average weight, number of users, number of processes, memory and CPU usage, connection distribution per hour, classification of servers and a series of top10 for CPU, VSZ, I/O, IPCS process information, etc.

The extracted server data includes the fields shown in Table 1 below:

Table 1

S1.2 parses and aggregates the original log data and extracts the TCP connection data, including the server names at both ends of the TCP connection, the type of the connection, the weight of the connection, the hourly distribution of the connection, and the sending and receiving queues.

The extracted TCP connection data includes the fields shown in Table 2 below:

Table 2

Numbering field name Field Description 1 id auto increment id 2 source_machine_id Server as server in tcp connection 3 target_machine_id The server that is the client in the tcp connection 4 vm_edge_service_class Connection class for tcp connection 5 weight tcp connection weight 6 date date 7 conn_dis_hour hourly distribution of tcp connections 8 type_conn Connection distribution for different connection classes 9 sum_recvq Receive queue for tcp connection 10 sum_sendq send queue for tcp connection

S1.3 Obtain business system information data from the CMDB, where the business system information data at least includes the business system name, the mapping relationship between the business system and the server, and the person in charge of the business system.

The externally introduced CMDB business system information data includes the fields shown in Table 3 below:

table 3

S2 aggregates the data information of the business system and the relational relationship data information between the business systems based on the data extracted by S1.

Among them, the data information of the business system specifically includes: the name of the business system, the person in charge of the business system, the number of servers included in the business system, the number of connections, the number of external connections, the number of internal connections, out-degree and in-degree, average number of connections, average weight, etc. information.

The data information of the aggregated business system includes the fields shown in Table 4 below:

Table 4

Numbering field name Field Description 1 id auto increment id 2 system_name business system name 3 header business system manager 4 vm_num The total number of servers included in the business system 5 edge_sum The number of connections included in the business system 6 outside_edge The number of external connections included in the business system 7 inside_edge The number of internal connections included in the business system 8 out_degree Out-degree of business system 9 in_degree In-degree of business system 10 avg_edge Average number of connections to business systems 11 avg_weight Average connection weight of business systems 12 data date

Wherein, the data information of the association relationship between the business systems includes information such as the association server business system, the association client business system, and the association weight.

The association relationship data information between the aggregated business systems includes the fields shown in Table 5 below:

table 5

Numbering field name Field Description 1 id auto increment id 2 source_system_id Server business system 3 target_system_id Client business system 4 weight Association weight 5 date date

S3 is the hierarchical division of the internal server of the business system based on the data information of S1 and S2.

S4 is based on the internal layer division result of the S3 service system, and then groups each layer of servers according to the TCP connection data extracted by S1.

S5 establishes the data information of the group and the data information of the association relationship of the group based on the layering of S3 and S4 and the service subsystem result of group discovery.

The data information of the group and the data information of the association relationship of the group include the fields shown in Table 6 and Table 7 below:

Table 6

Numbering field name Field Description 1 id auto increment id 2 date date 3 group_name group name 4 system_id The business system where the group is located 5 level_id Hierarchy grouped in business system 6 group_id group label 7 machine_num The number of servers included in the group 8 machine_list List of servers included in the group

Table 7

serial number field name Field Description 1 id auto increment id 2 date date 3 source_group_id Group as a server in an association relationship 4 target_group_id Group as client in relationship 5 weight Association weights in grouped associations

S6 builds a data center business system architecture diagram based on the grouping association relationship established by S5, as shown in Figure 2.

To sum up, the method for discovering large-scale data center business subsystems based on server application logic proposed in the embodiments of the present invention is used to construct a business system architecture diagram, which effectively overcomes the defects of traditional operation and maintenance methods, and does not need to spend a lot of manpower, material resources and financial resources to collect , statistical data; it is not necessary for the operation and maintenance personnel to have sufficient experience and knowledge of the application system architecture of the data center; it can automatically build an application-based business logic network, effectively assisting the operation and maintenance personnel to master the business system architecture and usage of the data center.

The business system architecture diagram constructed by the present invention can truly reflect the deployment situation of the business application system on the server and assist the management of operation and maintenance personnel. And the input of the invention only needs the snapshot data of the server log of the data center. Based on the snapshot data of the server log after interpretation and aggregation, the invention can automatically discover the business system architecture diagram, not only the result is accurate, but also there is no need for many people to operate, saving a lot of manpower , material expenses.

Claims (1)

1. A large data center service subsystem discovery method based on server application logic is characterized by comprising the following steps:

s1 parsing and aggregating server data, TCP connection data, and information data of an external CMDB service system from the original log data of the large-scale data center server, specifically including the following steps:

s1.1, analyzing and aggregating original log data of a large-scale data center server in a certain time period, and extracting server data from the original log data, wherein the server data at least comprises a server name, an IP list, an out-degree and an in-degree, an average weight, a user number, a process number, a memory and CPU use condition, hourly connection distribution, classification of the server, top10 process information of a series of CPUs, VSZ and I/O, IPCS and the like;

s1.2, extracting TCP connection data from original log data, wherein the TCP connection data at least comprises information such as server names at two ends of a TCP connection, connection types, connection weights, connection hour distribution, sending and receiving queues and the like;

s1.3, acquiring service system information data from the CMDB, wherein the service system information data at least comprises information such as a service system name, a mapping relation between a service system and a server, a service system person in charge and the like;

s2 aggregating the data information of the service system and the incidence relation data information between the service systems based on the data extracted in S1;

the data information of the service system specifically includes: name of service system, person in charge of service system, number of servers and number of connections S contained in service system_{edge_sum}External connection number S_{outside_edge}Internal connection number S_{inside_edge}And the degree of departure S_{out_degree}And degree of penetration S_{in_degree}Average number of connections S_{avg_edge}Average weight S_{avg_weight}And the like;

number of connections S_{edge_sum}Equal to the number of external connections S_{outside_edge}Number of connections to internal_{inside_edge}The formula is as follows:

S_{edge_sum}＝S_{outside_edge}+S_{inside_edge}

number of external connections S_{outside_edge}The server at one end of the TCP connection is shown in the service system, and the server at the other end does not belong to the service system; number of interconnections S_{inside_edge}The server at both ends of the TCP connection is shown in the service system;

output S_{out_degree}Indicating the number of other business systems accessed by the business system; s_{in_degree}Indicating the number of other business systems that have accessed the business system;

average number of connections S_{avg_edge}Indicating the number of connections S within a service system_{inside_edge}A ratio to the number of servers connected within the service system; average weight S_{avg_weight}The average value of the sum of the internal connection weights of the service system is represented by the following formula:

S_{avg_weight}＝Sum(W_{in_degree})/S_{inside_edge}

the incidence relation data information between the service systems comprises: associated server service system, associated client service system and associated weight S_weightAnd the like;

correlation weight S between service systems_weightEqual to the cumulative sum of the weights of the TCP connections between the two traffic systems;

s3, based on S1 and S2 data information, making hierarchy division for service system internal server;

s3.1 finding boundary server border inside service system_vmBorder Server Border_vmA non-database type server defined as having connection with the public network IP or having connection with an external non-monitoring service system; border border server_vmThe Level of (1) is marked as 0;

s3.2, taking a 0-layer boundary server as a starting point, according to TCP connection data of the server, finding a server accessed by the 0-layer boundary server from servers which are not divided into layers in the service system as the next layer of the 0-layer boundary server, wherein the layer Level is marked as 1, and if the server does not access other servers and only serves as a server side to provide services for other servers, the server is called a root server root_vmRoot server root_vmThe Level of (a) is marked as-1;

s3.3, repeating S3.2 by taking the server marked as Level +1 as a starting point until all servers in the service system complete hierarchical division;

s4 based on the internal hierarchy division result of the S3 service system, grouping each layer of server according to the TCP connection data extracted in S1; calculating whether the similarity between any two servers in each layer of servers reaches a threshold value, if so, dividing the servers into the same group, otherwise, dividing the servers into different groups;

similarity between two servers vm1 and vm2_vm1&vm2Defined as client Jacobian coefficient client_coefPlus server Jacobian coefficient server_coefThe formula is as follows:

similarity_vm1&vm2＝client_coef+server_coef

client-side Jacobian coefficient client_coefEqual to the server vm1 as the server set server accessed by the client_vm1Server set server accessed as client with server vm2_vm2Intersection and server of_vm1And server_vm2The union ratio, the formula is as follows:

client_coef＝(server_vm1∩server_vm2)/(server_vm1∪server_vm2)

jacobian coefficient server of server_coefEqual to the server vm1 serving as the server side_vm1Server set client served as server with server vm2_vm2Intersection of and client_vm1And client_vm2The union ratio, the formula is as follows:

server_coef＝(client_vm1∩client_vm2)/(client_vm1∪client_vm2)

s5 establishing grouped data information and grouped incidence relation data information based on the service subsystem result of the grouped discovery of S3 and S4;

s6 constructs the architecture diagram of the data center service system based on the grouping incidence relation established in S5.

Images