CN107885549A - Remove the method and system that process is remained in TORQUE computing cluster calculate nodes - Google Patents

Abstract

The invention belongs to the field of computer technology, and in particular relates to the management of high-performance computer clusters, in particular to a method and system for clearing residual processes in computing nodes of TORQUE computing clusters. The method for clearing the residual processes in the computing nodes of the TORQUE computing cluster includes: obtaining all user names from the user name list file and storing them in the user list; obtaining the user names of running tasks in all current computing nodes and storing them in the relational array; The computing node creates a file with the computing node name as the file name, and stores the user names that currently have no task execution on the computing node in the file; deletes the residual process for each computing node; automatically updates the user name list file. The system for clearing the residual processes in the computing nodes of the TORQUE computing cluster includes: a first acquisition module; a second acquisition module; a file creation module; a residual process deletion module; an update module. The invention can quickly and accurately clear the residual processes in the computing nodes of the TORQUE computing cluster.

Description

Method and system for clearing residual processes in TORQUE computing cluster computing nodes

technical field

The invention belongs to the field of computer technology, and in particular relates to the management of high-performance computer clusters, in particular to a method and system for clearing residual processes in computing nodes of TORQUE computing clusters.

Background technique

High-performance computing can be used for numerical analysis and simulation experiments. It is recognized as an important research method and an important means of scientific innovation. At present, high-performance computing capability is the embodiment of a country's comprehensive national strength and has an important impact on national strategy.

There are two ways to improve computing power: one is to increase the computing power of a single machine, such as increasing the main frequency of the CPU, adopting many-core technology, or using GPU computing, etc.; the other is to use a high-speed network (such as InfiniBand) to parallel multiple machines computer. At present, the construction of high-performance computing clusters often takes the above two factors into consideration, and chooses the most cost-effective solution. The computing software and computing results are stored in the file storage system to ensure the synchronization of data on each node. When the job is running, the management node assigns the task to the computing node for calculation, and a computing process is generated in the computing node. When the calculation is completed, the calculation process in the computing node ends, and the management node can no longer check the status of the task.

In the actual operation of the cluster, it is often found that individual computing nodes have computing processes running, but their running status cannot be found on the management node. For example, the sth computing node has a computing process running, but the management node thinks that the sth node is in an idle state. If there is a new task, the management node will continue to submit the task to the sth node, causing multiple computing processes of the sth node to run at the same time, which greatly reduces the computing efficiency. Here, we refer to the computing processes that cannot be found by the management node as residual processes. There are many reasons for the residual process, the most important is that the user program is not written in a standardized way, the second is that the user did not submit the computing task through the correct method, and it may be a system bug. In short, the residual computing process seriously affects the execution of normal computing tasks and should be removed. There is an urgent need for a fast and accurate method and system for clearing residual processes in computing nodes.

Contents of the invention

The purpose of the present invention is to overcome the above-mentioned deficiencies, and provide a method and system for clearing residual processes in computing nodes of TORQUE computing clusters, so as to solve the impact of residual processes on computing nodes and TORQUE computing clusters, and ensure fast execution of tasks.

In order to achieve the above object, the present invention adopts the following technical solutions:

A method for clearing residual processes in TORQUE computing cluster computing nodes, comprising the following steps:

Step 1: Obtain all user names from the user name list file and store them in the user list;

Step 2: Obtain the usernames of the running tasks in all current computing nodes and store them in the relational array;

Step 3: Create a file with the name of the computing node for each computing node, and store the user name in the file that currently has no task execution on the computing node;

Step 4: Delete residual processes for each computing node;

Step 5: Automatically update the user name list file.

Preferably, before said step 1, it also includes:

Create a username list file for the TORQUE compute cluster.

Preferably, after said step 4, it also includes:

Record the status of each computing node to delete residual processes.

Clear the system of residual processes in the computing nodes of the TORQUE computing cluster, including:

The first obtaining module is used to obtain all user names from the user name list file and store them in the user list;

The second obtaining module is used to obtain user names of running tasks in all current computing nodes and store them in a relational array;

The file building module is used to create a file named after the computing node for each computing node, and the file is stored in the user name that currently has no task execution on the computing node;

A residual process deletion module is used to delete residual processes for each computing node;

Update module for automatically updating username list files.

Preferably, it also includes:

Create a module for creating username list files for TORQUE compute clusters.

Preferably, it also includes:

The recording module is used to record the state of deleting residual processes of each computing node.

Compared with the prior art, the present invention has the beneficial effects:

The present invention obtains all user names from the user name list file and stores them in the user list, then obtains the user names of running tasks in all current computing nodes and stores them in a relational array, and then establishes a computing node for each computing node A file named file name, which stores the user name that currently has no task execution on the computing node, and finally deletes the residual process for each computing node, automatically updates the user name list file, and deletes each computing node of each TORQUE computing cluster Residual processes, so as to eliminate the impact of residual processes on computing tasks, so that computing tasks can be completed within the scheduled time, and the use of clusters is standardized.

Description of drawings

FIG. 1 is one of the basic flow diagrams of the method for clearing residual processes in computing nodes of a TORQUE computing cluster according to the present invention.

FIG. 2 is the second schematic flow diagram of the method for clearing residual processes in the computing nodes of the TORQUE computing cluster according to the present invention.

FIG. 3 is one of the structural schematic diagrams of the system for clearing residual processes in the computing nodes of the TORQUE computing cluster according to the present invention.

FIG. 4 is the second schematic structural diagram of the system for clearing residual processes in the computing nodes of the TORQUE computing cluster according to the present invention.

Detailed ways

For ease of understanding, the following explanations are given to some nouns that appear in the specific embodiments of the present invention:

TORQUE computing cluster: a common high-performance computing cluster job management system.

Management node: a node used to manage and allocate computing resources, generally there is only one cluster.

Computing node: The node that is mainly used for computing in the cluster, generally there are many clusters.

Residual process: the computing process that cannot be found in the management node.

The present invention will be further explained below in conjunction with accompanying drawing and specific embodiment:

Embodiment one:

As shown in Figure 1, a kind of method of clearing the residual process in the TORQUE computing cluster computing node of the present invention comprises the following steps:

Step S101: Obtain all user names from the user name list file and store them in the user list;

Step S102: Obtain the usernames of the running tasks in all current computing nodes and store them in the relational array;

Step S103: Create a file named after the computing node for each computing node, and store the user name in the file that currently has no task execution on the computing node;

Step S104: delete residual processes for each computing node;

Step S105: Automatically update the user name list file.

Embodiment two:

As shown in Figure 2, another method of clearing the residual process in the TORQUE computing cluster computing node of the present invention comprises the following steps:

Step S201: Create a user name list file user.list for the TORQUE computing cluster.

Step S202: Obtain all user names from the user name list file user.list and store them in the user list uli;

For example, suppose there are 10 users in the system:

uli=[user1, user2, user3, user4, user5, user6, user7, user8, user9, user10];

Step S203: Obtain the user names of the running tasks in all current computing nodes, and store them in the relational array idic in the format of idic[node name]=[user list];

For example, assume that the TORQUE computing cluster has 16 computing nodes, which are named node 1, node 2... node 16, taking node 1, node 2, and node 16 as examples:

idic[node1] = [user1, user3, user8];

idic[node2] = [user1, user4];

idic[node16] = [user5, user6].

Step S204: Create a file named after the computing node for each computing node, and store the user name in the file that currently has no task execution on the computing node;

E.g:

Store in the node1 file: user2, user4, user5, user6, user7, user9, user10;

Store in the node2 file: user2, user3, user5, user6, user7, user8, user9, user10;

Stored in the file of node16: user1, user2, user3, user4, user7, user8, user9, user10.

Step S205: According to the user name in each computing node file that currently has no task execution on the computing node, delete the residual process for each computing node;

E.g:

Delete all processes of user2, user4, user5, user6, user7, user9, user10 under node1;

Delete all processes of user2, user3, user5, user6, user7, user8, user9, user10 under node2;

Delete all processes of user1, user2, user3, user4, user7, user8, user9, user10 under node16.

Step S206: Record the status of deleting residual processes of each computing node, and write the status of deleting residual processes of each computing node into now.list until all residual processes are deleted.

Step S207: If the administrator newly adds a user, and the user is in the state of executing a task, automatically update the user name list file user.list.

Embodiment three:

As shown in Fig. 3, a kind of system of the present invention clears the residual process in the computing node of TORQUE computing cluster, comprises:

The first obtaining module 301 is used to obtain all user names from the user name list file and store them in the user list;

The second obtaining module 302 is used to obtain the usernames of running tasks in all current computing nodes and store them in a relational array;

The file creation module 303 is used to set up a file with the name of the computing node as the file name for each computing node, and store the user name currently having no task execution in the computing node in the file;

A residual process deletion module 304, configured to delete the residual process for each computing node;

An update module 305, configured to automatically update the user name list file.

Embodiment four:

As shown in Figure 4, another system of the present invention that removes residual processes in the computing nodes of the TORQUE computing cluster includes:

The creation module 401 is used for creating a user name list file for the TORQUE computing cluster.

The first obtaining module 402 is used to obtain all user names from the user name list file and store them in the user list;

The second obtaining module 403 is used to obtain the usernames of running tasks in all current computing nodes and store them in the relational array;

The file creation module 404 is used to set up a file with the computing node name as the file name for each computing node, and store the user name currently having no task execution in the computing node in the file;

Residual process deletion module 405, configured to delete residual processes for each computing node;

The recording module 406 is configured to record the state of deleting residual processes of each computing node.

An update module 407, configured to automatically update the user name list file.

What is shown above is only a preferred embodiment of the present invention, and it should be pointed out that for those of ordinary skill in the art, some improvements and modifications can also be made without departing from the principle of the present invention. It should be regarded as the protection scope of the present invention.

Claims (6)

1. the method for process is remained in a kind of removing TORQUE computing cluster calculate nodes, it is characterised in that comprise the following steps：

Step 1：All user names are obtained from user name listing file, are stored in user list；

Step 2：The user name of operation task in current all calculate nodes is obtained, is stored in relation array；

Step 3：A file with the entitled filename of calculate node is established for each calculate node, is stored in currently at this in file Calculate node does not have the user name of tasks carrying；

Step 4：Residual process is deleted for each calculate node；

Step 5：Automatically update user name listing file.

2. remaining the method for process in removing TORQUE computing cluster calculate nodes according to claim 1, its feature exists In also including before the step 1：

User name listing file is created for TORQUE computing clusters.

3. remaining the method for process in removing TORQUE computing cluster calculate nodes according to claim 1, its feature exists In also including after the step 4：

Record the situation that each calculate node deletes residual process.

4. based on the clear of the method that process is remained in any described removing TORQUE computing cluster calculate nodes of claim 1-3 Except the system that process is remained in TORQUE computing cluster calculate nodes, it is characterised in that including：

First acquisition module, for obtaining all user names from user name listing file, it is stored in user list；

Second acquisition module, for obtaining the user name of operation task in current all calculate nodes, it is stored in relation array；

File establishes module, for establishing a file with the entitled filename of calculate node for each calculate node, in file Deposit does not currently have the user name of tasks carrying in the calculate node；

Process-kill module is remained, for deleting residual process for each calculate node；

Update module, for automatically updating user name listing file.

5. remaining the system of process in removing TORQUE computing cluster calculate nodes according to claim 4, its feature exists In, in addition to：

Creation module, for creating user name listing file for TORQUE computing clusters.

6. remaining the system of process in removing TORQUE computing cluster calculate nodes according to claim 4, its feature exists In, in addition to：

Logging modle, the situation of residual process is deleted for recording each calculate node.