Disclosure of Invention
The invention aims to provide an initialization method of a communication support environment, which is beneficial to supporting a user to solve a larger problem scale and pursue higher performance, is simple and convenient to realize, does not influence the message performance, and has no extra overhead when the user calls a message interface.
In order to achieve the purpose, the invention adopts the technical scheme that: the method for initializing the communication support environment is provided, wherein cg _ per _ process or mapping _ file is set through an environment variable, and is used for a process list marked with communication requirements: the former indicates that the process with the (process number% cg _ per _ process) of 0 has a communication demand, and the other processes have no communication demand; the latter represents a process mapping file, and the content of the file is a resource serial number Sid with communication requirements;
the method comprises the following steps:
s1, calling a job management interface, analyzing the total number of resources tasksize of the job, the initial resource sequence number Sid of each process in the job, a resource name list nodeidlist, unique identifiers mpeldist in the resource slice, a pid list pidlist and a network equipment identifier list guidlist, and arranging according to the natural sequence of the resource list, wherein the ith element of each list is information of the ith resource;
s2, for the scenario with cg _ per _ process set, calculating Sid% cg _ per _ process according to the initial resource sequence Sid of each process, and extracting the relevant information of the process satisfying the modulus cg _ per _ process as 0 from the return information of the operating system, the specific operations are as follows:
s21, skipping initialization when the resource serial number Sid% cg _ per _ process is not 0, and directly returning an initialization completion identifier;
s22, reducing the total number tasksize of the resources to 1/cg _ per _ process, and setting Sid as Sid/cg _ per _ process;
s23, nodeidlist, mpeldlist, pidlist and guidllist structures adopt an in-place merging mode, an array space is multiplexed, an effective element is extracted every cg _ per _ process-1 elements, and the effective element is stored in the position where the tasksize elements are located in front of the original structures;
s3, for the scene with mapping _ file, calculating the process number after logical mapping according to the initial resource serial number Sid of each process, each process reading the mapping file, and storing the mapping file in the memory file system through the file system and the operating system interface, the specific operation is as follows:
s31, the resource serial number Sid is not in mapping _ file, skipping initialization, and directly returning to the initialization completion identifier;
s32, modifying the total number of the mapped processes, tasksize, according to the number of the mapping _ file items, and setting the natural sequence number of the item of the process with the initial resource sequence number Sid in the mapping _ file as the Sid value after logical mapping;
s33, extracting elements of nodeidlist, mpeldlist, pidlist and guidllist structures in sequence according to mapping _ file entry content;
s34, covering the front tasksize elements of the original information lists by the extracted information lists;
s4, setting an environment variable memory _ balance, setting a memory management function pointer according to a variable value, and being used for specifying that a memory space initialized by a communication support environment is mapped to a process private memory space or a processor shared memory space, and balancing memory overhead of each resource in the processor when the memory space is mapped to the shared memory;
and S5, initializing the subsequent communication support environment by using the appointed memory management function pointer, the updated tasksize and each information list, wherein the process logic number adopted in the communication is the serial number of each resource in the updated information list.
Due to the application of the technical scheme, compared with the prior art, the invention has the following advantages:
the invention can effectively reduce the process scale and memory waste, is beneficial to improving the expandability of a communication support environment, and is also beneficial to supporting users to solve larger problem scale and pursue higher performance; the method is simple and convenient to implement, the message performance is not influenced, and no extra overhead is caused when a user calls a message interface.
Detailed Description
Example (b): the invention provides an initialization method of a communication support environment, which sets cg _ per _ process or mapping _ file through an environment variable and is used for marking a process list with communication requirements: the former indicates that the process with the (process number% cg _ per _ process) of 0 has a communication demand, and the other processes have no communication demand; the latter represents a process mapping file, and the content of the file is a resource serial number Sid with communication requirements;
the method comprises the following steps:
s1, calling a job management interface, analyzing the total number of resources tasksize of the job, the initial resource sequence number Sid of each process in the job, a resource name list nodeidlist, unique identifiers mpeldist in the resource slice, a pid list pidlist and a network equipment identifier list guidlist, and arranging according to the natural sequence of the resource list, wherein the ith element of each list is information of the ith resource;
s2, for the scenario with cg _ per _ process set, calculating Sid% cg _ per _ process according to the initial resource sequence Sid of each process, and extracting the relevant information of the process satisfying the modulus cg _ per _ process as 0 from the return information of the operating system, the specific operations are as follows:
s21, skipping initialization when the resource serial number Sid% cg _ per _ process is not 0, and directly returning an initialization completion identifier;
s22, reducing the total number tasksize of the resources to 1/cg _ per _ process, and setting Sid as Sid/cg _ per _ process;
s23, nodeidlist, mpeldlist, pidlist and guidllist structures adopt an in-place merging mode, an array space is multiplexed, an effective element is extracted every cg _ per _ process-1 elements, and the effective element is stored in the position where the tasksize elements are located in front of the original structures;
s3, for the scene with mapping _ file, calculating the process number after logical mapping according to the initial resource serial number Sid of each process, each process reading the mapping file, and in order to reduce the pressure of large-scale concurrent reading on the global file system, storing the mapping file in the memory file system through the file system and the operating system interface, the specific operation is as follows:
s31, the resource serial number Sid is not in mapping _ file, skipping initialization, and directly returning to the initialization completion identifier;
s32, modifying the total number of the mapped processes, tasksize, according to the number of the mapping _ file items, and setting the natural sequence number of the item of the process with the initial resource sequence number Sid in the mapping _ file as the Sid value after logical mapping;
s33, extracting elements of nodeidlist, mpeldlist, pidlist and guidllist structures in sequence according to mapping _ file entry content;
s34, covering the front tasksize elements of the original information lists by the extracted information lists;
s4, setting an environment variable memory _ balance, setting a memory management function pointer according to a variable value, and being used for specifying that a memory space initialized by a communication support environment is mapped to a process private memory space or a processor shared memory space, and balancing memory overhead of each resource in the processor when the memory space is mapped to the shared memory;
and S5, initializing the subsequent communication support environment by using the appointed memory management function pointer, the updated tasksize and each information list, wherein the process logic number adopted in the communication is the serial number of each resource in the updated information list.
The above embodiments are further explained as follows:
the invention modifies the structure array submitted to the message support environment by the operating system, and continues the initialization process of the standard message support environment as much as possible, thereby realizing the balance of the on-demand initialization and the memory overhead by the change as little as possible.
Starting with the communication support environment initialization process modification, the resource information submitted to the communication support environment by the operating system is adjusted according to the environment variables and the configuration file, and the initialization of the communication support environment is completed as required, so that the communication requirement of a user program can be ensured, and the memory overhead can be effectively reduced; in the use aspect, a user only needs to specify environment variables and configuration files; when the number of the configuration file entries is consistent with the total number of the resources, the method and the system can realize the process mapping customized by the user.
The invention still realizes the initialization of each process under the default condition, and when the user confirms that some processes have no communication requirements in the whole process of program operation, the communication support environment can complete the message initialization customized by the user according to the requirements so as to reduce the process scale and the memory overhead; and the memory space occupied by initialization can be mapped to the private memory space or the shared memory space of the default designated process according to the designation of the user, so that the memory overhead of each process is balanced.
The method comprises the following specific steps:
1. the cg _ per _ process or mapping _ file is set by an environment variable, the process list marked with communication requirements: the former indicates that the process with the (process number% cg _ per _ process) of 0 has a communication demand, and the other processes have no communication demand; the latter represents a process mapping file, and the content of the file is a resource serial number Sid with communication requirements.
2. And calling a job management interface, analyzing the total resource number tasksize, the resource name list nodeidlist, the unique identifier mpeldist in the resource slice, the pid list pidlist, the network equipment identifier list guidlist and the like of the job, arranging the contents according to the natural sequence of the resource list, wherein the ith element in each list is the information of the ith resource.
3. For the scenario of setting cg _ per _ process, Sid% cg _ per _ process is calculated according to the initial resource sequence Sid of each process, and the relevant information of the process satisfying the module cg _ per _ process of 0 is extracted from the return information of the operating system, specifically the following operations are performed:
a) and if the initial resource sequence number Sid% cg _ per _ process is not 0, skipping initialization and directly returning to the initialization completion identifier.
b) The total number of resources tasksize is reduced to 1/cg _ per _ process.
c) The method comprises the steps of multiplexing an array space by adopting an in-place merging mode for nodeidlist, mpeeidlist, pidlist, guidllist and other structural bodies, extracting an effective element every cg _ per _ process-1 element, and storing the effective element in the position where the tasksize elements are located in front of the original structural body.
4. For the scene of setting mapping _ file, the process number after logical mapping is calculated according to the initial resource serial number Sid of each process, each process reads the mapping file, and the mapping file can be stored in the memory file system through the file system and the operating system interface in order to reduce the pressure of large-scale concurrent reading on the global file system. The specific operation is as follows:
a) and the initial resource sequence number Sid does not skip the initialization in the mapping _ file, and directly returns an initialization completion identifier.
b) And modifying the total number of the mapped processes, namely tasksize, according to the number of the mapping _ file entries, and setting the entry natural sequence number of the process with the initial resource sequence number of Sid in the mapping _ file as the logically mapped Sid value.
c) And extracting elements of nodeidlist, mpeeidlist, pidlist and guidllist structures in order according to the mapping _ file entry content.
d) And covering the front tasksize elements of the original information lists by the extracted information lists.
5. Setting an environment variable memory _ balance, setting a memory management function pointer according to a variable value, and being used for appointing the memory space initialized by the communication support environment to be mapped to a process private memory space or a processor shared memory space.
6. And (4) finishing the initialization of the subsequent communication support environment by using the appointed memory management function pointer, the updated tasksize and each information list, wherein the process logic number adopted in the communication is the serial number of each resource in the updated information list.
Through the initialization process, the invention can find that other steps except the steps of adjusting the total number of resources and the information list according to the requirements are not modified for initialization, the modification amount is less, and the implementation is easy; when cg _ per _ process =1 or mapping _ file is a resource natural sequence number table, the initialization result is the same as the original message support environment, and the standard initialization mode is compatible; the process which is not involved in communication and is appointed by the user does not participate in the specific initialization step, so that various hardware resources and memory buffers do not need to be applied; the process size is effectively compressed, and the buffer space overhead related to the process size is reduced. In addition, when mapping _ file is used, if the given mapping _ file content is any arrangement of the resource sequence number table Sid, the patent can implement process mapping, that is, any designated corresponding relationship between the process logic sequence number and the resource sequence number.
When the initialization method of the communication support environment is adopted, the process scale can be effectively reduced, the memory waste is reduced, the expandability of the communication support environment is improved, the support of users for solving larger problem scale and pursuing higher performance is facilitated; the method is simple and convenient to implement, the message performance is not influenced, and no extra overhead is caused when a user calls a message interface.
To facilitate a better understanding of the invention, the terms used herein will be briefly explained as follows:
an operation management system: the method is used for scheduling and allocating system resources in a large cluster environment, and specifically, enough resources are divided from a cluster system according to the requirements of programs to be operated, and the application programs are submitted to the divided resources to be operated.
The above embodiments are merely illustrative of the technical ideas and features of the present invention, and the purpose thereof is to enable those skilled in the art to understand the contents of the present invention and implement the present invention, and not to limit the protection scope of the present invention. All equivalent changes and modifications made according to the spirit of the present invention should be covered within the protection scope of the present invention.