Disclosure of Invention
The invention mainly solves the problems that the screening efficiency of the existing method for screening the log files is extremely low and the existing method cannot meet the production requirement of the server.
In order to solve the technical problems, the invention adopts a technical scheme that: there is provided a method of screening log files, the method comprising the steps of:
configuring a search instruction and a management instruction;
setting a screening item and a screening variable, and creating a screening instruction according to the screening item, the screening variable and the search instruction;
acquiring a first file, setting a screening value according to the management instruction, and dividing the first file according to the screening value to obtain a plurality of second files;
creating a screening process, screening the second file according to the screening instruction and the screening process, and outputting a screening result;
and judging whether the log file is qualified according to the screening result, and executing a repeated screening step.
Further, the step of creating screening instructions based on the screening terms, the screening variables, and the search instructions further comprises: assigning the screening item to the screening variable; and compiling the assigned screening variable and the search instruction in combination to obtain the screening instruction corresponding to the screening item.
Further, the step of obtaining the first file further comprises: setting a screening path; accessing the screening path and acquiring the log file in the screening path; and judging whether a first sub-path exists in the screening path, if not, summarizing the log file to obtain the first file.
Further, the step of setting a screening value according to the management instruction, and dividing the first file according to the screening value to obtain a plurality of second files further includes: calling the management instruction to acquire the number of the cores of the processor; defining the number of cores of the processor as the screening value; dividing the target file into screening intervals with the number corresponding to the screening value; and summarizing the log files according to the screening interval to obtain a plurality of second files.
Further, the step of creating a screening process, screening the second file according to the screening instruction and the screening process, and outputting a screening result further includes:
counting the number of the second files; creating the screening process corresponding to the number of the second files; calling the screening instructions through the screening process to screen the second files respectively; and after screening the second file, judging whether a first abnormal file is generated or not, if so, taking the screening result as a first result, and if not, taking the screening result as a second result.
Further, the step of determining whether the log file is qualified according to the screening result, and the step of performing the repeated screening step further includes: when the screening result is the first result, the repeated screening step is executed on the first abnormal file, and whether the log file is qualified or not is judged according to the repeated screening step; and when the screening result is the second result, judging that the log file is qualified.
Further, the screening terms include a first screening term and a second screening term; the screening instructions include first screening instructions and second screening instructions;
the step of screening the second document according to the screening instruction further comprises: calling the first screening instructions through the screening process to screen the second files respectively; and after the second file is screened, judging whether the first abnormal file corresponding to the first screening item is generated or not.
Further, the iterative screening step comprises: calling the second screening instruction to screen the first abnormal file, judging whether a second abnormal file corresponding to the second screening item is generated or not after the first abnormal file is screened, and if not, judging that the log file is unqualified; if yes, judging whether a third abnormal file except the second abnormal file exists in the first abnormal file, if yes, judging that the log file is unqualified, and if not, judging that the log file is qualified.
The invention also provides a log file screening system, which is applied to the server and comprises the following components:
the device comprises an initialization module, a command compiling module, a file dividing module, a screening module and a judging module;
the initialization module is used for configuring a search instruction and a management instruction and setting a screening item and a screening variable;
the command compiling module is used for creating screening commands according to the screening items, the screening variables and the searching commands;
the file dividing module is used for acquiring a first file and setting a screening value according to the management instruction, and the file dividing module divides the first file according to the screening value to obtain a plurality of second files;
the screening module is used for creating a screening process; the screening module screens the second file through the screening instruction and the screening process and outputs a screening result;
the judging module is used for judging whether the log file is qualified according to the screening result and executing repeated screening steps.
The present invention also provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the log file screening method.
The invention has the beneficial effects that:
1. the method for screening the log files can realize high-efficiency screening of large batches of log files under the specified requirement, has wide screening range and extremely high screening efficiency, can screen hundreds or even thousands of log files within a few minutes, greatly saves the efficiency of production test of the server, ensures the accuracy of the test result of the server and improves the reliability of the server.
2. The log file screening system can realize high-efficiency screening of large batches of log files under the specified requirement through the mutual cooperation of the initialization module, the command compiling module, the file dividing module, the screening module and the judging module, has wide screening range and extremely high screening efficiency, can screen hundreds or even thousands of log files within a few minutes, greatly saves the production and test efficiency of the server, ensures the accuracy of the test result of the server, and improves the reliability of the server.
3. The computer-readable storage medium can realize the cooperation of the guide initialization module, the command compiling module, the file dividing module, the screening module and the judging module, further realize the high-efficiency screening of large batch of log files under the specified requirement, has wide screening range and extremely high screening efficiency, can screen hundreds or even thousands of log files within a few minutes, greatly saves the production test efficiency of the server, ensures the accuracy of the test result of the server, not only improves the reliability of the server, but also effectively increases the operability of the log file screening method.
Detailed Description
The following detailed description of the preferred embodiments of the present invention, taken in conjunction with the accompanying drawings, will make the advantages and features of the invention easier to understand by those skilled in the art, and thus will clearly and clearly define the scope of the invention.
In the description of the present invention, it should be noted that the described embodiments of the present invention are a part of the embodiments of the present invention, and not all embodiments; all other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the description of the present invention, it should be noted that the terms "first" and "second" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
In the description of the present invention, it should be noted that, unless otherwise expressly specified and limited, the terms "screening command," "screening request," "screening value," "screening variable," "repeat screening step," and "looping step" are to be construed broadly. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
Example 1
The present embodiment provides a log file screening method, as shown in fig. 1 and fig. 2, including the following steps:
the embodiment is applied to the realization under the Linux system, but is not limited to the Linux system;
s100, setting a retrieval range of a log file; acquiring a log file in a retrieval range, and summarizing the log file into a target file (namely a first file);
step S100 further includes: in order to improve the retrieval efficiency and accuracy of the log files, the retrieval range of the corresponding log file can be specified according to the type to be analyzed in the server each time, and a screening path corresponding to the retrieval range can be set; accessing and acquiring all log files in the path, judging whether a sub-path exists under the path, if so, acquiring all log files under the sub-path, and executing a circulating step; the circulating steps are as follows: continuously judging whether a sub-path still exists in the sub-path, if so, acquiring the log files in the sub-path until other paths do not exist under the sub-path, and acquiring all the log files; after the acquisition is finished, all the log files are collected and stored into a target file.
S200, setting screening items; writing a screening order (i.e., screening instructions) according to the screening item;
step S200 further includes: setting a screening item according to a first keyword corresponding to a log file needing screening in the server; the screening item is the same as the first keyword and is a standard for judging whether the log file is abnormal or not; setting a screening variable "VAR"; assigning the screening item to a "VAR"; configuring and calling grep commands (namely search instructions) and regular expressions; compiling the grep command, the regular expression and the VAR in a combined manner to obtain a corresponding first screening command: similarly, a second screening command of a required second keyword can be compiled according to the principle, wherein the second keyword is a keyword which can be ignored and is also used as a standard for judging whether the log file is abnormal, and is not described herein again; for example: if the bmclog file in the target file needs to be screened whether the bmclog file contains the keywords of 'ecc, fail and error', VAR is 'ecc fail i error'; the screening order is: cat bmclog | grep-iE ecc | fail | error 'regular matching'; wherein, the smaller the search range, the faster the search speed.
S300, setting a screening value; dividing the target file according to a screening value;
step S300 further includes: in order to fully utilize resources, the number of search processes is set according to the number of CPU cores of the server; configuring and calling a management instruction: "cat/proc/cpuinfo | grep" processor "| wc-1" obtains the kernel number N, and defines the kernel number N as the screening value; dividing the target file into N screening intervals, if Q log files exist in the target file, P log files exist in each screening interval, and if a remainder O exists, O is Q- (P × N); therefore, the number of log files in the last screening interval is: o + P; respectively summarizing the log files in the screening interval to obtain a second file; and adding O to any interval, and if the remainder O is not available, not considering the step of adding, wherein the purpose of the step is to realize parallel processing of the target file by a plurality of processes.
S400, calling the screening command to screen the divided target file, and outputting a first screening result;
step S400 further includes: the first screening result includes null (i.e., second result) and non-null (i.e., first result); the null represents a log file which does not contain corresponding keywords in the target file; non-null represents a log file in the target file that contains the corresponding key.
S500, judging whether the log file is normal or not according to the first screening result, and further judging whether the server is qualified or not;
step S500 further includes: when the first screening result is empty, judging that the log file is qualified and the server is qualified; when the first screening result is non-empty, generating a first abnormal file corresponding to the first keyword, wherein the first abnormal file is a log file of a target file containing the first keyword, and calling the second screening command to screen the first abnormal file to obtain a second screening result; the second screening result comprises null and non-null; when the second screening result is empty, judging that the log file is unqualified and the server is unqualified; when the second screening result is non-empty, generating a second abnormal file; and judging whether other log files (namely third abnormal files) except the second abnormal file exist in the first abnormal file, if so, judging that the log files are unqualified and the server is unqualified, and if not, judging that the log files are qualified and the server is qualified.
Example 2
The present embodiment provides a log file screening system, as shown in fig. 3, including:
the device comprises an initialization module, a command compiling module, a file dividing module, a screening module and a judging module;
the initialization module is used for setting a retrieval range of the log files, acquiring the log files in the retrieval range and summarizing the log files into a target file;
the initialization module further comprises, during specific operations: in order to improve the retrieval efficiency and accuracy of the log files, the initialization module can specify the retrieval range of the corresponding log files according to the types to be analyzed in the server each time and set the screening paths corresponding to the retrieval range; the initialization module accesses and acquires all log files in the path, judges whether a sub-path exists under the path or not, and acquires all log files under the sub-path if the sub-path exists under the path until other paths do not exist under the sub-path; after the acquisition is finished, the initialization module collects all log files and stores the log files into a target file.
The command compiling module is used for setting screening items and compiling screening commands according to the screening items;
when the command compiling module is operated specifically, the command compiling module further comprises: the command compiling module sets a screening item according to a first keyword corresponding to a log file needing screening in the server; the screening item is the same as the first keyword and is a standard for judging whether the log file is abnormal or not; commanding the compiling module to set a screening variable "VAR" and assigning the screening item to "VAR"; the command compiling module configures a grep command and a regular expression, and combines and compiles the grep command, the regular expression and a screening variable to obtain a corresponding first screening command: similarly, according to the principle, a second screening command of a required second keyword can be compiled, wherein the second keyword is a keyword that can be ignored and is also used as a standard for judging whether the log file is abnormal, and details are not repeated herein.
The file dividing module is used for setting a screening factor and dividing the target file according to the screening factor;
when the file division module is in specific operation, the file division module further comprises: in order to fully utilize resources, the file dividing module sets the number of search processes according to the number of CPU cores of the server; the file division module configures and calls a management command: "cat/proc/cpuinfo | grep" processor "| wc-1" obtains the kernel number N, and defines the kernel number N as the screening value; the file dividing module divides the target file into N screening intervals, if the target file has Q log files, P log files exist in each screening interval, and correspondingly, if a remainder O exists, O is Q- (P x N); therefore, the number of log files in the last screening interval is: and O + P, wherein O can be added to any interval, and if the remainder O is not available, the step of adding can be omitted, and the purpose of the step is to realize parallel processing of the target file by a plurality of processes.
The screening module is used for calling the screening command to screen the divided target files and outputting a first screening result; the first screening result includes null and non-null; the null represents a log file which does not contain corresponding keywords in the target file; non-null represents a log file in the target file that contains the corresponding key.
The judging module is used for judging whether the log file is normal according to the first screening result so as to judge whether the server is qualified;
when the judging module is in specific operation, the judging module further comprises: when the first screening result is empty, a judging module judges that the log file is qualified and the server is qualified; when the first screening result is non-empty, the screening module generates a first abnormal file corresponding to the first keyword, wherein the first abnormal file is a log file containing the first keyword in a target file; at the moment, the judgment module calls the second screening command to screen the first abnormal file to obtain a second screening result; when the second screening result is empty, the judging module judges that the log file is unqualified and the server is unqualified; when the second screening result is non-empty, the judgment module generates a second abnormal file; the judging module judges whether other log files except the second abnormal file exist in the first abnormal file, if so, the judging module judges that the log files are unqualified and the server is unqualified, and if not, the judging module judges that the log files are qualified and the server is qualified.
Based on the same inventive concept as one of the log file screening methods in the foregoing embodiments, an embodiment of the present specification further provides a computer-readable storage medium having a computer program stored thereon, where the computer program, when executed by a processor, implements the steps of the log file screening method.
Different from the prior art, the log file screening method, the log file screening system and the log file screening medium can be used for efficiently screening a large batch of log files under the specified requirement, the screening range is wide, hundreds or even thousands of log files can be screened within a few minutes, the production test efficiency of a server is greatly saved, the log files are screened for the second time by compiling the screening command related to the omitted keywords, the accuracy of the test result of the server is further ensured, and the reliability of the server is improved.
The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.
It will be understood by those skilled in the art that all or part of the steps of implementing the above embodiments may be implemented by hardware, and a program that can be implemented by the hardware and can be instructed by the program to be executed by the relevant hardware may be stored in a computer readable storage medium, where the storage medium may be a read-only memory, a magnetic or optical disk, and the like.
The above description is only an embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes performed by the present specification and drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.