CN116303319A - A log collection method, device and storage medium - Google Patents

Abstract

The application discloses a log acquisition method, a log acquisition device and a storage medium, which are used for preventing log loss. The log acquisition method disclosed by the application comprises the following steps: storing the generated log in a file A; collecting the content in the file A in real time, and uploading the content to the cloud; cutting and rotating the logs in the file A, and cleaning out expired logs; setting a shared dictionary in a shared memory; and starting a timer task, wherein the timer task performs path switching operation according to the shared dictionary. The application also provides a log acquisition device and a storage medium.

Description

A log collection method, device and storage medium

technical field

The present application relates to the technical field of computing, and in particular to a log collection method, device and storage medium.

Background technique

With the rapid development of the Internet, more and more websites are facing the challenge of high concurrent requests. When using web services or dynamic gateways, log recording and analysis will be essential. Traditional log recording and collection solutions generally send logs directly to the cloud. When the network is interrupted, logs will be lost. When the logs are saved locally and then collected, when the logs with log rotation enabled are collected through the Agent, the logs will also be lost.

Contents of the invention

In view of the above technical problems, embodiments of the present application provide a log collection method, device, and storage medium for log collection.

In the first aspect, a log collection method provided in the embodiment of the present application includes:

Save the generated log in file A;

Collect the content in the file A in real time, and upload the content to the cloud;

Cut and rotate the logs in the file A, and clean up expired logs;

Set up a shared dictionary in shared memory;

A timer task is started, and the timer task performs a path switching operation according to the shared dictionary.

In the present invention, the log file A is uploaded to the cloud, and the file A is cut and rotated, the timing task is started, and whether to perform the path switching operation is judged according to the shared dictionary, thereby improving the security of the log record and preventing the loss of the log file.

Preferably, the real-time collection of the contents of the file A includes:

When the content in the file A is collected in real time, a cache with a predetermined length is set, and the cache is used for log retransmission when the network is interrupted and then restored.

Start timer tasks include:

Add a timer task to each process.

Preferably, the path switching operation performed by the timer task according to the shared dictionary includes:

When the timer task times out, it is judged according to the shared dictionary whether a log path switching operation needs to be triggered, and if a log path switching operation is required, the shared dictionary is reset after the log path switching operation is completed.

Preferably, said resetting said shared dictionary comprises:

The shared dictionary is reset through the newly added HTTP API interface.

Preferably, the cutting and rotation of the log in the file A includes:

When log rotation is required, each process calls the HTTP API interface to perform a log path switching operation;

Each process sleeps for a predetermined length of time, and renames the file A to a log file B after the dormancy ends;

Each process modifies the log storage path to a temporary file C.

When log rotation is complete, also include:

Convert the temporary file C to file A in MV mode, and continue to save the log in the file A.

In the second aspect, the embodiment of the present application also provides a log collection device, including:

The acquisition module is configured to save the generated log in file A;

The upload module is configured to collect the content in the file A in real time, and upload the content to the cloud;

The rotation module is configured to cut and rotate the logs in the file A, and clean up expired logs;

A dictionary module configured to set up a shared dictionary in shared memory;

The switching module is configured to start a timer task, and the timer task performs a path switching operation according to the shared dictionary.

In a third aspect, the embodiment of the present application further provides a log collection device, including: a memory, a processor, and a user interface;

The memory is used to store computer programs;

The user interface is configured to interact with the user;

The processor is configured to read the computer program in the memory, and when the processor executes the computer program, the log collection method provided by the present invention is realized.

In a fourth aspect, the embodiment of the present application further provides a processor-readable storage medium, the processor-readable storage medium stores a computer program, and the processor implements the log collection method provided by the present invention when executing the computer program .

Using the log collection method of the present invention, the renaming mode is adopted instead of the copy and truncation mode during log rotation, which can ensure that all logs can be recorded in files. The scheme of switching engine logs to temporary files before log rotation and renaming the temporary files to original files after log rotation can ensure that all logs are collected without loss of logs on the cloud.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For Those of ordinary skill in the art can also obtain other drawings based on these drawings without making creative efforts.

Fig. 1 provides a schematic flow chart of the log collection method in the application embodiment;

Fig. 2 is a log collection example 1 provided by the embodiment of the present application;

FIG. 3 is a log collection example 2 provided by the embodiment of the present application;

Fig. 4 is the third example of log collection provided by the embodiment of the present application;

FIG. 5 is a schematic diagram of a log collection device provided in an embodiment of the present application;

FIG. 6 is a schematic structural diagram of another log collection device provided in an embodiment of the present application.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments . Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

The following is an explanation of some words that appear in the text:

1. The term "and/or" in the embodiment of the present invention describes the association relationship of associated objects, indicating that there may be three relationships, for example, A and/or B, which may mean: A exists alone, A and B exist simultaneously, and There are three cases of B. The character "/" generally indicates that the contextual objects are an "or" relationship.

2. The term "multiple" in the embodiments of this application refers to two or more, and other quantifiers are similar.

The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

It should be noted that the display order of the embodiments of the present application only represents the sequence of the embodiments, and does not represent the advantages or disadvantages of the technical solutions provided by the embodiments.

Referring to Fig. 1, a schematic diagram of a log collection method provided in the embodiment of the present application, as shown in Fig. 1, the method includes steps S101 to S103:

S101. Save the generated log in file A;

S102. Collect the content in the file A in real time, and upload the content to the cloud;

S103. Cut and rotate the logs in the file A, and clean up expired logs;

S104, setting a shared dictionary in the shared memory;

S105. Start a timer task, where the timer task performs a path switching operation according to the shared dictionary.

Using the log collection method of the present invention, the renaming mode is adopted instead of the copy and truncation mode during log rotation, which can ensure that all logs can be recorded in files. The scheme of switching engine logs to temporary files before log rotation and renaming the temporary files to original files after log rotation can ensure that all logs are collected without loss of logs on the cloud.

As a preferred example, in S101, saving the generated log in file A may include: when collecting the content in file A in real time, setting a cache of a predetermined length, the cache is used for recovery when the network is interrupted When the log retransmission operation is performed.

In the present invention, file A refers to the specified file for saving the log, which is used to distinguish other files that will be used later in the present invention, indicating that file A is not the same file as other files.

Wherein, the file A is determined before the log collection starts or before the log file is to be recorded in the file, and the determined content includes the file name, the storage path, and the like. It may also include a module or process for saving the generated log into the file A, that is, which module or which process saves the generated log into the file A.

As a preferred example, in S102, collecting the content in the file A in real time and uploading the content to the cloud may include:

Use the log collection agent to monitor and collect log file A in real time, and upload the collected logs to the cloud, and the cloud will save the received logs for subsequent retrieval and analysis. As a preferred example, the proxy increases the cache to enable the proxy to perform log retransmission operations when the network is interrupted and restored to ensure that the logs on the cloud are not lost and complete.

Wherein, the cloud refers to a cloud server or a cloud storage;

As a preferred example, the log collection agent may be a separate process or a separate module. The cache size increased by the proxy can be predetermined as needed. When the network is interrupted, the logs in the cache are retransmitted to the cloud to ensure that the logs will not be lost and the integrity of the logs is guaranteed. The larger the cache, the longer the network interruption can be tolerated, but the more resources are consumed at the same time. Therefore, the size of the cache needs to be comprehensively determined according to resource conditions and security requirements.

As a preferred example, cutting and rotating the logs in the file A in S103, and clearing expired logs includes:

Rotate the generated log file A, and set a timer to call a script, detect whether the log file A needs to be rotated in the script, and clean up the expired log file, avoiding the speed and performance of writing the log and preventing the log from filling up the entire disk . Wherein, the determination of whether to expire is preset according to needs. As a preferred example, logs exceeding 7 days are determined to be expired.

As a preferred example, setting the shared dictionary in the shared memory in S104 includes:

Apply for a shared dictionary in the shared memory, so that each subsequent process can judge whether it needs to switch the log path, and initialize the shared dictionary during initialization, add a dictionary for each process (worker), and the process key (key) is Process ID (worker_id), the value is false. As a preferred example, a value of false indicates that the path switching operation is not required, and a value of true indicates that the path switching operation is not performed.

As a preferred example, the task of starting the timer includes:

Add a timer task to each process.

As a preferred example, starting a timer task in S105, and performing a path switching operation according to the shared dictionary by the timer task may include:

When the timer task times out, it is judged according to the shared dictionary whether a log path switching operation needs to be triggered, and if a log path switching operation is required, the shared dictionary is reset after the log path switching operation is completed.

As a preferred example, add a timed task to each process during the initialization phase. When the timer task times out, go to the shared dictionary applied for in step S104 to get the value corresponding to the current process number as the key key to determine whether to trigger the log Path switch operation, if it is true, switch the save path of the log to file C and reset the corresponding key in the shared dictionary to false.

Said resetting said shared dictionary includes:

The shared dictionary is reset through the newly added HTTP API interface.

Specifically, when it is detected that log rotation is required, the HTTP API interface will be called to allow each process to perform log path switching operations, and then sleep and wait for a predetermined length of time to ensure that each process triggers log path switching operations. If the interface returns not successful then Continue to request the HTTP API interface, retry the mechanism up to 5 times, and then perform the log rotation operation, that is, rename the log file A to the rotated log file B. At this time, the log collection agent will lose the collection target, that is, file A.

As a preferred example, the cutting and rotating the log in the file A includes:

When log rotation is required, each process calls the HTTP API interface to perform a log path switching operation;

Each process sleeps for a predetermined length of time, and renames the file A to a log file B after the dormancy ends;

Each process modifies the log storage path to a temporary file C.

When log rotation is complete, also include:

Convert the temporary file C to file A in MV mode, and continue to save the log in the file A.

The above methods from S101 to S105 will be described below with reference to specific examples. In this example, the log collection agent is Agent, the log collection platform is OpenResty, and the cloud stores the received logs in Elasticsearch:

Step 1: Add a Lua module in OpenResty to save the generated log to a file, support specifying the save path of the log file, and the log file is file A;

Step 2: Use the log collection Agent to monitor and collect log file A in real time, and upload the collected logs to the cloud, and the cloud will store the received logs in Elasticsearch for subsequent retrieval and analysis. The Agent increases the cache design for when the network is interrupted When recovering, the Agent can perform log retransmission operations to ensure that the logs on the cloud are not lost;

Step 3: Use the mv+nocreate mode of logrotate to rotate the log file A generated by OpenResty, and set crontab to call a bash script regularly, call logrotate in the script to detect whether the log file A needs to be rotated, and clean up the log file A that is more than 7 days old Log files, to avoid slowing down the speed and performance of writing logs and preventing logs from filling up the entire disk;

Step 4: Apply for a shared dictionary log_path_switch in the shared memory of OpenResty, so that each subsequent worker can determine whether a log path switching operation is required, and initialize the shared dictionary in the init_by_lua stage of OpenResty, and add a dictionary dict, key for each worker is worker_id, value is false;

Step 5: Add a scheduled task to each worker in the init_worker_by_lua stage of OpenResty, go to the shared dictionary applied in step 4 every 0.1 seconds to get the value corresponding to the current worker_id as the key to determine whether to trigger the log path switch operation, if If true, switch the save path of the log to file C and reset the corresponding key in the shared dictionary to false;

Step 6: Add an HTTP API interface in OpenResty, and after calling, modify the value of each key in the shared dictionary applied in step 4 to true;

Step 7: When logrotate detects that log rotation is required, it will call the HTTP API interface in step 6 in the prerotate stage to let each worker perform log path switching operations, and then sleep for 0.5 seconds to ensure that each worker triggers log path switching operations. If the interface return is not successful, continue to request the HTTP API interface, retry the mechanism up to 5 times, and then perform the log rotation operation, that is, the log file A mv will be converted into the rotated log file B. At this time, the log collection agent will lose the collection target. i.e. file A;

Step 8: When the scheduled task of each worker detects the need to switch the log path, it will switch the log saving path to the temporary file C, and the subsequent logs will be saved in the file C;

Step 9: When logrotate detects that the log rotation is complete, the temporary file Cmv is converted into the original log file A in the postrotate stage. At this time, the log collection Agent detects that the collection target appears and will continue to collect operations. According to the Linux file system feature, Openresty's new log will continue to be saved in log file A after mv.

The above is only an example. In order to better illustrate the solution of the present invention in combination with the above examples, and in combination with the usage scenarios, some examples are given to illustrate the use of the method of the present invention.

As shown in Figure 2, the client initiates a request to OpenResty, and OpenResty generates log data and saves it in file A. At the same time, the log collection agent collects the content in the log file and uploads it to the cloud.

As shown in Figure 3, 1. logrotate calls OpenResty’s HTTP API to perform log switching operations before log rotation; 2. OpenResty changes the log storage path from file A to file C after receiving the HTTP API request; 3. The client initiates access Request; 4. OpenResty saves the newly generated log to log file C. At this time, the log collection agent will stop collecting because the target file A does not exist; 5. Logrotate performs log rotation operation and renames log file A to log file B .

As shown in Figure 4, 1. Logrotate calls the MV command to rename the temporary file C to log file A after the log rotation is successful; 2. The client initiates an access request; 3. Due to the characteristics of the Linux file system, OpenResty will generate The log is written into the renamed file A; 4. Agent detects that the log file A exists, and will collect and upload the log to the cloud.

Using the log collection method of the present invention, the renaming mode is adopted instead of the copy and truncation mode during log rotation, which can ensure that all logs can be recorded in files. The scheme of switching engine logs to temporary files before log rotation and renaming the temporary files to original files after log rotation can ensure that all logs are collected without loss of logs on the cloud. Taking the above example as an example, the technical effects obtained by the present invention are as follows: 1) The mv mode is used instead of the copytruncate mode during logrotate log rotation, so as to ensure that all OpenResty logs can be recorded in the file, because the rotation mode of copytruncate is after the copy The new logs generated in this time slice before truncate will be cleared and lost. 2) In order to be compatible with the log collection by the Agent, switch the engine log to a temporary file before the logrotate log rotation, and mv the temporary file into the original file after the log rotation is completed, so as to ensure that the Agent collects all the logs without causing cloud Logs on are missing. 3) The Agent increases the cache design so that when the network is interrupted and then restored, the Agent can retransmit the logs generated during the network interruption to ensure that the logs on the cloud are not lost.

Based on the same inventive concept, the embodiment of the present invention also provides a log collection device, as shown in Figure 4, the device includes:

The acquisition module 401 is configured to save the generated log in file A;

The upload module 402 is configured to collect the content in the file A in real time, and upload the content to the cloud;

The rotation module 404 is configured to cut and rotate the logs in the file A, and clean up expired logs;

The dictionary module 403 is configured to set a shared dictionary in the shared memory;

The switching module 405 is configured to start a timer task, and the timer task performs a path switching operation according to the shared dictionary.

As a preferred example, the collection module 401 is also configured to set a buffer of a predetermined length when collecting the contents of the file A in real time, and the buffer is used for log retransmission when the network is interrupted and then restored.

As a preferred example, the switching module 405 is also configured to add a timer task to each process;

As a preferred example, the switching module 405 is also configured to judge whether a log path switching operation needs to be triggered according to the shared dictionary when the timer task times out, and if a log path switching operation is required, the log path switching operation is completed After the operation, the shared dictionary is reset.

Said resetting said shared dictionary includes:

The shared dictionary is reset through the newly added HTTP API interface.

The cutting and rotation of the logs in the file A includes:

When log rotation is required, each process calls the HTTP API interface to perform a log path switching operation;

Each process sleeps for a predetermined length of time, and renames the file A to a log file B after the dormancy ends;

Each process modifies the log storage path to a temporary file C.

As a preferred example, the switching module 405 is further configured to convert the temporary file C into file A in MV mode when the log rotation is completed, and continue to save the log in the file A.

It should be noted that the device described in Figure 4 belongs to the same inventive concept as the above-mentioned method embodiment, solves the same technical problem, achieves the same technical effect, and the device can realize all the methods, and the similarities will not be repeated.

Based on the same inventive concept, the embodiment of the present invention also provides a log collection device, as shown in Figure 5, the device includes:

Including memory 502, processor 501 and user interface 503;

The memory 502 is used to store computer programs;

The user interface 503 is configured to interact with the user;

The processor 501 is configured to read the computer program in the memory 502, and when the processor 501 executes the computer program, it realizes:

Save the generated log in file A;

Collect the content in the file A in real time, and upload the content to the cloud;

Cut and rotate the logs in the file A, and clean up expired logs;

Set up a shared dictionary in shared memory;

A timer task is started, and the timer task performs a path switching operation according to the shared dictionary.

Wherein, in FIG. 5 , the bus architecture may include any number of interconnected buses and bridges, specifically one or more processors represented by the processor 501 and various circuits of the memory represented by the memory 502 are linked together. The bus architecture can also link together various other circuits such as peripherals, voltage regulators, and power management circuits, etc., which are well known in the art and therefore will not be further described herein. The bus interface provides the interface. The processor 501 is responsible for managing the bus architecture and general processing, and the memory 502 can store data used by the processor 501 when performing operations.

The processor 501 may be a CPU, ASIC, FPGA or CPLD, and the processor 501 may also adopt a multi-core architecture.

When the processor 501 executes the computer program stored in the memory 502, any log collection method in the present invention is implemented.

It should be noted that the device and method shown in Figure 5 belong to the same inventive concept, solve the same technical problem, achieve the same technical effect, and the device can realize all the methods, and the similarities will not be repeated.

The present application also proposes a processor-readable storage medium. Wherein, the processor-readable storage medium stores a computer program, and when the processor executes the computer program, any log collection method in Embodiment 1 is implemented.

It should be noted that the division of units in the embodiment of the present application is schematic, and is only a logical function division, and there may be another division manner in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowcharts and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

Obviously, those skilled in the art can make various changes and modifications to the application without departing from the spirit and scope of the application. In this way, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims (10)

1. A log collection method, characterized in that, comprising: Save the generated log in file A; Collect the content in the file A in real time, and upload the content to the cloud; Cut and rotate the logs in the file A, and clean up expired logs; Set up a shared dictionary in shared memory; A timer task is started, and the timer task performs a path switching operation according to the shared dictionary. 2. The method according to claim 1, wherein said collecting the contents of said file A in real time comprises: When the content in the file A is collected in real time, a cache with a predetermined length is set, and the cache is used for log retransmission when the network is interrupted and then restored. 3. The method according to claim 2, wherein the starting timer task comprises: Add a timer task to each process. 4. The method according to claim 3, wherein said timer task performing a path switching operation according to said shared dictionary comprises: When the timer task times out, it is judged according to the shared dictionary whether a log path switching operation needs to be triggered, and if a log path switching operation is required, the shared dictionary is reset after the log path switching operation is completed. 5. The method according to claim 4, wherein said resetting said shared dictionary comprises: The shared dictionary is reset through the newly added HTTP API interface. 6. The method according to claim 5, wherein the cutting and rotating the log in the file A comprises: When log rotation is required, each process calls the HTTP API interface to perform a log path switching operation; Each process sleeps for a predetermined length of time, and renames the file A to a log file B after the dormancy ends; Each process modifies the log storage path to a temporary file C. 7. The method according to claim 6, further comprising: when the log rotation is completed: Convert the temporary file C to file A in MV mode, and continue to save the log in the file A. 8. A log collection device, characterized in that, comprising: The acquisition module is configured to save the generated log in file A; The upload module is configured to collect the content in the file A in real time, and upload the content to the cloud; The rotation module is configured to cut and rotate the logs in the file A, and clean up expired logs; A dictionary module configured to set up a shared dictionary in shared memory; The switching module is configured to start a timer task, and the timer task performs a path switching operation according to the shared dictionary. 9. A log collection device, comprising a memory, a processor and a user interface; The memory is used to store computer programs; The user interface is configured to interact with the user; The processor is configured to read the computer program in the memory, and when the processor executes the computer program, the log collection method according to any one of claims 1 to 7 is realized. 10. A processor-readable storage medium, wherein the processor-readable storage medium stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is realized. Log collection method.

Images