CN115102830B - Log reduction method, device, computer equipment and computer readable storage medium - Google Patents
Log reduction method, device, computer equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN115102830B CN115102830B CN202210731326.6A CN202210731326A CN115102830B CN 115102830 B CN115102830 B CN 115102830B CN 202210731326 A CN202210731326 A CN 202210731326A CN 115102830 B CN115102830 B CN 115102830B
- Authority
- CN
- China
- Prior art keywords
- preset
- log
- request message
- query
- storing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 230000009467 reduction Effects 0.000 title claims abstract description 41
- 230000004044 response Effects 0.000 claims abstract description 36
- 238000004364 calculation method Methods 0.000 claims abstract description 11
- 238000004590 computer program Methods 0.000 claims description 9
- 238000011835 investigation Methods 0.000 abstract description 8
- 238000013500 data storage Methods 0.000 abstract description 5
- 238000004519 manufacturing process Methods 0.000 abstract description 4
- 230000006835 compression Effects 0.000 description 5
- 238000007906 compression Methods 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 3
- 206010000060 Abdominal distension Diseases 0.000 description 2
- 208000024330 bloating Diseases 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000010295 mobile communication Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000003442 weekly effect Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/0246—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
- H04L41/0273—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols using web services for network management, e.g. simple object access protocol [SOAP]
- H04L41/0286—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols using web services for network management, e.g. simple object access protocol [SOAP] for search or classification or discovery of web services providing management functionalities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0604—Management of faults, events, alarms or notifications using filtering, e.g. reduction of information by using priority, element types, position or time
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a log reduction method, a log reduction device, computer equipment and a computer readable storage medium. The method comprises the following steps: according to a preset category model and a preset calculation and statistics formula, counting the number of query category requests sent by a received client in real time; based on the query request, generating a corresponding parameter entering request message and a response message, and sending the response message to the client; when the number of the query requests reaches a corresponding preset access amount threshold value within a preset time range, executing a preset operation, wherein the preset operation comprises the steps of storing a parameter entering request message corresponding to the query requests into a log table and not storing a response message. The invention can effectively reduce the data storage and the increment of the log list and the server log, avoid excessively bloated log list or server log caused by a large number of repeated logs, improve the running performance of the server, reduce invalid logs and greatly improve the effectiveness of log investigation and the response efficiency of production problem investigation.
Description
Technical Field
The present invention relates to the field of log file processing technologies, and in particular, to a log reduction method, a log reduction device, a computer device, and a computer readable storage medium.
Background
With the rapid development of the internet, the information data is accelerated obviously, and the data size required to be stored is larger and larger, so that a large amount of log information is generated to be recorded, and most of the records are repeated. In particular, in the financial industry, hundreds of millions of journal information are produced daily, growing at a rate of several tens of G. The conventional log recording system records all log information, and stores the log information into a log table and a server without modification, so that a large amount of data redundancy is caused, and the disk space of the server is quickly piled up by a large amount of log information, so that the running speed of the system and the access speed of the server are reduced, and the situation that the data writing of a database is slow due to too many log files is caused. Meanwhile, technicians are required to frequently clean and archive the logs, so that the history logs cannot be queried in real time.
At present, in order to solve the problem that the server cannot cause abnormal system operation because disk space is fully occupied by log information, a mode of reducing log writing standards is generally adopted, but the problem can cause serious information loss when a later related technician views a log file, and the integrity and accuracy of the log information are seriously affected.
Disclosure of Invention
In view of the above, the present invention aims to overcome the shortcomings in the prior art, and provide a log reduction method, a log reduction device, a computer device and a computer readable storage medium, which solve the problem that the access speed of a server is slow due to excessive log quantity.
The embodiment of the disclosure provides a log reduction method, a log reduction device, computer equipment and a computer readable storage medium.
In a first aspect, an embodiment of the present disclosure provides a log reduction method, applied to a server, where the method includes:
according to a preset category model and a preset calculation and statistics formula, counting the number of query category requests sent by a received client in real time;
generating a corresponding parameter entering request message and a response message based on the query type request, and sending the response message to the client;
and when the number of the query requests reaches a corresponding preset access amount threshold value within a preset time range, executing preset operation, wherein the preset operation comprises the steps of storing a parameter entering request message corresponding to the query requests into a log table and not storing the response message.
Further, the method further comprises:
storing the query type request in a preset interval time threshold to a server log;
and storing a parameter entering request message corresponding to the query type request exceeding a preset interval time threshold value into the log table.
Further, the method further comprises:
and when the size of the parameter entering request message reaches a preset threshold, compressing the parameter entering request message and storing the compressed parameter entering request message into the log table.
Further, the method further comprises:
comparing the content in the parameter entering request message with a preset invalid field;
if the fields which are the same as the preset invalid fields exist, deleting the fields which are the same as the preset invalid fields in the parameter entering request message.
Further, the method further comprises:
identifying the parameter entering request message by using a preset repeatability identification model;
if two request messages with the repetition degree reaching a preset repetition degree threshold exist, deleting one request message.
Further, the method further comprises:
judging whether a preset url address exists in the parameter entering request message stored in the log table or not;
if so, deleting the preset url address in the entry request message to be stored in the log table.
In a second aspect, in an embodiment of the present disclosure, there is provided a log reduction apparatus, including:
the statistics module is used for counting the number of query class requests sent by the received client in real time according to a preset category model and a preset calculation statistics formula;
the sending module is used for generating a corresponding parameter entering request message and a response message based on the query type request and sending the response message to the client;
and the first storage module is used for executing preset operation when the number of the query requests reaches a corresponding preset access amount threshold value within a preset time range, wherein the preset operation comprises the steps of storing a parameter entering request message corresponding to the query requests into a log table and not storing the response message.
Further, the apparatus further comprises:
the second storage module is used for storing the query type request in the preset interval time threshold to the server log;
and the third storage module is used for storing the parameter entering request message corresponding to the query type request exceeding the preset interval time threshold value into the log table.
In a third aspect, embodiments of the present disclosure provide a computer device, including a memory storing a computer program and a processor implementing the steps of the log reduction method of any one of the first aspects when the processor executes the computer program.
In a fourth aspect, in an embodiment of the present disclosure, there is provided a computer readable storage medium storing a computer program which, when executed by a processor, implements the steps of the log reduction method of any one of the first aspects.
Embodiments of the present invention have the following advantages:
according to the log reduction method provided by the embodiment of the application, the number of query type requests sent by the received client is counted in real time according to a preset category model and a preset calculation statistics formula; based on the query request, generating a corresponding parameter entering request message and a response message, and sending the response message to the client; and when the number of the query requests reaches a corresponding preset access amount threshold value within a preset time range, executing preset operation, wherein the preset operation comprises the steps of storing a parameter entering request message corresponding to the query requests into a log table and not storing the response message. The invention can effectively reduce the data storage and the increment of the log list and the server log, avoid excessively bloating the log list or the server log caused by a large number of repeated logs, improve the running performance of the server, reduce invalid logs, reduce the log length and greatly improve the validity of log investigation and the response efficiency of production problem investigation.
In order to make the above objects, features and advantages of the present invention more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 shows a flowchart of a log reduction method provided in an embodiment of the present application;
fig. 2 shows a schematic structural diagram of a log reduction device according to an embodiment of the present application;
fig. 3 shows a schematic hardware architecture of a computer device according to an embodiment of the present application.
Detailed Description
Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative only and are not to be construed as limiting the invention.
It will be understood that when an element is referred to as being "fixed to" another element, it can be directly on the other element or intervening elements may also be present. When an element is referred to as being "connected" to another element, it can be directly connected to the other element or intervening elements may also be present. In contrast, when an element is referred to as being "directly on" another element, there are no intervening elements present. The terms "vertical," "horizontal," "left," "right," and the like are used herein for illustrative purposes only.
In the present invention, unless explicitly specified and limited otherwise, the terms "mounted," "connected," "secured," and the like are to be construed broadly, and may be, for example, fixedly connected, detachably connected, or integrally formed; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communicated with the inside of two elements or the interaction relationship of the two elements. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art according to the specific circumstances.
Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. In the description of the present invention, the meaning of "a plurality" is two or more, unless explicitly defined otherwise.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the templates herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and/or" as used herein includes any and all combinations of one or more of the associated listed items.
Example 1
As shown in fig. 1, a flowchart of a log reduction method in an embodiment of the present application is shown, and the log reduction method provided in the embodiment of the present application is applied to a server, and the method includes the following steps:
specifically, according to a preset category model and a preset calculation statistical formula, the number of query class requests sent by a received client is counted in real time, and further, the number of query class requests at a daily level, the number of query class requests at a weekly level and the number of query class requests at a monthly level can be counted respectively. The hierarchical statistics can achieve the statistics of the number of the requests of the query class according to a certain time period, so that the log manager and the querier can manage and query conveniently.
specifically, based on the counted query requests, corresponding request-in messages and response messages are generated, and the response messages can be directly sent to the corresponding clients after the corresponding response messages are generated. The message body of the parameter entering request message comprises at least one log object, and the log object comprises corresponding data in the query type request. And the server side packs a plurality of log objects according to a preset parameter entering request message format to obtain a parameter entering request message.
In this embodiment, the header of the incoming request message includes a message length, a header identifier, a version number, a compression identifier, and the number of log objects. The message length of the incoming request message is the total byte number of the message (not including the length of the length value itself), that is, the number of bytes from the beginning of the header identification of the incoming request message to the end of the incoming request message, which occupies four bytes, and the maximum single incoming request length can be 4294967295 bytes by adopting a large-end mode. Head identification: the identification information of the header of the incoming request message may be a fixed character of one byte, for example, may be 'H' (0 x 72); version number: the current request message protocol version information occupies two bytes; compression identification: the message body used for marking the incoming request is compressed or not, and occupies one byte, the compression algorithm can adopt a deflate algorithm and the like, the compression can be represented by 'C' (0 x 67), and the compression can be represented by '0' (0 x 30); log object number: the number of log objects contained in the body of the reference request message occupies four bytes, and a large-end mode is adopted, so that a maximum of 4294967295 log objects can be placed in one log request message.
Further, the message body of the request message can include a plurality of log objects, the message body is a log object set, each log object can also include information such as the length of the log, the log serial number value and the like besides the content of the log, and the single log object content can be composed of three parts of the log object length, the log serial number and the log content, wherein the log object length is the total byte number (not including the length of the length value) of the current log object, occupies four bytes, and adopts a big end mode; the log sequence number represents the arrangement position of the current log object in the message body of the request message, occupies four bytes, adopts a big end mode, and has uniqueness in the log sequence number of each log object content in the same request message. The log sequence number may be from 1, with each increment of log sequence number thereafter incremented by 1; the log content is information in the query class request.
And 103, when the number of the query requests reaches a corresponding preset access amount threshold value within a preset time range, executing a preset operation, wherein the preset operation comprises storing a parameter entering request message corresponding to the query requests into a log table and not storing the response message.
Specifically, when the number of the query requests reaches a corresponding preset access amount threshold value within a preset time range, a preset operation is executed. If the number of the query class requests of the daily level reaches ten thousand times within 8 hours, the preset operation is executed, if the number of the query class requests of the weekly level reaches ten hundred thousand times within 48 hours, the preset operation is executed, and if the number of the query class requests of the monthly level reaches ten million times within 200 hours, the preset operation is executed. It can be appreciated that the above-mentioned preset time range and the preset access amount threshold may be set according to actual requirements, which is not limited in the embodiment of the present application.
Further, the preset operation includes storing the request message of the query corresponding to the query type request in a log table and not storing the response message. Because the request message is convenient for inquiring, only the request message is stored in the log table, and the response message is not stored, because the size of the data quantity stored in the log table can be effectively reduced.
In an alternative embodiment, the method may further include:
step 104, storing the query type request in a preset interval time threshold to a server log;
and 105, storing a request message of the query class corresponding to the query class request exceeding the preset interval time threshold into the log table.
Specifically, query requests of the same client within a preset interval time threshold are stored in a server log, wherein the server log is one or more log files automatically created and maintained by a server, and a parameter entering request message corresponding to the query requests exceeding the preset interval time threshold is stored in the log table. For example, a query class request of the same client in one hour is stored in a server log every minute, and a request-entering message corresponding to the query class request of the same client in more than one hour is stored in a log table every minute. The preset interval time threshold may be set according to actual requirements, which is not limited in the embodiment of the present application.
It can be understood that, one part of the query requests is stored in the log table, and the other part of the query requests is stored in the server log, so that the size of the data quantity stored in the log table can be effectively reduced, and the storage pressure of the log table is relieved.
In an alternative embodiment, the method may further include:
and 106, when the size of the parameter entering request message reaches a preset threshold, compressing the parameter entering request message and storing the compressed parameter entering request message into the log table.
In this embodiment, when the size of each incoming request message reaches a preset threshold, the incoming request message may be compressed to a preset size and then falls into a table. Specifically, if the size of the current request message for entering the reference is larger than a preset threshold, the current request message for entering the reference is truncated from the end so that the size of the current request message for entering the reference meets the preset threshold. The preset threshold may be set according to actual requirements, which is not limited in the embodiment of the present application.
It can be understood that the parameter entering request message is compressed first and then stored in the log table, so that the size of the data quantity stored in the log table can be effectively reduced, and the storage pressure of the log table can be relieved.
In an alternative embodiment, the method may further include:
step 107, comparing the content in the request message with a preset invalid field;
and step 108, if the fields which are the same as the preset invalid fields exist, deleting the fields which are the same as the preset invalid fields in the parameter entering request message.
Specifically, according to a preset invalid field, for example "= = = = = = = = = =" @ @ @ @ @ @ @ "# # # # -, or"%%%%% ", a large string of space occupying symbols is used, the content in the incoming request message is compared with the preset invalid field, and if the field identical to the preset invalid field exists, the field identical to the preset invalid field in the incoming request message is deleted. The preset invalid field may be set according to actual requirements, which is not limited in the embodiment of the present application.
It can be understood that deleting the invalid field in the entry request message can effectively remove invalid or unimportant content, and only the key and necessary content in the entry request message is reserved, so that the size of the data amount stored in the log table can be effectively reduced by storing the key and necessary content in the log table, and the storage pressure of the log table is relieved.
In an alternative embodiment, the method may further include:
step 109, identifying the parameter entering request message by using a preset repeatability identification model;
step 110, if there are two request messages with repetition degree reaching the preset repetition degree threshold, deleting one request message.
Specifically, the repetition degree identification can be performed on the incoming request messages by using a pre-trained repetition degree identification model, and if two incoming request messages with the repetition degree reaching a preset repetition degree threshold exist, one of the incoming request messages is deleted. For example, if it is detected that there are two incoming request messages with a repetition degree up to 90%, one of the incoming request messages is deleted. The preset repetition threshold may be set to 90%, 95%, 97%, or the like, and may be specifically set according to actual requirements, which is not limited in the embodiment of the present application.
It can be understood that deleting the request message with high repetition degree can effectively prevent the message with similar repetition degree from being stored in the log table, effectively reduce the size of the data quantity stored in the log table and relieve the storage pressure of the log table.
In an alternative embodiment, the method may further include:
step 111, judging whether a preset url address exists in the parameter entering request message stored in the log table;
step 112, if so, deleting the preset url address in the entry request message to be stored in the log table.
Specifically, according to preset url (uniform resource location, uniform resource locator) addresses, such as "/get,/fetch,/list,/query" and other keywords related to the request, it is determined whether the preset url addresses already stored in the entry request message of the log table exist, and if the preset url addresses already exist in the entry request message already stored in the log table, the preset url addresses to be stored in the entry request message of the log table are deleted. The preset url address may be set according to actual requirements, which is not limited in the embodiment of the present application.
It can be understood that, because the preset url addresses exist in the log table, all the addresses can be directly accessed when the addresses are accessed, and the preset url addresses to be stored in the log table entry request message are deleted, so that the stored address information can be effectively prevented from being repeatedly stored, the size of the data storage volume of the log table can be effectively reduced, and the storage pressure of the log table can be relieved.
Embodiments of the present invention have the following advantages:
according to the log reduction method provided by the embodiment of the application, the number of query type requests sent by the received client is counted in real time according to a preset category model and a preset calculation statistics formula; based on the query request, generating a corresponding parameter entering request message and a response message, and sending the response message to the client; and when the number of the query requests reaches a corresponding preset access amount threshold value within a preset time range, executing preset operation, wherein the preset operation comprises the steps of storing a parameter entering request message corresponding to the query requests into a log table and not storing the response message. The invention can effectively reduce the data storage and the increment of the log list and the server log, avoid excessively bloating the log list or the server log caused by a large number of repeated logs, improve the running performance of the server, reduce invalid logs, reduce the log length and greatly improve the validity of log investigation and the response efficiency of production problem investigation.
Example 2
As shown in fig. 2, a schematic structural diagram of a log reduction device 200 in an embodiment of the present application, where the log reduction device is applied to a server, the device includes:
the statistics module 201 is configured to combine a preset calculation statistics formula according to a preset category model, and count in real time the number of query category requests sent by the received client;
a sending module 202, configured to generate a corresponding request packet for entering a parameter and a response packet based on the query request, and send the response packet to the client;
and the storage module 203 is configured to execute a preset operation when the number of the query requests reaches a corresponding preset access amount threshold within a preset time range, where the preset operation includes storing a parameter entry request packet corresponding to the query request in a log table and not storing the response packet.
Optionally, the log reduction device may further include:
a first storage module 204, configured to store a query class request within a preset interval time threshold to a server log;
the second storage module 205 is configured to store a parameter entering request packet corresponding to a query type request that exceeds a preset interval time threshold in the log table.
Optionally, the log reduction device may further include:
and a third storage module 206, configured to compress the incoming request message and store the compressed incoming request message in the log table when the size of the incoming request message reaches a preset threshold.
Optionally, the log reduction device may further include:
a comparison module 207, configured to compare the content in the parameter entry request packet with a preset invalid field;
the first deleting module 208 is configured to delete the field identical to the preset invalid field in the incoming request message if the field identical to the preset invalid field exists.
Optionally, the log reduction device may further include:
the identifying module 209 is configured to identify the incoming request packet by using a preset repetition identification model;
the second deleting module 210 is configured to delete one of the incoming request messages if there are two incoming request messages with repetition degree reaching the preset repetition degree threshold.
Optionally, the log reduction device may further include:
a judging module 211, configured to judge whether a preset url address exists in the parameter entry request packet already stored in the log table;
the third deleting module 212 is configured to delete a preset url address in the entry request packet to be saved to the log table.
Embodiments of the present invention have the following advantages:
the log reduction device provided by the embodiment of the application can effectively reduce the data storage capacity and the increment of the log table and the server log, avoid excessively bloated log table or server log caused by a large number of repeated logs, improve the running performance of the server, reduce invalid logs, reduce the log length, and greatly improve the validity of log investigation and the response efficiency of production problem investigation.
Example 3
Fig. 3 shows a schematic hardware architecture of a computer device provided in the present application, where the computer device includes a memory and a processor, and the memory stores a computer program, and the processor implements the steps of the log reduction method described in embodiment 1 when executing the computer program.
In the present embodiment, the computer device 300 is a device capable of automatically performing numerical calculation and/or information processing in accordance with instructions set or stored in advance. For example, it may be a rack server, a blade server, a tower server, or a rack server (including an independent server or a server cluster composed of a plurality of servers), etc. As shown in fig. 3, computer device 300 includes at least, but is not limited to: memory 310, processor 320, and network interface 330 may be communicatively linked to each other by a system bus. Wherein:
memory 310 includes at least one type of computer-readable storage medium including flash memory, hard disk, multimedia card, card memory (e.g., SD or DX memory, etc.), random Access Memory (RAM), static Random Access Memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 may be an internal storage module of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 may also be an external storage device of the computer device 300, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card) or the like, which are provided on the computer device 300. Of course, memory 310 may also include both internal memory modules of computer device 300 and external memory devices. In this embodiment, the memory 310 is typically used to store an operating system and various types of application software installed on the computer device 300, such as program codes of a video playing method. In addition, the memory 310 may also be used to temporarily store various types of data that have been output or are to be output.
Processor 320 may be a central processing unit (Central Processing Unit, simply CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300, such as performing control and processing related to data interaction or communication with the computer device 300, and the like. In this embodiment, the processor 320 is configured to execute program codes or process data stored in the memory 310.
The network interface 330 may include a wireless network interface or a wired network interface, the network interface 330 typically being used to establish a communication link between the computer device 300 and other computer devices. For example, the network interface 330 is used to connect the computer device 300 to an external terminal through a network, establish a data transmission channel and a communication link between the computer device 300 and the external terminal, and the like. The network may be a wireless or wired network such as an Intranet (Intranet), the Internet (Internet), a global system for mobile communications (GlobalSystem of Mobile communication, abbreviated as GSM), wideband code division multiple access (Wideband Code DivisionMultiple Access, abbreviated as WCDMA), a 4G network, a 5G network, bluetooth (Bluetooth), wi-Fi, etc.
It should be noted that FIG. 3 only shows a computer device having components 310-330, but it should be understood that not all of the illustrated components are required to be implemented and that more or fewer components may be implemented instead.
In this embodiment, the log reduction method stored in the memory 310 may be further divided into one or more program modules and executed by one or more processors (the processor 320 in this embodiment) to complete the present invention.
Example 4
The present embodiment also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the log reduction method of the embodiment.
In this embodiment, the computer-readable storage medium includes a flash memory, a hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the computer readable storage medium may be an internal storage unit of a computer device, such as a hard disk or a memory of the computer device. In other embodiments, the computer readable storage medium may also be an external storage device of a computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card), etc. that are provided on the computer device. Of course, the computer-readable storage medium may also include both internal storage units of a computer device and external storage devices. In this embodiment, the computer-readable storage medium is typically used to store an operating system and various types of application software installed on a computer device. Furthermore, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or are to be output.
Any particular values in all examples shown and described herein are to be construed as merely illustrative and not a limitation, and thus other examples of exemplary embodiments may have different values.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.
The above examples merely represent a few embodiments of the present invention, which are described in more detail and are not to be construed as limiting the scope of the present invention. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the invention, which are all within the scope of the invention.
Claims (10)
1. A log reduction method, applied to a server, comprising:
according to a preset category model and a preset calculation and statistics formula, counting the number of query category requests sent by a received client in real time;
generating a corresponding parameter entering request message and a response message based on the query type request, and sending the response message to the client;
and when the number of the query requests reaches a corresponding preset access amount threshold value within a preset time range, executing preset operation, wherein the preset operation comprises the steps of storing a parameter entering request message corresponding to the query requests into a log table and not storing the response message.
2. The log reduction method of claim 1, wherein the method further comprises:
storing the query type request in a preset interval time threshold to a server log;
and storing a parameter entering request message corresponding to the query type request exceeding a preset interval time threshold value into the log table.
3. The log reduction method of claim 1, wherein the method further comprises:
and when the size of the parameter entering request message reaches a preset threshold, compressing the parameter entering request message and storing the compressed parameter entering request message into the log table.
4. The log reduction method of claim 1, wherein the method further comprises:
comparing the content in the parameter entering request message with a preset invalid field;
if the fields which are the same as the preset invalid fields exist, deleting the fields which are the same as the preset invalid fields in the parameter entering request message.
5. The log reduction method of claim 1, wherein the method further comprises:
identifying the parameter entering request message by using a preset repeatability identification model;
if two request messages with the repetition degree reaching a preset repetition degree threshold exist, deleting one request message.
6. The log reduction method of claim 1, wherein the method further comprises:
judging whether a preset url address exists in the parameter entering request message stored in the log table or not;
if so, deleting the preset url address in the entry request message to be stored in the log table.
7. A log reduction device, the device comprising:
the statistics module is used for counting the number of query class requests sent by the received client in real time according to a preset category model and a preset calculation statistics formula;
the sending module is used for generating a corresponding parameter entering request message and a response message based on the query type request and sending the response message to the client;
and the first storage module is used for executing preset operation when the number of the query requests reaches a corresponding preset access amount threshold value within a preset time range, wherein the preset operation comprises the steps of storing a parameter entering request message corresponding to the query requests into a log table and not storing the response message.
8. The log reduction device of claim 7, wherein the device further comprises:
the second storage module is used for storing the query type request in the preset interval time threshold to the server log;
and the third storage module is used for storing the parameter entering request message corresponding to the query type request exceeding the preset interval time threshold value into the log table.
9. A computer device comprising a memory storing a computer program and a processor implementing the steps of the log reduction method of any of claims 1-6 when the computer program is executed.
10. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program which, when executed by a processor, implements the steps of the log reduction method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210731326.6A CN115102830B (en) | 2022-06-24 | 2022-06-24 | Log reduction method, device, computer equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210731326.6A CN115102830B (en) | 2022-06-24 | 2022-06-24 | Log reduction method, device, computer equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115102830A CN115102830A (en) | 2022-09-23 |
| CN115102830B true CN115102830B (en) | 2023-07-14 |
Family
ID=83293657
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210731326.6A Active CN115102830B (en) | 2022-06-24 | 2022-06-24 | Log reduction method, device, computer equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115102830B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10210162B1 (en) * | 2010-03-29 | 2019-02-19 | Carbonite, Inc. | Log file management |
| CN112650753A (en) * | 2020-12-23 | 2021-04-13 | 平安普惠企业管理有限公司 | Log management method, device, system, equipment and readable storage medium |
| CN113497723A (en) * | 2020-03-20 | 2021-10-12 | 阿里巴巴集团控股有限公司 | Log processing method, log gateway and log processing system |
| CN113535654A (en) * | 2021-06-11 | 2021-10-22 | 安徽安恒数智信息技术有限公司 | Log processing method, system, electronic device and storage medium |
| CN113760849A (en) * | 2021-11-10 | 2021-12-07 | 深圳市明源云科技有限公司 | Log processing method, system, electronic device and computer readable storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10834113B2 (en) * | 2017-07-25 | 2020-11-10 | Netskope, Inc. | Compact logging of network traffic events |
-
2022
- 2022-06-24 CN CN202210731326.6A patent/CN115102830B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10210162B1 (en) * | 2010-03-29 | 2019-02-19 | Carbonite, Inc. | Log file management |
| CN113497723A (en) * | 2020-03-20 | 2021-10-12 | 阿里巴巴集团控股有限公司 | Log processing method, log gateway and log processing system |
| CN112650753A (en) * | 2020-12-23 | 2021-04-13 | 平安普惠企业管理有限公司 | Log management method, device, system, equipment and readable storage medium |
| CN113535654A (en) * | 2021-06-11 | 2021-10-22 | 安徽安恒数智信息技术有限公司 | Log processing method, system, electronic device and storage medium |
| CN113760849A (en) * | 2021-11-10 | 2021-12-07 | 深圳市明源云科技有限公司 | Log processing method, system, electronic device and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115102830A (en) | 2022-09-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10652265B2 (en) | Method and apparatus for network forensics compression and storage | |
| US9563531B2 (en) | Storage of mass data for monitoring | |
| CN102937926B (en) | Method and device for recovering deleted sqlite files on mobile terminal | |
| CN109815214B (en) | Database access method, system, device and storage medium | |
| CN111740868B (en) | Alarm data processing method and device and storage medium | |
| CN113472858B (en) | Buried point data processing method and device and electronic equipment | |
| CN110457255B (en) | Method, server and computer readable storage medium for archiving data | |
| CN113238912B (en) | Aggregation processing method for network security log data | |
| CN111309696A (en) | Log processing method and apparatus, electronic device, and readable medium | |
| CN110825598A (en) | Log real-time processing method and system | |
| CN111177094B (en) | Log data processing method and device, electronic equipment and storage medium | |
| EP4174675A1 (en) | On-board data storage method and system | |
| CN115102830B (en) | Log reduction method, device, computer equipment and computer readable storage medium | |
| CN118860449B (en) | DSP online upgrading processing method, system and medium based on CAN communication | |
| CN111818097B (en) | Traffic monitoring method and device based on behaviors | |
| CN113934747A (en) | Data processing method, device, equipment and storage medium | |
| CN117880389A (en) | Data monitoring method, device, equipment and medium for HTTP2 protocol | |
| CN117873393A (en) | Time sequence data storage method, device, equipment and storage medium | |
| CN113316125B (en) | Monitoring method, distributed vehicle-mounted system, vehicle and readable storage medium | |
| CN115470186A (en) | Data slicing method, device and system | |
| CN107589907A (en) | Data processing method, electronic equipment and computer-readable recording medium | |
| CN116015904A (en) | Blocking method and device for malicious scanning traffic | |
| CN117294578B (en) | Communication method, system, computer equipment and storage medium | |
| CN114048957A (en) | Risk score determination method, device, equipment and readable storage medium | |
| CN112486976A (en) | Data processing method, device, network equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |