CN119250025A

CN119250025A - Method for converting SNMP data into JSON data based on target requirements

Info

Publication number: CN119250025A
Application number: CN202411234680.3A
Authority: CN
Inventors: 余剑; 杨维敏; 赵庆亚; 马立珂; 王子骏
Original assignee: Anhui Dingjia Computer Technology Co ltd
Current assignee: Anhui Dingjia Computer Technology Co ltd
Priority date: 2024-09-04
Filing date: 2024-09-04
Publication date: 2025-01-03

Abstract

The present application relates to a method, device, computer equipment, computer-readable storage medium and computer program product for converting SNMP data into JSON data in accordance with target requirements, which can be used in the field of computer technology. The method comprises: calculating the word frequency information, inverse document frequency information and information gain information of SNMP data according to preset keywords; calculating the weight information of SNMP data according to the word frequency information, inverse document frequency information and information gain information; classifying SNMP data according to the weight information to obtain the classification result of SNMP data; storing the SNMP data in a preset storage unit according to the classification result; reading the target SNMP data from the SNMP data stored in the preset storage unit according to the received target requirements; converting the target SNMP data into JSON data according to the object identifier in the target SNMP data. The use of this method can improve the efficiency of data conversion.

Description

Method for converting SNMP data facing target demand into JSON data

Technical Field

The present application relates to the field of computer technology, and in particular, to a method, an apparatus, a computer device, a computer readable storage medium, and a computer program product for converting SNMP data oriented to a target requirement into JSON data.

Background

With the development of network management technology, simple Network Management Protocol (SNMP) has become a widely used standard for network monitoring and management. However, with the rise of internet of things and big data technologies, higher demands are put on the flexibility and readability of data formats. Therefore, how to efficiently convert SNMP data into JSON format data that is easier to process and analyze has become an important research direction.

The traditional technology generally converts SNMP data into JSON data by manually writing an analysis script, but the data conversion by the method requires more manual processing time, so that the data conversion efficiency is lower.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a method, apparatus, computer device, computer readable storage medium, and computer program product for converting SNMP data for object-oriented requirements to JSON data that can improve the efficiency of data conversion.

In a first aspect, the present application provides a method for converting SNMP data oriented to target requirements into JSON data. The method comprises the following steps:

according to the preset keywords, calculating word frequency information, inverse document frequency information and information gain information of SNMP data;

calculating weight information of the SNMP data according to the word frequency information, the inverse document frequency information and the information gain information;

Classifying the SNMP data according to the weight information to obtain a classification result of the SNMP data;

According to the classification result, classifying and storing the SNMP data into a preset storage unit;

Reading target SNMP data from the SNMP data stored in the preset storage unit according to the received target demand;

And converting the target SNMP data into JSON data according to the object identifier in the target SNMP data.

In one embodiment, the classifying the SNMP data according to the weight information to obtain a classification result of the SNMP data includes:

Classifying SNMP data with the weight information larger than a preset threshold value as hot data, and classifying SNMP data with the weight information smaller than or equal to the preset threshold value as cold data;

and determining classification results of the SNMP data according to the hot data and the cold data.

In one embodiment, the preset storage unit at least includes a preset cache and a preset database;

The classifying the SNMP data into a preset storage unit according to the classification result includes:

and storing the hot data into the preset cache, and storing the cold data into the preset database.

In one embodiment, the reading the target SNMP data from the SNMP data stored in the preset storage unit according to the received target requirement includes:

Under the condition that SNMP data meeting the target requirement is queried from the preset cache, reading the target SNMP data from the preset cache;

And under the condition that the SNMP data meeting the target requirement cannot be queried from the preset cache, reading the target SNMP data from the preset database.

In one embodiment, the calculating word frequency information, inverse document frequency information and information gain information of SNMP data according to a preset keyword includes:

setting a sliding window of the SNMP data;

In the sliding window, calculating word frequency of the preset keywords, inverse document frequency of the preset keywords and information gain of the preset keywords in the SNMP data;

And taking the word frequency of the preset keyword as the word frequency information, taking the inverse document frequency of the preset keyword as the inverse document frequency information, and taking the information gain of the preset keyword as the information gain information.

In one embodiment, before calculating word frequency information, inverse document frequency information and information gain information of SNMP data according to a preset keyword, the method further includes:

receiving SNMP messages;

Storing the SNMP message into a message queue;

preprocessing the SNMP message in the message queue to obtain preprocessed SNMP message as the SNMP data, wherein the preprocessing at least comprises duplication removal processing, format unification processing and data cleaning processing.

In a second aspect, the present application further provides a device for converting SNMP data facing to an object requirement into JSON data. The device comprises:

The first calculation module is used for calculating word frequency information, inverse document frequency information and information gain information of the SNMP data according to preset keywords;

The second calculation module is used for calculating weight information of the SNMP data according to the word frequency information, the inverse document frequency information and the information gain information;

the data classification module is used for classifying the SNMP data according to the weight information to obtain a classification result of the SNMP data;

The data storage module is used for storing the SNMP data in a preset storage unit in a classified manner according to the classification result;

the data reading module is used for reading target SNMP data from the SNMP data stored in the preset storage unit according to the received target demand;

and the data conversion module is used for converting the target SNMP data into JSON data according to the object identifier in the target SNMP data.

In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor which when executing the computer program performs the steps of:

In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:

In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of:

The method, the device, the computer equipment, the computer readable storage medium and the computer program product for converting the SNMP data facing the target requirement into the JSON data are used for calculating word frequency information, inverse document frequency information and information gain information of the SNMP data according to preset keywords, calculating weight information of the SNMP data according to the word frequency information, the inverse document frequency information and the information gain information, classifying the SNMP data according to the weight information to obtain a classification result of the SNMP data, classifying and storing the SNMP data into a preset storage unit according to the classification result, reading target SNMP data from the SNMP data stored in the preset storage unit according to the received target requirement, and converting the target SNMP data into the JSON data according to an object identifier in the target SNMP data. The method obtains weight information by calculating word frequency, inverse document frequency and information gain of SNMP data, facilitates identification of important data, facilitates optimization of data access by classifying and storing the SNMP data, facilitates reduction of unnecessary data processing by reading related SNMP data according to target requirements, and facilitates simplification of data conversion process by converting the SNMP data into JSON format by using an object identifier. The scheme can effectively reduce the processed data quantity, optimize data access and simplify data conversion steps, thereby being beneficial to improving the data conversion efficiency and enabling the system to respond to different target demands more quickly.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related art, the drawings that are required to be used in the embodiments or the related technical descriptions will be briefly described, and it is apparent that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to the drawings without inventive effort for those skilled in the art.

FIG. 1 is a flow diagram of a method for converting target-demand-oriented SNMP data into JSON data in one embodiment;

FIG. 2 is a schematic diagram of a backup storage system in a clustered environment in one embodiment;

FIG. 3 is a system architecture diagram of JSON message generation in one embodiment;

FIG. 4 is a flow chart of data cold and hot classification in one embodiment;

FIG. 5 is a flow diagram of computing information weights in one embodiment;

FIG. 6 is a flow diagram of JSON data generation in one embodiment;

FIG. 7 is a block diagram of an apparatus for converting target-demand-oriented SNMP data into JSON data in one embodiment;

fig. 8 is an internal structural diagram of a computer device in one embodiment.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.

It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are both information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data are required to meet the related regulations.

In an exemplary embodiment, as shown in fig. 1, a method for converting SNMP data facing a target requirement into JSON data is provided, and this embodiment is illustrated by applying the method to a terminal, and it will be understood that the method may also be applied to a server, and may also be applied to a system including a terminal and a server, and implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and the like; the server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud computing service. In this embodiment, the method includes the steps of:

Step S101, word frequency information, inverse document frequency information and information gain information of SNMP data are calculated according to preset keywords.

Step S102, calculating weight information of the SNMP data according to word frequency information, inverse document frequency information and information gain information.

Step S103, classifying the SNMP data according to the weight information to obtain a classification result of the SNMP data.

Step S104, classifying and storing the SNMP data into a preset storage unit according to the classification result.

Step S105, reading target SNMP data from SNMP data stored in a preset storage unit according to the received target requirement.

And step S106, converting the target SNMP data into JSON data according to the object identifier in the target SNMP data.

The SNMP (Simple Network Management Protocol ) data may be network device or system status information collected by the simple network management protocol, for example, SNMP data may be CPU (central processing unit) usage of a router, port status of a switch, or memory occupancy of a server.

The preset keyword may be a predefined word or phrase for screening and evaluating the importance of SNMP data, for example, the preset keyword may be a term related to network management such as "CPU", "memory", "error" or "traffic".

The word frequency information may be statistics of the frequency of occurrence of a preset keyword in SNMP data, for example, the word frequency information may be the number of times that a preset keyword occurs in a set of SNMP messages.

The inverse document frequency information may be an index for measuring importance of a preset keyword in the whole SNMP data set, for example, the inverse document frequency information may be an inverse of how widely a certain preset keyword appears in all SNMP messages.

The information gain information may be an index for measuring the contribution degree of the preset keyword to SNMP data classification, for example, the information gain information may be the variation of entropy values of SNMP data sets before and after using a certain preset keyword.

The weight information may be an SNMP data importance index calculated by integrating word frequency information, inverse document frequency information and information gain information, for example, the weight information may be a weighted sum of the three information.

The classification result may be a result of classifying the SNMP data into hot data and cold data according to the weight information, for example, the classification result may be that the SNMP data with the weight information higher than a preset threshold value is classified into hot data and the SNMP data with the weight information lower than the preset threshold value is classified into cold data.

The preset storage unit may be a storage device or a storage area for storing different kinds of SNMP data, for example, the preset storage unit may be a cache for storing hot data and a database for storing cold data.

The target requirement may be a query or a processing request of a user or a system for specific SNMP data, for example, the target requirement may be a CPU usage rate of a latest hour for querying a certain server.

The target SNMP data may be SNMP data meeting conditions read from a preset storage unit according to a target requirement, for example, the target SNMP data may be CPU usage data of a certain server for the last hour.

Wherein the object identifier may be a code for uniquely identifying a specific information item in the SNMP data.

Wherein, JSON (JavaScript Object Notation, javaScript object notation, a lightweight data exchange format) data can be JSON format data.

Optionally, the terminal receives the SNMP data and stores the SNMP data in a message queue, and then performs preprocessing on the SNMP data in the message queue, including deduplication, format unification, and data cleaning. The terminal sets a sliding window, and word frequency information, inverse document frequency information and information gain information of preset keywords are calculated in the sliding window. And the terminal calculates weight information of the SNMP data according to the calculated word frequency information, the inverse document frequency information and the information gain information. The terminal sets a weight threshold, classifies SNMP data with weight information larger than the weight threshold as hot data, and classifies SNMP data with weight information smaller than or equal to the weight threshold as cold data. The terminal stores the hot data in the cache and the cold data in the database. When receiving the target demand, the terminal preferentially reads SNMP data meeting the target demand from the cache, and if the SNMP data in the cache does not meet the target demand, reads the SNMP data meeting the target demand from the database. The terminal formats the read target SNMP data, and then converts the target SNMP data into JSON data according to the object identifier in the target SNMP data.

According to the method for converting the SNMP data facing the target requirement into the JSON data, word frequency information, inverse document frequency information and information gain information of the SNMP data are calculated according to preset keywords, weight information of the SNMP data is calculated according to the word frequency information, the inverse document frequency information and the information gain information, classification is carried out on the SNMP data according to the weight information to obtain classification results of the SNMP data, the SNMP data are classified and stored in a preset storage unit according to the classification results, the target SNMP data are read from the SNMP data stored in the preset storage unit according to the received target requirement, and the target SNMP data are converted into the JSON data according to an object identifier in the target SNMP data. The method obtains weight information by calculating word frequency, inverse document frequency and information gain of SNMP data, facilitates identification of important data, facilitates optimization of data access by classifying and storing the SNMP data, facilitates reduction of unnecessary data processing by reading related SNMP data according to target requirements, and facilitates simplification of data conversion process by converting the SNMP data into JSON format by using an object identifier. The scheme can effectively reduce the processed data quantity, optimize data access and simplify data conversion steps, thereby being beneficial to improving the data conversion efficiency and enabling the system to respond to different target demands more quickly.

In an exemplary embodiment, classifying the SNMP data according to the weight information to obtain a classification result of the SNMP data, specifically comprising classifying the SNMP data with the weight information larger than a preset threshold value as hot data and classifying the SNMP data with the weight information smaller than or equal to the preset threshold value as cold data, and determining the classification result of the SNMP data according to the hot data and the cold data.

The preset threshold may be a weight threshold for distinguishing hot data from cold data, for example, the preset threshold may be 0.5 or a value dynamically adjusted by the system according to the current resource status.

The hot data may be SNMP data with weight information higher than a preset threshold, which indicates that the data has higher importance or access frequency, for example, the hot data may be SNMP data with CPU usage rate exceeding 80% in the last 10 minutes.

The cold data may be SNMP data with weight information lower than or equal to a preset threshold, which indicates that the data is relatively less important or has a low access frequency, for example, the cold data may be SNMP data of a normal operation state of the network device before one week.

The classification result of the SNMP data may be a data set obtained by dividing the SNMP data into hot data and cold data.

Optionally, the terminal sets a preset threshold, where the preset threshold may be a fixed value or a value dynamically adjusted according to the system resource status. The terminal traverses weight information of all SNMP data, marks SNMP data with weight information larger than a preset threshold value as hot data, and marks SNMP data with weight information smaller than or equal to the preset threshold value as cold data. The terminal creates two lists for storing hot data and cold data, respectively. The terminal adds SNMP data labeled hot data to the hot data list and SNMP data labeled cold data to the cold data list. The terminal combines the hot data list and the cold data list into a data structure as a classification result of the SNMP data.

According to the technical scheme provided by the embodiment, the SNMP data are classified into the hot data and the cold data according to the weight information, so that the importance and the access frequency of the data can be distinguished, the data storage strategy can be optimized, and the overall system performance and the response speed can be improved.

In an exemplary embodiment, the preset storage unit at least comprises a preset cache and a preset database, and the SNMP data is classified and stored in the preset storage unit according to the classification result, and specifically comprises the following steps of storing hot data in the preset cache and storing cold data in the preset database.

The preset storage unit may be a set of different types of storage devices or storage areas for storing SNMP data, and for example, the preset storage unit may include a cache (preset cache) and a persistent storage database (preset database).

The preset cache can be a high-speed access storage area for storing hot data, and has the characteristics of high reading and writing speed and relatively small capacity.

The preset database can be a persistent storage system for storing cold data, and has the characteristics of large capacity and relatively low access speed.

Optionally, the terminal configures a preset storage unit, including a preset cache and a preset database. The terminal sets a preset cache as a high-speed memory storage system for storing hot data, and sets a preset database as a persistent storage system for storing cold data. The terminal acquires a classification result of SNMP data, extracts hot data and cold data in the SNMP data, stores each piece of hot data into a preset cache, and stores each piece of cold data into a preset database.

According to the technical scheme provided by the embodiment, hot data are stored in the preset cache, and cold data are stored in the preset database, so that layered management of data storage is facilitated, and data access efficiency is improved. The storage strategy can rapidly respond to frequently accessed hot data requests and ensure long-term storage of cold data, thereby being beneficial to optimizing system resource utilization and improving overall performance.

In an exemplary embodiment, the target SNMP data is read from SNMP data stored in a preset storage unit according to the received target requirement, and specifically comprises the steps of reading the target SNMP data from a preset cache when SNMP data meeting the target requirement is queried from the preset cache, and reading the target SNMP data from a preset database when SNMP data meeting the target requirement is not queried from the preset cache.

Optionally, the terminal receives the target requirement and analyzes the query condition in the target requirement. The terminal firstly accesses a preset cache, and searches SNMP data meeting the conditions in the preset cache by using the analyzed query conditions. If the SNMP data meeting the conditions are found in the preset cache, the terminal directly reads the data from the preset cache as target SNMP data. If no conditional SNMP data is found in the preset cache, the terminal accesses the preset database, searches the preset database for the conditional SNMP data by using the same query condition, and reads the data from the preset database as target SNMP data.

According to the technical scheme, the hierarchical query strategy that the preset cache is queried first and then the preset database is queried is beneficial to preferentially utilizing hot data in the cache, so that the data retrieval efficiency is improved, and when the cache is not hit, the preset database is accessed again, so that the integrity of the data is guaranteed.

In an exemplary embodiment, word frequency information, inverse document frequency information and information gain information of SNMP data are calculated according to preset keywords, and specifically the method comprises the steps of setting a sliding window of the SNMP data, calculating word frequency of the preset keywords, inverse document frequency of the preset keywords and information gain of the preset keywords in the SNMP data in the sliding window, taking the word frequency of the preset keywords as word frequency information, taking the inverse document frequency of the preset keywords as inverse document frequency information and taking the information gain of the preset keywords as information gain information.

The sliding window may be a dynamic SNMP data analysis range, and is used for defining a data set for calculating word frequency, inverse document frequency and information gain, for example, the sliding window may be the latest 100 pieces of SNMP data or the latest SNMP data within 1 hour.

The word frequency may be a frequency of occurrence of a preset keyword in SNMP data within a sliding window.

The inverse document frequency may be an index that measures the importance of a preset keyword in the whole SNMP data set, for example, the inverse document frequency may be a logarithmic value of the total number of SNMP data divided by the number of SNMP data containing the preset keyword.

The information gain may be an index for measuring the contribution degree of the preset keywords to the SNMP data classification, for example, the information gain may be a difference between the entropy of the SNMP data set and the uncertainty of the keywords in the SNMP data set.

Optionally, the terminal sets a sliding window of the SNMP data, and determines a window size and a sliding step size. The terminal calculates word frequency of preset keywords in SNMP data in a sliding window, takes the word frequency of the preset keywords as word frequency information, calculates inverse document frequency of the preset keywords in the SNMP data, takes the inverse document frequency of the preset keywords as inverse document frequency information, calculates information gain of the preset keywords in the SNMP data, and takes the information gain of the preset keywords as information gain information.

According to the technical scheme provided by the embodiment, the sliding window is set, the word frequency, the inverse document frequency and the information gain of the preset keywords are calculated, so that the importance and the distribution characteristics of the key information in the SNMP data can be dynamically evaluated, and the SNMP data can be classified more accurately.

In an exemplary embodiment, before calculating word frequency information, inverse document frequency information and information gain information of the SNMP data according to the preset keywords, the method further comprises the steps of receiving SNMP messages, storing the SNMP messages in a message queue, preprocessing the SNMP messages in the message queue to obtain preprocessed SNMP messages as SNMP data, wherein the preprocessing at least comprises duplication removal processing, format unification processing and data cleaning processing.

The SNMP message may be a data packet sent by the network device through a Simple Network Management Protocol (SNMP), and contains device status, performance, or configuration information, for example, the SNMP message may be information such as a CPU usage rate, a memory occupancy rate, or a network interface status sent by the router.

The message queue may be a data structure for storing and managing SNMP messages, for example, the message queue may be a distributed message system implemented using rubbi mq (open source message broker software) or Kafka (distributed stream processing platform).

The preprocessing may be a process of performing preliminary processing on the original SNMP message, so as to improve efficiency and accuracy of subsequent analysis.

Where the deduplication process may be a process of deleting duplicate SNMP messages in a message queue, for example, the deduplication process may be to identify and delete duplicate messages based on their unique identifiers.

The format unification process may be a process of converting SNMP messages of different sources or formats into a unified format.

The data cleaning process may be a process of removing invalid, erroneous or irrelevant data in the SNMP message.

Optionally, the terminal receives SNMP messages from various network devices through a network interface. The terminal stores the received SNMP message into a pre-configured message queue for subsequent processing. The terminal reads SNMP messages from the message queue and preprocesses the messages. The preprocessing process comprises the steps of de-duplication processing, namely deleting repeated SNMP messages, format unification processing, data cleaning processing and invalid or irrelevant data removal, wherein SNMP messages in different formats are converted into unified formats. The terminal takes the preprocessed SNMP message as SNMP data.

According to the technical scheme provided by the embodiment, through receiving SNMP information and storing the SNMP information into the information queue, and then carrying out preprocessing of duplication removal, format unification and data cleaning, the data quality and consistency are improved, and the accuracy of subsequent word frequency, inverse document frequency and information gain calculation is improved.

The method for converting the SNMP data facing the target requirement into the JSON data provided by the application is described below by using an application example, and the application example is applied to a terminal for illustration by using the method.

Networks in clustered environments are typically composed of many devices including, but not limited to, routers, hubs, switches, printers, scans, servers, or other network devices. These devices typically use both SNMP and JSON protocols for information exchange. SNMP is a protocol widely used for network management that allows network devices (e.g., routers, switches, servers, etc.) to report their status and performance data to a management system. JSON is a lightweight data exchange format that is easy for a user to read and write, and also convenient for machine parsing and generation.

As the variety and number of network devices increases, managing these devices becomes more and more complex. SNMP provides a unified interface as a standard protocol, but its data format is complex and not easy to use directly. SNMP data is usually represented in the form of OID (object identifier) and MIB (management information base), which are not intuitive and friendly for users and modern applications, whereas JSON format has a high degree of data structuring, is easy to parse and process, and is suitable for Web applications and modern systems.

In network management, real-time monitoring and fast response are critical requirements. The SNMP data is converted into the JSON format, so that the high-efficiency analysis and processing capacity of the JSON can be utilized, and the efficiency of data exchange is improved. Therefore, it is important to realize a method capable of efficiently performing information interaction between SNMP and JSON. The network equipment management system not only can improve the efficiency of network equipment management, but also can improve the expandability and flexibility of a network system, thereby being better suitable for complex and changeable network environments.

The application example specifically comprises the following key characteristics:

(1) And calculating the weight of the SNMP data based on the weight analysis of the keywords, and evaluating the complexity and information quantity of the data. And screening data with high information content according to TF-IDF (word frequency-inverse document frequency) and data information gain, and ensuring that the generated JSON message has high correlation and practicability.

(2) And combining and generating the screened SNMP data to generate the JSON format message meeting the target requirement. Flexible configuration options are provided that allow the user to customize keywords and weight thresholds.

Referring to fig. 2, fig. 2 is a schematic structural diagram of a backup storage system in a cluster environment (including a backup service center 101a, a data center 101b for backup metadata, a backup storage center 101c, a network management system 100, a network connector 102a, a network 103, a network connector 102b, a server center 104a, a virtualization center 104b, a storage center 104c, and a client cluster environment). The network environment may include deployment scenarios of devices/clients, network Management Systems (NMSs) and backup disaster recovery systems in a clustered environment. In general, one or more network devices 102a, 102b are now included in a clustered network environment in connection with one or more servers 104a, 104b, and 104c. Generally, the servers 104 a-104 c comprise various cloud services such as computing services, virtualization services, storage services and the like of enterprises, and the business servers construct productivity environments of the enterprises and are connected through 102 a-102 b network equipment to realize management, monitoring, data backup and disaster recovery.

The system may include a plurality of logically grouped servers 106. The logical group of servers may be referred to as a clustered environment of clients. The environment includes a plurality of server clusters 104 a-104 c, each of which represents cloud services of one or more enterprises.

Server 104a may be referred to as a file server, an application server, a web server, a proxy server, or a gateway server. The server 104a may have the capability to act as an application server or as a primary application server. Server 104a may include an active directory. The server 104a has the capability to act as a client node seeking access to applications on the server and as an application server providing access to hosted applications for other clients 104a-104n (n represents a sequence number). Server 104b provides enterprise virtualized cloud services including computing, container, and desktop applications, with installed virtualized services of the server being serviced by virtualization vendors. An enterprise may use multiple products of virtualization for multiple vendors at the same time. The enterprise uses a storage center 104c to store and manage important service data, and these storage centers use protocols such as SAN (storage area network), DAS (direct attached storage), and NFS (network file system) to communicate data. The storage center 104c provides storage services including structured data, semi-structured data, unstructured objects, and the like.

The backup disaster recovery servers 101a, 101b, 101c provide complete data protection functions for the user's cluster environment. Through the working mode with the agent, the data protection agent (agent program) deployment and the enterprise cloud services 104 a-104 c provide data monitoring and protection services under the same physical host or network environment. And providing data monitoring and protecting services through interfaces of related applications on the enterprise cloud services 104 a-104 c in a proxy-free mode.

The topology of network devices 102a and/or 102b can be a bus, star, or ring network topology and connected by network 103, including any of a point-to-point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communications network, a computer network, an ATM (asynchronous transfer mode) network, a SONET (synchronous optical network) network, an SDH (synchronous digital hierarchy) network, a wireless network, and a wired network. The network 103 may also include wireless links such as infrared channels or satellite bands.

The Network Management System (NMS) system 100, the backup disaster recovery systems 101 a-101 c, and the servers 104 a-104 c of the enterprise are respectively located in different networks 102a and 102b, but may be local area networks, metropolitan area networks, or wide area networks when actually deployed. In an actual deployment, network 102a may be a private network and network 102b may be a public network, whereas in an actual deployment, network 102b may be a private network and network 102a may be a public network. In an actual deployment, both networks 102a and 102b may be private networks, such as an enterprise's internal private network. In actual deployment, the NMS and the backup service center may be located in some management center of the enterprise, communicating with a cloud server 104 of the company data center or third party through a WAN (wide area network), e.g., 103, connection through a network 102 b.

For more flexible deployment and business expansion, cross-domain capacity expansion services are supported. The network WAN103 may also include multiple interface units or gateways to provide for more flexible and convenient deployment and management. For example, a branch office of an enterprise may deploy device storage center 106 at the branch office and network management system 100 at the management center.

In order to achieve maximum performance management, multiple network optimization devices may be used for data and network balancing and security considerations, and devices such as architecture network optimization, VPN (virtual private network), or security may be deployed for accelerating, optimizing, or otherwise improving the performance, operation, or quality of service of any type and form of network traffic (e.g., traffic to and/or from WAN connections), such as 105a and 105b. As an example, an enterprise may deploy WAN optimization device 105b at a service management center and VPN or firewall device 105a at a data center for data security. Device 105a may be located on network 102 b. The device 105a may be located on the network WLAN 103.

The device 105 provides any type and form of WAN optimization or acceleration device, referred to as a WAN optimization controller. The device 105 is a routing device product. The devices 105 may include any FC (fibre channel) acceleration device platform, or any WAN related devices and/or software manufactured, such as wide area network application service software and network modules and wide area network engine devices.

Fig. 3 is a system architecture diagram for JSON message generation, including JSON translator, key-value database service, cache service, message queue, data processing, user interface, automation operation and maintenance system, network management system, backup system, socket (transmission control protocol), SNMP agent. The system may consist of a message queue, a caching mechanism, a K-V (key value) database, a data processing service, a user interface.

SNMP agents are typically provided by a device vendor or monitoring service provider running on a network device or software system and are responsible for collecting and reporting the current state information of the device or system. It supports messages of the SNMP protocol, allowing the management system to obtain device information, modify configuration, and receive alarms.

In some cases, the management system will proactively deploy SNMP agents on the target device or platform to better monitor and manage these devices and services, ensuring their proper functioning and timely response to failures. For example, network devices, such as routers, switches, firewalls, and the like, deploy SNMP agents on the network devices to monitor network performance and faults. And the server and the host deploy SNMP agents on the server and the host to monitor the use condition (such as CPU, memory and magnetic disk) and the running state of the system resources. Application program-an SNMP agent is deployed on a critical application program to monitor the performance and health status of the application program. And the virtualization platform is used for deploying an SNMP agent on the virtualization platform so as to monitor the states of the virtual machine and the virtual network. And the Internet of things equipment is provided with an SNMP agent to monitor the running state and the environmental parameters of the equipment.

After receiving SNMP message, socket module stores message into message queue so as to provide basis for subsequent data processing and storage. Here the management queue may be implemented using either rabitmq or Kafka for passing messages between components, decoupling producers and consumers, providing an asynchronous communication mechanism.

After the message queue, the hot SNMP data is stored in the cache, so that temporary data can be quickly accessed and stored in the message processing process, the direct access pressure to the database is reduced, and the overall system performance is improved. The caching mechanism can adopt memory data for storing frequently accessed data, reduces the query times of a database, accelerates data reading and improves the response speed of the system.

The limited memory resources are generally limited, and the caching mechanism can relieve the pressure of the database, but the final data needs to be stored in the database for long-term use. In order to avoid the influence on the generation environment, after a message queue and a caching mechanism are carried out on cold data, the data is stored in a K-V unstructured database such as MongoDB (document database) and the like for persistence processing, so that the persistence and the queriability of the data are ensured.

The data processing service performs data processing and analysis based on the message queues, caches and database stores. The method needs to acquire data from a message queue, can utilize a caching mechanism to accelerate processing, determine the classification of cold and hot degrees of the data, keep hot data in a cache and store cold data in a database. Meanwhile, the data processing service may further process and analyze the data according to the requirements of the purpose, which may be requirements of keywords, formats, etc.

Fig. 4 is a flow chart of data cold and hot classification, which describes how to perform data classification by entropy analysis through keywords of a target client, including message queuing, SNMP data collection, preprocessing, deduplication, formatting, cleansing, calculation of weights, data classification, SNMP message active acquisition (optional), SNMP agent, data storage, hot data, cold data, caching, and persistence storage. The flow steps are mainly described as follows:

1. Data collection, i.e. acquiring an SNMP message which is not processed through a message queue (such as RabbitMQ or Kafka).

2. And data preprocessing, namely preprocessing and filtering the collected SNMP message according to the OID (object identification) message, thereby reducing the processing load and improving the data processing efficiency. Including removing the same SNMP message based on the deduplication of the message or the OID of the device, converting the SNMP message into a unified format, e.g., unifying time stamping for subsequent data processing analysis, cleaning and removing irrelevant fields or entire messages in the message, e.g., status of system timing reporting, but not of interest to the target management service, and optionally removing some fields or messages. Since this is a lossy data process, which steps are performed can be selected according to specific target requirements. For example, if the goal is to monitor device status, messages related to device status may be emphasized to preserve, and other unrelated messages removed.

3. And (3) weight calculation, namely respectively calculating entropy values of each SNMP message according to the keywords of the target management service so as to measure the information quantity and uncertainty of the SNMP message. The higher the entropy value, the more information the message contains, and the higher the uncertainty, the lower the entropy value, the less the amount of information the message is, and the less the uncertainty.

4. The SNMP information is actively acquired, when the condition of larger entropy value occurs, such as monitoring abnormality of task failure, system restarting and the like, the system can actively inquire corresponding or related equipment according to the source of SNMP in the current information, such as CPU occupancy rate, memory utilization rate, disk space condition and the like, so as to acquire more related information. This step is optional in practical use, as it requires equipment or service platform support.

5. Data storage and processing, the setting of which is dynamically adjustable, typically according to memory size, based on entropy values and predefined thresholds, such that data is classified into hot data with a larger entropy value and cold data with a smaller entropy value. Hot data is stored in a cache (e.g., redis, open-source memory data structure storage system) for fast access, and cold data is stored in a NoSQL (non-relational database management system) database (e.g., mongoDB) for long-term storage and querying.

Fig. 5 is a schematic flow chart of information weight calculation, including inputting a/group of SNMP information, sliding SNMP message window (counting only the most recent N messages or messages within a fixed time window), calculating word frequency, calculating frequency of inverse document, calculating information gain, calculating information weight. The flow steps are mainly described as follows:

1. the input of a/group SNMP message is retrieved from the cache. One or more SNMP messages are read from the cache as input data that has been pre-processed.

2. The SNMP message window is slid and the window size is set to the number of messages or a fixed length of time. The sliding window is used to divide the message stream, and the size of the window determines the cost of space and time of the algorithm flow.

3. Calculating word frequency TF:

。

4. Calculating the frequency IDF of the inverse document:

。

N represents the number of all messages and N _i represents the number of messages containing key.

5. For all keys TF-idf=tf×idf within each message.

6. Information gain IG (t) =h (D) -H (d|key) is calculated.

Representing the entropy of the dataset.

Representing the uncertainty of the key in the dataset.

7. Calculating the weight of the message:

n _key denotes the number of key occurrences in the message.

Fig. 6 is a flow chart of JSON data generation, which describes how to generate JSON data according to the requirements of a target end, including obtaining requirements of target management, determining whether relevant data is hit in a cache through a user third party interface, if not, reading relevant SNMP data from a database, if yes, reading SNMP data from the cache, and determining whether the data meets the requirements, if not, reading relevant SNMP data from the database, if yes, combining SNMP data sets, formatting SNMP data, and outputting JSON data. The flow steps are mainly described as follows:

1. The requirements of the target management end are transmitted in through a third-party data interface of the system.

2. According to the requirements of the target management end, for example, keywords, time periods, data types and the like, the data is tried to be searched in the cache. If the information such as the key words and the time are matched with the information in the data analysis system, the hit probability is high, otherwise, miss may occur, and at this time, the data is read from the K-V database and jumps to step 4.

3. And reading the data from the cache, analyzing whether the data meets the requirement, and if the data does not meet the requirement, attempting to read the data from the K-V database. For example, since only part of data is in the cache, but the target end needs a larger time period or a larger number of devices, the data may be obtained from the cache incompletely.

4. And generating a set of acquired SNMP data.

5. The SNMP data is formatted, where OID of each device needs to be simplified and unified, information of irrelevant fields in the query is removed, and for data with similar data structure, a storage mode is optimized, for example, storage is performed by using a dictionary or an array mode.

6. And converting the SNMP data into data in a JSON format according to the OID in the SNMP object and returning the data. For SNMP aggregation, a simple method is to group according to OID of device or message type to form JSON array, wherein OID is used as key value of JSON, value is converted into character string to be stored in JSON format.

Wherein NMS: NET MANAGER SYSTEM, abbreviation for network management system. A network management system is a tool or system for monitoring and managing network devices (e.g., routers, switches, servers, firewalls, etc.).

Wherein SNMP is an abbreviation for Simple Network Management Protocol (simple network management protocol). It is an internet standard protocol widely used in Network Management Systems (NMS) to monitor and manage network devices such as routers, switches, servers, firewalls, etc.

Wherein, JSON: javaScript Object Notation (JavaScript object notation) is a lightweight data exchange format. It is based on a subset of JavaScript (programming language), but it is a language independent text format, many programming languages supporting JSON.

Wherein, BS is Backup System, backup System.

Wherein, AOS is Automation Operations System, and the operation and maintenance system is automated.

The technical scheme provided by the application example realizes (1) high correlation and practicability, namely, the weight of SNMP data is calculated, TF-IDF and data information gain are used for screening out data with high information content, and the generated JSON message is ensured to have high correlation and practicability. Users can obtain more valuable and meaningful data rather than being inundated with large amounts of irrelevant or redundant information. (2) Data combination and generation, keywords of the system, window size, threshold, etc., allowing the user to customize and set. This allows the system to be tailored to different needs and scenarios to meet various specific business needs. (3) And the high-efficiency data processing is realized by effectively processing and filtering a large amount of SNMP data through a weight analysis and screening mechanism, so that the data processing efficiency is improved, and unnecessary data transmission and storage are reduced. Through the advantages and the advantages, the efficiency and the quality of converting SNMP data into JSON data can be remarkably improved, and more valuable data service is provided for users.

It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiment of the application also provides a device for converting the object-oriented SNMP data into JSON data, which is used for realizing the method for converting the object-oriented SNMP data into the JSON data. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation in the embodiment of the device for converting SNMP data for target requirements into JSON data provided below can be referred to the limitation in the method for converting SNMP data for target requirements into JSON data hereinabove, and will not be repeated here.

In an exemplary embodiment, as shown in fig. 7, there is provided an apparatus for converting object-oriented SNMP data into JSON data, and the apparatus 700 for converting object-oriented SNMP data into JSON data may include:

A first calculation module 701, configured to calculate word frequency information, inverse document frequency information, and information gain information of SNMP data according to a preset keyword;

a second calculation module 702, configured to calculate weight information of SNMP data according to word frequency information, inverse document frequency information, and information gain information;

The data classification module 703 is configured to classify SNMP data according to the weight information, to obtain a classification result of the SNMP data;

The data storage module 704 is configured to store SNMP data in a preset storage unit according to the classification result;

The data reading module 705 is configured to read target SNMP data from SNMP data stored in a preset storage unit according to a received target requirement;

The data conversion module 706 is configured to convert the target SNMP data into JSON data according to the object identifier in the target SNMP data.

In an exemplary embodiment, the data classifying module 703 is further configured to classify SNMP data having weight information greater than a preset threshold value as hot data and SNMP data having weight information less than or equal to the preset threshold value as cold data, and determine a classification result of the SNMP data according to the hot data and the cold data.

In an exemplary embodiment, the preset storage unit includes at least a preset cache and a preset database, and the data storage module 704 is further configured to store hot data in the preset cache and cold data in the preset database.

In an exemplary embodiment, the data reading module 705 is further configured to read the target SNMP data from the preset cache if the SNMP data meeting the target requirement is queried from the preset cache, and read the target SNMP data from the preset database if the SNMP data meeting the target requirement is not queried from the preset cache.

In an exemplary embodiment, the first calculation module 701 is further configured to set a sliding window of SNMP data, calculate, in the sliding window, a word frequency of a preset keyword, an inverse document frequency of the preset keyword, and an information gain of the preset keyword in the SNMP data, take the word frequency of the preset keyword as word frequency information, take the inverse document frequency of the preset keyword as inverse document frequency information, and take the information gain of the preset keyword as information gain information.

In an exemplary embodiment, the device 700 further includes a message processing module configured to receive the SNMP message, store the SNMP message in a message queue, perform preprocessing on the SNMP message in the message queue to obtain a preprocessed SNMP message as SNMP data, where the preprocessing includes at least a deduplication process, a format unification process, and a data cleaning process.

The above-mentioned respective modules in the device for converting SNMP data to JSON data for object-oriented requirements may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In an exemplary embodiment, a computer device, which may be a terminal, is provided, and an internal structure thereof may be as shown in fig. 8. The computer device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input means. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program, when executed by a processor, implements a method for converting object-oriented SNMP data into JSON data. The display unit of the computer device is used for forming a visual picture, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be a key, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.

It will be appreciated by those skilled in the art that the structure shown in FIG. 8 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.

In an exemplary embodiment, a computer device is also provided, comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method embodiments described above when the computer program is executed.

In one exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method embodiments described above.

In an exemplary embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magneto-resistive random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (PHASE CHANGE Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in various forms such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.

Claims

1. A method for converting SNMP data into JSON data based on target requirements, characterized in that the method comprises:

According to the preset keywords, the word frequency information, inverse document frequency information and information gain information of SNMP data are calculated;

According to the classification result, the SNMP data is classified and stored in a preset storage unit;

Reading target SNMP data from the SNMP data stored in the preset storage unit according to the received target requirement;

The target SNMP data is converted into JSON data according to the object identifier in the target SNMP data.

2. The method according to claim 1, characterized in that the step of classifying the SNMP data according to the weight information to obtain the classification result of the SNMP data comprises:

Classify the SNMP data whose weight information is greater than a preset threshold as hot data, and classify the SNMP data whose weight information is less than or equal to the preset threshold as cold data;

A classification result of the SNMP data is determined according to the hot data and the cold data.

3. The method according to claim 2, characterized in that the preset storage unit at least includes a preset cache and a preset database;

According to the classification result, the SNMP data is classified and stored in a preset storage unit, including:

The hot data is stored in the preset cache, and the cold data is stored in the preset database.

4. The method according to claim 3, characterized in that the step of reading target SNMP data from the SNMP data stored in the preset storage unit according to the received target requirement comprises:

When SNMP data meeting the target requirement is found in the preset cache, the target SNMP data is read from the preset cache;

When no SNMP data meeting the target requirement is found from the preset cache, the target SNMP data is read from the preset database.

5. The method according to claim 1, wherein the step of calculating the word frequency information, inverse document frequency information and information gain information of the SNMP data according to the preset keywords comprises:

Setting a sliding window for the SNMP data;

In the sliding window, the word frequency of the preset keyword in the SNMP data, the inverse document frequency of the preset keyword and the information gain of the preset keyword are calculated;

The word frequency of the preset keyword is used as the word frequency information, the inverse document frequency of the preset keyword is used as the inverse document frequency information, and the information gain of the preset keyword is used as the information gain information.

6. The method according to any one of claims 1 to 5, characterized in that before calculating the word frequency information, inverse document frequency information and information gain information of the SNMP data according to the preset keywords, it also includes:

Receive SNMP messages;

Storing the SNMP message in a message queue;

The SNMP messages in the message queue are preprocessed to obtain preprocessed SNMP messages as the SNMP data; the preprocessing at least includes deduplication processing, format unification processing and data cleaning processing.

7. A device for converting SNMP data into JSON data based on target requirements, characterized in that the device comprises:

A first calculation module, used to calculate word frequency information, inverse document frequency information and information gain information of SNMP data according to preset keywords;

A second calculation module, used for calculating the weight information of the SNMP data according to the word frequency information, the inverse document frequency information and the information gain information;

A data classification module, used for classifying the SNMP data according to the weight information to obtain a classification result of the SNMP data;

A data storage module, used for storing the SNMP data in a preset storage unit according to the classification result;

A data reading module, configured to read target SNMP data from the SNMP data stored in the preset storage unit according to the received target requirement;

The data conversion module is used to convert the target SNMP data into JSON data according to the object identifier in the target SNMP data.

8. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the steps of the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.