Disclosure of Invention
The invention aims to provide a data exchange monitoring method for an enterprise internal system, which can enable an enterprise to supervise and control data transmitted between business systems.
In order to solve the technical problem, the patent provides the following basic technical scheme:
the data exchange monitoring method of the enterprise internal system comprises the following steps:
a data acquisition step, wherein a service system sends a request and data to a service module through a data interface module;
a service processing step, in which a service module receives a service request and data, processes the data according to the service system request and sends the data to a corresponding service system;
a data monitoring step, wherein the data monitoring step is used for monitoring and recording data of the data acquisition step and the service processing step, and the data monitoring step specifically comprises the following contents:
an error detection step, in which an error detection submodule detects errors occurring in the data acquisition step, the service processing step and the data monitoring step;
a data statistics step, wherein a data statistics submodule counts the use condition of information resources in the data transmission process of each service system;
and a log recording step, wherein the log recording submodule generates log data according to the data in the data acquisition step, the service processing step, the error detection step and the data statistics step.
In the technical scheme of the invention, a data interface module is used for establishing data connection between the data exchange monitoring method of the internal system of the enterprise and each service system, each service system can send a request or data to a service module through the data interface module, the service module sends data to each service system through the data interface module, an error detection submodule can detect errors in the operation process of a platform in real time, such as data format errors, service system errors, network connection errors, data transmission errors and the like, and a log recording submodule can record log data generated in the process of processing transmission data by the service module, such as transmission time, transmission content, data exchange quantity, error information and the like, so that managers can check the log data conveniently in the later period; the data statistics submodule counts the use condition of each information resource in the data exchange process between each service system and the platform, such as the access quantity and the access frequency of one data, the data exchange quantity of one service system, the request quantity of one service system and the like, so that an administrator can allocate each service system conveniently according to the use condition of the information resource, for example, the load capacity of the service system with large access quantity is improved, and the data exchange quantity of the service system is controlled.
Further, the method also comprises an alarming step, wherein the alarming step specifically comprises the following steps:
the method comprises the following steps: the alarm submodule acquires the error level and the number;
step two: the alarm submodule calculates the alarm level according to the error level and the quantity and preset weight;
step three: and the alarm submodule sends an alarm to a manager corresponding to the alarm level according to the alarm level.
The alarm sub-module can send an alarm to the manager when an error is detected, the manager can be reminded to process the error in the first time, the alarm is sent to the corresponding manager according to the number of times of the error level, the manager corresponding to each error level can be informed more accurately, and the responsibility range of the manager is determined.
And further, the method further comprises a data temporary storage step, wherein the data temporary storage module stores the data to be sent when detecting that the service module sends the data to the service system in an error.
The data to be sent can be stored through the data temporary storage module so as to be sent again after the service system is normal. Data loss is avoided.
Further, the log data comprises a request service system, a target service system, time, request content, data content and error records.
According to the request service system, the target service system and the time, the flow direction of the data is convenient to track, and errors can be conveniently checked and repaired according to the data content, the request content and the error record.
Further, the service processing step includes:
and a data format checking step, wherein a data format checking submodule judges whether the data format sent by the service system is correct or not, if so, the data format is continuously processed, and if not, the data format is stopped being processed and a format error signal is generated.
And detecting whether the data format sent by the service system is correct or not, screening out error data, and avoiding sending the error data to other service systems.
Further, the data format verifying step specifically includes:
the method comprises the following steps: the data format checking submodule acquires the data type and the data name of the data to be detected;
step two: the data format checking submodule searches a standard data format of the data according to the data type and the data name, if the data format is not found, the data format checking step is ended, and if the standard data format is found, the next step is executed;
step three: and the data format checking submodule compares the number of the data with a standard data format and judges whether the data are the same or not, if so, the data format is judged to be correct, the check is passed, if not, the data format is judged to be wrong, the check is not passed, and the data format checking submodule ends the service processing step of the data.
Detailed Description
The present invention will be described in further detail below by way of specific embodiments:
as shown in fig. 1, the data exchange monitoring method for the internal system of the enterprise in this embodiment uses the following data monitoring system, where the data monitoring system includes a data interface module, a service module, a data monitoring module, and a data temporary storage module, where:
the data interface module is used for sending requests and data to the service module by each service system, and is also used for sending data to each service system by the service module;
the service module is used for processing the data according to the request of the service system and sending the data to the corresponding service system; the service module comprises a data format check submodule, the format check submodule is used for detecting and checking whether a data format sent by the service system is correct, the data format check submodule comprises a format matching unit and a format check unit, the format matching unit is used for obtaining the data format of the data standard according to the data type and the data name, and the format check unit is used for verifying whether the data format is correct according to the data format of the data standard.
The format matching unit acquires the format of the data standard according to the data type and the data name so as to verify the format of the data.
The data monitoring module comprises an error detection submodule, a log recording submodule, an alarm submodule and a data statistics submodule, and the error detection submodule is used for detecting errors in the data processing process of the service module.
The log recording submodule is used for recording log data generated in the data processing process of the service module, and the log data comprises a request service system, a target service system, time, request content, data content and error records. According to the request service system, the target service system and the time, the flow direction of the data is convenient to track, and errors can be conveniently checked and repaired according to the data content, the request content and the error record.
The data statistics submodule is used for counting the use condition of information resources in the data transmission process of each service system, and comprises a processing duration statistics unit, a data volume statistics unit, an access frequency statistics unit and an access volume statistics unit, wherein the processing duration statistics unit is the same as the total duration spent by the service module for processing each service system, the data volume statistics unit is used for counting the total data volume processed and transmitted by each service system through the service module, the access frequency statistics unit is used for counting the access frequency of each service system and data, and the access volume statistics unit is used for counting the access volume of each service system and data.
The activity and the importance degree of each business system can be judged by counting the total data amount, the total data processing time, the access frequency and the access amount of the business systems and the data, and data support is provided for managers when the business systems and the data are adjusted, so that enterprises can reasonably configure and manage resources of the enterprises.
The alarm submodule is used for sending an alarm to the corresponding manager according to the error grade and the number of times when the error detection submodule detects an error. The management personnel can be reminded to process at the first time, and the alarm is sent to the corresponding management personnel according to the number of times of the error level, so that the management personnel corresponding to each error level can be more accurately notified, and the responsibility range of the management personnel is determined.
The platform also comprises a data cleaning module, the data cleaning module is used for detecting, matching and cleaning the data content sent by each service system, and the data cleaning module comprises:
the main data management module comprises a main data storage submodule approval submodule and a main data newly-built submodule, wherein the main data storage submodule stores a main data list and a data cleaning rule list, main data in the main data list corresponds to cleaning rules in the cleaning rule list one by one, and the main data newly-built submodule is used for newly building main data and data cleaning rules corresponding to the main data and respectively storing the main data and the data cleaning rules in the main data list and the cleaning rule list;
and the data operation module is used for matching the data sent by each service system according to the data cleaning rule of the main data and cleaning and replacing the matched data.
In many cases, the same data may have different expression modes, especially between different regional companies, due to the difference of regions, different calls may be made for the same content, even if the same company is used, the cognition of the same content may also change in different development periods, and the problem of data non-uniformity often occurs in each business system of a company, for example, for the name of a client company, there is a short name and a full name in the business system; for the name of a product, along with the expansion of the product function, the product needs to be changed, so that a plurality of names and the like exist in the product in a business system;
the main data list is stored and managed through the main data management module, each main data in the main data list comprises a corresponding data cleaning rule, the data operation module cleans data sent by each business system according to the main data in the main data list, and each data meeting the data cleaning rule is replaced, so that key data finally returned to each business system by the service module are standard main data, and the problem of data confusion caused by the fact that the same data in the enterprise business system are different in existing form is solved.
The main data newly-built sub-module comprises a main data input unit, a main data searching unit, a keyword matching unit and a data cleaning rule generating unit.
The main data searching unit is used for searching and verifying whether the main data input by the user exists in the main data list or not, and repeated establishment of the main data is avoided.
The keyword matching unit can automatically acquire data similar to the main data input by the user from the log data in the data content records transmitted among all the service systems as the cleaning keywords for the user to select, so that the user can know the data similar to the main data in the data transmitted by all the service systems;
the data cleansing rule generating unit can generate a data cleansing rule according to the main data input by the user and the selected cleansing keyword. The data cleaning rule comprises cleaning keywords and matching associated keywords, the matching associated keywords are used for matching data with the cleaning keywords during data cleaning, the data cleaning rule generating unit can extract the matching associated keywords according to the cleaning keywords selected by a user for the user to select, for example, for a company name, after the user selects the cleaning keywords containing the company name, the data cleaning rule generating unit automatically groups the associated data according to the common other associated data of the keywords selected by the user, such as the company address, according to the similarity for the user to select, and the selection of the user is used as the matching associated keywords.
The data operation module comprises a data replacement sub-module and a backup reduction sub-module, the data replacement sub-module is used for matching data in data transmitted by each service system through the data read-write module according to a data cleaning rule of main data, specifically, firstly, cleaning keywords are used as matching keywords to select all matched data, then, matching associated keywords are used for screening all matched data to select data matched with the matching associated keywords, and then, the matched data are cleaned and replaced according to the data cleaning rule. The data operation module is used for cleaning the data transmitted by each business system and replacing each data which accords with the data cleaning rule, so that the problem of data confusion caused by inconsistent existing forms of the same data in the data transmitted by the enterprise business system is solved.
For example, two data in the data transmitted by the service system are both ' X companies ', but the data refer to different companies, have different addresses and operate services, then a Chongqing X technology Limited liability company ' is established through the system to clean the data of the service system, and the manager takes the ' X company ' as a cleaning keyword, and at the moment, the manager needs to select and confirm the corresponding matching associated keyword, for example, the company address is taken as the matching associated keyword, so that the system can correctly distinguish the two ' X companies ' and avoid misoperation.
The log data further comprises a main data cleaning record, the platform further comprises a data cleaning reminding module, the data cleaning reminding module can acquire the category corresponding to the cleaned data according to the data content transmitted according to the cleaning record in the log data, and sends reminding information to a responsible person in the corresponding category, for example, the data content is related to a contract, and then informs a financial department and a business department, and the data is related to a product, and then informs a production department and a sales department, so that the main data is rapidly popularized and applied, and the main data is convenient for a relevant person to trace after being modified by mistake.
The method comprises the following steps:
a data acquisition step, wherein a service system sends a request and data to a service module through a data interface module;
a service processing step, in which a service module receives a service request and data, processes the data according to the service system request and sends the data to a corresponding service system;
a data monitoring step, wherein the data monitoring step is used for monitoring and recording the data of the data acquisition step and the service processing step, and the data monitoring step specifically comprises the following contents:
an error detection step, in which an error detection submodule detects errors occurring in the data acquisition step, the service processing step and the data monitoring step;
a data statistics step, wherein a data statistics submodule counts the use condition of information resources in the data transmission process of each service system;
and a log recording step, wherein the log recording submodule generates log data according to the data in the data acquisition step, the service processing step, the error detection step and the data statistics step.
The method further comprises an alarming step, wherein the alarming step specifically comprises the following steps:
the method comprises the following steps: the alarm submodule acquires the error level and the number;
step two: the alarm submodule calculates the alarm level according to the error level and the quantity and preset weight;
step three: and the alarm submodule sends an alarm to a manager corresponding to the alarm level according to the alarm level.
The data temporary storage module stores the data to be sent when detecting that the service module sends the data to the service system in error.
The service processing step comprises:
and a data format checking step, wherein a data format checking submodule judges whether the data format sent by the service system is correct or not, if so, the data format is continuously processed, and if not, the data format is stopped being processed and a format error signal is generated.
The data format checking step specifically comprises the following steps:
the method comprises the following steps: the data format checking submodule acquires the data type and the data name of the data to be detected;
step two: the data format checking submodule searches a standard data format of the data according to the data type and the data name, if the data format is not found, the data format checking step is ended, and if the standard data format is found, the next step is executed;
step three: and the data format checking submodule compares the number of the data with a standard data format and judges whether the data are the same or not, if so, the data format is judged to be correct, the check is passed, if not, the data format is judged to be wrong, the check is not passed, and the data format checking submodule ends the service processing step of the data.
The method also comprises a data cleaning step, wherein the data cleaning step is used for cleaning and filtering the data input by the service system, and the data cleaning step comprises the following steps:
a main data newly building step, wherein a user builds new main data through a main data newly building submodule and stores the main data into a main data list of a main data storage submodule, wherein the main data comprises a data cleaning rule;
and a data cleaning step, namely reading the data sent by each service system by a data operation module, matching the read data with the main data in the main data list, and replacing the matched data according to the data cleaning rule of the corresponding main data.
The main data new building step comprises a main data checking step and a main data generating step, wherein the main data checking step specifically comprises the following steps:
the method comprises the following steps: a main data newly-built sub-module acquires main data keywords input by a user;
step two: the main data newly-built sub-module reads a main data list in the main data storage sub-module and matches the main data keywords input by the user with the main data in the main data list;
step three: the main data newly-built sub-module judges whether main data with completely the same key words exists in the main data input by the user, if yes, the main data is built, and if not, the next step is executed;
step four: the main data newly-built submodule screens out main data with the similarity exceeding a preset value with the main data keyword input by a user;
step five: and the main data newly-built sub-module executes the steps from six to eight on each screened main data:
step six: the main data newly-built sub-module obtains the difference between the main data key words input by the user and the main data transmitted by screening;
step seven: the main data newly building module obtains the corresponding score of each difference and calculates the total score of the differences;
step eight: the main data newly-built sub-module judges whether the total score of the difference is within a preset difference value range, if so, the main data is marked as suspect data, and if not, the main data is skipped;
step nine: and the master data newly building module screens out all the master data marked as suspect data for the user to check and confirm.
The main data generating step specifically comprises:
the method comprises the following steps: the main data newly-establishing module acquires data contents transmitted among all the service systems from the log data and matches main data keywords input by a user with the data of all the service systems;
step two: the main data newly-built sub-module screens out data with the similarity exceeding a preset value with the main data keyword input by a user as a cleaning keyword for the user to select;
step three: the main data newly-built sub-module screens out data with high relevance to the cleaning keywords from the contents transmitted by the service system according to the cleaning keywords selected by the user, classifies the data according to the similarity, and generates relevant matching keywords for the user to select;
step four: and the main data newly-built sub-module generates a data cleaning rule according to the main data selected by the user and input by the user, the selected cleaning keywords and the matched associated keywords.
The foregoing are merely exemplary embodiments of the present invention, and no attempt is made to show structural details of the invention in more detail than is necessary for the fundamental understanding of the art, the description taken with the drawings making apparent to those skilled in the art how the several forms of the invention may be embodied in practice with the teachings of the invention. It should be noted that, for those skilled in the art, without departing from the structure of the present invention, several changes and modifications can be made, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicability of the patent. The scope of the claims of the present application shall be determined by the contents of the claims, and the description of the embodiments and the like in the specification shall be used to explain the contents of the claims.