CN110491106B

CN110491106B - Data early warning method and device based on knowledge graph and computer equipment

Info

Publication number: CN110491106B
Application number: CN201910661958.8A
Authority: CN
Inventors: 柳洋
Original assignee: OneConnect Financial Technology Co Ltd Shanghai
Current assignee: OneConnect Financial Technology Co Ltd Shanghai
Priority date: 2019-07-22
Filing date: 2019-07-22
Publication date: 2022-03-18
Anticipated expiration: 2039-07-22
Also published as: WO2021012745A1; CN110491106A

Abstract

The present application discloses a data early warning method, device, computer equipment and storage medium based on knowledge graph. The method includes: generating a first data function of the first data changing with time; according to the formula: H(t)= min(G(t),m), where

E(t)=F(t)‑f(t), obtain the function H(t); obtain the first time length when the function H(t) is not equal to m and the second time length when it is equal to m on the time axis , and calculate the normal data time ratio; if the normal data time ratio is greater than the preset ratio threshold, obtain the association that has a direct connection relationship with the designated member from the knowledge graph including the designated member member; obtain the second data of the associated member, and determine whether the second data is abnormal; if the second data is abnormal, generate warning information, and attach the second data of the associated member to the warning information . Thus, the accuracy of early warning can be improved.

Description

Data early warning method and device based on knowledge graph and computer equipment

Technical Field

The present application relates to the field of computers, and in particular, to a method and an apparatus for data early warning based on a knowledge graph, a computer device, and a storage medium.

Background

The prior society mostly has the need of early warning in every aspect, and how to obtain accurate early warning results is pursued by people and is difficult to realize. The traditional technology can only collect and analyze the data of one main body generally, and draw the conclusion whether early warning is needed or not. However, in real production life, the main bodies are not isolated, and one main body is influenced by the main body with strong association relationship, so that the early warning conclusion obtained by analyzing the data of a single main body only is inaccurate. Therefore, the traditional technology lacks a comprehensive accurate early warning scheme.

Disclosure of Invention

The application mainly aims to provide a data early warning method and device based on a knowledge graph, computer equipment and a storage medium, and aims to improve the accuracy of early warning.

In order to achieve the above object, the present application provides a data early warning method based on a knowledge graph, which includes the following steps:

acquiring first data of a designated member by adopting a preset data acquisition technology, carrying out noise reduction processing on the first data, and generating a first data function of the first data changing along with time according to the first data subjected to noise reduction processing;

according to the formula: h (t) ═ min (g (t), m), where

(e) (t) f (t) -f (t), obtaining a function h (t), where f (t) is the first data function, f (t) is a function of the predetermined standard data over time, e (t) is a difference function of the first data function and the function of the standard data over time,

the differential function of the difference function to the time is adopted, min refers to a minimum function, t refers to the time, and m refers to a preset error parameter value larger than 0;

obtaining a first time length of the function H (t) on a time axis, wherein the first time length is not equal to m, and a second time length is equal to m, and according to a formula: calculating the normal data time ratio, namely the first time length/(the first time length + the second time length);

judging whether the time ratio of the normal data is greater than a preset ratio threshold value or not;

if the normal data time ratio is larger than a preset ratio threshold, calling a knowledge graph comprising the designated member from a preset knowledge graph library, and acquiring an associated member having a direct connection relation with the designated member from the knowledge graph comprising the designated member;

acquiring second data of the associated member, and judging whether the second data is abnormal according to a preset data abnormality judgment algorithm;

and if the second data are abnormal, generating early warning information, and attaching the second data of the associated member to the early warning information.

Further, the step of acquiring first data of a designated member by using a preset data acquisition technology and performing noise reduction processing on the first data includes:

crawling first data of a specified member in a preset website by adopting a Scapy frame of a Python language;

and combining the values of the first data into a specified value group, and adopting a preset formula:

calculating the global variance of the mth value in the specified set of values

Wherein N is a total number of values in the specified set of values, Am is an mth value of the specified set of values, and B is an average of the specified set of values;

judging the total variance

Whether all are smaller than a preset variance threshold value;

if the total variance

If the unevenness is less than the preset variance threshold value, the total variance is calculated

And taking the first data which is not less than the preset variance threshold value as noise and carrying out removal processing.

Further, the method comprises the following steps of:

h (t) ═ min (g (t), m), where

E (t) ═ f (t) — (t), obtain function h (t), where f (t) is the first data function, f (t) is a function of the predetermined standard data over time, e (t) isA difference function of the first data function and a function of the standard data over time,

before the step of obtaining a differential function of the difference function with respect to time, where min refers to a minimum function, t is time, and m is a preset error parameter value greater than 0, the method includes:

obtaining an inverse function F of the first data function^-1(y), wherein f (t) is a first data function and y is the first data;

according to the formula:

calculating a time value L, and judging whether the time value L is greater than a preset time threshold value or not, wherein p is a preset parameter value, and p is greater than 0;

and if the time value L is larger than a preset time threshold, generating a function H (t) to obtain an instruction.

Further, the method comprises the following steps of:

calculating a time value L, and judging whether the time value L is greater than a preset time threshold value or not, wherein p is a preset parameter value, and p is greater than 0, the method comprises the following steps:

acquiring historical data of the same type as the first data from a preset database, wherein a dangerous data threshold value is recorded in the historical data, and the dangerous data threshold value refers to a boundary line for dividing the historical data into normal data and abnormal data;

setting the value of the parameter value p as the hazard data threshold.

Further, before the step of retrieving the knowledge graph including the designated member from a preset knowledge graph library and obtaining the associated member having a direct connection relationship with the designated member from the knowledge graph including the designated member, if the normal data time ratio is greater than a preset ratio threshold, the method includes:

identifying an initial entity from pre-collected specified information by adopting a preset knowledge graph construction tool, wherein the specified information at least records the specified members, and the initial entity at least comprises the specified members;

carrying out duplicate removal processing on the initial entity so as to obtain a final entity;

and extracting the relation between final entities from the specified information so as to form a triple, and generating the knowledge graph comprising the specified members according to the triple.

Further, the step of acquiring second data of the associated member and determining whether the second data is abnormal according to a preset data abnormality determination algorithm includes:

acquiring second data of the associated member, and extracting a maximum numerical value and a minimum numerical value from the second data;

judging whether the time point of the maximum numerical value is within a first preset time range or not and judging whether the time point of the minimum numerical value is within a second preset time range or not;

and if the time point of the maximum numerical value is within a first preset time range and the time point of the minimum numerical value is within a second preset time range, judging that the second data is normal.

Further, the step of generating early warning information and attaching the second data of the associated member to the early warning information if the second data is abnormal includes:

if the second data is abnormal, acquiring the influence trend of the second data on the designated member according to the mutual influence relationship of the knowledge nodes of the knowledge graph comprising the designated member;

and generating early warning information, and attaching second data of the associated member and the influence trend of the second data on the specified member to the early warning information.

The application provides a data early warning device based on knowledge map includes:

the system comprises a first data function generating unit, a first data function generating unit and a second data function generating unit, wherein the first data function generating unit is used for acquiring first data of an appointed member by adopting a preset data acquisition technology, carrying out noise reduction processing on the first data and generating a first data function of the first data changing along with time according to the first data subjected to noise reduction processing;

a function h (t) generating unit for:

h (t) ═ min (g (t), m), where

a normal data time ratio obtaining unit, configured to obtain a first time length when the function h (t) is not equal to m and a second time length when the function h (t) is equal to m on a time axis, and according to a formula: calculating the normal data time ratio, namely the first time length/(the first time length + the second time length);

a preset ratio threshold judgment unit, configured to judge whether the normal data time ratio is greater than a preset ratio threshold;

the knowledge graph calling unit is used for calling a knowledge graph comprising the designated member from a preset knowledge graph library and obtaining an associated member which has a direct connection relation with the designated member from the knowledge graph comprising the designated member if the normal data time ratio is greater than a preset ratio threshold;

the second data judgment unit is used for acquiring second data of the associated member and judging whether the second data is abnormal or not according to a preset data abnormality judgment algorithm;

and the early warning information generating unit is used for generating early warning information if the second data are abnormal, and attaching the second data of the associated member to the early warning information.

The present application provides a computer device comprising a memory storing a computer program and a processor implementing the steps of any of the above methods when the processor executes the computer program.

The present application provides a computer-readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the method of any of the above.

According to the data early warning method and device based on the knowledge graph, the computer equipment and the storage medium, a first data function of the first data changing along with time is generated; according to the formula: h (t) ═ min (g (t), m), where

E (t) ═ f (t) — (t), the function h (t) is obtained; acquiring a first time length of the function H (t) on a time axis, wherein the first time length is not equal to m, and a second time length is equal to m, and calculating the normal data time ratio; judging whether the time ratio of the normal data is greater than a preset ratio threshold value or not; if the normal data time ratio is larger than a preset ratio threshold, acquiring associated members in direct connection relation with the specified members from the knowledge graph comprising the specified members; acquiring second data of the associated member, and judging whether the second data is abnormal according to a preset data abnormality judgment algorithm; and if the second data are abnormal, generating early warning information, and attaching the second data of the associated member to the early warning information. Therefore, the accuracy of early warning is improved.

Drawings

Fig. 1 is a schematic flowchart of a data early warning method based on a knowledge graph according to an embodiment of the present application;

FIG. 2 is a block diagram illustrating a data pre-warning apparatus based on knowledge-graph according to an embodiment of the present disclosure;

fig. 3 is a block diagram illustrating a structure of a computer device according to an embodiment of the present application.

The implementation, functional features and advantages of the objectives of the present application will be further explained with reference to the accompanying drawings.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

Referring to fig. 1, an embodiment of the present application provides a data early warning method based on a knowledge graph, including the following steps:

s1, acquiring first data of a designated member by adopting a preset data acquisition technology, performing noise reduction processing on the first data, and generating a first data function of the first data changing along with time according to the first data subjected to noise reduction processing;

s2, according to the formula:

h (t) ═ min (g (t), m), where

s3, obtaining a first time length of the function H (t) on a time axis, wherein the first time length is not equal to m, and a second time length is equal to m, and according to the formula: calculating the normal data time ratio, namely the first time length/(the first time length + the second time length);

s4, judging whether the time ratio of the normal data is larger than a preset ratio threshold value or not;

s5, if the normal data time ratio is larger than a preset ratio threshold, calling a knowledge graph comprising the designated member from a preset knowledge graph library, and acquiring an associated member having a direct connection relation with the designated member from the knowledge graph comprising the designated member;

s6, acquiring second data of the associated member, and judging whether the second data is abnormal according to a preset data abnormality judgment algorithm;

and S7, if the second data are abnormal, generating early warning information, and attaching the second data of the associated members to the early warning information.

As described in step S1, the first data of the designated member is obtained by using a preset data acquisition technique, the first data is subjected to noise reduction processing, and a first data function of the first data changing with time is generated according to the first data subjected to noise reduction processing. The first data can be acquired through the internet, the mobile internet and the internet of things, data including pictures, videos and text information can be processed to be acquired, data processing can be performed by adopting an open-source Storm (a distributed and fault-tolerant real-time computing system), a script framework of a Python language can also be adopted, and crawling is performed in a preset website, so that the first data of the designated member can be acquired. Wherein the first data may be any form of data, such as flow data, financial data, and the like. And noise reduction processing is carried out to ensure that the data is more accurate. And generating a first data function of the first data changing along with time according to the first data after the noise reduction processing, so as to analyze whether the first data is abnormal or not in the follow-up process.

As stated in step S2 above, according to the formula:

h (t) ═ min (g (t), m), where

E (t) ═ f (t) — (t), obtain function h (t), where f (t) is the first data function, f (t) is the preset standard data with timeA function of quantization, E (t) being a function of the difference of said first data function and a function of said standard data over time,

and a differential function of the difference function to time is adopted, min refers to a minimum function, t is time, and m is a preset error parameter value larger than 0. And acquiring a function H (t) according to a formula to represent the fitting degree of the first data function and the function of the standard data changing along with time. If the fitting degree of the first data function and the function of the standard data changing along with the time is small, the first data are normal, otherwise, the first data are abnormal.

As described in the above step S3, a first time length of the function h (t) on the time axis is not equal to m and a second time length of the function h (t) on the time axis is obtained, and according to the formula: and calculating the normal data time ratio, namely the first time length/(the first time length + the second time length). When the value of the function H (t) is m, the value of the first data is over large and is in an abnormal state; when the value of the function H (t) is not m, the first data is normal and is in a normal state, and the normal data time ratio is calculated according to the normal data time ratio. Therefore, whether the first data is in an abnormal state or not can be judged through normal data time ratio.

As described in step S4 above, it is determined whether the normal data time ratio is greater than a preset ratio threshold. If the time ratio of the normal data is larger than a preset ratio threshold, the first data is generally normal, and therefore the first data is judged to be normal; if the normal data time ratio is not larger than a preset ratio threshold, the first data are generally abnormal, and therefore the first data are judged to be abnormal.

As described in step S5, if the normal data time ratio is greater than the preset ratio threshold, the knowledge graph including the designated member is called from the preset knowledge graph library, and the associated member having a direct connection relationship with the designated member is obtained from the knowledge graph including the designated member. And if the time ratio of the normal data is greater than a preset ratio threshold, indicating that the first data is normal. However, in order to analyze the data more accurately and obtain an accurate early warning conclusion, the data of the associated members are also analyzed. Wherein, a plurality of knowledge maps are prestored in a preset knowledge map library. The knowledge graph is a series of different graphs displaying the relationship between the knowledge development process and the structure, describes knowledge resources and carriers thereof by using a visualization technology, excavates, analyzes, constructs, draws and displays knowledge and the mutual relation among the knowledge resources and the carriers, and consists of a plurality of knowledge nodes (or knowledge bodies and bodies) and the mutual relationship among the knowledge nodes. Accordingly, the related members having direct connection relation with the designated member are obtained from the knowledge graph comprising the designated member. Wherein the associated member is related to the designated member and the knowledge graph, for example, when the designated member is one server in the server cluster, the associated member is, for example, a database server establishing direct contact with the server, etc.; when a designated member is a natural human subject, the associated member is, for example, an immediate relative of the natural human subject.

As described in step S6, the second data of the associated member is obtained, and whether the second data is abnormal is determined according to a preset data abnormality determination algorithm. The second data of the associated member may be obtained by any means, for example, by a data acquisition technique from a network, or may be directly retrieved from a database. The preset data abnormality determination algorithm may be the same as the aforementioned method for determining whether the first data is abnormal, or may be other determination methods, for example: extracting a maximum value and a minimum value from the second data; judging whether the time point of the maximum numerical value is within a first preset time range or not and judging whether the time point of the minimum numerical value is within a second preset time range or not; and if the time point of the maximum numerical value is within a first preset time range and the time point of the minimum numerical value is within a second preset time range, judging whether the second data is normal or not. Thereby determining whether the second data is abnormal.

As described in step S7, if the second data is abnormal, an early warning message is generated, and the second data of the associated member is attached to the early warning message. If the second data is abnormal, although the first data is in a normal state, the second data of the associated member may affect the designated member. Therefore, early warning information is still generated, and the second data of the associated member is attached to the early warning information.

In an embodiment, the step S1 of acquiring first data of a designated member by using a preset data acquisition technique and performing noise reduction processing on the first data includes:

s101, crawling first data of a specified member in a preset website by adopting a Scapy frame of a Python language;

s102, combining the numerical values of the first data into a specified numerical value group, and adopting a preset formula:

calculating the global variance of the mth value in the specified set of values

s103, judging the total variance

Whether all are smaller than a preset variance threshold value;

s104, if the total variance

As has been described above, in the above-mentioned,the noise reduction processing is realized by using a preset noise reduction algorithm, so that the specified data is obtained. Wherein adopt the first data of predetermined data acquisition technique acquisition appointed member to include, adopt the script frame of Python language to crawl information in presetting the website, wherein the script frame of Python language mainly includes: engines, schedulers, downloaders, crawlers, project pipes, downloader middleware, crawler middleware, scheduling middleware, and the like. The specific crawling process comprises the following steps: the engine fetches a link from the scheduler for the next fetch; the engine encapsulates the link into a request and transmits the request to the downloader; downloading the resource by the downloader; the crawler analyzes the entity and gives the entity to the entity pipeline for further processing. Because inaccurate data may exist in the crawled numerical value, the method adopts a preset formula:

computing a global variance of the mth data in the specified set of values

Judging the total variance

Whether all are smaller than a preset variance threshold value; if the total variance

And taking the first data not less than a preset variance threshold value as noise and performing removal processing. Thereby avoiding the problem of data processing misalignment caused by noise data.

In one embodiment, the method further comprises:

h (t) ═ min (g (t), m), where

E (t) ═ f (t) — (t), orTaking a function H (t), wherein F (t) is the first data function, f (t) is a function of the time variation of the preset standard data, E (t) is a difference function of the first data function and the function of the time variation of the standard data,

before step S2, which is a differential function of the difference function with respect to time, where min refers to a minimum function, t is time, and m is a preset error parameter value greater than 0, the method includes:

s11, obtaining an inverse function F of the first data function^-1(y), wherein f (t) is a first data function and y is the first data;

s12, according to the formula:

and S13, if the time value L is larger than a preset time threshold, generating a function H (t) to obtain an instruction.

As described above, the generating function h (t) fetch instruction is implemented. In order to reduce the calculation consumption, the method also adopts a preprocessing mode to judge whether the first data is normal in advance, and generates a function H (t) to obtain an instruction under the condition that the first data is judged to be possibly abnormal. Specifically, an inverse function F of the first data function is obtained^-1(y), wherein f (t) is a first data function and y is the first data; according to the formula:

calculating a time value L, and judging whether the time value L is greater than a preset time threshold value or not, wherein p is a preset parameter value, and p is greater than 0; if the time value L is greater than a preset time threshold, it indicates that the first data may be abnormal, and a function h (t) is generated accordingly to obtain an instruction.

In one embodiment, the method further comprises:

calculating a time value L, and determining whether the time value L is greater than a preset time threshold, where p is a preset parameter value, and before step S12 when p is greater than 0, the method includes:

s111, acquiring historical data of the same type as the first data from a preset database, wherein a dangerous data threshold is recorded in the historical data, and the dangerous data threshold refers to a boundary line for dividing the historical data into normal data and abnormal data;

and S112, setting the value of the parameter value p as the dangerous data threshold value.

As mentioned above, setting the value of the parameter value p to the hazard data threshold is achieved. Wherein the parameter value p is used for measuring whether the first data has abnormal suspicion. Since the historical data with the same type as the first data already draw an accurate conclusion, including the specific numerical value of the dangerous situation data threshold value in the historical data, the method of reusing the historical data is adopted, so that even if the data is fully utilized, the setting of the parameter value p is more based and more accurate by setting the numerical value of the parameter value p as the dangerous data threshold value.

In one embodiment, before step S5, if the normal data time ratio is greater than a preset ratio threshold, the method includes:

s41, identifying an initial entity from pre-collected specified information by adopting a preset knowledge graph construction tool, wherein the specified information at least records the specified members, and the initial entity at least comprises the specified members;

s42, carrying out duplicate removal processing on the initial entity so as to obtain a final entity;

and S43, extracting the relation between final entities from the specified information to form a triple, and generating the knowledge graph comprising the specified members according to the triple.

As described above, building the knowledge-graph that includes the specified members is enabled. The preset knowledge graph constructing tool may be any tool, such as the existing SPSS, Sci2 Tools, ucinetraw, Pajek, VOSviewer, and the like, and is not described herein again because the tool is an existing knowledge graph constructing tool. The entity is a knowledge node in the knowledge graph, and the initial entity refers to the knowledge node which is not subjected to the past reprocessing. The process of identifying the initial entity is for example: performing word segmentation processing on the specified information to obtain a word sequence consisting of a plurality of words, and inputting the word sequence into a preset sentence structure model to obtain an initial entity in the word sequence. And then carrying out deduplication processing on the initial entity so as to obtain a final entity. The process of the deduplication processing is, for example: and carrying out synonym judgment on all initial entities, and replacing the initial entities belonging to the same synonym group with one vocabulary in the synonym group. And extracting the relation between final entities from the specified information so as to form a triple, and generating the knowledge graph comprising the specified members according to the triple. Where a triple refers to, for example, a relationship between two entities. The method for extracting the relationship between the final entities from the specific information includes, for example: and sleeving the specified information into a preset sentence structure, so as to extract the vocabulary expressing the relationship among the plurality of entities through the sentence structure.

In an embodiment, the step S6 of acquiring the second data of the associated member and determining whether the second data is abnormal according to a preset data abnormality determination algorithm includes:

s601, acquiring second data of the associated member, and extracting a maximum numerical value and a minimum numerical value from the second data;

s602, judging whether the time point of the maximum numerical value is within a first preset time range or not, and judging whether the time point of the minimum numerical value is within a second preset time range or not;

s603, if the time point of the maximum numerical value is within a first preset time range and the time point of the minimum numerical value is within a second preset time range, judging that the second data is normal.

As described above, whether the second data is abnormal or not is determined according to a preset data abnormality determination algorithm. The method comprises the steps of extracting a maximum value and a minimum value from second data, judging whether a time point of the maximum value is within a first preset time range, and judging whether the time point of the minimum value is within a second preset time range. Wherein, since the second data (such as the flow amount) is fluctuated with time and generally has periodicity, the maximum value and the minimum value of the second data should respectively appear in the first preset time range and the second preset time range. Therefore, if the time point of the maximum value is within a first preset time range and the time point of the minimum value is within a second preset time range, the second data is determined to be normal. And if the time point of the maximum numerical value is not within a first preset time range, or the time point of the minimum numerical value is not within a second preset time range, judging that the second data is abnormal. Further, the method for determining whether the second data is abnormal may be the same as the method for determining whether the first data is abnormal, regardless of the consumption of computing resources.

In one embodiment, the step S7 of generating warning information and attaching the second data of the associated member to the warning information if the second data is abnormal, includes:

s701, if the second data are abnormal, acquiring the influence trend of the second data on the designated member according to the mutual influence relationship of the knowledge nodes of the knowledge graph comprising the designated member;

s702, generating early warning information, and attaching second data of the associated member and the influence trend of the second data on the specified member to the early warning information.

As described above, attaching the second data of the associated member and the influence trend of the second data on the designated member to the early warning information is realized. The knowledge graph comprises the interaction relationship between the designated member and the associated member. And generating early warning information, and attaching second data of the associated member and the influence trend of the second data on the specified member to the early warning information. Further, an influence formula of the associated member on the designated member is recorded in the knowledge graph, and then an influence value of second data of the associated member on the designated member is obtained according to the influence formula, and the influence value is attached to the early warning information.

According to the data early warning method based on the knowledge graph, a first data function of the first data changing along with time is generated; according to the formula: h (t) ═ min (g (t), m), where

Referring to fig. 2, an embodiment of the present application provides a data early warning device based on a knowledge graph, including:

the first data function generating unit 10 is configured to acquire first data of an appointed member by using a preset data acquisition technology, perform noise reduction processing on the first data, and generate a first data function in which the first data changes with time according to the first data after the noise reduction processing;

a function h (t) generating unit 20 for generating, according to the formula:

h (t) ═ min (g (t), m), where

a normal data time ratio obtaining unit 30, configured to obtain a first time length when the function h (t) is not equal to m and a second time length when the function h (t) is equal to m on a time axis, and according to a formula: calculating the normal data time ratio, namely the first time length/(the first time length + the second time length);

a preset ratio threshold value judging unit 40, configured to judge whether the normal data time ratio is greater than a preset ratio threshold value;

a knowledge graph retrieving unit 50, configured to retrieve a knowledge graph including the specified member from a preset knowledge graph library if the normal data time ratio is greater than a preset ratio threshold, and retrieve an associated member having a direct connection relationship with the specified member from the knowledge graph including the specified member;

a second data determining unit 60, configured to obtain second data of the associated member, and determine whether the second data is abnormal according to a preset data abnormality determining algorithm;

an early warning information generating unit 70, configured to generate early warning information if the second data is abnormal, and attach the second data of the associated member to the early warning information.

As described in the above unit 10, the first data of the designated member is obtained by using a preset data acquisition technology, the first data is subjected to noise reduction processing, and a first data function of the first data changing with time is generated according to the first data subjected to noise reduction processing. The first data can be acquired through the internet, the mobile internet and the internet of things, data including pictures, videos and text information can be processed to be acquired, data processing can be performed by adopting an open-source Storm (a distributed and fault-tolerant real-time computing system), a script framework of a Python language can also be adopted, and crawling is performed in a preset website, so that the first data of the designated member can be acquired. Wherein the first data may be any form of data, such as flow data, financial data, and the like. And noise reduction processing is carried out to ensure that the data is more accurate. And generating a first data function of the first data changing along with time according to the first data after the noise reduction processing, so as to analyze whether the first data is abnormal or not in the follow-up process.

As described above in element 20, according to the formula:

h (t) ═ min (g (t), m), where

and a differential function of the difference function to time is adopted, min refers to a minimum function, t is time, and m is a preset error parameter value larger than 0. And acquiring a function H (t) according to a formula to represent the fitting degree of the first data function and the function of the standard data changing along with time. If the first data function and the standard data change with timeThe fitness of the function (b) is small, which indicates that the first data is normal, otherwise, the first data is abnormal.

As described in the above unit 30, a first time length of the function h (t) on the time axis is not equal to m and a second time length of the function h (t) on the time axis is obtained, and according to the formula: and calculating the normal data time ratio, namely the first time length/(the first time length + the second time length). When the value of the function H (t) is m, the value of the first data is over large and is in an abnormal state; when the value of the function H (t) is not m, the first data is normal and is in a normal state, and the normal data time ratio is calculated according to the normal data time ratio. Therefore, whether the first data is in an abnormal state or not can be judged through normal data time ratio.

As described above with respect to unit 40, it is determined whether the normal data time occupancy is greater than a preset occupancy threshold. If the time ratio of the normal data is larger than a preset ratio threshold, the first data is generally normal, and therefore the first data is judged to be normal; if the normal data time ratio is not larger than a preset ratio threshold, the first data are generally abnormal, and therefore the first data are judged to be abnormal.

As described in the above unit 50, if the normal data time ratio is greater than the preset ratio threshold, the knowledge graph including the specified member is called from the preset knowledge graph library, and the associated member having a direct connection relationship with the specified member is obtained from the knowledge graph including the specified member. And if the time ratio of the normal data is greater than a preset ratio threshold, indicating that the first data is normal. However, in order to analyze the data more accurately and obtain an accurate early warning conclusion, the data of the associated members are also analyzed. Wherein, a plurality of knowledge maps are prestored in a preset knowledge map library. The knowledge graph is a series of different graphs displaying the relationship between the knowledge development process and the structure, describes knowledge resources and carriers thereof by using a visualization technology, excavates, analyzes, constructs, draws and displays knowledge and the mutual relation among the knowledge resources and the carriers, and consists of a plurality of knowledge nodes (or knowledge bodies and bodies) and the mutual relationship among the knowledge nodes. Accordingly, the related members having direct connection relation with the designated member are obtained from the knowledge graph comprising the designated member. Wherein the associated member is related to the designated member and the knowledge graph, for example, when the designated member is one server in the server cluster, the associated member is, for example, a database server establishing direct contact with the server, etc.; when a designated member is a natural human subject, the associated member is, for example, an immediate relative of the natural human subject.

As described in the foregoing unit 60, the second data of the associated member is obtained, and whether the second data is abnormal is determined according to a preset data abnormality determination algorithm. The second data of the associated member may be obtained by any means, for example, by a data acquisition technique from a network, or may be directly retrieved from a database. The preset data abnormality determination algorithm may be the same as the aforementioned method for determining whether the first data is abnormal, or may be other determination methods, for example: extracting a maximum value and a minimum value from the second data; judging whether the time point of the maximum numerical value is within a first preset time range or not and judging whether the time point of the minimum numerical value is within a second preset time range or not; and if the time point of the maximum numerical value is within a first preset time range and the time point of the minimum numerical value is within a second preset time range, judging whether the second data is normal or not. Thereby determining whether the second data is abnormal.

As described in the above-mentioned unit 70, if the second data is abnormal, the warning information is generated, and the second data of the associated member is attached to the warning information. If the second data is abnormal, although the first data is in a normal state, the second data of the associated member may affect the designated member. Therefore, early warning information is still generated, and the second data of the associated member is attached to the early warning information.

In one embodiment, the first data function generating unit includes:

the first data crawling subunit is used for crawling first data of a specified member in a preset website by adopting a Scapy framework of a Python language;

a variance calculating subunit, configured to combine the values of the first data into a specified value group, and adopt a preset formula:

calculating the global variance of the mth value in the specified set of values

a variance threshold judgment subunit for judging the total variance

Whether all are smaller than a preset variance threshold value;

a de-noising subunit for determining the global variance

As described above, it is realized that the noise reduction processing is performed using the preset noise reduction algorithm, thereby obtaining the specified data. Wherein adopt the first data of predetermined data acquisition technique acquisition appointed member to include, adopt the script frame of Python language to crawl information in presetting the website, wherein the script frame of Python language mainly includes: engines, schedulers, downloaders, crawlers, project pipes, downloader middleware, crawler middleware, scheduling middleware, and the like. The specific crawling process comprises the following steps: the engine fetches a link from the scheduler for the next fetch; the engine encapsulates the link into a requestTransmitting to a downloader; downloading the resource by the downloader; the crawler analyzes the entity and gives the entity to the entity pipeline for further processing. Because inaccurate data may exist in the crawled numerical value, the method adopts a preset formula:

computing a global variance of the mth data in the specified set of values

Judging the total variance

In one embodiment, the apparatus comprises:

an inverse function obtaining unit for obtaining an inverse function F of the first data function^-1(y), wherein f (t) is a first data function and y is the first data;

a time value L calculation unit configured to:

and the generating function H (t) acquiring instruction unit is used for generating a function H (t) acquiring instruction if the time value L is greater than a preset time threshold.

As described aboveThe generate function h (t) fetch instruction is implemented. In order to reduce the calculation consumption, the method also adopts a preprocessing mode to judge whether the first data is normal in advance, and generates a function H (t) to obtain an instruction under the condition that the first data is judged to be possibly abnormal. Specifically, an inverse function F of the first data function is obtained^-1(y), wherein f (t) is a first data function and y is the first data; according to the formula:

In one embodiment, the apparatus comprises:

a historical data acquiring unit, configured to acquire historical data of the same type as the first data from a preset database, where a dangerous data threshold is recorded in the historical data, and the dangerous data threshold is a boundary between normal data and abnormal data obtained by dividing the historical data;

and the parameter value setting unit is used for setting the value of the parameter value p as the dangerous data threshold value.

In one embodiment, the apparatus comprises:

the initial entity identification unit is used for identifying an initial entity from pre-collected specified information by adopting a preset knowledge graph construction tool, wherein the specified information at least records the specified members, and the initial entity at least comprises the specified members;

an obtaining final entity unit, configured to perform deduplication processing on the initial entity, so as to obtain a final entity;

and the knowledge graph generating unit is used for extracting the relationship between final entities from the specified information so as to form a triple and generating the knowledge graph comprising the specified members according to the triple.

In one embodiment, the second data determining unit 60 includes:

the numerical value extraction subunit is used for acquiring second data of the associated member and extracting a maximum numerical value and a minimum numerical value from the second data;

a numerical value judging subunit, configured to judge whether the time point at which the maximum numerical value appears is within a first preset time range, and judge whether the time point at which the minimum numerical value appears is within a second preset time range;

and the second data normality judging subunit is used for judging that the second data is normal if the time point of the maximum numerical value is within a first preset time range and the time point of the minimum numerical value is within a second preset time range.

In one embodiment, the warning information generating unit 70 includes:

an influence trend acquiring subunit, configured to acquire, if the second data is abnormal, an influence trend of the second data on the designated member according to the mutual influence relationship between the knowledge nodes of the knowledge graph including the designated member;

and the early warning information generating subunit is used for generating early warning information and attaching second data of the associated member and the influence trend of the second data on the specified member to the early warning information.

The data early warning device based on the knowledge graph generates a first data function of the first data changing along with time; according to the formula: h (t) ═ min (g (t), m), where

E (t) ═ f (t) — (t), the function h (t) is obtained; acquiring a first time length of the function H (t) on a time axis, wherein the first time length is not equal to m, and a second time length is equal to m, and calculating the normal data time ratio; judging whether the time ratio of the normal data is greater than a preset ratio threshold value or not; if the normal data time ratio is larger than a preset ratio threshold, acquiring associated members in direct connection relation with the specified members from the knowledge graph comprising the specified members; acquiring second data of the associated member, and judging and calculating according to preset data abnormityJudging whether the second data is abnormal or not; and if the second data are abnormal, generating early warning information, and attaching the second data of the associated member to the early warning information. Therefore, the accuracy of early warning is improved.

Referring to fig. 3, an embodiment of the present invention further provides a computer device, where the computer device may be a server, and an internal structure of the computer device may be as shown in the figure. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the computer designed processor is used to provide computational and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for storing data used by the data early warning method based on the knowledge graph. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of data forewarning based on a knowledge-graph.

The processor executes the data early warning method based on the knowledge graph, and comprises the following steps: acquiring first data of a designated member by adopting a preset data acquisition technology, carrying out noise reduction processing on the first data, and generating a first data function of the first data changing along with time according to the first data subjected to noise reduction processing; according to the formula: h (t) ═ min (g (t), m), where

is a differential function of the difference function with respect to time, min is a minimum function, and t is timeM is a preset error parameter value larger than 0; obtaining a first time length of the function H (t) on a time axis, wherein the first time length is not equal to m, and a second time length is equal to m, and according to a formula: calculating the normal data time ratio, namely the first time length/(the first time length + the second time length); judging whether the time ratio of the normal data is greater than a preset ratio threshold value or not; if the normal data time ratio is larger than a preset ratio threshold, calling a knowledge graph comprising the designated member from a preset knowledge graph library, and acquiring an associated member having a direct connection relation with the designated member from the knowledge graph comprising the designated member; acquiring second data of the associated member, and judging whether the second data is abnormal according to a preset data abnormality judgment algorithm; and if the second data are abnormal, generating early warning information, and attaching the second data of the associated member to the early warning information.

In one embodiment, the step of acquiring first data of a designated member by using a preset data acquisition technology and performing noise reduction processing on the first data includes: crawling first data of a specified member in a preset website by adopting a Scapy frame of a Python language; and combining the values of the first data into a specified value group, and adopting a preset formula:

calculating the global variance of the mth value in the specified set of values

Wherein N is a total number of values in the specified set of values, Am is an mth value of the specified set of values, and B is an average of the specified set of values; judging the total variance

In one embodiment, the method further comprises: h (t) ═ min (g (t), m), where

before the step of obtaining a differential function of the difference function with respect to time, where min refers to a minimum function, t is time, and m is a preset error parameter value greater than 0, the method includes: obtaining an inverse function F of the first data function^-1(y), wherein f (t) is a first data function and y is the first data; according to the formula:

calculating a time value L, and judging whether the time value L is greater than a preset time threshold value or not, wherein p is a preset parameter value, and p is greater than 0; and if the time value L is larger than a preset time threshold, generating a function H (t) to obtain an instruction.

In one embodiment, the method further comprises:

calculating a time value L, and judging whether the time value L is greater than a preset time threshold value or not, wherein p is a preset parameter value, and p is greater than 0, the method comprises the following steps: obtaining historical data with the same type as the first data from a preset database, wherein dangerous data threshold values are recorded in the historical dataThe dangerous data threshold refers to a boundary line for dividing the historical data into normal data and abnormal data; setting the value of the parameter value p as the hazard data threshold.

In one embodiment, the step of retrieving the knowledge graph including the designated member from a preset knowledge graph library and retrieving the associated member having a direct connection relationship with the designated member from the knowledge graph including the designated member, if the normal data time ratio is greater than a preset ratio threshold, includes: identifying an initial entity from pre-collected specified information by adopting a preset knowledge graph construction tool, wherein the specified information at least records the specified members, and the initial entity at least comprises the specified members; carrying out duplicate removal processing on the initial entity so as to obtain a final entity; and extracting the relation between final entities from the specified information so as to form a triple, and generating the knowledge graph comprising the specified members according to the triple.

In one embodiment, the step of acquiring the second data of the associated member and determining whether the second data is abnormal according to a preset data abnormality determination algorithm includes: acquiring second data of the associated member, and extracting a maximum numerical value and a minimum numerical value from the second data; judging whether the time point of the maximum numerical value is within a first preset time range or not and judging whether the time point of the minimum numerical value is within a second preset time range or not; and if the time point of the maximum numerical value is within a first preset time range and the time point of the minimum numerical value is within a second preset time range, judging that the second data is normal.

In one embodiment, the step of generating warning information and attaching the second data of the associated member to the warning information if the second data is abnormal includes: if the second data is abnormal, acquiring the influence trend of the second data on the designated member according to the mutual influence relationship of the knowledge nodes of the knowledge graph comprising the designated member; and generating early warning information, and attaching second data of the associated member and the influence trend of the second data on the specified member to the early warning information.

It will be understood by those skilled in the art that the structures shown in the drawings are only block diagrams of some of the structures associated with the embodiments of the present application and do not constitute a limitation on the computer apparatus to which the embodiments of the present application may be applied.

The computer equipment generates a first data function of the first data changing along with time; according to the formula: h (t) ═ min (g (t), m), where

An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method for performing a data pre-warning based on a knowledge graph includes the following steps: acquiring first data of a designated member by adopting a preset data acquisition technology, carrying out noise reduction processing on the first data, and generating a first data function of the first data changing along with time according to the first data subjected to noise reduction processing;

according to the formula: h (t) ═ min (g (t), m), where

the differential function of the difference function to the time is adopted, min refers to a minimum function, t refers to the time, and m refers to a preset error parameter value larger than 0; obtaining a first time length of the function H (t) on a time axis, wherein the first time length is not equal to m, and a second time length is equal to m, and according to a formula: calculating the normal data time ratio, namely the first time length/(the first time length + the second time length); judging whether the time ratio of the normal data is greater than a preset ratio threshold value or not; if the normal data time ratio is larger than a preset ratio threshold, calling a knowledge graph comprising the designated member from a preset knowledge graph library, and acquiring an associated member having a direct connection relation with the designated member from the knowledge graph comprising the designated member; acquiring second data of the associated member, and judging whether the second data is abnormal according to a preset data abnormality judgment algorithm; and if the second data are abnormal, generating early warning information, and attaching the second data of the associated member to the early warning information.

calculating the global variance of the mth value in the specified set of values

Wherein N is as definedSpecifying a total number of values in a set of values, Am being an mth value of the set of values, B being an average of the set of values; judging the total variance

In one embodiment, theAccording to the formula:

calculating a time value L, and judging whether the time value L is greater than a preset time threshold value or not, wherein p is a preset parameter value, and p is greater than 0, the method comprises the following steps: acquiring historical data of the same type as the first data from a preset database, wherein a dangerous data threshold value is recorded in the historical data, and the dangerous data threshold value refers to a boundary line for dividing the historical data into normal data and abnormal data; setting the value of the parameter value p as the hazard data threshold.

A computer-readable storage medium of the present application, generating a first data function of the first data over time; according to the formula: h (t) ═ min (g (t), m), where

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium provided herein and used in the examples may include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), double-rate SDRAM (SSRSDRAM), Enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and bus dynamic RAM (RDRAM).

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, apparatus, article, or method. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, apparatus, article, or method that includes the element.

The above description is only a preferred embodiment of the present application, and not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings of the present application, or which are directly or indirectly applied to other related technical fields, are also included in the scope of the present application.

Claims

1. a data early warning method based on knowledge graph, is characterized in that, comprises:

The first data of the designated member is acquired by using a preset data collection technology, noise reduction processing is performed on the first data, and a first data function of the first data changing with time is generated according to the first data after the noise reduction processing. ;

According to the formula: H(t)=min(G(t), m), where

The acquisition function H(t), where F(t) is the first data function, f(t) is the function of the preset standard data changing with time, and E(t) is the first data function and the the difference function of the function of the standard data over time,

is the differential function of the difference function to time, min refers to the minimum value function, t is the time, and m is the preset error parameter value greater than 0;

Obtain the first time length when the function H(t) is not equal to m and the second time length when it is equal to m on the time axis, and according to the formula: normal data time ratio=the first time length/(the (the first time length+the second time length), calculate the normal data time ratio;

judging whether the normal data time proportion is greater than a preset proportion threshold;

If the normal data time proportion is greater than the preset proportion threshold, the knowledge graph including the designated member is retrieved from the preset knowledge graph library, and the knowledge graph including the designated member is obtained from the knowledge graph including the designated member. The designated member has an associated member with a direct connection relationship;

Acquiring the second data of the associated member, and judging whether the second data is abnormal according to a preset data abnormality judging algorithm;

If the second data is abnormal, generate early warning information, and attach the second data of the associated member to the early warning information.

2. The data early warning method based on a knowledge graph according to claim 1, wherein the step of obtaining the first data of a designated member by using a preset data collection technology, and performing noise reduction processing on the first data ,include:

Using the Scrapy framework of the Python language, crawl the first data of the specified member in the preset website;

The numerical value of the first data is formed into a specified numerical value group, and a preset formula is used:

computes the population variance of the mth value in the specified group of values

Wherein N is the total number of values in the specified numerical value group, Am is the mth value of the specified numerical value group, and B is the average value of the specified numerical value group;

determine the population variance

Are all less than the preset variance threshold;

If the population variance

If the unevenness is less than the preset variance threshold, the overall variance will be

The first data corresponding to not less than the preset variance threshold is regarded as noise and removed.

3. the data early warning method based on knowledge graph according to claim 1, is characterized in that, described according to formula:

H(t)=min(G(t),m), where

is the differential function of the difference function with respect to time, min refers to the minimum value function, t is the time, m is the preset error parameter value greater than 0 before the step, including:

Obtain the inverse function F ^-1 (y) of the first data function, wherein F(t) is the first data function, and y is the first data;

According to the formula:

Calculate the time value L, and determine whether the time value L is greater than a preset time threshold, where p is a preset parameter value, and p is greater than 0;

If the time value L is greater than a preset time threshold, the generation function H(t) acquires an instruction.

4. the data early warning method based on knowledge graph according to claim 3, is characterized in that, described according to formula:

Calculate the time value L, and judge whether the time value L is greater than a preset time threshold, where p is a preset parameter value, and before the step where p is greater than 0, includes:

Acquire historical data of the same type as the first data from a preset database, wherein a threshold of dangerous data is recorded in the historical data, and the threshold of dangerous data refers to dividing the historical data into normal data and abnormal data the demarcation line;

The value of the parameter value p is set as the risk data threshold.

5. The data early warning method based on knowledge graph according to claim 1, characterized in that, if the normal data time ratio is greater than a preset ratio threshold, then fetching from a preset knowledge graph library includes: The knowledge graph of the designated member, and before the step of acquiring an associated member that has a direct connection relationship with the designated member from the knowledge graph including the designated member, including:

Identify an initial entity from pre-collected specified information by using a preset knowledge graph construction tool, wherein the specified information records at least the specified member, and the initial entity includes at least the specified member;

performing deduplication processing on the initial entity to obtain the final entity;

The relationship between the final entities is extracted from the specified information to form a triple, and the knowledge graph including the specified member is generated according to the triple.

6. The data early warning method based on a knowledge graph according to claim 1, wherein the second data of the associated member is obtained, and according to a preset data abnormality judgment algorithm, it is judged whether the second data is Unusual steps, including:

Obtain the second data of the associated member, and extract the maximum value and the minimum value from the second data;

Judging whether the time point at which the maximum value appears is within the first preset time range, and whether the time point at which the minimum value appears is within the second preset time range;

If the time point at which the maximum value appears is within the first preset time range, and the time point at which the minimum value appears is within the second preset time range, it is determined that the second data is normal.

7. The data early warning method based on knowledge graph according to claim 1, wherein, if the second data is abnormal, generate early warning information, and attach the information of the associated member in the early warning information. The second data steps include:

If the second data is abnormal, obtain the influence trend of the second data on the designated member according to the mutual influence relationship of the knowledge nodes including the knowledge graph of the designated member;

Early warning information is generated, and the second data of the associated member and the influence trend of the second data on the designated member are attached to the early warning information.

8. A data early warning device based on knowledge graph, characterized in that, comprising:

The first data function generation unit is used to obtain the first data of a designated member by using a preset data collection technology, perform noise reduction processing on the first data, and generate the first data according to the noise reduction processed first data. a first data function of the data over time;

The function H(t) generates cells for use according to the formula:

H(t)=min(G(t),m), where

The normal data time proportion acquisition unit is used to obtain the first time length when the function H(t) is not equal to m on the time axis and the second time length when it is equal to m, and according to the formula: the normal data time proportion = the first time length/(the first time length + the second time length), and the normal data time ratio is calculated;

a preset proportion threshold judgment unit, configured to judge whether the normal data time proportion is greater than the preset proportion threshold;

A knowledge graph retrieval unit, configured to retrieve a knowledge graph including the specified member from a preset knowledge graph library if the normal data time proportion is greater than a preset proportion threshold, and retrieve the knowledge graph including the specified member from the Obtain an associated member that has a direct connection relationship with the specified member from the knowledge graph of the specified member;

a second data judging unit, configured to obtain the second data of the associated member, and judge whether the second data is abnormal according to a preset data abnormality judgment algorithm;

An early warning information generating unit is configured to generate early warning information if the second data is abnormal, and attach the second data of the associated member to the early warning information.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when the processor executes the computer program .

10. A computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.