JP2015028700A - Failure detection device, failure detection method, failure detection program and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a failure detection device capable of detecting a failure with high accuracy by observing and analyzing a plurality of kinds of states in parallel in a monitoring system including a monitoring server for monitoring a silent failure.SOLUTION: The failure detection device for monitoring an occurrence of a silent failure of a monitoring object host includes: an abnormality determination part for comparing a system log in the monitoring object host with a normal state model being a log transition of a system log existing in the past in the monitoring object host to determine an abnormality; and a failure estimation part for taking a negative event due to at least one observation data among SNS information, call center information in a carrier, user behavior information, and the number of accesses to a service into consideration, and comparing a word appearance distribution of the preceding system log with a result estimation model being a word appearance distribution within a fixed time related to a new transition appearance time of the past system log to thereby estimate a silent failure in the case that the abnormality determination part determines an abnormality.

Description

  The present invention relates to a failure detection system including a monitoring server that monitors a silent failure (a failure that does not cause a symptom even if it occurs) of a plurality of monitored hosts, and in particular, a failure detection device that efficiently detects the occurrence of a silent failure. The present invention relates to a failure detection method, a failure detection program, and a recording medium.

  Conventionally, with respect to silent failures in server systems such as web servers and mail servers, a syslog, which is a text log output from a server that is a monitored host, is monitored, CPU load, memory usage, I / O A method for determining whether or not a silent failure has occurred has been performed by checking resource information such as the number of O waits and the number of packets.

For example, according to abnormality detection by threshold monitoring, designated monitoring items such as CPU usage rate and memory usage are periodically monitored, and abnormality is detected by whether or not the threshold value set by the observed value is raised or lowered. It corresponds to general monitoring tools (Zabbix, Nagios, etc.) that use life and death monitoring and resource monitoring technology.
In addition, according to abnormality detection by correlation analysis, the existence of a time-series universal relationship such as resource information and performance information is discovered, modeled, and monitored, and abnormality is detected by investigating behavior different from the normal model.
According to anomaly detection by state transition pattern analysis, time series transition patterns of log records and resource information are modeled and monitored, and anomalies are detected by investigating transition patterns not found in the normal model.
Techniques for detecting an abnormal operation of the system or process described above are disclosed in, for example, Patent Literature 1 to Patent Literature 3 and Non-Patent Literature 1 and 2.

Japanese Patent Application No. 2012-275113 JP 2012-094046 A JP 2010-250502 A

The Institute of Electronics, Information and Communication Engineers Vol100, P50-P60 NEC Technical Journal Vol.63 No.2 / 2010. "Data center operation in the cloud era realized by WebSAM Ver.8"

However, with the method described above, the presence / absence of a failure is determined based only on resource information, so accurate determination cannot be made regarding the occurrence of a failure (the rate of detecting abnormalities unrelated to failures is high, and detected abnormalities There was a problem that it was difficult to understand what kind of obstacle was caused by this.
In addition, since it is necessary to periodically monitor resource information for all monitored hosts, there is a problem that the amount of traffic to be monitored increases and the monitoring load also increases.

In conventional detection techniques such as correlation analysis and transition pattern analysis, the monitoring system maintains a model indicating the normal state inside the system, and compares the current state that is regularly observed with the normal state model to determine whether there is an abnormality. If it is abnormal, an alarm is issued. Therefore, in order to prevent the state that is determined to be abnormal from continuing and alarms from being issued continuously, the abnormality once detected is learned to the model as a normal state, and thereafter even if a similar state occurs, Has a mechanism that does not recognize (automatic learning function).
However, when this automatic learning function is used, even if the same type of abnormality occurs multiple times, it is detected only once, so the scale is small (the number of cases is small), but the abnormality continues to occur. There was a problem that could not be detected.

  The present invention has been proposed in view of the above circumstances, and in a monitoring system including a monitoring server for monitoring a silent failure, a plurality of types of states can be set in parallel while reducing the monitoring load when monitoring a monitored host. It is an object of the present invention to provide a failure detection device, a failure detection method, a failure detection program, and a recording medium that can accurately detect a failure by performing observation analysis.

In order to achieve the above object, the present invention is characterized in that a failure detection apparatus for monitoring the occurrence of a silent failure of a monitored host includes the following configuration.
An abnormality determination unit that determines an abnormality by comparing a system log in the monitored host with a normal state model that is a log transition of a system log that has existed in the past in the monitored host.
When an abnormality is determined by the abnormality determination unit, a negative event due to at least one observation data among the SNS information, call center information in the carrier, user behavior information, and the number of accesses to the service is taken into account. A fault estimator that estimates a silent fault by comparing a word appearance distribution in a log with a result prediction model that is a word appearance distribution within a certain time linked to a new transition appearance in a past system log.

According to a second aspect of the present invention, a failure detection apparatus for monitoring the occurrence of a silent failure in a monitored host includes the following configuration.
An initial setting unit that reads a log transition of a system log that has existed in the past in the monitored host as a normal state model, and reads a word appearance distribution within a certain time associated with the appearance of a new transition in the past system log as a result prediction model.
An information collection unit that reads a system log and SNS information, call center information in a carrier, user behavior information, and at least one observation data among access counts by log transition monitoring in the monitoring target host.
A data information processing unit that processes the system log into a format comparable to each model.
An abnormality determination unit that determines an abnormality by comparing the system log and the normal state model.
A failure for estimating a silent failure by comparing a word appearance distribution of the immediately preceding system log with the result prediction model while considering a negative event based on the observation data when an abnormality is determined by the abnormality determination unit Estimator.
A model generation unit that generates and holds a normal state model based on a learning effect and generates and holds a result prediction model in which a silent failure is estimated by the failure estimation unit.
A timer management unit that manages a non-learning period in which the abnormality determination unit observes a new transition of the system log without learning for a certain period in the model creation unit.

Claim 3 is a method for monitoring the occurrence of a silent failure of a monitored host.
A normal state model is generated and held for each of the monitored host system log and at least one of the SNS information, call center information in the carrier, user behavior information, and number of accesses to the service, and a plurality of types of observation items A procedure for observing whether there is an abnormality by combining
A word appearance distribution within a certain period of time associated with the appearance of a new transition in the past system log of the monitored host is retained as a result prediction model, and a word appearance distribution of the immediately preceding system log when abnormality is simultaneously observed in a plurality of observation items And a procedure for determining the presence or absence of silent failure by comparing the result prediction model,
It is characterized by having.

  4. The failure detection method according to claim 3, wherein the result prediction model includes word appearance distribution (appearance rate) information associated with the number of detections of new transitions, or word appearances associated with category elements of new transitions. It is formed by distribution (appearance rate) information.

  According to a fifth aspect of the present invention, in the failure detection method of the third aspect, the result prediction model is compared with the state of the immediately preceding system log by an analysis method such as Mahalanobis distance, Z test, or T test.

  According to a sixth aspect of the present invention, in the failure detection method of the third aspect, learning is not performed for a period specified by the operator from when the number of new transition pairs detected by the log transition monitoring exceeds a threshold specified by the operator. By observing whether the new transition continues for several minutes exceeding the threshold every hour, noise is removed, and the state exceeding the threshold that continues inconspicuously is watched as an abnormality and the possibility of silent failure is It is characterized by doubt.

  According to a seventh aspect of the present invention, in the fault detection method of the sixth aspect, all new transitions exceeding the threshold value are continuously observed.

  The fault detection method according to claim 6 is characterized in that, in the fault detection method according to claim 6, only new transitions including a specific keyword are continuously observed as new transitions exceeding the threshold.

  A ninth aspect is characterized in that, in the failure detection method according to the third aspect, an operator can designate a plurality of observation targets at the start of system operation or during operation.

  Claim 10 stores the state determined as a failure as a failure state model in the failure detection method according to claim 3, and if a similar state is detected thereafter, information extracted from a past failure state model is stored. It is characterized in that it can be presented as a reference value.

  The failure detection method according to claim 11 is characterized in that, in the failure detection method according to claim 3, it is possible to cause a failure model to learn a state determined as a failure at the start of system operation or during operation by an instruction from an operator. .

  A twelfth aspect is characterized by a failure detection program capable of executing, by a computer, each procedure of the failure detection method for a monitored host according to the third aspect.

  A thirteenth aspect is a computer-readable recording medium storing the fault detection program according to the twelfth aspect.

  According to the present invention, the system log from the server system (monitoring target host), SNS (sowing information, etc.), inquiry status from customers (call center information in the carrier, etc.), behavior information (Web access information, GPS information, etc.) , Observing a combination of multiple types of observation data for the number of accesses to the service, and suspecting a failure when anomalies are simultaneously found in multiple types of observation items, Silent failure is estimated by comparing the result prediction model, which is the word appearance distribution within a certain time linked to the appearance of new transitions in the system log, so the server can be analyzed by considering abnormalities in other observation results along with log analysis. It is possible to improve the accuracy of determining whether or not a failure has occurred in the system.

  Also, the period of time specified by the operator (for example, several hours or two days) is not learned from the time when the number of abnormalities detected in the observation target exceeds the threshold specified by the operator, and the abnormality continues for a certain period of time. By observing whether or not it is possible to prevent a situation in which the presence or absence of abnormality cannot be determined by learning dynamically.

It is the whole system block diagram in the case of collecting the system log of various server systems and the tweet information by SNS, and detecting a failure using the failure detection device of the present invention. It is a block diagram which shows the structure of the failure detection apparatus of this invention. It is a block diagram which shows the structure of the abnormality determination part of a failure detection apparatus. It is explanatory drawing at the time of modeling by clustering with respect to the log information read by the monitoring apparatus. It is a model figure which shows the example of generation | occurrence | production of the abnormal pattern in the server system which a failure detection apparatus monitors.

An example of an embodiment of a monitoring system including a failure detection device that monitors the occurrence of a silent failure of a monitored host will be described with reference to FIGS.
As shown in FIG. 1, the failure detection apparatus (monitoring server) 10 collects system log information from a plurality of server systems (monitoring target hosts) 1 to be monitored and information from the social network (SNS) 2. Are compared with the normal model, and if both are abnormal, a silent failure is suspected, and a result prediction model that is a word appearance distribution within a certain time linked to the appearance of a new transition in the past system log and The presence of a silent failure is estimated by performing verification, and the failure detection result is notified to the operator of each server system (monitored host) 1 by issuing an alarm by e-mail, screen display, voice, etc. It is.

  The failure detection device (monitoring server) 10 stores a basic program including a general operating system (OS) such as Linux (registered trademark) and a ROM that stores various basic devices, and various programs and data. A hard disk drive (HDD), a media drive that reads programs and data from a storage medium such as a CR-ROM or DVD, a CPU that executes the program, a RAM that provides a work area for the CPU, and a communication with an external device It is configured by a computer having a parallel / serial IF as a main component, and a monitoring program is stored in the HDD via a recording medium or the like so that silent failure of various server systems 1 to be monitored can be detected. It is configured.

  As shown in FIG. 2, the failure detection apparatus (monitoring server) 10 includes an initial setting unit 11 that reads initial setting parameters and past models, an information collection unit 12 that reads a plurality of observation data, and data information that processes the read data. The processing unit 13, the abnormality determination unit 14, the failure estimation unit 15, the model generation unit 16, the timer management unit 17, and the alarm issue unit 18 are configured.

The initial setting unit 11 reads a log transition of a system log that has existed in the past in various server systems (monitoring target hosts) 1 as a normal state model, and a word appearance within a certain time associated with a new transition appearance of a past system log. Load the distribution as a result prediction model.
As a result prediction model, for example, if a new transition of such a pattern appears, a prediction such as “the word appearance rate in the log in a certain period before that is expected to have such a distribution” It is composed of the distribution model of the word appearance rate created by performing based on the above data.

The specific result prediction model X (word appearance distribution) is
The word distribution model is
Word 1: 60%
Word 2: 50%
Word 3: 35%
...
Word N: ~%
As shown in FIG. 5, the word is composed of data obtained by picking up the top N words having an appearance rate.

  When comparing the result prediction model with the actual system log distribution model, if the actual system log distribution model does not match the distribution model predicted by the result prediction model, an unknown failure (silent failure) ) Can be suspected.

If there is information on a fault state model generated in the past in the initial setting unit 11, it is read as a fault state model.
For example, (1) log transition patterns in various server systems 1 when it is determined that a failure has occurred in the past,
(2) List of the number (rate) of negative whispering in SNS information 2
(3) word appearance distribution in the latest log message in various server systems 1;
Is stored as a fault condition model.
Each information read into the initial setting unit 11 is shared on a readable memory among the information collection unit 12, the data information processing unit 13, the abnormality determination unit 14, the failure estimation unit 15, and the model generation unit 16. .
Also, initial setting parameters relating to various settings when performing failure detection are read.

  The information collection unit 12 includes a system log by log transition monitoring in each server system (monitoring target host) 1, SNS information such as whispering, call center information in a carrier, user behavior information such as Web access information and GPS information, and service information Load multiple types of observation data such as number of accesses.

  The data information processing unit 13 is a system log and observation data collected by the information collecting unit 12 that can be compared with the normal state model, the failure state model, and the result prediction model read into the initial setting unit 11. Processing into a form is performed.

The abnormality determination unit 14 determines an abnormality by comparing the system log (cause information source) with the normal state model (cause information source normal model) and the observation data (result information source) with the normal state model (result information source normal model). To do.
The abnormality determination unit 14 includes a log reading unit 21, a log clustering unit 22, and a log analysis unit 23, as shown in FIG. 3, in order to determine whether there is an abnormality by analyzing the system log. ing. In the log reading unit 21, various log information (syslog) generated every day in each server system 1 is periodically read for each server system via the information collecting unit 12 and the information processing unit 13.

  The log clustering unit 22 classifies the irregular log information (syslog) read for one server system (virtual host) 1 for each category. The log information (syslog) includes information such as time, host name, and program name. The categories are classified into a plurality of categories according to the number and types of items constituting the log information (syslog). For example, as shown in FIG. 4, for each log information read from one server system (virtual host), the number and type of items (log information omitted from Aug719: 00: 04 to Vpxa are different) ) Are classified into categories A, B, C, D.

The model generation unit 16 creates a transition model of log information at the normal time based on the log information classification result. For example, in the case of FIG. 4 in which log information is classified into categories A, B, C, and D, “category A to B” “category A to D” “category B to C” “category” C to A "are registered (modeled).
The log information transition model at the normal time is registered in advance as a plurality of transition models from one month of log information having no failure, for example.
Further, the transition model of log information without a failure may be configured to be always added as a normal model.

The log analysis unit 23 compares the log classification result of the newly read log information with the transition model created by the model generation unit 16, and recognizes that the log transition based on the log classification result is not in the transition model as a log change. Processing is performed. For example, when “category A to B”, “category A to D”, “category B to C”, and “category C to A” are registered (modeled) as normal models, “category B to C” “category C” To B ”,“ Category A to B ”,“ Category B to A ”, and“ Category C to A ”are detected because“ Category C to B ”and“ Category B to A ”are not in the normal model. This means that a transition (abnormality detection by authorization of log change) has been detected.
When the log analysis unit 23 detects a log change (when abnormality is determined), a failure estimation instruction signal is output to the failure estimation unit 15, and the failure estimation unit 15 that has received this signal has a plurality of types. A process for estimating a fault from the observed data and the result prediction model is performed.

That is, the failure estimation unit 15 considers a negative event based on the observation data when the abnormality determination unit 14 determines abnormality, and compares the word appearance distribution of the immediately preceding system log with the result prediction model described above. Thus, the presence or absence of silent failure is estimated.
A result prediction model that is a word appearance distribution within a certain time associated with a new transition appearance in the past system log is read in the initial setting unit 11 in advance, but a pattern in which a new transition is formed is ( There are a) a new transition formed by a combination of existing categories, and (b) a new transition including a new category.
In the case of the combination of existing categories in (a), according to the above-described example, “category A to B”, “category A to D”, “category B to C”, and “category C to A” are registered as normal models (modeled). "Category C to B" and "Category B to A" which are not in the normal model.
(B) A new transition including a new category corresponds to a combination including a new category.

  In this example, the result prediction model, which is a word appearance distribution within a certain period of time associated with the appearance of a new transition in the past system log, is formed by word appearance distribution (appearance rate) information associated with the category component of the new transition. However, you may make it form with the word appearance distribution (appearance rate) information linked | related with the detection number of new transition.

The model generation unit 16 generates and holds (learns) a normal state model and also generates and holds a result prediction model in which the silent failure is estimated by the failure estimation unit 15.
When the model generation unit 16 holds the prediction model as a result of the silent failure estimated by the failure estimation unit 15, the abnormality determination unit 14 determines that the log transition is the same as the log transition pattern of the failure state model At times, the number of occurrences (rate) of negative whispering in a certain section is greater than or equal to the fault state model, and the word distribution by the general distribution similarity calculation method in the most recent log message collected is similar to the fault state model In this case, the failure estimation unit 15 unconditionally estimates a silent failure.
Further, for the three items (1) to (3) described in paragraph number 0031, a failure may be estimated using an AND condition.

The timer management unit 17 manages a non-learning period in which the abnormality determination unit 14 observes a new transition of the system log without learning for a certain period in the model creation unit 16.
That is, the number of new transition pairs detected by log transition monitoring does not learn for the period specified by the operator from the time when the threshold exceeds the threshold specified by the operator, and continues to appear for the number of times when the new transition exceeds the threshold every hour. By observing this, the possibility of a silent failure is suspected by removing the noise and observing the state exceeding the threshold that is inconspicuous and observing it as an abnormality.
As a result, by not performing learning for a certain period of time, noise (abnormality due to temporary work or the like that is not a failure) can be ignored by observing continuously without immediately considering it as a failure. In addition, since the silent failure symptom is inconspicuous and often worsens little by little, it is possible to easily recognize the failure due to such an event.

Next, a procedure for detecting a silent failure of various server systems (monitoring target hosts) 1 using the above-described failure detection apparatus (monitoring server) 10 will be described with reference to FIG.
As a procedure 1, the number (rate) of negative whispers appearing from the SNS information 2 is regularly observed, and the observation results are accumulated.
As procedure 2, a normal state log from each server system 1 is read, and a normal state model of log transition is created.
As a procedure 3, a log is periodically read from each server system 1 and new transitions are monitored (observed whether they occur continuously for a certain period of time).

As Step 4, if new transitions are detected continuously and the number of negative applications (rate) within the detected period exceeds the threshold, a failure is suspected from both abnormalities and the process proceeds to the next step .
As procedure 5, the word appearance distribution (result prediction model) associated with the detected new transition pattern and the word appearance distribution in the latest log message are checked.

The collation between the result prediction model and the latest system log is determined, for example, when the corresponding result prediction model is the result prediction model X (word appearance distribution) described above and the word distribution model is not different. If they are different, an event that cannot be explained by the previous explanatory variable is detected, and a silent failure is suspected.
The determination of whether the result prediction model X (word appearance distribution) is different from the word appearance rate distribution model of the immediately preceding log is based on, for example, the appearance rate of each word in the distribution model of the word appearance rate of the previous log. Judgment is made based on the degree of coincidence with the appearance rate of each word in the model X. The determination criterion based on the degree of coincidence (deviation) can be set in advance by input to the initial setting unit 11.
Further, when the result prediction model is compared with the latest system log, it may be compared with the state of the immediately preceding system log by an analysis method such as Mahalanobis distance, Z test, T test, or the like.

As step 6, if the observed recent word appearance distribution is far from the result prediction model (if the prediction accuracy is low), an unknown failure is suspected and appears unexpectedly in the actual system log distribution model Or by notifying the operator of the words that did not appear, it can be provided as reference information for investigating the cause of silent failure.
As a procedure 7, the result prediction model is learned by the model generation unit 16 as a known failure.
Further, the state determined to be a failure may be learned by manual operation in the failure model at the start of the system operation or during operation according to an instruction from the operator.

FIG. 5 shows an example in which a new log transition and SNS information (appearance of negative whispering) 2 are monitored for a certain period when a silent failure of various server systems (monitoring target hosts) 1 is detected.
In FIG. 5, the number of failures in each system log detected by the bar graph is below a threshold value that is suspected of being a failure in a single occurrence, but an abnormality that is lower than the threshold value is continuously occurring (state A). In that period, if the number of negative whispers (or appearance rate) is equal to or greater than the threshold (state B), a silent failure is suspected from state A and state B.
That is, in the state B, in the SNS information 2 scatter information, if the number of negative words (or appearance rate) is less than the threshold set by the operator in a certain period, the normal state is assumed. Judge.

  In addition, the number of new transitions exceeding the threshold may be continuously observed, or only new transitions including a specific keyword may be continuously observed.

In FIG. 5, the state B in the SNS information 2 is an example of Twitter information (what kind of whispering), but as an alternative (or added) criterion,
・ Call center information (what kind of inquiries are received from customers)
・ Web access log (what kind of website is being accessed)
・ GPS information (where you are moving)
・ The number of accesses to the service can also be considered.
These pieces of information can also be one of the state determination elements for realizing the failure detection method of the present invention because the abnormality can be identified by defining the normal state.
In addition, regarding which of a plurality of observation targets to use as observation data, the operator may be able to specify when the system operation of the failure detection apparatus starts (when initial setting parameters are input) or during operation. .

In these, the method in the case of judging normality and abnormality is demonstrated below.
・ In the case of call center information (what kind of inquiries are received from customers) The number of negative words (or appearance rate) set by the operator in a certain period in the memo and text data of responders If it is less than the threshold, it is determined to be normal, and if it is greater than or equal to the threshold, it is determined to be abnormal.

・ When using Web access logs (what type of website is being accessed) Modeling the ranking of the number of access to the services provided using the observation target system and the homepage related to the company providing the service in a normal state In addition, the number of accesses to each home page in a certain period is compared with the model to determine an abnormality.
For example, the number of accesses to each homepage provided by KDDI is modeled in a normal state, and when there is a sudden increase in access to HPs that are rarely accessed, it is determined that there is an abnormality.
In addition, in the keyword searched in the related home page, it is also determined as abnormal when the number of negative words (or appearance rate) is equal to or greater than a threshold set by the operator in a certain period.

・ In the case of location information (where you are moving) Modeling normal movement (GPS history) from GPS information of the terminal connected to the service provided using the observation target system In addition, when the number of terminals that have moved to a position not included in the model in a certain period is equal to or greater than a threshold set by the operator, it is determined as abnormal.

・ In the case of the number of access to the service The number of connection requests for the service provided using the observation target system is modeled at normal time, and the number of connection requests to the service in a certain period is set by the operator in a certain period When the value is above or below the upper and lower thresholds, it is determined as abnormal.

  According to the above-described failure detection device, the word appearance distribution of the immediately preceding system log is compared with the result prediction model that is the word appearance distribution within a certain period of time associated with the new transition appearance of the past system log, and a plurality of types Therefore, it is possible to improve the accuracy of determining the detected abnormality as a silent failure.

  Even when there is an abnormality in the log transition of the system log, noise (abnormality due to temporary work etc. that is not an obstacle) can be ignored by observing continuously without being regarded as an obstacle immediately. In addition, since the symptom of silent failure is not conspicuous and often deteriorates little by little, the failure can be easily recognized.

  By identifying the outcome prediction model and co-occurrence variables, it is possible to identify a new cause (an unknown cause) correlated with the event. That is, an unknown state can be detected by finding an abnormal state that is far from expected.

  DESCRIPTION OF SYMBOLS 1 ... Server system (monitoring target host), 2 ... SNS information, 10 ... Failure detection apparatus (monitoring server), 11 ... Initial setting part, 12 ... Information collection part 12, 13 ... Data information processing part, 14 ... Abnormality determination part 15 ... Fault estimation unit, 16 ... Model generation unit, 17 ... Timer management unit, 18 ... Alarm issuing unit, 21 ... Log reading unit, 22 ... Log clustering unit, 23 ... Log analysis unit.

Claims (13)

In the failure detection device that monitors the occurrence of silent failure on the monitored host,
An abnormality determination unit that determines an abnormality by comparing a system log in the monitored host with a normal state model that is a log transition of a system log that has existed in the past in the monitored host;
When an abnormality is determined by the abnormality determination unit, a negative event due to at least one observation data among the SNS information, call center information in the carrier, user behavior information, and the number of accesses to the service is taken into account. A failure estimation unit that estimates silent failure by comparing a word appearance distribution of a log with a result prediction model that is a word appearance distribution within a certain period of time associated with a new transition appearance of a past system log;
A failure detection apparatus comprising: In the failure detection device that monitors the occurrence of silent failure on the monitored host,
An initial setting unit that reads a log transition of a system log that existed in the past in the monitored host as a normal state model, and that reads a word appearance distribution within a certain time associated with a new transition appearance of a past system log as a result prediction model; ,
An information collection unit that reads a system log, SNS information, call center information in a carrier, user behavior information, and at least one observation data of the number of accesses to a service by log transition monitoring in the monitored host;
A data information processing unit that processes the system log into a format comparable to the models;
An abnormality determination unit that determines an abnormality by comparing the system log and the normal state model;
A failure for estimating a silent failure by comparing a word appearance distribution of the immediately preceding system log with the result prediction model while considering a negative event based on the observation data when an abnormality is determined by the abnormality determination unit An estimation unit;
A model generation unit that generates and holds a normal state model based on a learning effect, and generates and holds a result prediction model in which a silent failure is estimated by the failure estimation unit;
A timer management unit that manages a non-learning period in which a new transition of the system log is observed in the abnormality determination unit without learning for a certain period in the model creation unit;
A failure detection apparatus comprising: In the method for monitoring the occurrence of a silent failure on a monitored host,
A normal state model is generated and held for each of the monitored host system log and at least one of the SNS information, call center information in the carrier, user behavior information, and number of accesses to the service, and a plurality of types of observation items A procedure for observing whether there is an abnormality by combining
A word appearance distribution within a certain period of time associated with the appearance of a new transition in the past system log of the monitored host is retained as a result prediction model, and a word appearance distribution of the immediately preceding system log when abnormality is simultaneously observed in a plurality of observation items And a procedure for determining the presence or absence of silent failure by comparing the result prediction model,
A failure detection method characterized by comprising:   4. The result prediction model is formed by word appearance distribution (appearance rate) information associated with the number of new transitions detected or word appearance distribution (appearance rate) information associated with a category component of a new transition. Failure detection method.   The failure detection method according to claim 3, wherein the result prediction model is compared with the state of the immediately preceding system log by an analysis method such as Mahalanobis distance, Z test, or T test.   The number of sets of new transitions detected by the log transition monitoring exceeds the threshold specified by the operator and is not learned for the period specified by the operator, and continues to appear for the number of times the new transition exceeds the threshold every hour. The failure detection method according to claim 3, wherein the possibility of a silent failure is suspected by observing the state exceeding the threshold value that is not conspicuously continued by removing noise and capturing it as an abnormality.   The fault detection method according to claim 6, wherein all new transitions exceeding the threshold are continuously observed.   The fault detection method according to claim 6, wherein new transitions exceeding the threshold are continuously observed only for new transitions including a specific keyword.   The failure detection method according to claim 3, wherein an operator can designate a plurality of observation targets at the start of system operation or during operation.   The failure according to claim 3, wherein a state determined as a failure is stored as a failure state model, and when a similar state is detected thereafter, information extracted from a past failure state model can be presented as a reference value. Detection method.   The failure detection method according to claim 3, wherein the failure model can be made to learn a state determined to be a failure at the start of system operation or during operation according to an instruction from an operator.   A failure detection program capable of executing, by a computer, each procedure of the failure detection method for a monitored host according to claim 3.   A computer-readable recording medium in which the failure detection program according to claim 12 is stored.