WO2023042301A1

WO2023042301A1 - Information processing device, information processing method, and recording medium

Info

Publication number: WO2023042301A1
Application number: PCT/JP2021/033932
Authority: WO
Inventors: 紗和子梅津
Original assignee: 日本電気株式会社
Priority date: 2021-09-15
Filing date: 2021-09-15
Publication date: 2023-03-23
Also published as: JPWO2023042301A1

Abstract

An input data acquiring means that acquires a plurality of pieces of input data, and an input data evaluating means that detects outliers and abnormal values in the input data. A prediction data acquiring means that acquires prediction data generated from the input data by using a trained model, and a prediction data evaluating means that detects abnormal values in the prediction data. A prediction accuracy acquiring means that acquires a prediction accuracy from a trained model. A display means that displays diagnostic information including whether the input data has outliers or abnormal values, whether the prediction data has abnormal values, and the prediction accuracy.

Description

Information processing device, information processing method, and recording medium

This disclosure relates to monitoring machine learning models.

In recent years, in various fields, work is being done using prediction models obtained by machine learning. As time passes from the creation of the initial model and the tendency of the data used for prediction changes, the prediction accuracy decreases and the model needs to be re-learned. Patent Literature 1 discloses a method of generating an inspector model for detecting deterioration in accuracy of an operational model and using this to detect changes in output results of the operational model due to temporal changes in data trends.

International publication WO2021/079458

　In order to appropriately determine the timing of re-learning for the model in operation, it is preferable to create and present diagnostic information on multiple factors that affect the prediction accuracy of the model.

One object of the present disclosure is to provide an information processing device capable of creating and presenting diagnostic information regarding a plurality of factors affecting prediction accuracy for a model in operation.

In one aspect of the present disclosure, an information processing device includes:
input data acquisition means for acquiring a plurality of input data;
input data evaluation means for detecting outliers and abnormal values in the input data;
prediction data acquisition means for acquiring prediction data generated from the input data using the trained model;
prediction data evaluation means for detecting abnormal values in the prediction data;
prediction accuracy acquisition means for acquiring prediction accuracy by the trained model;
display means for displaying diagnostic information including the presence or absence of outliers and abnormal values in the input data, the presence or absence of abnormal values in the prediction data, and the prediction accuracy;
Prepare.

In another aspect of the present disclosure, an information processing method includes:
get multiple input data,
detecting outliers and abnormal values in the input data;
Obtaining prediction data generated from the input data using the trained model,
detecting abnormal values in the predicted data;
Obtaining the prediction accuracy of the trained model,
Diagnosis information including presence/absence of outliers and abnormal values in the input data, presence/absence of abnormal values in the predicted data, and the prediction accuracy is displayed.

In yet another aspect of the present disclosure, the recording medium comprises
get multiple input data,
detecting outliers and abnormal values in the input data;
Obtaining prediction data generated from the input data using the trained model,
detecting abnormal values in the predicted data;
Obtaining the prediction accuracy of the trained model,
A program for causing a computer to execute a process of displaying diagnostic information including the presence or absence of outliers and abnormal values in the input data, the presence or absence of abnormal values in the predicted data, and the prediction accuracy is recorded.

According to this disclosure, it is possible to create and present diagnostic information regarding multiple factors that affect prediction accuracy for models in operation.

It is a block diagram showing the whole monitoring system composition concerning a 1st embodiment. It is a block diagram which shows the hardware constitutions of the monitoring apparatus of 1st Embodiment. It is a block diagram which shows the functional structure of the monitoring apparatus of 1st Embodiment. It is a flow chart of monitoring processing by a monitoring device. 4 shows an example of a diagnostic report included in display data. An example of an analysis result screen is shown. 13 shows another example of the analysis result screen. 13 shows another example of the analysis result screen. 13 shows another example of the analysis result screen. 13 shows another example of the analysis result screen. 13 shows another example of the analysis result screen. It is a block diagram which shows the functional structure of the information processing apparatus of 2nd Embodiment. 9 is a flowchart of processing by the information processing apparatus of the second embodiment;

Preferred embodiments of the present disclosure will be described below with reference to the drawings.
<First embodiment>
[overall structure]
FIG. 1 is a block diagram showing the overall configuration of a monitoring system 1 according to the first embodiment. The monitoring system 1 is a system that monitors the state of a machine learning model that has been learned in advance and is in operation. In this embodiment, the monitoring system 1 includes a prediction device 2 and a monitoring device 100 .

The prediction device 2 is a device that makes predictions using a prediction model. A prediction model is an example of a machine learning model to be monitored by the monitoring system 1, and is a model that has already been trained using learning data. The prediction device 2 makes a prediction based on the input data D1, generates prediction data D2 as a prediction result, and outputs the prediction data D2 to the monitoring device 100. FIG.

The monitoring device 100 evaluates whether the input data D1 is normal and evaluates whether the prediction data D2 generated by the prediction device 2 is normal. Also, the monitoring device 100 is input with performance data D3. The performance data D3 is data corresponding to the input data D1 and data actually obtained in the real world. The monitoring device 100 evaluates the prediction accuracy of the prediction model by calculating the error rate between the prediction data D2 generated by the prediction device 2 and the performance data D3. Then, the monitoring device 100 generates display data including the input data D1, the prediction data D2, and the prediction accuracy evaluation result. By looking at the displayed data, the user can know the operation status of the prediction model and consider the necessity of re-learning the prediction model.

[Monitoring device]
(Hardware configuration)
FIG. 2 is a block diagram showing the hardware configuration of the monitoring device 100. As shown in FIG. As illustrated, the monitoring device 100 includes an interface (I/F) 11, a processor 12, a memory 13, a recording medium 14, a database (DB) 15, a display device 16, and an input device 17. Prepare.

The interface 11 performs data input/output with an external device. Specifically, the input data D1, the prediction data D2, and the performance data D3 are input to the monitoring device 100 through the interface 11. FIG.

The processor 12 is a computer such as a CPU (Central Processing Unit), and controls the entire monitoring device 100 by executing a program prepared in advance. The processor 12 may be a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). The processor 12 executes monitoring processing, which will be described later.

The memory 13 is composed of ROM (Read Only Memory), RAM (Random Access Memory), and the like. Memory 13 is also used as a working memory during execution of various processes by processor 12 .

The recording medium 14 is a non-volatile, non-temporary recording medium such as a disk-shaped recording medium or a semiconductor memory, and is configured to be detachable from the monitoring device 100 . The recording medium 14 records various programs executed by the processor 12 . When the monitoring device 100 executes various processes, a program recorded on the recording medium 14 is loaded into the memory 13 and executed by the processor 12 . DB15 memorize|stores the input data D1, the prediction data D2, and the performance data D3 which were input through I/F11 as needed.

The display device 16 is, for example, a liquid crystal display device, etc., and displays the monitoring results generated by the monitoring device 100 . The input device 17 is, for example, a mouse, a keyboard, etc., and is used by the user to make necessary instructions and inputs in the monitoring process.

(Functional configuration)
FIG. 3 is a block diagram showing the functional configuration of the monitoring device 100 of the first embodiment. The monitoring device 100 functionally includes an input data evaluation unit 21 , a prediction data evaluation unit 22 , a prediction accuracy evaluation unit 23 , and a display data generation unit 24 .

Input data D1 is input to the input data evaluation unit 21 . The input data evaluation unit 21 detects outliers and abnormal values in the input data D1. Here, the "outlier" is a value within a predetermined range that is defined as an impossible value for the input data D1. On the other hand, an "abnormal value" is a value within a predetermined range that deviates from the normal value when the value of the input data with which the prediction model can make an appropriate prediction is assumed to be the normal value. The input data evaluation unit 21 adds information indicating an outlier or an abnormal value to the input data D1 and outputs the data to the display data generation unit 24 .

Prediction data D2, which is the result of prediction by the prediction device 2, is input to the prediction data evaluation unit 22. The predicted data evaluation unit 22 detects abnormal values in the predicted data D2. Here, the "abnormal value" is a value within a predetermined range outside the normal value when the value of the prediction data output when the prediction model makes a proper prediction is assumed to be the normal value. The prediction data evaluation unit 22 adds information indicating an abnormal value to the prediction data D2 and outputs the data to the display data generation unit 24 .

The prediction accuracy evaluation unit 23 receives the prediction data D2 and the performance data D3. The prediction accuracy evaluation unit 23 calculates an error rate, which is an index indicating prediction accuracy, based on the prediction data D2 and the performance data D3. Here, the error rate is calculated by the following formula using the absolute value of the difference between the predicted data and the actual data.
(error rate) = | (predicted data value) - (actual data value) | / (actual data value)

Since the actual data D3 is the actual value corresponding to the input data D1, the error rate is a value that indicates the deviation of the prediction result from the actual value, and is used as an index that indicates the prediction accuracy of the prediction model. The prediction accuracy evaluation section 23 outputs the calculated error rate to the display data generation section 24 . In addition, in the present embodiment, the prediction accuracy evaluation unit 23 uses the error rate as an index indicating the prediction accuracy. may be evaluated.

The display data generation unit 24 uses the input data D1 input from the input data evaluation unit 21, the prediction data D2 input from the prediction data evaluation unit 22, and the error rate input from the prediction accuracy evaluation unit 23. , to generate the display data. The display data is data for displaying to the user the presence/absence of outliers and abnormal values in the input data D1, the presence/absence of abnormal values in the prediction data D2, and the prediction accuracy, and is output to the display device 16. FIG. The display device 16 displays the input display data on the screen. This allows the user to easily obtain information on the performance and status of the prediction model currently in operation.

(monitoring process)
FIG. 4 is a flow chart of processing by the monitoring device. This processing is realized by executing a program prepared in advance by the processor 12 shown in FIG. 2 and operating as each element shown in FIG.

First, the input data evaluation unit 21 acquires the input data D1, the prediction data evaluation unit 22 acquires the prediction data D2, and the prediction accuracy evaluation unit 23 acquires the prediction data D2 and the performance data D3 (step S11). Next, the input data evaluation unit 21 evaluates the input data D1, adds information indicating an outlier or an abnormal value to the input data D1, and outputs the evaluation result to the display data generation unit 24 ( step S12). Next, the prediction data evaluation unit 22 evaluates the prediction data, adds information indicating an abnormal value to the prediction data D2, and outputs the evaluation result to the display data generation unit 24 (step S13). Next, the prediction accuracy evaluation unit 23 calculates prediction accuracy (error rate) based on the prediction data D2 and the performance data D3, and outputs the evaluation result to the display data generation unit 24 (step S14).

Next, the display data generation unit 24 generates display data using the evaluation results of the input data evaluation unit 21, the prediction data evaluation unit 22, and the prediction accuracy evaluation unit 23, and outputs the display data to the display device 16 (step S15). The display device 16 displays the display data (step S16). Then the process ends.

(display data)
Next, the display data presented to the user will be described in detail. The display data includes diagnostic reports and, if necessary, analysis result screens. That is, as the monitoring result regarding the prediction model, only the above diagnostic report may be presented to the user, or the following analysis result screen may be presented in addition to the diagnostic report. For example, assume that a certain company A generates a prediction model and delivers it to a company B, and the company B uses the prediction model to operate business. In this case, the person in charge at Company A displays the diagnostic report and the analysis result screen, continues detailed monitoring of the forecast model delivered to Company B, and presents only the diagnostic report to Company B. A report on the state of the predictive model may be made.

(1) Diagnosis Report FIG. 5 shows an example of a diagnosis report included in display data. The diagnostic report is created based on the evaluation results by the input data evaluation unit 21, the prediction data evaluation unit 22, and the prediction accuracy evaluation unit 23, and other various data obtained by monitoring the prediction model, and is used to describe the current state of the prediction model. This report briefly describes the situation of

In the example of FIG. 5, the diagnostic report is a report on 10 diagnostic items. The diagnostic report includes an ID, description, evaluation result, countermeasures, and priority of countermeasures for each diagnostic item. "ID" is the identification information of each diagnostic item reported by the diagnostic report, and "Description" is the description of the diagnostic item. The "evaluation result" is the diagnosis result for each diagnostic item, and the "countermeasure" is the action proposed to improve the condition when the evaluation result of the diagnostic item is not good. "Response priority" indicates the order of priority of each countermeasure when a plurality of countermeasures are proposed.

Each diagnostic item will be explained in detail below. The diagnostic item 001, “existence of outlier in input data” indicates whether or not the input data is an outlier. An "outlier" is a value that cannot exist in reality. For example, in the real estate rent prediction model described later, "required time from the station" is used as input data. Negative values are set as outliers for "required time from station". In the example of FIG. 5, since the evaluation result of the diagnostic item 001 is "none", the diagnostic report indicates that there are no outliers in the input data D1.

"Presence or absence of abnormal value in input data" in the diagnostic item 002 indicates whether or not the input data is an abnormal value. The "abnormal value" here is a value that indicates that the tendency of the input data has changed, and specifically refers to a value that does not belong to the range of the input data used during learning of the prediction model. If the input data value corresponds to an abnormal value, it is determined that the input data value input to the prediction model has changed during actual prediction. In the example of FIG. 5, since the evaluation result of the diagnostic item 002 is "none", the diagnostic report indicates that there were no abnormal values in the input data.

"Presence or absence of abnormal value in prediction data" in the diagnostic item 003 indicates whether or not the prediction data generated by the prediction model is an abnormal value. The "abnormal value" here refers to a value that does not belong to the range of the actually obtained performance data or the correct data used during the learning of the prediction model. If the predicted data value corresponds to an abnormal value, it is suspected that the input data is an outlier or an abnormal value, or that the prediction accuracy of the prediction model is low. In the example of FIG. 5, since the evaluation result of the diagnostic item 003 is "none", the diagnostic report indicates that there is no abnormal value in the prediction data.

The "prediction accuracy" of the diagnosis item 004 indicates the accuracy of prediction by the prediction model, that is, the reliability of the model, and is indicated by the above-mentioned error rate, for example. In the example of FIG. 5, the evaluation result of the diagnostic item 004 is "40% lower than the initial", and the diagnostic report shows that the accuracy of the prediction model currently in use has decreased by 40% compared to when it was first used. It is shown that. For this reason, the diagnosis report proposes "re-learning" as a countermeasure, and the priority of this countermeasure is set to "1", which is the highest.

The diagnostic item 005 "whether exceptions occurred in data processing" indicates whether or not any problem occurred in the processing when the input data was processed before being input to the prediction model. "Data processing" means that, for example, instead of inputting certain input data as it is to a prediction model, when calculating and inputting the average value of a predetermined number or the moving average value of the last 7 days, those values means to calculate In addition to the case where the data cannot be processed for some reason, the defect includes the case where the value obtained by the data processing corresponds to an outlier of the input data. In the example of FIG. 5, the evaluation result of the diagnostic item 005 is "none", so the diagnostic report indicates that no exception occurred during data processing.

"Model creation time" in diagnostic item 006 indicates the time required to create a prediction model, that is, to learn a prediction model using learning data. The creation of the model here includes updating (relearning) of the model in addition to the creation of the initial model. Normally, the time required to create a prediction model can be predicted to some extent according to the amount of learning data to be used, conditions for terminating the learning process, and the like. Therefore, if the prediction model creation time is much shorter or longer than usual, it is doubtful whether the learning process was performed correctly. Therefore, it is diagnosed whether or not the model creation time was appropriate. In the example of FIG. 5, the evaluation result of the diagnostic item 006 is "within the regulation", so the diagnostic report indicates that the model creation time was appropriate.

Diagnosis item 007 "cause of fluctuation in prediction accuracy" indicates an item presumed to be the cause when the diagnosis item 004 diagnoses that the prediction accuracy has deteriorated. For example, in the case of the forecast model for real estate rents mentioned above, factors such as ``stagnation of economic activity due to the epidemic of infectious diseases'' and ``construction of luxury condominiums and shopping malls nearby'' can be considered as causes of fluctuations in forecast accuracy. be done. Note that the cause of the prediction accuracy variation may be estimated using an algorithm for evaluating a prediction model or an evaluation model, or may be estimated by humans. In the example of FIG. 5, the evaluation result of diagnostic item 007 is presumed to be "because XX has occurred and the tendency has changed". In addition, regarding this, the diagnostic report proposes a countermeasure to add YY to the explanatory variable, and the priority of the countermeasure is set to "2."

The diagnostic item 008 "whether or not appropriate system output is implemented" indicates whether or not the prediction data is correctly output from the prediction device using the prediction model. Prediction results using prediction models are usually input to other related systems and used. Therefore, it is diagnosed whether or not the prediction device is correctly outputting prediction data to another system. In the example of FIG. 5, the evaluation result of the diagnostic item 008 is "implemented", so the diagnostic report indicates that the prediction data is correctly output from the prediction device.

"Response time to system" of diagnostic item 009 indicates whether or not the prediction data is correctly exchanged between the prediction device using the prediction model and another system that receives and uses the prediction data. . Specifically, if the prediction device is designed to transmit prediction data to another system, the response time to the system is the response from the other system indicating receipt after the prediction device has transmitted the prediction data to the other system. It is time to receive. In addition, in the case of specifications in which another system requests prediction data from the prediction device, the response time to the system is the time from when the prediction device receives the request from the other system to when it transmits the prediction data to the other system. It's time. It is diagnosed whether or not these response times are within the specified time. In the example of FIG. 5, the evaluation result of the diagnostic item 009 is "within specification", and the diagnostic report indicates that the prediction data is correctly transmitted and received.

The diagnostic item 010 "display of caution information" displays various alerts such as alerts related to input data, prediction data and prediction accuracy, which will be described later, and alerts output when prediction data is not output from the prediction device. 16 indicates whether or not it is displayed correctly. In the example of FIG. 5, the evaluation result of the diagnostic item 010 is "displayed", and the diagnostic report indicates that various alerts are correctly displayed.

Of the above diagnostic items 001 to 010, diagnostic items 001 to 004 are diagnostic items mainly related to the state of the prediction model, and diagnostic items 005 to 010 are the overall operation of the prediction device that performs prediction using the prediction model. It is an item related to the situation.

As described above, the diagnostic report displays a list of diagnostic results for predetermined diagnostic items, so the overall status of the prediction model can be easily grasped. In the example of FIG. 5, the diagnosis item for which the diagnosis result is determined to be abnormal is highlighted by, for example, changing the color of the characters or the background, so as to alert the user that there is an abnormality. good too.

(2) Analysis Result Screen Next, the analysis result screen will be described in detail. The analysis result screen displays the diagnostic results by graphically showing the numerical grounds for the diagnostic items 001 to 004, which are particularly related to the prediction model, among the diagnostic items described in the diagnostic report.

In the explanation of the analysis result screen below, the prediction model will predict real estate rents. Specifically, the prediction model predicts the monthly rent of a rental property based on input data such as the size of the property, the floor plan, and the time required from the station. Note that the prediction model is a trained model that has been trained using input data such as the size of the actual rental property, the floor plan, and the time required from the station, and the actual rent of the rental property as correct data.

Fig. 6 shows an example of the analysis result screen. The analysis result screen 200 includes input data, prediction data, and an error rate indicating prediction accuracy as display items. As described above, there are multiple pieces of input data such as the size of the property, the floor plan, and the time required from the station. can be made Note that the example of FIG. 6 shows analysis results in a steady operation state, that is, in a state in which all of the input data, prediction data, and error rate are normal.

The analysis result screen 200 includes

graphs

201 and 204 regarding input data,

graphs

202 and 205 regarding predicted data, and

graphs

203 and 206 regarding error rates. A graph 201 is a box plot showing transition of input data. In the graph 201, the horizontal axis indicates the date when the input data was input, and the vertical axis indicates the required time from the station, which is one of the input data. The graph 201 is a box plot showing the minimum, maximum, median, average, and interquartile range (25% to 75% of the total) of the required time from the station for each day when the input data is entered. distribution of values belonging to ).

The graph 202 is a boxplot of forecast data, with the horizontal axis showing the date when the forecast data was obtained and the vertical axis showing the rent as the forecast result. A graph 202 shows the minimum value, maximum value, median value, average value, interquartile range, etc. of the rent for each day on which forecast data is obtained by means of a box plot.

The graph 203 is a boxplot of the error rate, the horizontal axis indicates the date when the forecast data was obtained, and the vertical axis indicates the error rate of the forecast data. A graph 203 shows the minimum value, maximum value, median value, average value, interquartile range, etc. of the error rate that indicates the prediction accuracy for each day on which the prediction is made, using a box plot.

A graph 204 is a histogram of the required time from the station, which is one of the input data, and shows the distribution of the required time from the station over a certain period of time, for example, the period shown in the graph 201 . The horizontal axis indicates the required time from the station, and the vertical axis indicates the frequency. In graph 204, diagonally hatched bins indicate the distribution of values used as input data when training the prediction model, and gray bins indicate the distribution of values input as input data when making predictions using the prediction model. The value "8.03" in the figure is the average value of input data during learning, and the value "5.91" is the average value of input data during prediction.

Graph 205 is a histogram of forecast data, and shows the distribution of rent over a certain period of time, for example, the period shown in graph 202. The horizontal axis indicates rent, and the vertical axis indicates frequency. In graph 205, obliquely hatched bins indicate the distribution of rents used as correct data when learning the prediction model, and gray bins indicate the distribution of rents obtained as prediction results when making predictions using the prediction model. Similar to the graph 204, the average value of correct data during learning is "90,364" and the average value of prediction data during prediction is "70,036".

A graph 206 is a histogram of error rates, showing the distribution of error rates over a certain period of time, for example, the period shown in graph 203 . The horizontal axis indicates the error rate, and the vertical axis indicates the frequency. In the graph 206, obliquely hatched bins indicate the distribution of error rates calculated during prediction model learning, and gray bins indicate the distribution of error rates calculated during prediction using the prediction model. Similar to the graph 204, the average value of the error rate during learning is “24.977” and the average value of the error rate during prediction is “13.15”.

In this way, the analysis result screen shows the analysis results of input data, prediction data, and error rate (prediction accuracy) for a certain period of time in graphs such as box plots and histograms. The state of the error rate can be grasped based on a concrete numerical value. In addition, as shown in graphs 204 to 206, for the input data, prediction data, and error rate, by displaying the values at the time of learning of the prediction model and the values at the time of prediction in the same graph, It is possible to visualize the extent and trend of changes in each data.

Fig. 7 shows an example of the analysis result screen on another day. The analysis result screen 210 includes boxplots 211-213 and histograms 214-216 for input data, predicted data and error rate, as in the example of FIG. The view of each graph is the same as in the example of FIG. FIG. 7 shows analysis results when input data includes outliers. In the example of FIG. 7, it is assumed that "a value less than 0" is set as an outlier for the input data "required time from the station". In FIG. 7, the minimum value of the required time from the station on August 13 is "-1", which corresponds to an outlier. Therefore, in the graph 211, a circle AL1 is displayed as an alert. In the graph 214 as well, the minimum value of the required time from the station is "-1". As described above, on the analysis result screen, when the input data includes an outlier, an alert indicating that fact is displayed.

Fig. 8 shows an example of the analysis result screen on yet another day. The analysis result screen 220 includes boxplots 221-223 and histograms 224-226 for input data, prediction data and error rate, as in the example of FIG. The view of each graph is the same as in the example of FIG. In the example of FIG. 8, first, the input data is an abnormal value. Here, for the input data "required time from the station", the case where "the difference between the average value at the time of learning and the average value at the time of prediction is 3 or more" is set as an abnormal value. As shown in the graph 224, the difference between the average value “8.03” during learning of the input data “required time from the station” and the average value “14.13” during prediction is 3 or more. The data are judged to be outliers. Therefore, an arrow AL2 is displayed in the graph 224 as an alert to that effect.

Also, in the example of FIG. 8, the predicted data is an abnormal value. Here, the prediction data is set as an abnormal value when it is "260,000 or more" or "the difference between the average value at the time of learning and the average value at the time of prediction is 30,000 or more". and In the example of FIG. 8, as shown in the graph 222, the maximum value of forecast data for September 12 exceeds 260,000, and a circle AL3 is displayed as an alert to that effect.

Furthermore, in the example of FIG. 8, the error rate is an abnormal value. Here, it is assumed that an error rate of "40% or more" is set as an abnormal value. In the example of FIG. 8, as shown in graph 223, there are days when the error rate exceeds 40%, and a rectangle AL4 is displayed as an alert to that effect.

In this way, the analysis result screen displays alerts for outliers and abnormal values in the input data, abnormal values in the predicted data, and abnormal values in the error rate (prediction accuracy). Abnormal state of the error rate can be easily known.

In the example of FIG. 8, the alerts (circle AL3, rectangle AL4) related to the prediction data and the error rate are displayed in the boxplot graphs 222 and 223, but these alerts are displayed in the

histogram graphs

225 and 226. You may That is, alerts regarding input data, predicted data, and error rates may be displayed on one or more of the multiple graphs.

　In FIGS. 6 to 8, the input data, predicted data, and error rate are shown by box plots and histograms, but line graphs may be used instead of box plots. 9 to 11 show examples of displaying input data, predicted data, and error rates using line graphs and histograms.

Specifically, on the analysis result screen 200a shown in FIG. 9, line graphs 201a to 203a are displayed instead of the boxplots 201 to 203 shown in FIG. Further, on the analysis result screen 210a shown in FIG. 10, line graphs 211a to 213a are displayed instead of the boxplots 211 to 213 shown in FIG. Note that in the graph 211a, a circle AL5 is displayed as an alert indicating an outlier in the input data.

Also, on the analysis result screen 220a shown in FIG. 11, line graphs 221a to 223a are displayed instead of the boxplots 221 to 223 shown in FIG. On the analysis result screen 220a, an arrow AL6 is displayed as an alert in the graph 224 as in FIG. Furthermore, in the graph 222a, a circle AL7 is displayed as an alert indicating an abnormal value of predicted data, and in the graph 223a a rectangle AL8 is displayed as an alert indicating an abnormal value of the error rate.

In the above example, a box plot or a line graph is used on the analysis result screen. Other types of graphs may be used.

(Modification)
In the above example, as shown in FIGS. 6 to 11, the input data, prediction data and error rate for one prediction model are displayed on one analysis result screen, but if there are multiple prediction models , input data, prediction data and error rates for a plurality of prediction models may be simultaneously displayed on one analysis result screen. For example, when there are two target models, graphs of input data, prediction data, and error rates as shown in FIG. 6 may be prepared for each model and displayed side by side on one analysis result screen. Alternatively, on one analysis result screen, the values of the two models may be displayed simultaneously in different colors and superimposed on each graph of the input data, prediction data, and error rate. In this case, it may be determined arbitrarily whether to use a box plot, a line graph, or a histogram.

<Second embodiment>
FIG. 12 is a block diagram showing the functional configuration of the information processing apparatus according to the second embodiment. The information processing device 70 includes input data acquisition means 71 , input data evaluation means 72 , prediction data acquisition means 73 , prediction data evaluation means 74 , prediction accuracy acquisition means 75 , and display means 76 .

FIG. 13 is a flowchart of processing by the information processing apparatus of the second embodiment. First, the input data obtaining means 71 obtains a plurality of input data (step S71). Next, the input data evaluation means 72 detects outliers and abnormal values in the input data (step S72). Next, the prediction data acquisition unit 73 acquires prediction data generated from the input data using the trained model (step S73). Next, the predicted data evaluation means 74 detects abnormal values in the predicted data (step S74). Next, the prediction accuracy acquisition unit 75 acquires the prediction accuracy of the learned model (step S75). Note that steps S71 to S72, steps S73 to S74, and step S75 may be performed in a different order from the above, or may be performed in parallel in terms of time. Then, the display means 76 displays diagnostic information including the presence or absence of outliers and abnormal values in the input data, the presence or absence of abnormal values in the prediction data, and the prediction accuracy (step S76).

According to the information processing device 70 of the second embodiment, it is possible to create and present diagnostic information regarding multiple factors that affect the prediction accuracy for the model in operation.

Some or all of the above embodiments can also be described as the following additional remarks, but are not limited to the following.

(Appendix 1)
input data acquisition means for acquiring a plurality of input data;
input data evaluation means for detecting outliers and abnormal values in the input data;
prediction data acquisition means for acquiring prediction data generated from the input data using the trained model;
prediction data evaluation means for detecting abnormal values in the prediction data;
prediction accuracy acquisition means for acquiring prediction accuracy by the trained model;
display means for displaying diagnostic information including the presence or absence of outliers and abnormal values in the input data, the presence or absence of abnormal values in the prediction data, and the prediction accuracy;
Information processing device.

(Appendix 2)
The diagnostic information includes:
a diagnostic report describing the presence or absence of outliers and abnormal values in the input data, the presence or absence of abnormal values in the prediction data, and the prediction accuracy;
an analysis result screen displaying the input data, the prediction data, and the prediction accuracy on a graph, respectively;
The information processing device according to appendix 1, comprising:

(Appendix 3)
The information according to appendix 2, wherein the analysis result screen displays the presence or absence of outliers and abnormal values in the input data, the presence or absence of abnormal values in the prediction data, and the prediction accuracy, respectively, on time-series graphs. processing equipment.

(Appendix 4)
The information processing according to

appendix

2 or 3, wherein the analysis result screen displays the presence or absence of outliers and abnormal values in the input data, the presence or absence of abnormal values in the prediction data, and the prediction accuracy, respectively, on histograms. Device.

(Appendix 5)
In the analysis result screen, for at least one of the input data, the prediction data, and the prediction accuracy, the value at the time of learning of the learned model and the value at the time of prediction using the learned model are the same. The information processing device according to appendix 4, which is displayed in the graph of

(Appendix 6)
The analysis result screen includes a plurality of graphs showing the presence or absence of outliers and abnormal values in the input data for each of the plurality of trained models, and the presence or absence of abnormal values in the prediction data for each of the plurality of trained models. 6. The information processing apparatus according to any one of appendices 2 to 5, wherein a plurality of graphs and a plurality of graphs showing the prediction accuracy for each of the plurality of trained models are displayed side by side.

(Appendix 7)
The analysis result screen includes one graph that simultaneously displays the presence or absence of outliers and abnormal values in the input data for a plurality of trained models, and the presence or absence of abnormal values in the prediction data for each of the plurality of trained models. 6. The information processing apparatus according to any one of Appendices 2 to 5, including one graph that simultaneously displays , and one graph that simultaneously displays the prediction accuracy for each of a plurality of trained models.

(Appendix 8)
If the input data has an outlier or abnormal value, the predicted data has an abnormal value, or the prediction accuracy is less than or equal to a predetermined value, the diagnostic report includes countermeasures and corresponding countermeasures. 8. The information processing apparatus according to any one of Appendices 2 to 7, including a description of the priority of measures.

(Appendix 9)
get multiple input data,
detecting outliers and abnormal values in the input data;
Obtaining prediction data generated from the input data using the trained model,
detecting abnormal values in the predicted data;
Obtaining the prediction accuracy of the trained model,
An information processing method for displaying diagnostic information including the presence or absence of outliers and abnormal values in the input data, the presence or absence of abnormal values in the predicted data, and the prediction accuracy.

(Appendix 10)
get multiple input data,
detecting outliers and abnormal values in the input data;
Obtaining prediction data generated from the input data using the trained model,
detecting abnormal values in the predicted data;
Obtaining the prediction accuracy of the trained model,
A recording medium recording a program for causing a computer to execute processing for displaying diagnostic information including the presence or absence of outliers and abnormal values in the input data, the presence or absence of abnormal values in the predicted data, and the prediction accuracy.

Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above embodiments and examples. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

2 prediction device 12 processor 16 display device 21 input data evaluation unit 22 prediction data evaluation unit 23 prediction accuracy evaluation unit 24 display data generation unit 100

monitoring device

200, 200a, 210, 210a, 220, 220a Analysis result screen

Claims

input data acquisition means for acquiring a plurality of input data;
input data evaluation means for detecting outliers and abnormal values in the input data;
prediction data acquisition means for acquiring prediction data generated from the input data using the trained model;
prediction data evaluation means for detecting abnormal values in the prediction data;
prediction accuracy acquisition means for acquiring prediction accuracy by the trained model;
display means for displaying diagnostic information including the presence or absence of outliers and abnormal values in the input data, the presence or absence of abnormal values in the prediction data, and the prediction accuracy;
Information processing device.
The diagnostic information includes:
a diagnostic report describing the presence or absence of outliers and abnormal values in the input data, the presence or absence of abnormal values in the prediction data, and the prediction accuracy;
an analysis result screen displaying the input data, the prediction data, and the prediction accuracy on a graph, respectively;
The information processing apparatus according to claim 1, comprising:
3. The analysis result screen according to claim 2, wherein the presence or absence of outliers and abnormal values in the input data, the presence or absence of abnormal values in the prediction data, and the prediction accuracy are each displayed on a time-series graph. Information processing equipment.
4. The information according to claim 2, wherein the analysis result screen displays the presence or absence of outliers and abnormal values in the input data, the presence or absence of abnormal values in the predicted data, and the prediction accuracy, respectively, on histograms. processing equipment.
In the analysis result screen, for at least one of the input data, the prediction data, and the prediction accuracy, the value at the time of learning of the learned model and the value at the time of prediction using the learned model are the same. 5. The information processing apparatus according to claim 4, wherein the graph of .
The analysis result screen includes a plurality of graphs showing the presence or absence of outliers and abnormal values in the input data for each of the plurality of trained models, and the presence or absence of abnormal values in the prediction data for each of the plurality of trained models. 6. The information processing apparatus according to any one of claims 2 to 5, wherein a plurality of graphs and a plurality of graphs showing the prediction accuracy for each of the plurality of trained models are displayed side by side.
The analysis result screen includes one graph that simultaneously displays the presence or absence of outliers and abnormal values in the input data for a plurality of trained models, and the presence or absence of abnormal values in the prediction data for each of the plurality of trained models. 6. The information processing apparatus according to any one of claims 2 to 5, comprising: one graph that simultaneously displays , and one graph that simultaneously displays the prediction accuracy for each of a plurality of trained models.
If the input data has an outlier or abnormal value, the predicted data has an abnormal value, or the prediction accuracy is less than or equal to a predetermined value, the diagnostic report includes countermeasures and corresponding countermeasures. 8. The information processing apparatus according to any one of claims 2 to 7, comprising a description of priority of measures.
get multiple input data,
detecting outliers and abnormal values in the input data;
Obtaining prediction data generated from the input data using the trained model,
detecting abnormal values in the predicted data;
Obtaining the prediction accuracy of the trained model,
An information processing method for displaying diagnostic information including the presence or absence of outliers and abnormal values in the input data, the presence or absence of abnormal values in the predicted data, and the prediction accuracy.
get multiple input data,
detecting outliers and abnormal values in the input data;
Obtaining prediction data generated from the input data using the trained model,
detecting abnormal values in the predicted data;
Obtaining the prediction accuracy of the trained model,
A recording medium recording a program for causing a computer to execute processing for displaying diagnostic information including the presence or absence of outliers and abnormal values in the input data, the presence or absence of abnormal values in the predicted data, and the prediction accuracy.