TWI860923B - Model reconstruction method and system - Google Patents

Abstract

A model reconstruction system includes a processing unit, which performs the following operations: obtaining a target demand forecasting model and corresponding multiple demand forecasting results to be updated based on multiple training data sets, multiple test data sets and multiple training combinations; obtaining a first-level arrangement result based on a first-level arrangement method and the demand forecasting results to be updated; obtaining a second-level arrangement result using a demand forecasting model to be compared based on the test data sets and a second-level arrangement method; obtaining a first indicator value and a second indicator value based on the first-level arrangement result and the second-level arrangement result respectively; and determining whether to replace the demand forecasting model to be compared with the target demand forecasting model based on the first indicator value and the second indicator value.

Description

Model reconstruction method and system

The present invention relates to a reconstruction method and system, and more particularly to a model reconstruction method and system.

For machine learning models that have been deployed online, when the model's validity has dropped significantly due to changes in the real situation, the deployed model needs to be optimized. However, if a new model is rebuilt by changing the training data or using a different machine learning algorithm, the prediction results of the new model may be very different from those of the old model, and cannot be measured by a unified standard, so it is impossible to compare the validity of the two models.

Therefore, how to improve the existing model optimization methods and measure the pros and cons of new and old models with a unified standard has become one of the issues that the relevant technical fields want to solve.

Therefore, an object of the present invention is to provide a model reconstruction method and system, which can overcome at least one disadvantage of the prior art.

Therefore, a model reconstruction method provided by the present invention is applicable to optimizing a demand forecasting model. The demand forecasting model is used to obtain a demand forecasting result based on the interaction record data to be analyzed related to the interaction between a customer to be analyzed and a financial institution. The demand forecasting result includes a probability value indicating that the customer to be analyzed may purchase a financial product. The model reconstruction method is executed by a computer system that stores multiple customer behavior data sets corresponding to multiple customers and multiple training combinations. Each customer behavior data set includes multiple interaction record data related to the corresponding customer's interaction with the financial institution in multiple different time periods, and multiple purchase results indicating whether the corresponding customer purchased the financial product within the time period. Each training set includes a corresponding machine learning algorithm and a model parameter set. The model reconstruction method includes the following steps: (A) selecting multiple training data sets from the customer behavior data sets according to a first access rule, and selecting multiple test data sets from the customer behavior data sets according to a second access rule, wherein each training data set includes at least one interactive training data and one purchase training result selected from the interactive record data and the purchase results of the corresponding customer, and each test data set includes at least one interactive training data and one purchase training result selected from the interactive record data and the purchase results of the corresponding customer. and at least one interactive test data and a purchase test result selected from the purchase results; (B) obtaining a target demand prediction model based on the training data sets, the test data sets and the training combinations; (C) for each test data set, obtaining a to-be-updated demand prediction result including a probability value of the corresponding customer possibly purchasing the financial product based on the test data set using the target demand prediction model; (D) obtaining a to-be-updated demand prediction result based on a first set of first grouping level threshold values corresponding to the ... (E) for each test data set, a current demand forecast result including a probability value of the corresponding customer to purchase the financial product is obtained based on the test data set using a demand forecast model to be compared; (F) all current demand forecast results obtained in step (E) are grouped according to a second grouping class threshold value corresponding to a plurality of second grouping class threshold values. (G) obtaining a first indicator value corresponding to the target demand forecast model and a second indicator value corresponding to the demand forecast model to be compared according to the first-level arrangement result and the second-level arrangement result; (H) determining whether the first indicator value is greater than the second indicator value; and (I) when it is determined that the first indicator value is greater than the second indicator value, replacing the demand forecast model to be compared with the target demand forecast model.

Therefore, the present invention provides a model reconstruction system, which is suitable for optimizing a demand forecasting model. The demand forecasting model is used to obtain a demand forecasting result based on the interaction record data to be analyzed related to the interaction between a customer to be analyzed and a financial institution. The demand forecasting result includes a probability value indicating that the customer to be analyzed may purchase a financial product. The model reconstruction system includes a storage unit and a processing unit.

The storage unit stores a plurality of customer behavior data sets corresponding to a plurality of customers, and a plurality of training combinations. Each customer behavior data set includes a plurality of interaction record data related to the corresponding customer's interaction with the financial institution in a plurality of different time periods, and a plurality of purchase results indicating whether the corresponding customer purchased the financial product in the time periods. Each training combination includes a corresponding machine learning algorithm and a model parameter set.

The processing unit signal is connected to the storage unit, and is used to select multiple training data sets from the customer behavior data sets according to a first access rule, and select multiple test data sets from the customer behavior data sets according to a second access rule. Each training data set includes at least one interactive training data and one purchase training result selected from the interactive record data and the purchase results of the corresponding customer, and each test data set includes at least one interactive test data and one purchase test result selected from the interactive record data and the purchase results of the corresponding customer. A target demand forecasting model is obtained based on the training data sets, the test data sets and the training combinations. For each test data set, a to-be-updated demand forecasting result including a probability value of a corresponding customer who may purchase the financial product is obtained based on the test data set using the target demand forecasting model. All the to-be-updated demand forecasting results obtained are grouped according to a first-level interval arrangement method corresponding to a plurality of first grouping level thresholds to obtain a first-level interval arrangement result. For each test data set, a to-be-compared demand forecasting model is used to obtain a current demand forecasting result including a probability value of a corresponding customer who may purchase the financial product based on the test data set. All current demand forecast results obtained by a second level arrangement method corresponding to a plurality of second grouping level thresholds are grouped to obtain a second level arrangement result. A first index value corresponding to the target demand forecast model and a second index value corresponding to the demand forecast model to be compared are obtained according to the first level arrangement result and the second level arrangement result. It is determined whether the first index value is greater than the second index value. When it is determined that the first index value is greater than the second index value, the target demand forecast model replaces the demand forecast model to be compared.

The effect of the present invention is that all the demand forecast results to be updated of the target demand forecast model are grouped by adopting a grouping method with the same level standard to obtain the first level arrangement result, and all the current demand forecast results of the demand forecast model to be compared are grouped to obtain the second level arrangement result, so that each probability value in the first level arrangement result and the second level arrangement result can be divided and sorted by the same level standard. Even if the probability value distribution range of the demand forecast results generated by different demand forecast models is greatly increased or decreased due to changes in external factors, the advantages and disadvantages of different demand forecast models can still be judged by a common evaluation standard.

Before the present invention is described in detail, it should be noted that similar elements are represented by the same reference numerals in the following description.

Referring to FIG. 1 , a model reconstruction system 1 of an embodiment of the present invention is suitable for optimizing a demand prediction model. The demand prediction model is used to obtain a demand prediction result based on the interaction record data to be analyzed related to the interaction between a customer to be analyzed and a financial institution, and the demand prediction result includes a probability value indicating that the customer to be analyzed may purchase a financial product (such as but not limited to a fund, insurance policy, foreign currency). In this embodiment, the demand prediction model is used to predict the probability value of each customer to be analyzed to purchase the financial product after a specified time interval N. Each interactive record data to be analyzed includes, for example but not limited to, the number, frequency, and amount of each transaction of the corresponding customer to be analyzed in the financial institution, as well as the frequency of browsing the financial institution's web pages and the duration of each page browsing, etc. The model reconstruction system 1 includes a storage unit 11 and a processing unit 12 connected to the storage unit 11 by signal.

The storage unit 11 stores a plurality of customer behavior data sets corresponding to a plurality of customers, and a plurality of training combinations. Each customer behavior data set includes a plurality of interaction record data related to the corresponding customer's interaction with the financial institution in a plurality of different time periods (e.g., each month), and a plurality of purchase results indicating whether the corresponding customer purchased the financial product in the time period. Each training combination includes a corresponding machine learning algorithm (e.g., but not limited to, algorithms such as XGBoost, LightGBM, CatBoost, etc.) and a model parameter set.

1 and 2 , it is exemplarily explained how the processing unit 12 of the embodiment executes a model reconstruction procedure. The model reconstruction procedure includes the following steps 201-210.

First, in step 201, the processing unit 12 determines whether the current time T has reached a predetermined update time. The update time is, for example, the end of each month, so that the processing unit 12 executes the model reconstruction program once a month to achieve the effect of automatic model reconstruction on a regular basis. When the processing unit 12 determines that the current time T has reached the update time, the process proceeds to step 202; otherwise, the process returns to step 201.

In step 202, the processing unit 12 selects a plurality of training data sets from the customer behavior data sets according to a first access rule, and selects a plurality of test data sets from the customer behavior data sets according to a second access rule. Each training data set includes at least one interactive training data and one purchase training result selected from the interactive record data and the purchase results of the corresponding customer; each test data set includes at least one interactive test data and one purchase test result selected from the interactive record data and the purchase results of the corresponding customer.

For example, if the demand forecasting model is used to predict the probability value of each analyzed customer purchasing the financial product one month later (i.e., N=1), and a predetermined data acquisition window size (window-size) K is set to 3 months (i.e., K=3), the first acquisition rule is set to, for example, acquire a customer behavior data set within a first time interval of [T-N-K, T-N) as the training data set, and the second acquisition rule is set to, for example, acquire a customer behavior data set within a second time interval of [T-N-K, T-N) as the training data set. The customer behavior data sets within the time interval [T-1,T) are used as the test data sets: assuming that the current time T is the end of November (i.e., T=11), then each training data set uses all interaction record data from July to October (excluding October) as the at least one interaction training data and the purchase result in October as the purchase training result, and each test data set uses the interaction record data in October as the at least one interaction test data and the purchase result in November as the purchase test result.

In step 203, the processing unit 12 obtains a target demand prediction model according to the training data sets, the test data sets and the training combinations. More specifically, step 203 includes the following sub-steps 301-307 (see FIG. 3).

In sub-step 301, the processing unit 12 groups the training combinations into a plurality of training combinations to be evaluated, sets a training count value to 1, and sets a total number of trainings to M (M>1). More specifically, the total number of trainings represents that a total of M rounds of training will be performed for the training combinations, and the training count value is used to represent which round of training the current training is.

In sub-step 302, for each training combination to be evaluated, the processing unit 12 selects i training data sets from the training data sets, and uses the training combination to be evaluated to perform training according to the i training data sets to be trained to obtain a demand forecasting model to be evaluated corresponding to the training combination to be evaluated. i is the ratio of the training count value to the total number of training data sets multiplied by the total number of training data sets. For example, assuming that there are 100 training data sets in total, and the total number of training data sets is set to 5 (i.e., M=5): when the first round of training is performed (i.e., the training count value=1), i=20 (i.e., ), the processing unit 12 selects 20 training data sets from the training data sets; when the second round of training is performed (i.e., the training count value = 2), i = 40 (i.e., ), the processing unit 12 selects 40 sets of data to be trained from the training data sets; and so on.

In sub-step 303, for each demand forecasting model to be evaluated, the processing unit 12 tests the demand forecasting model to be evaluated according to the test data sets to obtain an index value to be evaluated. More specifically, for each test data set, the processing unit 12 obtains a demand prediction result to be evaluated including the probability value of the corresponding customer who may purchase the financial product according to the test data set using the demand prediction model to be evaluated; then the processing unit 12 sorts the probability values of all the obtained demand prediction results to be evaluated from high to low to obtain a total prediction result after sorting to be evaluated; finally, the processing unit 12 obtains the proportion of the number of people who actually purchase the financial product among the customers who are ranked within a target percentage of the total prediction result after sorting to be evaluated as the index value to be evaluated. For example, assuming that the total forecast results after the evaluation sorting contain 100 demand forecast results to be evaluated from 100 customers, and the target percentage is 30%, then the value of the index to be evaluated is the proportion of the number of people who actually purchased the financial product among the 30 customers with the highest probability values in the top 30% of the 100 demand forecast results to be evaluated. If 27 of these 30 customers actually purchased the financial product, the value of the index to be evaluated is 0.9.

In sub-step 304, the processing unit 12 selects j candidate demand forecast models from the demand forecast models to be evaluated according to the evaluation index value. j is 1/M of the total number of the demand forecast models to be evaluated. For example, assuming that the total number of training is set to 5 (i.e., M=5), if a total of 50 demand forecast models to be evaluated are obtained in this round, the processing unit 12 selects 10 (i.e., j= ) candidate demand forecast models. The selection method is, for example, to select the demand forecast models to be evaluated that correspond to the top 10 highest values of the indicators to be evaluated as the candidate demand forecast models.

It is worth mentioning that, since each round only selects 1/M of the total number of demand forecast models to be evaluated in that round as candidate demand forecast models, each subsequent round will select fewer candidate demand forecast models than the previous round; in addition, since not all training data sets are used for training at once during model training, but only part of the training data sets are used as the waiting training data sets in each round, and the number of the waiting training data sets is gradually increased, by combining this method of selecting the waiting training data sets with the aforementioned method of selecting candidate demand forecast models, it helps to reduce the time for model screening and make it less than the time spent on the commonly used method of training all candidate models with complete training data.

In sub-step 305, the processing unit 12 determines whether the training count value is equal to M. If the processing unit 12 determines that the training count value is not equal to M, the process proceeds to sub-step 306; if the processing unit 12 determines that the training count value is equal to M, the process proceeds to sub-step 307.

In sub-step 306, the processing unit 12 combines the training combinations corresponding to the j candidate demand forecasting models into j training combinations to be evaluated, and increases the training count value by one, and the process returns to sub-step 302 to perform the next round of training.

In sub-step 307, the processing unit 12 selects the candidate forecasting model corresponding to the highest value of the to-be-evaluated index from the j candidate demand forecasting models as the target demand forecasting model.

Next, in step 204, for each test data set, the processing unit 12 uses the target demand prediction model according to the test data set to obtain a demand prediction result to be updated, which includes the probability value of the corresponding customer possibly purchasing the financial product.

In step 205, the processing unit 12 groups all the demand forecast results to be updated according to a first-level arrangement method corresponding to a plurality of first grouping level thresholds to obtain a first-level arrangement result. More specifically, step 205 includes the following sub-steps 401-404 (see FIG. 4).

In sub-step 401, the processing unit 12 sorts the probability values of all the demand forecast results to be updated from high to low to obtain a total forecast result to be updated after sorting.

In sub-step 402, the processing unit 12 divides the total forecast results to be updated after sorting into multiple groups according to the purchase results of the test data sets and multiple grouping accuracy rates. The proportion of the demand forecast results to be updated in each group that meet the corresponding purchase results is equal to one of the grouping accuracy rates. For example, suppose the total forecast results after the update sorting contain 10 forecast results of 10 customers (customers #1~#10) to be updated (as shown in Table 1 below), and three desired grouping accuracy rates are predetermined, which are 80%, 50% and 30% respectively: customers #1~#5 are divided into the group corresponding to the desired grouping accuracy rate of 80%, customers #6~#7 are divided into the group corresponding to the desired grouping accuracy rate of 50%, and customers #8~#10 are divided into the group corresponding to the desired grouping accuracy rate of 30%. Table 1 customer Demand forecast results to be updated Purchase Results Grouping Customer #1 98% Buy The accuracy of the desired grouping is 80% Customer #2 95% Buy Client #3 88% Buy Client #4 87% Didn't buy Customer #5 85% Buy Customer #6 80% Didn't buy The accuracy of grouping is 50% Client #7 75% Buy Customer #8 74% Didn't buy The accuracy of grouping is 30% Customer #9 70% Didn't buy Client #10 64% Buy

In sub-step 403, for each cluster, the processing unit 12 uses the demand forecast result to be updated that is the last in the sorting in the cluster as the first clustering level threshold value of the cluster. Continuing with the example of Table 1 above, 0.85 is the first clustering level threshold value corresponding to the cluster with an accuracy of 80%, 0.75 is the first clustering level threshold value corresponding to the cluster with an accuracy of 50%, and 0.64 is the first clustering level threshold value corresponding to the cluster with an accuracy of 80%.

In sub-step 404, the processing unit 12 groups all the demand forecast results to be updated according to the first grouping level thresholds to obtain the first level arrangement result. The first level arrangement result includes the groupings. More specifically, for each first grouping level threshold, the processing unit 12 adds all the demand forecast results to be updated that are greater than or equal to the first grouping level threshold and do not belong to another group to the group corresponding to the first grouping level threshold. It is worth noting that the difference between the re-grouping in sub-step 404 and the grouping in sub-step 402 is that sub-step 404 groups all the unsorted demand forecast results to be updated, while sub-step 402 groups the sorted total forecast results to be updated. Continuing with the example of Table 1 above, the first level arrangement result is shown in Table 2 below: Table 2 customer Demand forecast results to be updated Purchase Results Grouping Customer #5 85% Buy The accuracy of the desired grouping is 80% Client #3 88% Buy Customer #2 95% Buy Client #4 87% Didn't buy Customer #1 98% Buy Customer #6 80% Didn't buy The accuracy of grouping is 50% Client #7 75% Buy Customer #8 74% Didn't buy The accuracy of grouping is 30% Client #10 64% Buy Customer #9 70% Didn't buy

Then, in step 206, for each test data set, the processing unit 12 obtains a current demand forecast result including a probability value of the corresponding customer to purchase the financial product using a demand forecast model to be compared according to the test data set. The demand forecast model to be compared is the demand forecast model currently deployed online that needs to be optimized.

In step 207, the processing unit 12 groups all the current demand forecast results obtained according to a second level arrangement method corresponding to a plurality of second grouping level thresholds to obtain a second level arrangement result. It is worth noting that the second grouping level thresholds are the first grouping level thresholds obtained when the demand forecast model to be compared was previously selected as the target demand forecast model. The details of obtaining the second level arrangement result are roughly the same as sub-step 404, and no further details are given here.

In step 208, the processing unit 12 obtains a first index value corresponding to the target demand forecast model and a second index value corresponding to the demand forecast model to be compared according to the first-level ranking result and the second-level ranking result. In this embodiment, the first index value is the proportion of the number of people who actually purchase the financial product among the customers who are ranked within the target percentage of the first-level ranking result, and the second index value is the proportion of the number of people who actually purchase the financial product among the customers who are ranked within the target percentage of the second-level ranking result. The way of calculating the first index value and the second index value here is roughly the same as the way of calculating the index value to be evaluated in sub-step 303. The difference is that the sorting method of sub-step 303 is to put the demand forecast results corresponding to the higher probability value in the front, while here the demand forecast results corresponding to the higher accuracy of the desired grouping are put in the front. For example, if the target percentage is 30%, continuing the example of Table 2 above, the top 3 customers (top 30%) are customer #5, customer #3, and customer #2, and these three people actually purchased the financial product, then the first index value is 1.

In step 209, the processing unit 12 determines whether the first index value is greater than the second index value. When the processing unit 12 determines that the first index value is greater than the second index value, the process proceeds to step 210; otherwise, the process returns to step 201.

In step 210, the processing unit 12 replaces the demand forecast model to be compared with the target demand forecast model.

In summary, by adopting a grouping method with the same level standard to group all the demand forecast results to be updated of the target demand forecast model to obtain the first level ranking result, and grouping all the current demand forecast results of the demand forecast model to be compared to obtain the second level ranking result, the probability values in the first level ranking result and the second level ranking result can be divided and sorted by the same level standard. Even if the probability value distribution range of the demand forecast results generated by different demand forecast models is greatly increased or decreased due to changes in external factors, the advantages and disadvantages of different demand forecast models can still be judged by a common evaluation standard. Secondly, the model training method adopted in this case is to gradually reduce the number of candidate models and increase the amount of model training data in each round of model training, which can save more time than previous model training methods, thereby more effectively screening the training combination to obtain the target demand prediction model. In addition, regular reconstruction and redeployment of the model can also enable the online model to always maintain the best validity and reduce the impact of model validity due to changes in actual conditions. Therefore, the purpose of this invention can indeed be achieved.

However, the above is only an example of the implementation of the present invention, and it should not be used to limit the scope of the implementation of the present invention. All simple equivalent changes and modifications made according to the scope of the patent application of the present invention and the content of the patent specification are still within the scope of the patent of the present invention.

1: Model reconstruction system 11: Storage unit 12: Processing unit 201~210: Steps 301~307: Sub-steps 401~404: Sub-steps

Other features and functions of the present invention will be clearly presented in the implementation method with reference to the drawings, wherein: FIG. 1 is a block diagram, exemplarily illustrating a model reconstruction system of an embodiment of the present invention; FIG. 2 is a flow chart, exemplarily illustrating how a processing unit of the embodiment executes a model reconstruction procedure; FIG. 3 is a flow chart, exemplarily illustrating how the processing unit of the embodiment obtains a target demand prediction model; and FIG. 4 is a flow chart, exemplarily illustrating how the processing unit of the embodiment obtains a first-order interval arrangement result.

1: Model reconstruction system

11: Storage unit

12: Processing unit

Claims (12)

A model reconstruction method is used for optimizing a demand prediction model. The demand prediction model is used to obtain a demand prediction result based on interaction record data related to the interaction between a customer to be analyzed and a financial institution. The demand prediction result includes a probability value indicating that the customer to be analyzed may purchase a financial product. The model reconstruction method stores a plurality of customer behavior data sets corresponding to a plurality of customers, and a plurality of training The model reconstruction method is executed by a computer system of a training combination. Each customer behavior data set includes multiple interaction record data related to the corresponding customer's interaction with the financial institution in multiple different time periods, and multiple purchase results indicating whether the corresponding customer purchased the financial product in the time period. Each training combination includes a corresponding machine learning algorithm and a model parameter set. The model reconstruction method includes the following steps: (A) Selecting multiple training data sets from the customer behavior data sets according to a first access rule, and selecting multiple test data sets from the customer behavior data sets according to a second access rule, wherein each training data set includes at least one interactive training data and one purchase training result selected from the corresponding customer interaction record data and the purchase results, and each test data set includes at least one interactive test data and one purchase test result selected from the corresponding customer interaction record data and the purchase results; (B) Obtaining a target demand prediction model based on the training data sets, the test data sets and the training combinations; (C) For each test data set, obtain a to-be-updated demand forecast result including the probability value of the corresponding customer who may purchase the financial product based on the test data set using the target demand forecast model; (D) Group all to-be-updated demand forecast results obtained in step (C) according to a first-level interval arrangement method corresponding to a plurality of first-level grouping interval thresholds to obtain a first-level interval arrangement result; (E) For each test data set, obtain a current demand forecast result including the probability value of the corresponding customer who may purchase the financial product based on the test data set using a to-be-compared demand forecast model; (F) grouping all current demand forecast results obtained in step (E) according to a second level arrangement method corresponding to a plurality of second grouping level thresholds to obtain a second level arrangement result; (G) obtaining a first index value corresponding to the target demand forecast model and a second index value corresponding to the demand forecast model to be compared according to the first level arrangement result and the second level arrangement result; (H) determining whether the first index value is greater than the second index value; and (I) when it is determined that the first index value is greater than the second index value, replacing the demand forecast model to be compared with the target demand forecast model. The model reconstruction method as described in claim 1, wherein step (B) comprises the following sub-steps: (B-1) the training combinations are grouped into a plurality of training combinations to be evaluated, and a training count value is set to 1, and a total number of training is set to M (M>1); (B-2) for each training combination to be evaluated, i training data sets to be trained are selected from the training data sets, and the training combination to be evaluated is used to perform training according to the i training data sets to be trained, so as to obtain a demand forecasting model to be evaluated corresponding to the training combination to be evaluated, i being the ratio of the training count value to the total number of training times the total number of the training data sets; (B-3) For each demand forecasting model to be evaluated, the demand forecasting model to be evaluated is tested according to the test data sets to obtain an evaluation index value; (B-4) According to the evaluation index value, j candidate demand forecasting models are selected from the demand forecasting models to be evaluated, where j is 1/M of the total number of the demand forecasting models to be evaluated; (B-5) Determine whether the training count value is equal to M; (B-6) When it is determined that the training count value is not equal to M, the training combinations corresponding to the j candidate demand forecasting models are grouped as j training combinations to be evaluated, and the training count value is increased by one, and the process returns to step (B-2); and (B-7) When it is determined that the training count value is equal to M, the candidate prediction model corresponding to the highest value of the index to be evaluated is selected from the j candidate demand prediction models as the target demand prediction model. The model reconstruction method as described in claim 2, wherein, for each demand prediction model to be evaluated, step (B-3) includes the following sub-steps: (B-3-1) for each test data set, using the demand prediction model to be evaluated to obtain a demand prediction result to be evaluated including the probability value of the corresponding customer to purchase the financial product according to the test data set; (B-3-2) sorting the probability values of all demand prediction results to be evaluated obtained in step (B-3-1) to obtain a total prediction result after sorting; and (B-3-3) obtaining the proportion of the number of people who actually purchase the financial product among the customers who are within a target percentage of the total prediction result after sorting as the evaluation index value. The model reconstruction method as described in claim 3, wherein step (D) includes the following sub-steps: (D-1) sorting the probability values of all demand forecast results to be updated obtained in step (C) to obtain a total forecast result to be updated after sorting; (D-2) dividing the total forecast result to be updated after sorting into multiple clusters according to the purchase results of the test data sets and multiple clustering accuracies, and the proportion of the demand forecast results to be updated in each cluster that meet the corresponding purchase results is equal to one of the clustering accuracies; (D-3) for each cluster, the demand forecast result to be updated corresponding to the last sorting in the cluster is used as the first clustering level threshold value of the cluster; and (D-4) Group all the demand forecast results to be updated obtained in step (C) according to the first grouping level thresholds to obtain the first level arrangement result, and the first level arrangement result includes the groupings. The model reconstruction method as described in claim 4, wherein in step (G): The first indicator value is the proportion of the number of customers who actually purchase the financial product among the customers who are ranked within the target percentage of the first-level ranking result; and The second indicator value is the proportion of the number of customers who actually purchase the financial product among the customers who are ranked within the target percentage of the second-level ranking result. The model reconstruction method as described in claim 5, before step (A), further comprises the following steps: (J) determining whether the current time has reached a predetermined update time; wherein, when it is determined that the current time has reached the update time, step (A) is executed. A model reconstruction system is used for optimizing a demand forecasting model, wherein the demand forecasting model is used to obtain a demand forecasting result based on interaction record data to be analyzed related to the interaction between a customer to be analyzed and a financial institution, wherein the demand forecasting result includes a probability value indicating that the customer to be analyzed may purchase a financial product. The model reconstruction system includes: A storage unit storing a plurality of customer behavior data sets corresponding to a plurality of customers, and a plurality of training combinations, each customer behavior data set comprising a plurality of interaction record data related to the corresponding customer's interaction with the financial institution in a plurality of different time periods, and a plurality of purchase results indicating whether the corresponding customer purchased the financial product in the time periods, and each training combination comprising a corresponding machine learning algorithm and a model parameter set; and A processing unit, signal-connected to the storage unit, for selecting a plurality of training data sets from the customer behavior data sets according to a first access rule, and selecting a plurality of test data sets from the customer behavior data sets according to a second access rule, wherein each training data set includes at least one interactive training data and one purchase training result selected from the interactive record data and the purchase results of the corresponding customer, and each test data set includes at least one interactive test data and one purchase test result selected from the interactive record data and the purchase results of the corresponding customer, A target demand forecasting model is obtained based on the training data sets, the test data sets and the training combinations. For each test data set, a to-be-updated demand forecasting result including the probability value of the corresponding customer who may purchase the financial product is obtained based on the test data set using the target demand forecasting model. All the to-be-updated demand forecasting results are grouped according to a first-level interval arrangement method corresponding to a plurality of first grouping level thresholds to obtain a first-level interval arrangement result. For each test data set, a current demand forecasting result including the probability value of the corresponding customer who may purchase the financial product is obtained based on the test data set using a to-be-compared demand forecasting model. All current demand forecast results obtained by a second-level arrangement method corresponding to a plurality of second grouping level thresholds are grouped to obtain a second-level arrangement result. A first index value corresponding to the target demand forecast model and a second index value corresponding to the demand forecast model to be compared are obtained according to the first-level arrangement result and the second-level arrangement result, and whether the first index value is greater than the second index value is determined. When it is determined that the first index value is greater than the second index value, the target demand forecast model replaces the demand forecast model to be compared. A model reconstruction system as described in claim 7, wherein: The processing unit combines the training combinations into a plurality of training combinations to be evaluated, and sets a training count value to one, and sets a total number of training to M (M>1), For each training combination to be evaluated, the processing unit selects i training data sets to be trained from the training data sets, and uses the training combination to be evaluated to perform training based on the i training data sets to be trained to obtain a demand forecast model to be evaluated corresponding to the training combination to be evaluated, i is the ratio of the training count value to the total number of training times the total number of training data sets, For each demand forecasting model to be evaluated, the processing unit tests the demand forecasting model to be evaluated according to the test data sets to obtain an evaluation index value. The processing unit selects j candidate demand forecasting models from the demand forecasting models to be evaluated according to the evaluation index value, where j is 1/M of the total number of the demand forecasting models to be evaluated. The processing unit determines whether the training count value is equal to M. When it is determined that the training count value is not equal to M, the processing unit combines the training combinations corresponding to the j candidate demand forecast models into j training combinations to be evaluated, and increases the training count value by one, and repeatedly obtains the demand forecast model to be evaluated and its to-be-evaluated index value corresponding to each training combination to be evaluated, and selects j candidate demand forecast models therefrom until the training count value is equal to M. When it is determined that the training count value is equal to M, the processing unit selects the candidate forecast model corresponding to the highest to-be-evaluated index value from the j candidate demand forecast models as the target demand forecast model. A model reconstruction system as described in claim 8, wherein for each demand prediction model to be evaluated: For each test data set, the processing unit obtains a demand prediction result to be evaluated including a probability value of the corresponding customer to purchase the financial product according to the test data set using the demand prediction model to be evaluated, The processing unit sorts the probability values of all the obtained demand prediction results to be evaluated to obtain a total prediction result after sorting, The processing unit obtains the proportion of the number of people who actually purchase the financial product among the customers who are ranked within a target percentage of the total prediction result after sorting as the evaluation index value. A model reconstruction system as described in claim 9, wherein: The processing unit sorts the probability values of all the demand forecast results to be updated to obtain a total forecast result after sorting to be updated, The processing unit divides the total forecast result to be updated after sorting into multiple groups according to the purchase results of the test data sets and multiple grouping accuracy rates, and the proportion of the demand forecast results to be updated in each group that meet the corresponding purchase results is equal to one of the grouping accuracy rates, For each group, the processing unit uses the demand forecast result to be updated corresponding to the last sorting in the group as the first grouping level threshold value of the group, The processing unit groups all the demand forecast results to be updated according to the first grouping level thresholds to obtain the first level arrangement result, and the first level arrangement result includes the groupings. A model reconstruction system as described in claim 10, wherein: The first indicator value is the proportion of the number of customers who actually purchase the financial product among the customers who are ranked within the target percentage of the first-level ranking result; and The second indicator value is the proportion of the number of customers who actually purchase the financial product among the customers who are ranked within the target percentage of the second-level ranking result. A model reconstruction system as described in claim 11, wherein: The processing unit determines whether the current time has reached a predetermined update time, When the processing unit determines that the current time has reached the update time, the processing unit selects the training data sets and the test data sets.

Images