CN109558543A - Sample sampling method, sample sampling device, server and storage medium - Google Patents

Abstract

Embodiments of the present invention relate to the field of data statistics, and disclose a sample sampling method, a sample sampling device, a server and a storage medium. The sample sampling method in the present invention includes: respectively acquiring scale factors and corresponding data amounts at a plurality of predetermined times; performing model training on the acquired scale factors and corresponding data amounts to generate a scale factor prediction model; according to the scale factors The prediction model and the data volume predict the scale factor; the data is sampled using the predicted scale factor. The invention also discloses a sample sampling device, a server and a non-volatile storage medium respectively, so that the scale factor changes dynamically and is more suitable for the data characteristics of the sampling moment.

Description

A sample sampling method, sample sampling device, server and storage medium

technical field

Embodiments of the present invention relate to the field of data statistics, and in particular, to a sample sampling method, a sample sampling device, a server, and a storage medium.

Background technique

Due to the large amount of data generated in the e-commerce industry, it is suitable for solutions such as personalized recommendations to improve user experience. In this process, statistical analysis of the data is required. One of the existing methods is to use historical data to train a model. In this method, the accuracy of the model is related to the proportion of positive and negative samples in the data samples. Generally, positive and negative samples are required. The proportion of samples is higher than that of negative samples, but in practical applications, there are often unbalanced samples such as too high negative samples, which greatly reduces the accuracy of the model obtained.

The inventor of the present application found that, in order to solve the above problems, a sampling scale factor is set in the prior art, and the actual data volume is sampled to adjust the ratio of positive and negative samples. However, the value of the set sampling scale factor remains static, and because the amount of data at each sampling time is not the same, the sample proportion obtained by sampling with the same scale factor is still difficult to control, especially when the amount of data changes. larger application scenarios.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present invention is to provide a sample sampling method, a sample sampling device, a server and a storage medium, so that the scale factor changes dynamically and is more suitable for the data characteristics at the sampling moment.

In order to solve the above-mentioned technical problems, an embodiment of the present invention provides a sample sampling method, which includes: acquiring scale factors and corresponding data volumes at multiple predetermined times; performing model training according to the acquired scale factors and corresponding data volumes, and generating a scale factor prediction model; predict a scale factor according to the scale factor prediction model and the amount of data; use the predicted scale factor to sample the data.

An embodiment of the present invention also provides a sample sampling device, comprising: an acquisition module for acquiring scale factors and corresponding data amounts at multiple predetermined times; a model generation module for acquiring scale factors and corresponding data according to the acquired scale factors A prediction module is used to predict a scale factor according to the scale factor prediction model and the data amount; a sampling module is used to sample the data by using the predicted scale factor.

Embodiments of the present invention also provide a server, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a program executable by the at least one processor instructions, the instructions are executed by the at least one processor, so that the at least one processor can perform the following steps: acquiring scale factors and corresponding data amounts at a plurality of predetermined moments; according to the acquired scale factors and corresponding data Predict the scale factor according to the scale factor prediction model and the data amount, and use the predicted scale factor to sample the data.

Embodiments of the present invention also provide a non-volatile storage medium for storing a computer-readable program, the computer-readable program being used for a computer to execute the above-mentioned sample sampling method.

Compared with the prior art, the main difference and effect of the embodiments of the present invention are: according to a plurality of scale factors at a predetermined time in the historical time, a prediction model for generating the scale factor is trained, and then the required ratio of the current sampling is predicted according to the prediction model. factor, the resulting scale factor, even in the same statistical period, will dynamically change according to the different moments of sampling, and the change factor will cover the change of historical data, so that the scale factor is more in line with the data characteristics of the current moment, thus Get a more appropriate sample size when sampling. It can be seen that the embodiment of the present invention can also sample an appropriate amount of sample data in a scenario where the amount of data varies greatly, so that the positive and negative sample amounts required for sample application are more balanced.

As a further improvement, the model training is regression fitting training based on the Xgboost model. Regression fitting training using the Xgboost model can obtain prediction models more quickly and accurately.

As a further improvement, the predetermined time includes: a comparable time to the time to be sampled, and/or a chain time compared to the time to be sampled.

As a further improvement, the comparable moments of the to-be-sampling moments include: the moments corresponding to the to-be-sampled moments in the first N statistical cycles, where N is a natural number greater than 0; the chain-to-sample moments of the to-be-sampled moments include: The time to be sampled is within the same statistical period and M times earlier than the time to be sampled, where M is a natural number greater than 0.

As a further improvement, the statistical period is one day. One day is used as the statistical period to avoid untimely data update due to too long period, and at the same time to avoid too short period and too frequent data calculation, which is more in line with the user's usage frequency.

As a further improvement, the data to be sampled is negative sample data.

As a further improvement, the data volume is traffic data.

The above description is only an overview of the technical solutions of the present invention, in order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand , the following specific embodiments of the present invention are given.

Description of drawings

One or more embodiments are exemplified by the pictures in the corresponding drawings, and these exemplifications do not constitute limitations of the embodiments, and elements with the same reference numerals in the drawings are denoted as similar elements, Unless otherwise stated, the figures in the accompanying drawings do not constitute a scale limitation.

1 is a flowchart of a sample sampling method according to an embodiment of the present invention;

2 is a schematic diagram of a sample sampling device according to another embodiment of the present invention;

FIG. 3 is a schematic structural diagram of a server provided according to another embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those of ordinary skill in the art can appreciate that, in the various embodiments of the present invention, many technical details are set forth in order for the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can be realized. The following divisions of the various embodiments are for the convenience of description, and should not constitute any limitation on the specific implementation of the present invention, and the various embodiments may be combined with each other and referred to each other on the premise of not contradicting each other.

A first embodiment of the present invention relates to a sample sampling method. The process is shown in Figure 1, and the details are as follows:

Step 101 , acquiring scale factors and corresponding data amounts at multiple predetermined times.

Take the predetermined time including the time-to-sampling time as an example for illustration. Specifically, the year-on-year time to be sampled includes: the time corresponding to the time to be sampled in the first N statistical cycles, and N is a natural number greater than 0, such as: The sampling time is 10:00 today, and the statistical period is 1 day, so the corresponding year-on-year time can include: 10:00 yesterday, 10:00 the day before yesterday, 10:00 the day before yesterday, etc. In practical applications, in order to ensure the convenience of data statistics, you can Expand the year-on-year time of the sampling time into a time period, such as: 10:00-11:00 yesterday, 10:00-11:00 the day before yesterday, 10:00-11:00 the day before yesterday, etc.

In the embodiment, it can also be described by taking the predetermined time including the time to be sampled as an example. Specifically, the chain time of the time to be sampled includes: within the same statistical period as the time to be sampled, and earlier than the time to be sampled. M times, M is a natural number greater than 0. , such as: the time to be sampled is 10:00 today, then the corresponding chain ratio time can include: 9:50 today, 9:40 today, 9:30 today, etc.

In practical applications, the plurality of predetermined moments may include: a year-on-year moment of the to-be-sampling moment and a chain-comparison moment of the to-be-sampled moment, that is to say, the above two parts of data are included at the same time, which will not be repeated here.

Continuing to explain, after a plurality of predetermined moments are determined, the scale factor and the corresponding data volume at the predetermined moment are obtained, wherein the scale factor at the predetermined moment may be a preset value, or may be the result of the last prediction, the corresponding data volume It is the amount of data generated at a predetermined time, such as click-through rate.

Step 102: Perform model training according to the obtained scale factor and the corresponding data amount.

Specifically, the regression fitting training of the scale factor can be performed based on the Xgboost model. More specifically, the CART (Classification and Regression Trees) classification and regression tree can be used in the Xgboost model, and the value corresponding to the leaf node of the CART tree is an actual score. Whether the scale factor has an impact, so as to filter out which classes (times) of the scale factor have an impact on the prediction result. It should be further explained that after using the Xgboost model for training, a scale factor prediction model is generated.

Step 103: Predict the scale factor according to the scale factor prediction model and the amount of data.

Specifically, the scale factor prediction model in this step is the scale factor prediction model obtained in step 102, and the amount of data can also be obtained according to the historical prediction model, so that the estimated value is more accurate, and it is possible to predict earlier The amount of data is conducive to real-time sampling in practical applications.

Step 104, sampling the data using the predicted scale factor.

Specifically, this step uses the scale factor predicted in step 103 to perform sampling.

The above steps 101 and 102 are equivalent to the model accuracy stage. The generated model can be used for a long time and may not be performed every time. Steps 103 and 104 are equivalent to the actual sampling process and are steps that need to be performed each time sampling.

Compared with the prior art, the main difference and effect of this embodiment are: according to a plurality of scale factors at a predetermined time in the historical time, a prediction model for generating the scale factor is trained, and then the scale factor required for the current sampling is predicted according to the prediction model. , the resulting scale factor, even in the same statistical period, will dynamically change according to the different moments of sampling, and the change factor will cover the change of historical data, so that the scale factor is more in line with the data characteristics of the current moment, so that in the A more suitable sample size is obtained when sampling. It can be seen that this embodiment can also sample an appropriate amount of sample data in a scenario where the amount of data varies greatly, so that the amount of positive and negative samples required for sample application is more balanced.

A second embodiment of the present invention relates to a sample sampling method. This embodiment is specifically applied to the negative sample sampling process of the click-through rate of products.

Specifically, in the training of the click-through rate estimation model, the user's click on the product can be regarded as a positive sample, and the product displayed to the user without being clicked can be regarded as a negative sample. In the embodiment, the number of negative samples is often five times or even more than ten times more than the number of positive samples. Therefore, in this embodiment, the sample sampling method is used to sample the negative samples and reduce the number of negative samples, so that the training of the click-through rate estimation model is performed. positive and negative sample balance.

Continuing to explain, also according to steps 101 to 104 in the first embodiment, the amount of data may be flow data, and after a scale factor is predicted, the above flow data is negatively sampled using the scale factor.

It can be seen that when this embodiment is applied to the negative sample sampling of the click rate of products, the negative samples required in the training of the click rate estimation model can be reduced, so that the positive and negative samples are balanced, so that the product click rate model to be trained later is more accurate. In line with the actual situation, the CTR estimate is more accurate.

The steps of the above various methods are divided only for the purpose of describing clearly. During implementation, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as the same logical relationship is included, they are all within the protection scope of this patent. ;Adding insignificant modifications to the algorithm or process or introducing insignificant designs, but not changing the core design of the algorithm and process are all within the scope of protection of this patent.

The third embodiment of the present invention relates to a sample sampling device, as shown in FIG. 2 , which specifically includes:

The acquiring module is used to acquire the scale factors and corresponding data amounts of multiple predetermined moments.

The model generation module is used to perform model training according to the obtained scale factor and the corresponding data amount, and generate a scale factor prediction model.

Prediction module for predicting the scale factor based on the scale factor prediction model and the amount of data.

A sampling module for sampling the data with the predicted scale factor.

In one example, the model is trained as a regression fit training based on the Xgboost model.

In one example, the predetermined time includes: a comparable time to the time to be sampled, and/or a chain time compared to the time to be sampled. Specifically, the year-on-year time of the time to be sampled includes: the time corresponding to the time to be sampled in the first N statistical periods, where N is a natural number greater than 0; the chain-comparison time of the time to be sampled includes: within the same statistical period as the time to be sampled, And M times earlier than the time to be sampled, M is a natural number greater than 0.

In one example, the statistical period is one day.

In one example, the data to be sampled is negative sample data.

In one example, the amount of data is traffic data.

Compared with the prior art, this embodiment trains a prediction model for generating a scale factor according to a plurality of scale factors at a predetermined time in the historical time, and then predicts the scale factor required for the current sampling according to the prediction model. , even in the same statistical period, it will change dynamically according to the different times of sampling, and the change factor will cover the change of historical data, so that the scale factor is more in line with the data characteristics of the current time, so that more suitable sampling can be obtained during sampling quantity. It can be seen that this embodiment can also sample an appropriate amount of sample data in a scenario where the amount of data varies greatly, so that the amount of positive and negative samples required for sample application is more balanced.

It is not difficult to find that this embodiment is a device example corresponding to the first embodiment, and this embodiment can be implemented in cooperation with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and are not repeated here in order to reduce repetition. Correspondingly, the related technical details mentioned in this embodiment can also be applied to the first embodiment.

It is worth mentioning that each module involved in this embodiment is a logical module. In practical applications, a logical unit may be a physical unit, a part of a physical unit, or multiple physical units. A composite implementation of the unit. In addition, in order to highlight the innovative part of the present invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other units in this embodiment.

The fourth embodiment of the present invention relates to a server. As shown in FIG. 3, the server includes: at least one processor 301; and a memory 302 communicatively connected to the at least one processor 301; Component 303, the communication component 303 receives and sends data under the control of the processor 301; wherein, the memory 302 stores instructions executable by the at least one processor 301, and the instructions are executed by the at least one processor 301 to achieve:

Obtain scale factors and corresponding data volumes at multiple predetermined moments.

Perform model training according to the obtained scale factor and the corresponding amount of data to generate a scale factor prediction model.

Predict the scale factor based on the scale factor prediction model and the amount of data.

The data is sampled using the predicted scale factor.

Specifically, the server includes: one or more processors 301 and a memory 302, and one processor 301 is taken as an example in FIG. 3 . The processor 301 and the memory 302 may be connected by a bus or in other ways, and the connection by a bus is taken as an example in FIG. 3 . As a non-volatile computer-readable storage medium, the memory 302 can be used to store non-volatile software programs, non-volatile computer-executable programs and modules. The processor 301 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions and modules stored in the memory 302, that is, to implement the above-mentioned sample sampling method.

The memory 302 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function; the storage data area may store an option list and the like. Additionally, memory 302 may include high speed random access memory, and may also include nonvolatile memory, such as at least one magnetic disk storage device, flash memory device, or other nonvolatile solid state storage device. In some embodiments, the memory 302 may optionally include memory 302 located remotely from the processor 301, and these remote memories 302 may be connected to external devices via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

One or more modules are stored in the memory 302, and when executed by the one or more processors 301, perform the sample sampling method in any of the above-described method embodiments.

The above product can execute the method provided by the embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present application.

The fifth embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program, and the computer-readable program is used for a computer to execute some or all of the above method embodiments.

That is, those skilled in the art can understand that all or part of the steps in the method for implementing the above embodiments can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium and includes several instructions to make a device ( It may be a single chip microcomputer, a chip, etc.) or a processor (processor) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: U disk, removable hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes.

Those skilled in the art can understand that the above-mentioned embodiments are specific examples for realizing the present invention, and in practical applications, various changes in form and details can be made without departing from the spirit and the spirit of the present invention. scope.

Embodiments of the present application provide A1. a sample sampling method, comprising:

Obtain the scale factors and corresponding data volumes of multiple predetermined moments;

Perform model training according to the obtained scale factor and the corresponding amount of data to generate a scale factor prediction model;

Predict the scale factor according to the scale factor prediction model and the amount of data;

The data is sampled using the predicted scale factor.

A2. The sample sampling method according to A1, wherein the model training is regression fitting training based on the Xgboost model.

A3. According to the sample sampling method described in A1, the predetermined time includes: a time-comparison time of the time to be sampled, and/or a chain-comparison time of the time to be sampled.

A4. According to the sample sampling method described in A3, the year-on-year moments of the to-be-sampled moments include: the moments corresponding to the to-be-sampled moments in the first N statistical cycles, and the N is a natural number greater than 0;

The chain ratio of the time to be sampled includes: M times that are within the same statistical period as the time to be sampled and earlier than the time to be sampled, where M is a natural number greater than 0.

A5. The sample sampling method according to A4, wherein the statistical period is one day.

A6. The sample sampling method according to any one of A1 to A5, the data to be sampled is negative sample data.

A7. The sample sampling method according to any one of A1 to A5, wherein the data volume is flow data.

Embodiments of the present application also provide B8. A sample sampling device, comprising:

an acquisition module, used to acquire scale factors and corresponding data volumes at multiple predetermined moments;

The model generation module is used to perform model training according to the obtained scale factor and the corresponding data amount, and generate a scale factor prediction model;

a prediction module, used for predicting the scale factor according to the scale factor prediction model and the amount of data;

A sampling module for sampling the data with the predicted scale factor.

Embodiments of the present application also provide C9. A server, comprising:

at least one processor; and,

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of:

Obtain the scale factors and corresponding data volumes of multiple predetermined moments;

Perform model training according to the obtained scale factor and the corresponding amount of data to generate a scale factor prediction model;

Predict the scale factor according to the scale factor prediction model and the amount of data;

The data is sampled using the predicted scale factor.

C10. The server according to C9, wherein the model training is regression fitting training based on the Xgboost model.

C11. The server according to C9, wherein the predetermined time includes: a time-comparison time of the time to be sampled, and/or a chain-comparison time of the time to be sampled.

C12. According to the server described in C11, the comparable time of the time to be sampled includes: the time corresponding to the time to be sampled in the first N statistical periods, and the N is a natural number greater than 0;

The chain ratio of the time to be sampled includes: M times that are within the same statistical period as the time to be sampled and earlier than the time to be sampled, where M is a natural number greater than 0.

C13. The server according to C12, wherein the statistical period is one day.

C14. The server according to any one of C9 to C14, wherein the data to be sampled is negative sample data.

C15. The server according to any one of C9 to C14, wherein the data volume is traffic data.

Embodiments of the present application also provide D16. A non-volatile storage medium for storing a computer-readable program, the computer-readable program being used for a computer to execute the sample sampling method as described in any one of A1 to A7 .

Claims (10)

1. A method of sampling a sample, comprising:

acquiring a plurality of scale factors at preset time and corresponding data volumes;

performing model training according to the obtained scale factors and the corresponding data quantity to generate a scale factor prediction model;

predicting a scale factor according to the scale factor prediction model and the data volume;

the data is sampled using the predicted scale factor.

2. The sample sampling method of claim 1, wherein the model training is a regression fit training based on an Xgboost model.

3. The sample sampling method as claimed in claim 1, wherein said predetermined time instants comprise: the same-ratio time of the time to be sampled and/or the ring-ratio time of the time to be sampled.

4. A method for sampling samples according to claim 3, characterized in that said times of parity of the times to be sampled comprise: the time corresponding to the time to be sampled in the first N statistical periods is a natural number greater than 0;

the ring ratio time of the time to be sampled comprises: and the time to be sampled are in the same statistical period and are earlier than M times of the time to be sampled, wherein M is a natural number larger than 0.

5. The sample sampling method of claim 4, wherein the statistical period is one day.

6. The sample sampling method according to any one of claims 1 to 5, wherein the data to be sampled is negative sample data.

7. The sample sampling method according to any one of claims 1 to 5, wherein the data volume is flow data.

8. A sample sampling device, comprising:

the acquisition module is used for acquiring the scale factors and the corresponding data volumes at a plurality of preset moments;

the model generation module is used for carrying out model training according to the obtained scale factors and the corresponding data quantity to generate a scale factor prediction model;

the prediction module is used for predicting the scale factor according to the scale factor prediction model and the data volume;

a sampling module for sampling the data using the predicted scale factor.

9. A server, comprising:

at least one processor; and the number of the first and second groups,

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the steps of:

acquiring a plurality of scale factors at preset time and corresponding data volumes;

performing model training according to the obtained scale factors and the corresponding data quantity to generate a scale factor prediction model;

predicting a scale factor according to the scale factor prediction model and the data volume;

the data is sampled using the predicted scale factor.

10. A non-volatile storage medium storing a computer-readable program for causing a computer to perform the sample sampling method of any one of claims 1 to 7.