CN114791877A - Recommendation algorithm testing method and advertisement recommendation system - Google Patents
Recommendation algorithm testing method and advertisement recommendation system Download PDFInfo
- Publication number
- CN114791877A CN114791877A CN202210116486.XA CN202210116486A CN114791877A CN 114791877 A CN114791877 A CN 114791877A CN 202210116486 A CN202210116486 A CN 202210116486A CN 114791877 A CN114791877 A CN 114791877A
- Authority
- CN
- China
- Prior art keywords
- algorithm
- recommendation
- target user
- test
- experiment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Prevention of errors by analysis, debugging or testing of software
- G06F11/3668—Testing of software
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/02—Marketing; Price estimation or determination; Fundraising
- G06Q30/0241—Advertisements
- G06Q30/0251—Targeted advertisements
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/02—Marketing; Price estimation or determination; Fundraising
- G06Q30/0241—Advertisements
- G06Q30/0251—Targeted advertisements
- G06Q30/0255—Targeted advertisements based on user history
- G06Q30/0256—User search
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Development Economics (AREA)
- Strategic Management (AREA)
- Finance (AREA)
- Theoretical Computer Science (AREA)
- Accounting & Taxation (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Game Theory and Decision Science (AREA)
- Entrepreneurship & Innovation (AREA)
- Economics (AREA)
- Marketing (AREA)
- General Business, Economics & Management (AREA)
- General Engineering & Computer Science (AREA)
- Quality & Reliability (AREA)
- Computer Hardware Design (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
本申请提供一种推荐算法的测试方法和广告推荐系统，所述推荐算法的测试方法，包括：基于待测试的推荐算法，构造相互之间为正交关系的至少两个算法测试实验，其中，每个算法测试实验中包括需相互比较的至少两种推荐算法；在每一个算法测试实验中，将参与实验人群中的目标用户以随机方式平均分配至所述算法测试实验的各个推荐算法中，其中，每个算法测试实验基于相同的实验人群；根据所述目标用户对应的推荐算法，生成向各个所述目标用户推送的推荐内容；搜集所述实验人群中各个目标用户针对所述推荐内容的反馈操作的数据，对各推荐算法的算法质量进行评估。
The present application provides a method for testing a recommendation algorithm and an advertising recommendation system. The method for testing a recommendation algorithm includes: constructing at least two algorithm testing experiments that are orthogonal to each other based on the recommendation algorithm to be tested, wherein, Each algorithm testing experiment includes at least two recommendation algorithms that need to be compared with each other; in each algorithm testing experiment, the target users in the experimental population are equally distributed to each recommendation algorithm in the algorithm testing experiment in a random manner, Wherein, each algorithm test experiment is based on the same experimental population; according to the recommendation algorithm corresponding to the target user, the recommended content to be pushed to each of the target users is generated; the target users in the experimental population are collected for the recommended content Feedback operation data to evaluate the algorithm quality of each recommendation algorithm.
Description
Technical Field
The application relates to the field of data processing, in particular to a recommendation algorithm testing method and an advertisement recommendation system, and also relates to a recommendation algorithm testing device, electronic equipment and a computer storage medium.
Background
With the development of internet technology, the amount of information carried on the network is more and more abundant, and related information recommendation is gradually becoming a trend.
In the course of a particular application, the information recommended to the user is typically calculated by some recommendation algorithm. In order to obtain a suitable recommendation algorithm to send recommendation information to a user, the prior art generally adopts an AB test experiment method to test the recommendation algorithm, and evaluates the quality of the recommendation algorithm according to the test result. Specifically, the user is randomly divided into a group A and a group B, and two recommendation algorithms which are mutually contrasted are respectively used for distinguishing the advantages and disadvantages of the recommendation algorithms through comparison of test experiments, so that the more excellent recommendation algorithm is selected.
At present, various recommendation strategies and recommendation algorithms generated along with the recommendation strategies are greatly emerged, and the recommendation algorithms need to be tested and tested; in addition, each recommendation algorithm may need to adjust some key parameters, and each adjustment often needs to be tested; under the condition that the test experiment demand is so vigorous, the test through a special test platform becomes a conventional choice, however, how to quickly realize the test of various recommended algorithms is difficult to realize under the current technical conditions.
Disclosure of Invention
The application provides a method for testing a recommendation algorithm and an advertisement recommendation system, which are used for solving the technical problems in the background art.
The method for testing the recommendation algorithm provided by the application comprises the following steps:
constructing at least two algorithm test experiments which are in an orthogonal relation with each other based on the recommendation algorithms to be tested, wherein each algorithm test experiment comprises at least two recommendation algorithms which need to be compared with each other;
in each algorithm test experiment, target users participating in an experiment group are evenly distributed to each recommendation algorithm of the algorithm test experiment in a random mode, wherein each algorithm test experiment is based on the same experiment group;
generating recommended contents pushed to each target user according to a recommendation algorithm corresponding to the target user;
and collecting data of feedback operation of each target user aiming at the recommended content in the experimental population, and evaluating the algorithm quality of each recommendation algorithm.
Optionally, in the step of averagely allocating the target users in the experiment group to each recommendation algorithm of the algorithm test experiment, algorithm labels are allocated to the target users according to the recommendation algorithm to which the target users are allocated;
the step of generating the recommended content pushed to each target user according to the recommendation algorithm corresponding to the target user includes:
obtaining an algorithm label of the target user;
classifying the target user into a target group of a corresponding recommendation algorithm according to the algorithm label of the target user; for a target user, more than one recommended algorithm target population which can belong to different algorithm test experiments;
and each advertisement recommendation algorithm pushes the recommended content in the target crowd belonging to the recommendation algorithm.
Optionally, the method further includes:
obtaining a promotion budget set for recommended content, and averagely distributing the promotion budget to each recommendation algorithm of each algorithm test experiment, wherein each algorithm test experiment corresponds to the total promotion budget;
and correspondingly deducting the promotion cost of the recommendation algorithm corresponding to the target user corresponding to the feedback operation of the target user on the recommendation content.
Optionally, the method includes:
obtaining a preset popularization plan;
selecting the recommendation algorithm to be tested according to the promotion plan;
constructing a corresponding algorithm test experiment according to the selected to-be-tested recommendation algorithm corresponding to the promotion plan and the constraint conditions provided by the promotion plan; the algorithm test experiment can be one or more corresponding to the promotion plan.
Optionally, the collecting feedback operations of the target users in the experimental population for the recommended content includes obtaining a recommendation algorithm corresponding to the target users.
Optionally, each algorithm test experiment includes a reference recommendation algorithm.
Optionally, the generating, according to the recommendation algorithm corresponding to the target user, the recommendation content pushed to each target user includes:
determining a preset time period of an algorithm test experiment;
randomly distributing each target user in the participating experiment crowd to each time point of the preset time period;
and when the test time reaches the time point corresponding to the target user, generating recommendation content pushed to the target user according to a recommendation algorithm corresponding to the target user.
The present application also provides an advertisement recommendation system, comprising:
the advertisement main module is used for constructing at least two algorithm test experiments which are in an orthogonal relation with each other based on recommendation algorithms to be tested, wherein each algorithm test experiment comprises at least two recommendation algorithms which need to be compared with each other;
the flow distribution module is used for distributing target users participating in the experimental population to each recommended algorithm of the algorithm testing experiment in each algorithm testing experiment in an average manner, wherein each algorithm testing experiment is based on the same experimental population;
and the advertisement recommending module is used for generating recommended advertisements pushed to the target users according to the recommending algorithm corresponding to the target users.
The application also provides a testing device for the recommendation algorithm, which comprises:
the system comprises an experiment construction unit, a test unit and a test unit, wherein the experiment construction unit is used for constructing at least two algorithm test experiments which are in an orthogonal relation with each other based on recommendation algorithms to be tested, and each algorithm test experiment comprises at least two recommendation algorithms which need to be compared with each other;
the system comprises a user distribution unit, a recommendation unit and a recommendation unit, wherein the user distribution unit is used for distributing target users participating in an experimental group to recommended algorithms of the algorithm testing experiment in each algorithm testing experiment in an average manner, and each algorithm testing experiment is based on the same experimental group;
the content recommendation unit is used for generating recommendation content pushed to each target user according to a recommendation algorithm corresponding to the target user;
and the algorithm evaluation unit is used for collecting data of feedback operation of each target user aiming at the recommended content in the experimental population and evaluating the algorithm quality of each recommended algorithm.
This application provides an electronic equipment simultaneously, includes:
a processor;
a memory for storing a program of the method, which program, when read and executed by the processor, performs the steps of: constructing at least two algorithm test experiments which are in an orthogonal relation with each other based on the recommendation algorithms to be tested, wherein each algorithm test experiment comprises at least two recommendation algorithms which need to be compared with each other; in each algorithm test experiment, target users participating in the experiment population are evenly distributed to recommended algorithms of the algorithm test experiment, wherein each algorithm test experiment is based on the same experiment population; generating recommended contents pushed to each target user according to a recommendation algorithm corresponding to the target user; and collecting data of feedback operation of each target user aiming at the recommended content in the experimental population, and evaluating the algorithm quality of each recommendation algorithm.
The present application also provides a computer storage medium storing a computer program that, when executed, performs the steps of:
constructing at least two algorithm test experiments which are in an orthogonal relation with each other based on the recommendation algorithms to be tested, wherein each algorithm test experiment comprises at least two recommendation algorithms which need to be compared with each other; in each algorithm test experiment, target users participating in the experiment population are evenly distributed to recommended algorithms of the algorithm test experiment, wherein each algorithm test experiment is based on the same experiment population; generating recommended contents pushed to each target user according to a recommendation algorithm corresponding to the target user; and collecting data of feedback operation of each target user aiming at the recommended content in the experimental population, and evaluating the algorithm quality of each recommendation algorithm.
Compared with the prior art, the method has the following advantages:
according to the method for testing the recommendation algorithm, a plurality of algorithm testing experiments which are orthogonal to each other are constructed to compare different recommendation algorithms, and the plurality of different recommendation algorithms can be tested in the same time period, so that the testing experiments have concurrency, and the testing efficiency is guaranteed; in addition, the groups of people participating in the experiment are respectively and evenly distributed to each recommendation algorithm of the algorithm test experiment in a random mode in different algorithm test experiments, so that the experimental results of the different algorithm test experiments cannot be influenced by some correlation among experimental objects in the different algorithm test experiments, and the accuracy of each algorithm test experiment is ensured.
Drawings
FIG. 1 is a schematic structural diagram of an advertisement recommendation system according to an embodiment of the present application scenario;
FIG. 2 is a diagram illustrating interaction between a target user and an advertisement recommendation system according to an embodiment of the present application scenario;
FIG. 3 is a flowchart of a method for testing a recommendation algorithm provided in a first embodiment of the present application;
fig. 4 is a structural diagram of a testing apparatus for a recommendation algorithm according to a third embodiment of the present application;
fig. 5 is a schematic structural diagram of an electronic device according to a third embodiment of the present application.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. This application is capable of embodiments in many different forms than those described herein, and it will be appreciated by those skilled in the art that similar language may be used without departing from the spirit and scope of the application, and therefore the application is not limited to the specific embodiments disclosed below.
The embodiment of the application provides a method for testing a recommendation algorithm and an advertisement recommendation system, which will be described in detail in the following embodiments one by one.
In order to facilitate understanding of the recommendation algorithm testing method provided by the present application, the present application first introduces the advertisement recommendation system in combination with a specific use scenario. It should be noted that although the specific usage scenario of the present application focuses on advertisement recommendation, the technical solution can also be applied to recommendation of various other contents. The following describes embodiments of the present application mainly in terms of advertisement recommendation.
Referring to fig. 1, fig. 1 is a schematic structural diagram of an advertisement recommendation system according to an embodiment of the present application scenario.
Fig. 1 includes: advertisement main module 101, flow distribution module 102, advertisement recommendation module 103.
The advertisement main module 101 is configured to construct at least two algorithm test experiments that are orthogonal to each other based on recommendation algorithms to be tested, where each algorithm test experiment includes at least two recommendation algorithms to be compared with each other.
Specifically, the algorithm test experiment specifically refers to an AB test experiment set for the recommended algorithm. The recommendation algorithm to be tested comprises: the method comprises a recommendation algorithm 1, a recommendation algorithm 2, a recommendation algorithm 3 and a recommendation algorithm 4, wherein the algorithm test experiment 1 and the algorithm test experiment 2 can be constructed according to the algorithms, the algorithm test experiment 1 comprises an A1 group and a B1 group, the A1 group uses the recommendation algorithm 1, the B1 group uses the recommendation algorithm 2, the recommendation algorithm 1 and the recommendation algorithm 2 are associated algorithms which can be compared with each other, in general, one algorithm can be a reference algorithm, and the other algorithm is an algorithm which actually needs to be tested; similarly, the algorithm test experiment 2 includes a group a2 and a group B2, wherein the group a2 uses the recommendation algorithm 3, and the group B2 uses the recommendation algorithm 4; the recommendation algorithm 3 and the recommendation algorithm 4 are correlation algorithms that can be compared with each other, and in general, one of the algorithms may be a reference algorithm and the other algorithm may be an algorithm that actually needs to be tested. The reference algorithm may be an algorithm whose effect has been known as a reference for comparison of the algorithms to be tested, and in the present embodiment, it is assumed that the B1 group and the B2 group are reference algorithms of the algorithm test experiment 1 and the algorithm test experiment 2, respectively.
Further, the advertisement main module further comprises: budget allocation module 1011.
The budget allocation module 1011 is configured to obtain a promotion budget set for a recommended advertisement, and evenly allocate the promotion budget to each recommendation algorithm of each algorithm test experiment, where each algorithm test experiment corresponds to a full amount of the promotion budget.
For example, assuming that the promotion budget that an advertiser intends to promote for this advertisement is 1000 yuan, the a1 and B1 groups in experiment 1 were tested by the algorithm to be able to allocate 500 yuan (50%) of budget, respectively. Also, the a1 and B1 groups in simultaneous algorithmic test experiment 2 each also allocated a budget of 500 dollars (50%). Budgets in the two groups of algorithm test experiments are calculated independently, and the budgets are consumed for promoting 1000 yuan.
The flow distribution module 102 is configured to, in each algorithm test experiment, evenly distribute target users participating in the experiment population to each recommended algorithm of the algorithm test experiment, where each algorithm test experiment is based on the same experiment population.
For example, assuming that the experimental population involved in the experiment totals 1000 persons, for the algorithm test experiment 1, these 1000 persons are randomly assigned to the a1 group and the B1 group of the algorithm test experiment 1; likewise, for algorithmic test experiment 2, these 1000 persons were randomly assigned to groups a2 and B2 of algorithmic test experiment 2.
Further, the target user 1 in the experimental population carries the algorithm label corresponding to the algorithm assigned thereto, for example, if the target user 1 is assigned to the group B1 in the algorithm test experiment 1, and is assigned to the group a2 in the algorithm test experiment 2, the algorithm label of the target user 1 is B1a 2. The algorithm label has the effect that once each target user is extracted after the algorithm label is marked on each target user, the recommendation algorithm adopted by the target user can be known through the algorithm label. For example, for the target user 1, recommendation algorithm 2 corresponding to B1 and recommendation algorithm 3 corresponding to a2 are adopted; because the recommendation algorithm 2 and the recommendation algorithm 3 are algorithms in two algorithm experiments, and are orthogonal to each other, i.e. they do not affect each other, the target user 1 can recommend advertisements by using two recommendation algorithms at the same time. For example, the advertisement recommended by both recommendation algorithms may be provided to the target user 1, or recommendation values may be calculated according to the same weight for the recommendation degrees of both recommendation algorithms, and recommended to the user according to the recommendation values.
And the advertisement recommending module 103 is configured to generate recommended advertisements pushed to the target users according to the recommending algorithm corresponding to the target users.
Further, the advertisement recommendation module 103 is specifically configured to determine, according to the algorithm tag corresponding to the target user, a recommendation algorithm corresponding to the algorithm tag; and recommending the advertisement to the target user according to the recommendation algorithm.
For example, for directly recommended advertisements, the recommendation algorithm may determine advertisements that meet the user preferences according to the preference information of the target user, and for example, for advertisements recommended based on search terms, the recommendation algorithm may recommend advertisements to the target user according to keywords output by the target user in a search engine.
Further, the advertisement recommendation module 103 includes: an algorithm access layer, a recommended content recall layer, a recommended content sequencing layer and the like.
The algorithm access layer is used for importing a corresponding recommendation algorithm in the advertisement recommendation module according to the algorithm label of the target user; the recommended content recalling layer is used for obtaining recommended content to be recommended according to the recommendation algorithm; and the recommended content sequencing layer is used for sequencing the recommended content.
In an embodiment of the present application scenario, in order to verify the quality of the recommendation algorithm, the advertisement recommendation system further includes: and the algorithm evaluation module 104 is used for collecting feedback operation data of each target user in the experimental population aiming at the recommended content and evaluating the algorithm quality of each recommendation algorithm.
Specifically, the evaluation of the algorithm quality of each recommendation algorithm is specifically obtained according to the residual budget of the test algorithm corresponding to each recommendation algorithm test experiment in the budget allocation module 1011.
In a specific and applicable process, if the target user 1 with the algorithm tag B1a2 obtains the advertisement recommended by the recommendation algorithm corresponding to the algorithm tag B1a2 and the target user 1 browses or clicks the recommended advertisement, the budget amount corresponding to the B1 group of primary advertisement of algorithm test experiment 1 and the budget amount corresponding to the a2 group of primary advertisement of algorithm test experiment 2 may be deducted respectively.
Further, in order to facilitate understanding of the interaction relationship between the target user and the advertisement recommendation system in the scenario described in the foregoing scenario embodiment, an interaction process between the target user and the advertisement recommendation system is described below with reference to fig. 2.
Referring to fig. 2, fig. 2 is a diagram of an interaction relationship between a target user and an advertisement recommendation system according to an embodiment of the present application scenario.
As shown in FIG. 2, the target user performs steps S201 to S205 in the process of interacting with the advertisement recommendation system, which performs steps S206 to S208.
First, before the interactive flow starts, a test experiment in the advertisement recommendation system is first created by the following steps.
Step S206, according to a promotion plan preset by an advertiser, setting a promotion budget and a recommendation algorithm to be tested required by the current advertisement promotion;
step S207, constructing at least two algorithm test experiments which are in an orthogonal relation with each other based on the recommendation algorithms to be tested, wherein each algorithm test experiment comprises at least two recommendation algorithms which need to be compared with each other;
step S208, based on the obtained promotion budget set for the recommended content, averagely distributing the promotion budget to each recommendation algorithm of each algorithm test experiment in a random mode, wherein each algorithm test experiment corresponds to the total promotion budget.
After the advertisement recommendation system is established, the terminal equipment of the target user and the advertisement recommendation system sequentially execute the following steps:
step S201, the target user sends first request information for obtaining an algorithm label corresponding to the target user to an advertisement recommendation system through terminal equipment;
step S202, the advertisement recommendation system returns an algorithm label corresponding to the target user to the terminal equipment of the target user in response to the first request information;
step S203, the target user sends second request information of advertisement recommendation carrying the algorithm label to the advertisement recommendation system through terminal equipment;
step S204, the advertisement recommending system analyzes the second request information to obtain the algorithm label, and sends a corresponding recommended advertisement to the terminal equipment of the target user according to the algorithm label;
step S205, the terminal device of the target user receives and displays the recommended advertisement, and sends fee deduction request information carrying the algorithm label to the advertisement recommendation system after the target user selects to click the recommended advertisement.
And S206, the advertisement recommendation system analyzes the fee deduction request information to obtain the algorithm labels, and determines the testing algorithms to be deducted in each algorithm testing experiment according to the algorithm labels to finish fee deduction operation.
At the end of the experiment, the staff judges the quality of the testing algorithms corresponding to different groups by comparing the advertisement budgets remaining in the different groups of the experiment through algorithm testing, for example, for algorithm testing experiment 1, if the remaining advertisement budget of group a1 is lower than the remaining advertisement budget of group B1, it indicates that the algorithm quality of algorithm 1 corresponding to group a1 is higher than that of algorithm 2 corresponding to group B1; for Algorithm test experiment 2, if the remaining advertising budget for group A2 is higher than the remaining advertising budget for group B2, it indicates that the algorithm quality of Algorithm 3 for group A2 is lower than that of Algorithm 4 for group B2. In addition, through budget allocation, after the budget of a certain party is deducted, the test can be quitted, and the test is finished; thus, the conditions for the end test were obtained.
It should be noted that the above description of the scenario embodiment of the present application is not used to limit the application scenario of the recommended algorithm test method provided by the present application, for example, in other scenario embodiments, the construction of the algorithm test experiment is not limited to that two algorithm test experiments are provided by the present application, and for the case that more groups of test algorithms need to be tested simultaneously, the number of the algorithm test experiments may be determined according to the number of the recommended algorithms; for another example, the method for testing a recommendation algorithm provided by the present application may also be used to test a recommendation algorithm in a search engine, and provide corresponding content recommendation for a user while testing the algorithm, which is not limited in the present application.
A first embodiment of the present application provides a method for testing a recommendation algorithm, please refer to fig. 3, and fig. 3 is a flowchart of the method for testing the recommendation algorithm provided in the first embodiment of the present application. The method for testing the recommendation algorithm includes the following steps S301 to S304. The execution subject of this embodiment is an advertisement recommendation system.
Step S301, constructing at least two algorithm test experiments which are in an orthogonal relation with each other based on the recommendation algorithms to be tested, wherein each algorithm test experiment comprises at least two recommendation algorithms which need to be compared with each other.
The test algorithm to be recommended may be understood as an algorithm for recommending content for a user in a computer system, and in a specific application configuration, the form of the recommendation algorithm may be various, for example: the recommendation algorithm may be a program deployed in a computer system for recommending content to a target user, or may be a neural network or the like that is obtained through machine learning and is capable of outputting recommended content according to a search keyword input by the target user in a search engine or a browsing record of the target user. The present application is not limited thereto.
In an implementation scenario of the present application, the algorithm test experiment is specifically used for recommending advertisements to a user and sorting recommended advertisements; of course, algorithmic testing experiments may also be used to make recommendations for other content.
In the first embodiment of the present application, the algorithm test experiments include multiple sets of algorithm test experiments performed simultaneously, each algorithm test experiment has the same test period, specifically, the test period may be in units of days/weeks, and the present application is not limited thereto. Furthermore, the different algorithm test experiments are orthogonal to each other, that is, the algorithms tested by the different algorithm test experiments do not interfere with each other, that is, one algorithm test experiment affects the test effect of the other algorithm test experiment. When constructing an algorithm test experiment of an orthogonal relation, the principle of the tested algorithm needs to be deeply understood, and the test with mutual influence of results in the simultaneously-performed algorithm test experiment is avoided, so that the distortion of the test result is avoided.
In the first embodiment of the present application, the algorithm test specifically refers to an AB test set for the recommended algorithm. The AB test experiment is to make two (A/B) or a plurality of (A/B/n) different versions for Web (World Wide Web) or APP (Application, mobile phone software) interfaces or processes, make the access groups with the same (or similar) composition randomly access the versions in the same time dimension, then collect the user experience data and business data of the access users of each version, and finally obtain the best version for formal adoption through analysis and evaluation.
For example, the algorithmic test experiment may include: an algorithm test experiment 1, an algorithm test experiment 2 and an algorithm test experiment 3.
The algorithm test experiment 1 comprises the following steps: group a1 and group B1, where group a1 was used for test algorithm 1 and group a2 was used for test algorithm 2;
algorithm test experiment 2 included: group a2 and group B2, where group a2 was used for test algorithm 3 and group a2 was used for test algorithm 4;
algorithm test experiment 3 included: group A3 and group B3, with group A3 being used for test algorithm 5 and group A3 being used for test algorithm 6.
In the construction of practical application, the construction of the algorithm test experiment is related to a specific application scenario, for example: in an optional embodiment of the present application, the algorithm test experiment is mainly used for selecting a suitable recommendation algorithm to recommend advertisements to advertisers.
The advertiser refers to a legal person, an economic organization or an individual who designs, makes or issues advertisements for a promoter or provides services, or entrusts others, and is an initiator of an advertisement campaign, and the method comprises the following steps: in merchants selling or advertising their products and services.
Specifically, if an advanced test needs to be performed for promotion of a certain advertiser so as to select the most appropriate algorithm for the advertiser, the above algorithm test experiment designed for the advertiser can be obtained by the following method:
firstly, obtaining a preset popularization plan;
secondly, selecting a candidate recommendation algorithm according to the promotion plan;
and finally, constructing corresponding algorithm test experiments according to the selected candidate recommendation algorithms corresponding to the promotion plans and the constraint conditions provided by the promotion plans, wherein each promotion plan corresponds to one or more algorithm test experiments.
Wherein the preset promotion plan may include: user group information for publicizing products in the advertisement, advertisement putting strategy information, advertisement publicity form information, advertisement total budget information and other information closely related to advertisement publicity. Because the content of the preset promotion plan is not the key point of the protection requested by the application, the content of the promotion plan is not limited in the embodiment of the application.
Further, the candidate recommendation algorithm is selected according to the promotion plan, and the candidate recommendation algorithm can be obtained according to a candidate algorithm selection model with promotion plan information as input and recommendation algorithm information as output. In a specific and applied process, the candidate algorithm selection model may be obtained by Machine Learning (ML) training, and specifically, the Learning sample of the Machine Learning may be obtained by using specific content of the promotion plan and a suitable recommendation algorithm corresponding to the promotion plan. Machine learning (a multi-field cross subject, which relates to multi-subjects such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and the like) is specially used for researching how a computer simulates or realizes the learning behavior of human beings so as to obtain new knowledge or skills and reorganize the existing knowledge structure to continuously improve the performance of the existing knowledge structure. Machine learning generally includes techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and the like. And feature machine learning belongs to a branch of Artificial Intelligence (AI) technology.
It should be noted that the above manner of obtaining the candidate recommendation algorithm is only an optional implementation manner of the present application, and in other implementation manners of the present application, the candidate recommendation algorithm may also be obtained in other manners, for example, a suitable candidate recommendation algorithm may be selected according to the characteristics of the promotion plan. The present application is not limited thereto.
In addition, for algorithmic testing in the context of recommending advertisements, the promotion plan typically includes the promotion budget for the advertisement, i.e., the advertiser's investment in its advertising campaign. In a specific application process, if an advertisement is recommended to a user and the user browses the advertisement by clicking the advertisement or the like, the advertisement is successfully advertised once, and advertisement promotion cost required by the advertisement once needs to be deducted from the total promotion budget. Further, for the specific application of the popularization budget in the first embodiment of the present application, reference may be made to the description of step S203, which is not described herein again.
Further, after an algorithm test experiment with a recommended algorithm as a main body is constructed, corresponding constraint conditions need to be added to the algorithm test experiment, and in the embodiment of the present application, the constraint conditions include: experimental population, popularization budget and algorithm testing time.
The above constraints are described below in connection with a specific testing procedure for the algorithm.
In the first embodiment of the present application, the method in step S302 is adopted to allocate target users in the experimental population, that is, the number of testing people for different testing algorithms in the algorithm experiment is limited by means of step S302.
Step S302, in each algorithm test experiment, target users participating in the experiment crowd are evenly distributed to each recommended algorithm of the algorithm test experiment, wherein each algorithm test experiment is based on the same experiment crowd;
for example, the algorithm test experiment 1 includes: group a1 and group B1, where group a1 was used for test algorithm 1 and group B1 was used for test algorithm 2; test experiment 2 included: group a2 and group B2, where group a2 was used for test algorithm 3 and group a2 was used for test algorithm 4.
Then, for the algorithmic test experiment 1, if the experimental population totals 100, then the 100 people will be randomly divided into 2 groups, assigned to groups a1 and B1, respectively. For algorithmic test experiment 2, the 100 persons were re-randomized and assigned to the a2 and B2 groups, respectively. (that is, the algorithm test experiment 1 and the algorithm test experiment 2 are two parallel experiments, wherein the algorithm test experiment 1 is divided into two groups to test the algorithm 1 and the algorithm 2 by using half of the total number of the experimental population, respectively, and the algorithm test experiment 2 is also divided into two groups to test the algorithm 3 and the algorithm 4 by using half of the total number of the experimental population, respectively).
For the promotion budget: in embodiments of the present application, the promotion plan typically includes a promotion budget for the advertisement, i.e., the advertiser's placement of its ad campaigns. For example, for a scenario of advertisement recommendation, if an advertisement is recommended to a user and the user browses the advertisement by clicking the advertisement, it means that the advertisement is successfully advertised once, and the advertisement promotion cost required for one advertisement needs to be deducted from the total promotion budget.
In the specific application process, the method similar to the above step S202 is adopted to limit the advertisement budgets corresponding to different test algorithms in different algorithm test experiments in the algorithm test experiments.
Specifically, the allocation of the promotion budget includes the following steps S1 and S2.
Step S1, obtaining a promotion budget set for recommended content, and averagely distributing the promotion budget to each recommendation algorithm of each algorithm test experiment, wherein each algorithm test experiment corresponds to the total promotion budget;
for example, the algorithm test experiment 1 includes: group a1 and group B1, where group a1 was used for test algorithm 1 and group a2 was used for test algorithm 2; algorithm test experiment 2 included: group a2 and group B2, with group a2 being used for test algorithm 3 and group a2 being used for test algorithm 4.
Then, if the promotion budget is 1000 yuan, the total budget allocated to the algorithm test experiment 1 is 1000 yuan, and the total budget allocated to the algorithm test experiment 2 is also 1000 yuan;
the popularization budget allocated to the A1 group experiment of the algorithm test experiment 1 is 500 yuan, and the popularization budget allocated to the B1 group experiment is also 500 yuan;
the popularization budget allocated to the a2 group of experiments of the algorithm test experiment 2 is 500 yuan, and the popularization budget allocated to the B2 group of experiments is also 500 yuan (that is, the algorithm test experiment 1 and the algorithm test experiment 2 are two experiments performed in parallel, where the algorithm test experiment 1 is used for testing the algorithm 1 and the algorithm 2, and the algorithm test experiment 2 is used for testing the algorithm 3 and the algorithm 4).
And step S2, corresponding to the feedback operation of the target user on the recommended content, carrying out corresponding deduction operation in the promotion cost of the recommendation algorithm corresponding to the target user.
Specifically, for the related description of the step S2, reference may be made to the following description of the step S203, and details are not repeated here.
Further, in order to accurately determine the algorithm corresponding to the target user, the step S202 further includes allocating an algorithm label to the target user according to the recommended algorithm allocated to the target user.
Specifically, the algorithm label of the target user is mainly used for representing an algorithm adopted when providing recommended content for the target user. For example, still taking the above algorithm test experiment 1 and algorithm test experiment 2 as an example, assuming that, in the process of performing target user allocation for the algorithm test experiment 1, the target user 1 is allocated to the test algorithm 2 in the algorithm test experiment 1 (i.e., allocated to the group B1 in the algorithm test experiment 1), and, in the process of performing target user allocation for the non-algorithm test experiment 2, the target user 1 is allocated to the test algorithm 3 in the algorithm test experiment 2 (i.e., allocated to the group a2 in the algorithm test experiment 2), then the algorithm label corresponding to the target user 1 is B1a 2.
Step S303, generating recommendation contents pushed to each target user according to the recommendation algorithm corresponding to the target user.
Specifically, the above step S303 includes the following steps S3 to S5.
Step S3, obtaining the algorithm label of the target user;
step S4, classifying the target user into a target crowd of a corresponding recommendation algorithm according to the algorithm label of the target user; for a target user, the target user can belong to more than one recommended algorithm target crowd of different algorithm test experiments;
and step S5, each advertisement recommendation algorithm pushes the recommendation content in the target crowd belonging to the recommendation algorithm.
For example, assuming that the algorithm tag of the target user 1 is B1a2, the target user 1 should be classified into group B1 in algorithm test experiment 1 and group a2 in algorithm test experiment 2.
In an alternative embodiment of the present application, the step S203 further includes assigning each target user in the experiment population in a time dimension, so as to make the algorithmic test experiment closer to the real situation.
Specifically, the step S303 further includes the following steps S6 to S8.
Step S6, determining a preset time period of an algorithm test experiment;
the time period of the algorithm test experiment, that is, the test period preset for each algorithm test experiment, specifically, the preset time period may be in units of days/weeks, which is not limited in the present application.
Step S7, randomly distributing each target user in the participating experimental population to each time point of the preset time period;
for example, assuming that the experimental population includes 100 target users and the preset time period is 1 day, the 100 target users may be randomly allocated to each time point of the preset time period.
And step S8, when the test time reaches the time point corresponding to the target user, generating the recommended content pushed to the target user according to the recommendation algorithm corresponding to the target user.
Step S304, collecting data of feedback operation of each target user aiming at the recommended content in the experimental population, and evaluating the algorithm quality of each recommendation algorithm.
In an alternative embodiment of the present application, the feedback operation includes an effective operation and an ineffective operation, and still taking the above algorithm test experiment 1 and algorithm test experiment 2 as an example, it is assumed that the algorithm label B1a2 of the target user 1 is provided, and the recommended content recommended to the target user 1 according to the algorithm label includes advertisement 1 and advertisement 2. If the target user clicks any one of the advertisement 1 and the advertisement 2, the clicking operation is an effective operation, and if the target user selects to close the advertisement or perform other operations without clicking the advertisement 1 or the advertisement 2, the operation is an ineffective operation.
It should be noted that, the introduction of the feedback operation is limited to a scene of recommending an advertisement, and for other scenes, the definition of the effective operation and the ineffective operation of the feedback operation may be set according to actual situations, for example, if the recommendation algorithm is specifically used for recommending content related to a keyword to a target user according to the keyword input by the target user in a search engine; if the user selects to click the recommended content, after the user enters an introduction interface of the recommended content, and when the stay time length of the introduction interface is greater than or equal to a preset time threshold value, the click operation of the target user is considered as effective operation; and if the target user selects to click the recommended content and enters an introduction interface of the recommended content, the staying time length of the target user on the introduction interface is smaller than a preset time threshold, or the target user does not click the recommended content, the operation of the target user is considered as invalid operation.
Further, corresponding to the effective operation in the feedback operation of the target user on the recommended content, corresponding deduction operation is performed in the promotion cost of the recommendation algorithm corresponding to the target user.
Specifically, the following still explains the operation of deducting the promotion cost in the advertisement scene by taking the above algorithm test experiment 1 and algorithm test experiment 2 as examples.
Algorithm test experiment 1 included: group a1 and group B1, where group a1 was used for test algorithm 1 and group a2 was used for test algorithm 2; algorithm test experiment 2 included: group a2 and group B2, where group a2 was used for test algorithm 3 and group a2 was used for test algorithm 4, while group a1 and group B1 of test algorithm 1, and group a2 and group B2 of test algorithm were each assigned a budget of 500 dollars, the total number of experimental population being 500 people, where group a1 of test algorithm 1 was randomly assigned 250 people and group B1 was randomly assigned 250 people, respectively, and group a2 and group B2 of test algorithm 2 were randomly assigned 250 people, respectively.
Assuming that the algorithm label of the target user 1 in the experimental population is B1a2, the test algorithms corresponding to the algorithm label B1a2 are test algorithm 2 and test algorithm 3.
Further, the advertisement 1 and the advertisement 2 are recommended to the target user 1 according to the test algorithm 2 and the test algorithm 3, and the recommended advertisements are displayed in the terminal device of the target user, at this time, if the target user 1 performs a click operation on the advertisement 1 and the advertisement 2, it indicates that 1 successful advertisement recommendation is performed to the target user 1 according to the test algorithm 2 and the test algorithm 3.
Further, after a successful advertisement recommendation, the advertisement budgets in group B1 of algorithm test experiment 1 and group a2 of algorithm test experiment 2 are deducted according to algorithm label B1a 2. For example, if the cost of successful one-time consumption of the advertisement recommendation is 1 yuan, the budgets of 1 yuan of the B1 group and the A2 group are deducted respectively.
For each algorithm test experiment, the number of people and budget allocated to the recommendation algorithms which are compared with each other are the same, so that the different recommendation algorithms are compared in the same way, and different recommendation algorithm qualities can be determined only by the residual budget amounts of the different recommendation algorithms after the algorithm test experiment is finished.
For example, after the algorithm test experiment is finished, if 100-element budgets remain in the group a1 and 50-element budgets remain in the group B1 corresponding to the algorithm test experiment 1, it may be determined that the recommendation quality of the recommendation algorithm 2 in the algorithm test experiment 1 is higher than that of the recommendation algorithm 1; similarly, if the algorithm tests the budget of the remaining 25 elements of the group a2 and the budget of the remaining 35 elements of the group B2 corresponding to experiment 2, it can be determined that the recommendation quality of the recommendation algorithm 3 in the algorithm test experiment 2 is higher than that of the recommendation algorithm 4.
Based on the description of the first embodiment of the present application, it can be seen that the method for testing a recommendation algorithm provided by the present application compares different recommendation algorithms by constructing a plurality of algorithm test experiments in an orthogonal relationship with each other, so as to ensure the high efficiency of testing a plurality of different recommendation algorithms; in addition, the groups of people participating in the experiment are averagely distributed to the recommended algorithms of the algorithm test experiment, so that the problem that the test algorithms which are mutually compared in the algorithm test experiment compete for experimental objects is solved, and the test accuracy is greatly improved.
A third embodiment of the present application provides a testing apparatus for a recommendation algorithm, please refer to fig. 4, where fig. 4 is a structural diagram of the testing apparatus for the recommendation algorithm provided in the second embodiment of the present application.
The device for testing the recommendation algorithm comprises:
the experiment construction unit 401 is configured to construct at least two algorithm test experiments in an orthogonal relationship with each other based on the recommended algorithms to be tested, where each algorithm test experiment includes at least two recommended algorithms to be compared with each other;
the user allocation unit 402 is configured to, in each algorithm test experiment, evenly allocate target users participating in the experiment population to respective recommended algorithms of the algorithm test experiment in a random manner, where each algorithm test experiment is based on the same experiment population;
a content recommending unit 403, configured to generate recommended content pushed to each target user according to a recommendation algorithm corresponding to the target user;
and an algorithm evaluation unit 404, configured to collect data of feedback operations of the target users in the experimental population for the recommended content, and evaluate the algorithm quality of each recommendation algorithm.
Optionally, in the step of averagely allocating the target users participating in the experiment population to each recommendation algorithm of the algorithm test experiment, algorithm labels are allocated to the target users according to the recommendation algorithm allocated to the target users;
the step of generating the recommended content pushed to each target user according to the recommendation algorithm corresponding to the target user includes:
obtaining an algorithm label of the target user;
classifying the target user into a target group of a corresponding recommendation algorithm according to the algorithm label of the target user; for a target user, more than one recommended algorithm target population which can belong to different algorithm test experiments;
and each advertisement recommendation algorithm pushes the recommended content in the target crowd belonging to the recommendation algorithm.
Optionally, the apparatus is further configured to:
obtaining a promotion budget set for recommended content, and averagely distributing the promotion budget to each recommendation algorithm of each algorithm test experiment, wherein each algorithm test experiment corresponds to the total promotion budget;
and correspondingly deducting the promotion cost of the recommendation algorithm corresponding to the target user corresponding to the feedback operation of the target user on the recommendation content.
Optionally, the apparatus is further configured to:
obtaining a preset promotion plan;
selecting a candidate recommendation algorithm according to the promotion plan;
constructing a corresponding algorithm test experiment according to the selected candidate recommendation algorithm corresponding to the promotion plan and the constraint conditions provided by the promotion plan; the algorithm test experiment can be one or more corresponding to the promotion plan.
Optionally, the collecting feedback operations of the target users in the experimental population for the recommended content includes obtaining a recommendation algorithm corresponding to the target users.
Optionally, each algorithm test experiment includes a reference recommendation algorithm.
Optionally, the generating, according to the recommendation algorithm corresponding to the target user, the recommendation content pushed to each target user includes:
determining a preset time period of an algorithm test experiment;
randomly distributing each target user in the participating experiment population to each time point of the preset time period;
and when the test time reaches the time point corresponding to the target user, generating recommendation content pushed to the target user according to the recommendation algorithm corresponding to the target user.
Referring to fig. 5, fig. 5 is a schematic structural diagram of an electronic device according to a third embodiment of the present application.
The electronic device includes: a processor 501;
the memory 502 is used for storing a program of the method, which is read and executed by the processor 501 to perform the method according to the first embodiment of the present application.
Another embodiment of the present application provides a computer storage medium storing a computer program that, when executed, performs the method of the first embodiment of the present application.
It should be noted that, for detailed descriptions of devices and computer storage media provided in the embodiments of the present application, reference may be made to the relevant description of the first embodiment provided in the present application, and details are not described here again.
Although the present application has been described with reference to the preferred embodiments, it is not intended to limit the present application, and those skilled in the art can make variations and modifications without departing from the spirit and scope of the present application, therefore, the scope of the present application should be determined by the claims that follow.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
1. Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory computer readable media (transient media), such as modulated data signals and carrier waves.
2. It will be apparent to those skilled in the art that embodiments of the present application may be provided as a system or an electronic device. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
Claims (11)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210116486.XA CN114791877A (en) | 2022-02-07 | 2022-02-07 | Recommendation algorithm testing method and advertisement recommendation system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210116486.XA CN114791877A (en) | 2022-02-07 | 2022-02-07 | Recommendation algorithm testing method and advertisement recommendation system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114791877A true CN114791877A (en) | 2022-07-26 |
Family
ID=82460529
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210116486.XA Pending CN114791877A (en) | 2022-02-07 | 2022-02-07 | Recommendation algorithm testing method and advertisement recommendation system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114791877A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115391184A (en) * | 2022-08-09 | 2022-11-25 | 北京百度网讯科技有限公司 | A method, device and equipment for AB testing |
| CN119201729A (en) * | 2024-09-20 | 2024-12-27 | 前锦网络信息技术(上海)有限公司 | Testing method, device, equipment and storage medium for recruitment recommendation system |
| CN120163614A (en) * | 2025-02-25 | 2025-06-17 | 阿里巴巴华东有限公司 | Data processing method, device, program product, storage medium and electronic device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110413533A (en) * | 2019-09-03 | 2019-11-05 | 江苏满运软件科技有限公司 | AB test method, system, equipment and storage medium |
| CN113127328A (en) * | 2019-12-31 | 2021-07-16 | 国信优易数据股份有限公司 | Test system and test method for recommendation algorithm |
| CN113315672A (en) * | 2020-02-27 | 2021-08-27 | 腾讯科技（深圳）有限公司 | Flow distribution method and device, electronic equipment and computer readable storage medium |
-
2022
- 2022-02-07 CN CN202210116486.XA patent/CN114791877A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110413533A (en) * | 2019-09-03 | 2019-11-05 | 江苏满运软件科技有限公司 | AB test method, system, equipment and storage medium |
| CN113127328A (en) * | 2019-12-31 | 2021-07-16 | 国信优易数据股份有限公司 | Test system and test method for recommendation algorithm |
| CN113315672A (en) * | 2020-02-27 | 2021-08-27 | 腾讯科技（深圳）有限公司 | Flow distribution method and device, electronic equipment and computer readable storage medium |
Non-Patent Citations (1)
| Title |
|---|
| （美）罗恩·科哈维等: "《关键迭代 可信赖的线上对照实验》", vol. 1, 30 April 2021, 机械工业出版社, pages: 222 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115391184A (en) * | 2022-08-09 | 2022-11-25 | 北京百度网讯科技有限公司 | A method, device and equipment for AB testing |
| CN119201729A (en) * | 2024-09-20 | 2024-12-27 | 前锦网络信息技术(上海)有限公司 | Testing method, device, equipment and storage medium for recruitment recommendation system |
| CN120163614A (en) * | 2025-02-25 | 2025-06-17 | 阿里巴巴华东有限公司 | Data processing method, device, program product, storage medium and electronic device |
| CN120163614B (en) * | 2025-02-25 | 2026-01-27 | 阿里巴巴华东有限公司 | Data processing methods, apparatus, software products, storage media and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| AU2010201495B2 (en) | Touchpoint customization system | |
| US9183562B2 (en) | Method and system for determining touchpoint attribution | |
| US8255563B2 (en) | Method and system for determining overall content values for content elements in a web network and for optimizing internet traffic flow through the web network | |
| US20130085838A1 (en) | Incentive optimization for social media marketing campaigns | |
| CN114791877A (en) | Recommendation algorithm testing method and advertisement recommendation system | |
| US20150046223A1 (en) | Trend-factored rfm scores to improve campaign performance | |
| CN111178983A (en) | User gender prediction method, device, equipment and storage medium | |
| US11704705B2 (en) | Systems and methods for an intelligent sourcing engine for study participants | |
| US20110251889A1 (en) | Inventory clustering | |
| Zhao et al. | Shapley value methods for attribution modeling in online advertising | |
| CN111260416A (en) | Method and device for determining associated user of object | |
| CN112435070A (en) | Method, device and equipment for determining user age and storage medium | |
| CN109949089A (en) | A kind of method, apparatus and terminal of determining displaying rate | |
| Yin et al. | Modeling behavioral dynamics in digital content consumption: An attention-based neural point process approach with applications in video games | |
| Zhang et al. | Mining target users for mobile advertising based on telecom big data | |
| CN115345635B (en) | Recommended content processing method, recommended content processing device, computer equipment and storage medium | |
| US20160342699A1 (en) | Systems, methods, and devices for profiling audience populations of websites | |
| Däs et al. | Customer lifetime network value: customer valuation in the context of network effects: Däs M. et al. | |
| CN101449284A (en) | Scoring traffic quality to network sites using cross-correlated traffic parameters | |
| CN118710336A (en) | An advertising consulting service method and system based on AI | |
| Breur | How to evaluate campaign response—The relative contribution of data mining models and marketing execution | |
| Adikari et al. | Adaptive Ad Network Selection for Publisher‐Return Optimization in Mobile‐App Advertising | |
| CN111291795A (en) | Crowd characteristic analysis method and device, storage medium and computer equipment | |
| CN115860907A (en) | Loan recommendation mode determination method, device, equipment and storage medium | |
| Chen et al. | Multi-state open opinion model based on positive and negative social influences |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20240308 Address after: Room 303, 3rd Floor, Building 5, No. 699 Wangshang Road, Changhe Street, Binjiang District, Hangzhou City, Zhejiang Province, 310056 Applicant after: Hangzhou Alibaba Overseas Internet Industry Co.,Ltd. Country or region after: China Address before: 310052 room 508, 5th floor, building 4, No. 699 Wangshang Road, Changhe street, Binjiang District, Hangzhou City, Zhejiang Province Applicant before: Alibaba (China) Co.,Ltd. Country or region before: China |
|
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220726 |