CN109800441A - A kind of model output recommended method and device, model export recommender system - Google Patents
A kind of model output recommended method and device, model export recommender system Download PDFInfo
- Publication number
- CN109800441A CN109800441A CN201910105297.0A CN201910105297A CN109800441A CN 109800441 A CN109800441 A CN 109800441A CN 201910105297 A CN201910105297 A CN 201910105297A CN 109800441 A CN109800441 A CN 109800441A
- Authority
- CN
- China
- Prior art keywords
- task
- model
- feature
- parameter
- weight
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a kind of model output recommended method and device, model output recommender system, wherein, model output recommended method includes: that task input feature vector is separately input into few two task models, obtains the task output feature and the corresponding confidence level of task output feature of each task model；The task of corresponding at least two task model of task input feature vector is exported into feature and the corresponding confidence level of task output feature inputs parameter search model, so that parameter search model exports the weight parameter of the corresponding confidence level of feature and parameter search model according to the task of each task model, obtain the confidence weight of each task model, and feature is exported according to the task that the confidence weight of each task model is recommended, to have difference to different sentence adaptability in the same task model, in the case that multiple tasks model exports multiple results, after parameter search model is to the evaluation of task model, it filters out and more preferably exports result.
Description
Technical field
This application involves field of artificial intelligence, in particular to a kind of model output recommended method and device, model are defeated
Recommender system and its training method and device, calculating equipment, storage medium and chip out.
Background technique
Hyper parameter is the parameter being arranged before task model starts learning process, rather than trained by task model
The supplemental characteristic arrived.Under normal conditions, it needs to optimize hyper parameter in machine-learning process, to task model selection one
The optimal hyper parameter of group, to improve the performance and effect of machine learning.
In the prior art, the hyper parameter of task model is set by artificial experience, and in the follow-up process step by step
Adjustment, this needs takes a long time and can just obtain one group of optimal task model hyper parameter, is then based on multiple tasks mould
Type obtains optimal output as a result, delivery efficiency is low.
Summary of the invention
In view of this, the embodiment of the present application provides a kind of model output recommended method and system is recommended in device, model output
System and its training method and device calculate equipment, storage medium and chip, to solve technological deficiency existing in the prior art.
The embodiment of the present application discloses a kind of model output recommended method, which comprises
Task input feature vector is separately input into few two task models, obtains the task output feature of each task model
Confidence level corresponding with task output feature；
The task of corresponding at least two task model of the task input feature vector is exported into feature and task exports feature
Corresponding confidence level inputs parameter search model, so that the parameter search model exports spy according to the task of each task model
The weight parameter for levying corresponding confidence level and the parameter search model obtains the confidence weight of each task model, and root
Feature is exported according to the task that the confidence weight of each task model is recommended, wherein the parameter search model has optimal
Weight parameter.
Optionally, the parameter search model according to the task of each task model export the corresponding confidence level of feature and
The weight parameter of the parameter search model, obtains the confidence weight of each task model, comprising:
The parameter search model exports the corresponding confidence level of feature and each task according to the task of each task model
The product of the weight parameter of the corresponding parameter search model of model, obtains the confidence weight of each task model.
Optionally, feature is exported according to the task that the confidence weight of each task model is recommended, comprising:
The corresponding task model of optimal confidence weight is determined according to the confidence weight of each task model；
Feature is exported using the task of the corresponding task model of optimal confidence weight output feature as recommending for task.
The embodiment of the present application discloses a kind of training method of model output recommender system, which comprises
Task input feature vector is separately input into few two task models, obtains the task output feature of each task model
Confidence level corresponding with task output feature；
The task of corresponding at least two task model of the task input feature vector is exported into feature and task exports feature
Corresponding confidence level is inputted as training sample, using the corresponding task verifying feature of the task input feature vector as training label
Parameter search model is trained parameter search model, so that the training sample is associated with the trained label, and makes
The parameter search model has optimal weight parameter.
Optionally, the task of corresponding at least two task model of the task input feature vector is exported into feature and task is defeated
The corresponding confidence level of feature is marked as training sample, using the corresponding task verifying feature of the task input feature vector as training out
Label input parameter search model, is trained parameter search model, comprising:
The corresponding confidence level of feature is exported according to the task of each task model and each task model is corresponding described
The weight parameter of parameter search model obtains the confidence weight of each task model；
Feature is exported according to the task of at least two task models and the corresponding task verifying of the task input feature vector is special
Sign, obtains the corresponding evaluation coefficient of each task model；
The weight parameter for adjusting the corresponding parameter search model of each task model, makes the task that confidence weight is optimal
Model is the optimal task model of evaluation coefficient.
Optionally, the weight parameter of the corresponding parameter search model of each task model is adjusted, comprising:
Determine the search parameter of the parameter search model；
By Gaussian function stochastical sampling, the weight ginseng of the corresponding parameter search model of each task model is generated
Number, wherein the Gaussian function is with the weight parameter of the parameter search model corresponding before the adjustment of the task model
Mean value, using search parameter as variance.
The embodiment of the present application discloses a kind of model output recommendation apparatus, and described device includes:
First input module is configured as task input feature vector being separately input into few two task models, obtain each
The task output feature and the corresponding confidence level of task output feature of task model；
Evaluation module is configured as the task output of corresponding at least two task model of the task input feature vector is special
Task of seeking peace exports the corresponding confidence level of feature and inputs parameter search model, so that the parameter search model is according to each task
The task output corresponding confidence level of feature of model and the weight parameter of the parameter search model, obtain each task model
Confidence weight, and being recommended according to the confidence weight of each task model for task exports feature, wherein the parameter is searched
Rope model has optimal weight parameter.
Optionally, the evaluation module is specifically configured to: making the parameter search model according to each task model
Task exports the product of the weight parameter of the corresponding confidence level of feature and the corresponding parameter search model of each task model,
Obtain the confidence weight of each task model.
Optionally, the evaluation module is specifically configured to:
The corresponding task model of optimal confidence weight is determined according to the confidence weight of each task model；
Feature is exported using the task of the corresponding task model of optimal confidence weight output feature as recommending for task.
The embodiment of the present application discloses a kind of training device of model output recommender system, and described device includes:
Second input module is configured as task input feature vector being separately input into few two task models, obtain each
The task output feature and the corresponding confidence level of task output feature of task model；
Training module is configured as the task output of corresponding at least two task model of the task input feature vector is special
Task of seeking peace exports the corresponding confidence level of feature and verifies feature as training sample, by the corresponding task of the task input feature vector
Parameter search model is inputted as training label, parameter search model is trained, so that the training sample and the instruction
It is associated to practice label, and makes the parameter search model that there is optimal weight parameter.
The embodiment of the present application discloses a kind of model output recommender system, comprising:
At least two task models, each task model receive the task input feature vector of input, obtain each task
The task output feature and the corresponding confidence level of task output feature of model；
Parameter search model, the receiving corresponding at least two task model of the task input feature vector of input of the task are defeated
Feature and the corresponding confidence level of task output feature out, so that the parameter search model is defeated according to the task of each task model
The corresponding confidence level of feature and the weight parameter of the parameter search model out, obtain the confidence weight of each task model,
And feature is exported according to the task that the confidence weight of each task model is recommended, wherein the parameter search model has
Optimal weight parameter.
The embodiment of the present application discloses a kind of calculating equipment, including memory, processor and storage are on a memory and can
The computer instruction run on a processor, the processor realize that model output as described above is recommended when executing described instruction
The training method or model of system export the step of recommended method.
The embodiment of the present application discloses a kind of computer readable storage medium, is stored with computer instruction, the instruction quilt
Processor realizes the step of training method or model output recommended method of model output recommender system as described above when executing.
The embodiment of the present application discloses a kind of chip, is stored with computer instruction, and it is as above which is performed realization
The step of training method or model output recommended method of the model output recommender system.
Model provided by the present application exports recommended method, and task input feature vector is separately input into few two task models,
The task output feature and the corresponding confidence level of task output feature of each task model are obtained, then by task input feature vector pair
The task output feature and the corresponding confidence level of task output feature at least two task models answered input parameter search model,
So that parameter search model exports the corresponding confidence level of feature and parameter search model according to the task of each task model
Weight parameter obtains the confidence weight of each task model, and being recommended according to the confidence weight of each task model for task
Feature is exported, to have difference to different sentence adaptability in the same task model, multiple tasks model exports multiple results
In the case where, after parameter search model is to the evaluation of task model, filters out and more preferably export result.
The training method of model output recommender system provided by the present application is few by the way that task input feature vector to be separately input into
Two task models obtain the task output feature and the corresponding confidence level of task output feature of each task model, by task
The task output feature and the corresponding confidence level of task output feature of corresponding at least two task model of input feature vector are as instruction
Practice sample, parameter search model is trained using the corresponding task verifying feature of task input feature vector as training label, from
And make the parameter search model that there is optimal weight parameter, improve the regulated efficiency of weight parameter.
Secondly, the application is by being side with search parameter using the weight parameter of the parameter search model before adjusting as mean value
The Gaussian function stochastical sampling of difference generates the weight parameter of parameter search model, to be made with the optimal weight parameter before adjusting
The search that weight parameter is guided for heuristic information is effectively reduced the search range of weight parameter, improves parameter search
Speed, while more preferably result can be obtained.Under conditions of searching times are enough, this algorithm can be searched more efficiently entirely
Office's optimal solution.
Detailed description of the invention
Fig. 1 is the structural schematic diagram of the calculating equipment of the embodiment of the present application；
Fig. 2 is the flow diagram of the training method of the model output recommender system of the embodiment of the present application；
Fig. 3 is the flow diagram of the training method of the model output recommender system of the embodiment of the present application；
Fig. 4 a and Fig. 4 b are the schematic diagrames that parameter adjustment is carried out based on Gaussian function of the embodiment of the present application；
Fig. 5 is the flow diagram of the model output recommended method of the embodiment of the present application；
Fig. 6 is the structural schematic diagram of the model output recommender system of the embodiment of the present application；
Fig. 7 is the exemplary structural schematic diagram of concrete application of the model output recommender system of the embodiment of the present application；
Fig. 8 is the structural schematic diagram of the training device of the model output recommender system of the embodiment of the present application；
Fig. 9 is the structural schematic diagram of the model output recommendation apparatus of the embodiment of the present application.
Specific embodiment
Many details are explained in the following description in order to fully understand the application.But the application can be with
Much it is different from other way described herein to implement, those skilled in the art can be without prejudice to the application intension the case where
Under do similar popularization, therefore the application is not limited by following public specific implementation.
The term used in this specification one or more embodiment be only merely for for the purpose of describing particular embodiments,
It is not intended to be limiting this specification one or more embodiment.In this specification one or more embodiment and appended claims
The "an" of singular used in book, " described " and "the" are also intended to including most forms, unless context is clearly
Indicate other meanings.It is also understood that term "and/or" used in this specification one or more embodiment refers to and includes
One or more associated any or all of project listed may combine.
It will be appreciated that though may be retouched using term first, second etc. in this specification one or more embodiment
Various information are stated, but these information should not necessarily be limited by these terms.These terms are only used to for same type of information being distinguished from each other
It opens.For example, first can also be referred to as second, class in the case where not departing from this specification one or more scope of embodiments
As, second can also be referred to as first.Depending on context, word as used in this " if " can be construed to
" ... when " or " when ... " or " in response to determination ".
Firstly, the vocabulary of terms being related to one or more embodiments of the invention explains.
Gauss sampling: also known as Gauss sampling or Gaussian Profile sampling is referred to and is taken out with Gaussian Profile (normal distribution)
Sample, mean value and the variance for being specifically described as the random number X generated are consistent with Gaussian function.
Bleu value (bilingual evaluation understudy): i.e. bilingual intertranslation quality evaluation auxiliary tool.It
It is the tool for assessing mechanical translation quality, for judging the similarity degree between two sentences.
In this application, a kind of model output recommended method and device, model output recommender system and its training are provided
Method and apparatus calculate equipment, storage medium and chip, are described in detail one by one in the following embodiments.
Fig. 1 is to show the structural block diagram of the calculating equipment 100 according to one embodiment of this specification.The calculating equipment 100
Component include but is not limited to memory 110 and processor 120.Processor 120 is connected with memory 110 by bus 130,
Database 150 is for saving data.
Calculating equipment 100 further includes access device 140, access device 140 enable calculate equipment 100 via one or
Multiple networks 160 communicate.The example of these networks includes public switched telephone network (PSTN), local area network (LAN), wide area network
(WAN), the combination of the communication network of personal area network (PAN) or such as internet.Access device 140 may include wired or wireless
One or more of any kind of network interface (for example, network interface card (NIC)), such as IEEE802.11 wireless local area
Net (WLAN) wireless interface, worldwide interoperability for microwave accesses (Wi-MAX) interface, Ethernet interface, universal serial bus (USB) connect
Mouth, cellular network interface, blue tooth interface, near-field communication (NFC) interface, etc..
In one embodiment of this specification, unshowned other component in above-mentioned and Fig. 1 of equipment 100 is calculated
It can be connected to each other, such as pass through bus.It should be appreciated that calculating device structure block diagram shown in FIG. 1 is merely for the sake of example
Purpose, rather than the limitation to this specification range.Those skilled in the art can according to need, and increase or replace other portions
Part.
Calculating equipment 100 can be any kind of static or mobile computing device, including mobile computer or mobile meter
Calculate equipment (for example, tablet computer, personal digital assistant, laptop computer, notebook computer, net book etc.), movement
Phone (for example, smart phone), wearable calculating equipment (for example, smartwatch, intelligent glasses etc.) or other kinds of shifting
Dynamic equipment, or the static calculating equipment of such as desktop computer or PC.Calculating equipment 100 can also be mobile or state type
Server.
Wherein, processor 120 can execute the step in method shown in Fig. 2.Fig. 2 is to show to be implemented according to the application one
The schematic flow chart of the training method of the model output recommender system of example, including step 201 is to step 202.
201, task input feature vector is separately input into few two task models, obtains the task output of each task model
Feature and the corresponding confidence level of task output feature.
It should be noted that the task input feature vector in this step has multiple, each task input feature vector is inputted respectively
At least two task models, so as to obtain the corresponding at least two tasks output feature of each task input feature vector and task
Export the corresponding confidence level of feature.
Wherein, task input feature vector and task verifying feature are to obtain in advance, and the mode of acquisition can pass through internet search
It obtains, it can also be by being obtained in pre-stored task-set.
Task model can be to be a variety of, such as translation model, image detection model etc..
By taking translation model as an example, task input feature vector can be sentence to be translated, and task, which exports feature, to be translation language
Sentence, it can be standard translation sentence that task, which verifies feature,.It accordingly, include multiple sentences to be translated and multiple standards in task-set
Translate sentence, sentence and standard translation sentence to be translated translation relationship each other.For example, sentence to be translated is " I likes China ", mark
Quasi- translation sentence is " I love China ".
By taking image detection model as an example, task input feature vector can be image to be detected, and task, which exports feature, to be inspection
Measurement information, it can be standard detection information that task, which verifies feature,.It accordingly, include multiple image to be detected and multiple in task-set
Detection information.
For example, task input feature vector can be facial image to be detected by taking human face recognition model as an example, task exports feature
It can be detection facial image, it can be standard faces image that task, which verifies feature,.Evaluation index can be the phase of facial image
Like degree.
202, the task of corresponding at least two task model of the task input feature vector is exported into feature and task exports
The corresponding confidence level of feature is as training sample, using the corresponding task verifying feature of the task input feature vector as training label
Parameter search model is inputted, parameter search model is trained, so that the training sample is associated with the trained label,
And make the parameter search model that there is optimal weight parameter.
Optionally, step 202 specifically includes:
S1, the corresponding confidence level of feature and the corresponding institute of each task model are exported according to the task of each task model
The weight parameter for stating parameter search model obtains the confidence weight of each task model.
S2, feature and the corresponding task verifying of the task input feature vector are exported according to the task of at least two task models
Feature obtains the corresponding evaluation coefficient of each task model；
The weight parameter of S3, the corresponding parameter search model of each task model of adjustment, keep confidence weight optimal
Task model is the optimal task model of evaluation coefficient.
Specifically, referring to Fig. 3, step 202 includes the following steps 301~308:
301, the initial weight parameter and searching times threshold value of parameter search model are determined.
Wherein, initial weight parameter can be artificial setting.
302, feature is exported according to the task of at least two task models and the corresponding task of the task input feature vector is tested
Characteristics of syndrome obtains corresponding first evaluation coefficient of each task model, to determine the optimal task model of the first evaluation coefficient.
Specifically, it can be compared by the way that task output feature and task are verified feature, obtain the first evaluation coefficient.
By taking translation model as an example, it is translation sentence that task, which exports feature, and it is standard translation sentence that task, which verifies feature, can pass through translation
The bleu value that sentence and standard translation sentence are calculated is as the first evaluation coefficient.Bleu value is higher, then translates sentence
Closer to standard translation sentence, the translation effect of the corresponding translation model of translation sentence is better.
Certainly, the first evaluation coefficient can also be realized by other parameters, such as by NIST value as the first evaluation
Coefficient.
303, weight parameter, the optimal task mould of the first evaluation coefficient of task model for keeping confidence weight optimal are adjusted
Type, wherein the confidence weight of each task model is according to the corresponding confidence level of the task of each task model output feature and often
The weight parameter of the corresponding parameter search model of a task model obtains.
Specifically, the corresponding confidence level of feature and each task model pair can be exported according to the task of each task model
The product of the weight parameter for the parameter search model answered obtains the confidence weight of each task model.
It should be noted that by confidence weight as the evaluation index to task model in the present embodiment, then, first
The optimal task model of evaluation coefficient, confidence weight also should be optimal.The process of training parameter search model, substantially adjusts
The process of confidence weight.
Optionally, there are many ways to adjusting weight parameter, such as grid search, random search etc..
Grid search is most widely used hyper parameter searching algorithm, and grid search is by searching for all in search range
Point determine optimal value.Generally by providing biggish search range and lesser step-length, grid search is can to find
The overall situation is compared with the figure of merit, if grid can find global optimum when infinitely intensive.But the maximum defect of grid search
Be: it very consumes computing resource, in particular for tuning parameter it is more when.
Compared with grid search, random search does not attempt all parameter values, but samples and fix from specified distribution
The parameter setting of quantity.The theoretical foundation of random search is: if sample point set is sufficiently large immediately, can also find the overall situation
Optimal value or their approximation.By the grab sample to search range, random search generally can be faster than grid search
It is some.This method has that precision is poor, is generally used for roughing or generaI investigation.
The present embodiment adjusts weight parameter by the following method: determining the search parameter of the parameter search model, then
By Gaussian function stochastical sampling, the weight parameter of the corresponding parameter search model of each task model is generated, wherein institute
Gaussian function is stated using the weight parameter of the corresponding parameter search model before the adjustment of the task model as mean value, to search
Rope parameter is variance.
The adjustment of weight parameter is carried out by Gaussian function, to realize using the optimal weight parameter before adjusting as opening
Photos and sending messages guide the search of weight parameter, effectively reduce the search range of weight parameter, while can obtain more preferably
As a result.Under conditions of searching times are enough, this algorithm can more efficiently search globally optimal solution.With grid search and with
Machine search is compared, and the adjustment weight parameter method of the present embodiment is more high-efficient than random search, and has more preferably result.
By practical test, it is optimal the parameter adjustment number of result, the method for the present embodiment is about 500 times, and
Random search is about 10000 times.
A and Fig. 4 b referring to fig. 4, following examples are the exemplary illustration that parameter adjustment is carried out based on Gaussian function.
Mission statement: fitting quadratic equation with one unknown y=ax2+ bx+c, wherein setting a=0.5, b=0.1, c=1, section is
[- 1,1].
According to actual task, Fig. 4 a is the result that parameter adjustment is carried out in the case of being added without Gaussian noise: ' a':
0.4990,'b':0.0994,'c':1.0002}；Fig. 4 b is the result that parameter adjustment is carried out after Gaussian noise is added: ' a':
0.4856,'b':0.0946,'c':1.0073}。
As can be seen from the above results being based on Gaussian function in the case where being added without Gaussian noise and Gaussian noise being added
The result for carrying out parameter adjustment is closer to actual result.
304, judge whether searching times reach searching times threshold value, if so, step 308 is executed, if it is not, executing step
305。
305, according to the task of at least two task models inputted again output feature and the task input feature vector pair
The task verifying feature answered, obtains corresponding second evaluation coefficient of each task model.
Specifically, it can be compared by the way that task output feature and task are verified feature, obtain the second evaluation coefficient.
By taking translation model as an example, it is translation sentence that task, which exports feature, and it is standard translation sentence that task, which verifies feature, can be with
The bleu value being calculated by translation sentence and standard translation sentence is as the second evaluation coefficient.Bleu value is higher, then
Sentence is translated closer to standard translation sentence, the translation effect of the corresponding translation model of translation sentence is better.
306, the second evaluation coefficient and the first evaluation coefficient are compared, if the second evaluation coefficient is better than the first evaluation system
Number executes step 307, if the first evaluation coefficient is better than the second evaluation coefficient, executes step 304.
307, the second evaluation coefficient is assigned to the first evaluation coefficient, and adjusts weight parameter, keep confidence weight optimal
Task model is the optimal task model of the first evaluation coefficient, executes step 304.
Wherein, the confidence weight of each task model exports the corresponding confidence level of feature according to the task of each task model
The weight parameter of the parameter search model corresponding with each task model obtains.
It is better than the situation of the first evaluation coefficient for the second evaluation coefficient, is commented it may be considered that task model corresponds to second
The task output feature that the task output feature of valence coefficient is better than the first evaluation coefficient needs to search parameter in such cases
The weight parameter of rope model is readjusted, so that the optimal task model of confidence weight is the optimal task mould of the first evaluation coefficient
Type.
The method for adjusting weight parameter is interpreted in abovementioned steps, is just no longer described in detail herein.
308, using the corresponding weight parameter adjusted of each task model as optimal weight parameter.
The training method of model output recommender system provided by the present application is few by the way that task input feature vector to be separately input into
Two task models obtain the task output feature and the corresponding confidence level of task output feature of each task model, by task
The task output feature and the corresponding confidence level of task output feature of corresponding at least two task model of input feature vector are as instruction
Practice sample, parameter search model is trained using the corresponding task verifying feature of task input feature vector as training label, from
And make the parameter search model that there is optimal weight parameter, improve the regulated efficiency of weight parameter.
Secondly, the application is by being side with search parameter using the weight parameter of the parameter search model before adjusting as mean value
The Gaussian function stochastical sampling of difference generates the weight parameter of parameter search model, to be made with the optimal weight parameter before adjusting
The search that weight parameter is guided for heuristic information is effectively reduced the search range of weight parameter, improves parameter search
Speed, while more preferably result can be obtained.Under conditions of searching times are enough, this algorithm can be searched more efficiently entirely
Office's optimal solution.
The embodiment of the present application also discloses a kind of model output recommended method, and referring to Fig. 5, the method includes the following steps
501~502:
501, task input feature vector is separately input into few two task models, obtains the task output of each task model
Feature and the corresponding confidence level of task output feature.
502, the task of corresponding at least two task model of the task input feature vector is exported into feature and task exports
The corresponding confidence level of feature inputs parameter search model, so that the parameter search model is defeated according to the task of each task model
The corresponding confidence level of feature and the weight parameter of parameter search model out, obtain the confidence weight of each task model, and root
Feature is exported according to the task that the confidence weight of each task model is recommended, wherein the parameter search model has optimal
Weight parameter.
Specifically, parameter search model exports the corresponding confidence level of feature and parameter according to the task of each task model
The weight parameter of search model obtains the confidence weight of each task model, comprising:
The parameter search model exports the corresponding confidence level of feature and each task according to the task of each task model
The product of the weight parameter of the corresponding parameter search model of model, obtains the confidence weight of each task model.
Specifically, feature, packet are exported according to the task that the confidence weight of each task model is recommended in step 502
It includes:
The corresponding task model of optimal confidence weight is determined according to the confidence weight of each task model；
Feature is exported using the task of the corresponding task model of optimal confidence weight output feature as recommending for task.
In a specific example, by taking 4 task models as an example, the 1st corresponding confidence weight of task model is 0.8,
The 2nd corresponding confidence weight of task model is that the 0.7, the 3rd corresponding confidence weight of task model is the 0.77, the 4th task
The corresponding confidence weight of model is 0.5, then, it is exported using the task of the 1st task model output feature as recommending for task
Feature.
Model provided by the present application exports recommended method, and task input feature vector is separately input into few two task models,
The task output feature and the corresponding confidence level of task output feature of each task model are obtained, then by task input feature vector pair
The task output feature and the corresponding confidence level of task output feature at least two task models answered input parameter search model,
So that parameter search model exports the corresponding confidence level of feature and parameter search model according to the task of each task model
Weight parameter obtains the confidence weight of each task model, and being recommended according to the confidence weight of each task model for task
Feature is exported, to have difference to different sentence adaptability in the same task model, multiple tasks model exports multiple results
In the case where, after parameter search model is to the evaluation of task model, filters out and more preferably export result.
It is specific with one below in order to be further understood that the model of the embodiment of the present application exports the technical solution of recommended method
Example is illustrated.
By taking translation model as an example, the embodiment of the present application discloses a kind of translation model output recommended method, comprising:
1) sentence A is inputted into Google's translation model, improved Google's translation model respectively and improves CNN
(Convolutional Neural Network, convolutional neural networks) model, obtains corresponding translation sentence a1 and confidence level
Wa1, translation sentence a2 and confidence level wa2 and translation sentence a3 and confidence level wa3.
It 2) will translation sentence a1 and confidence level wa1, translation sentence a2 and confidence level wa2 and translation sentence a3 and confidence level
Wa3 is input to parameter search model, obtains the confidence weight of each translation model.
Wherein, parameter search model includes the corresponding optimal weight parameter of each translation model, according to weight parameter and
The product of confidence level, the corresponding confidence weight of available translation model.
3) optimal confidence weight is determined, and using the corresponding translation sentence of optimal confidence weight as optimal translation language
Sentence output.
If the confidence weight of Google's translation model is 0.6, the confidence weight of improved Google's translation model is 0.8, improve
The confidence weight of CNN model is 0.7, then, using the corresponding translation sentence of improved Google's translation model as optimal translation
Sentence output.
The embodiment of the present application also discloses a kind of model output recommender system, referring to Fig. 6, comprising:
At least two task models, each task model receive the task input feature vector of input, obtain each task model
Task output feature and the corresponding confidence level of task output feature；
Parameter search model, the receiving corresponding at least two task model of the task input feature vector of input of the task are defeated
Feature and the corresponding confidence level of task output feature out, so that the parameter search model is defeated according to the task of each task model
The corresponding confidence level of feature and the weight parameter of parameter search model out, obtain the confidence weight of each task model, and root
Feature is exported according to the task that the confidence weight of each task model is recommended, wherein the parameter search model has optimal
Weight parameter.
Wherein, obtaining for feature is exported for the task of the explanation of task input feature vector, the calculating of confidence weight and recommendation
Method is taken, has been described in detail in the aforementioned embodiment, just it is no longer repeated herein.
It is illustrated below with a specific model output recommender system.Referring to Fig. 7, system is recommended in the model output in Fig. 7
System includes 4 translation models so that task model is translation model as an example altogether.
This exemplary model exports recommended method
S701, sentence to be translated is inputted to four translation models respectively, respectively obtains the translation sentence of four translation models
The corresponding confidence level p1~p4 of j1~j4 and translation sentence j1~j4.
S702, the task input feature vector corresponding translation sentence j1~j4 and confidence level p1~p4 are inputted into parameter search
Model, so that parameter search model is according to translation sentence j1~j4 corresponding confidence level p1~p4 and weight parameter w1~w4,
Obtain the confidence weight of four translation models.
Wherein, the confidence weight of first translation model is w1*p1, and the confidence weight of second translation model is w2*p2,
The confidence weight of third translation model is w3*p3, and the confidence weight of the 4th translation model is w4*p4.
S703, the corresponding translation model of optimal confidence weight is determined according to the confidence weight of each translation model.
S704, spy is exported using the task of the corresponding translation model of optimal confidence weight output feature as recommending for task
Sign.
For example, the 1st corresponding confidence weight of translation model is that the 0.8, the 2nd corresponding confidence weight of translation model is
It is 0.5 that 0.7, the 3rd corresponding confidence weight of translation model, which is the 0.77, the 4th corresponding confidence weight of translation model, then,
Feature is exported using the task of the 1st translation model output feature as recommending for task.
The embodiment of the present application also discloses a kind of training device of model output recommender system, referring to Fig. 8, comprising:
First input module 801 is configured as task input feature vector being separately input into few two task models, obtain every
The task output feature and the corresponding confidence level of task output feature of a task model；
Training module 802 is configured as the task of corresponding at least two task model of the task input feature vector is defeated
Feature and the corresponding confidence level of task output feature verify the corresponding task of the task input feature vector as training sample out
Feature is trained parameter search model as training label, so that the training sample is associated with the trained label,
And make the parameter search model that there is optimal weight parameter.
Optionally, training module 802 is specifically configured to:
The corresponding confidence level of feature is exported according to the task of each task model and each task model is corresponding described
The weight parameter of parameter search model obtains the confidence weight of each task model；
Feature is exported according to the task of at least two task models and the corresponding task verifying of the task input feature vector is special
Sign, obtains the corresponding evaluation coefficient of each task model；
The weight parameter for adjusting the corresponding parameter search model of each task model, makes the task that confidence weight is optimal
Model is the optimal task model of evaluation coefficient.
Optionally, training module 802 includes:
Initial value determining module is configured to determine that the initial weight parameter and searching times threshold of parameter search model
Value；
First evaluation coefficient determining module is configured as exporting feature and described according to the task of at least two task models
The corresponding task of task input feature vector verifies feature, obtains corresponding first evaluation coefficient of each task model, to determine first
The optimal task model of evaluation coefficient；
First processing module, is configured as adjustment weight parameter, and the task model first for keeping confidence weight optimal is evaluated
The optimal task model of coefficient, wherein the confidence weight of each task model exports feature according to the task of each task model
Corresponding confidence level and the weight parameter of the corresponding parameter search model of each task model obtain；
Judgment module is configured as judging whether searching times reach searching times threshold value, if so, executing optimal weights ginseng
Number determining module, if it is not, executing the second evaluation coefficient determining module；
Second evaluation coefficient determining module is configured as the task output according at least two task models inputted again
Feature and the corresponding task of the task input feature vector verify feature, obtain corresponding second evaluation coefficient of each task model；
Comparison module is configured as the second evaluation coefficient and the first evaluation coefficient being compared, if the second evaluation coefficient
Better than the first evaluation coefficient, Second processing module is executed, if the first evaluation coefficient is better than the second evaluation coefficient, execution judges mould
Block；
Second processing module is configured as the second evaluation coefficient being assigned to the first evaluation coefficient, and adjusts weight parameter,
The optimal task model of the first evaluation coefficient of task model for keeping confidence weight optimal executes judgment module, wherein Mei Geren
The confidence weight of business model exports the corresponding confidence level of feature according to the task of each task model and each task model is corresponding
The weight parameter of the parameter search model obtain；
Optimal weights parameter determination module is configured as the corresponding weight parameter adjusted of each task model
As optimal weight parameter.
Optionally, training module 802 is specifically configured to:
The corresponding confidence level of feature and the corresponding ginseng of each task model are exported according to the task of each task model
The product of the weight parameter of number search model, obtains the confidence weight of each task model.
Optionally, training module 802 is specifically configured to:
Determine the search parameter of the parameter search model；
By Gaussian function stochastical sampling, the weight ginseng of the corresponding parameter search model of each task model is generated
Number, wherein the Gaussian function is with the weight parameter of the corresponding parameter search model before the adjustment of the task model
For mean value, using search parameter as variance.
A kind of above-mentioned model for the present embodiment exports the exemplary scheme of the training device of recommender system.It needs to illustrate
It is that the model exports the technical solution and the training method of above-mentioned model output recommender system of the training device of recommender system
Technical solution belongs to same design, in the details that the technical solution of the training device of model output recommender system is not described in detail
Hold, may refer to the description of the technical solution of the training method of above-mentioned model output recommender system.
The embodiment of the present application also discloses a kind of model output recommendation apparatus, referring to Fig. 9, comprising:
Second input module 901 is configured as task input feature vector being separately input into few two task models, obtain every
The task output feature and the corresponding confidence level of task output feature of a task model；
Evaluation module 902 is configured as the task of corresponding at least two task model of the task input feature vector is defeated
Feature and the corresponding confidence level of task output feature input parameter search model out, so that the parameter search model is according to each
The corresponding confidence level of task output feature of task model and the weight parameter of parameter search model, obtain each task model
Confidence weight, and being recommended according to the confidence weight of each task model for task exports feature, wherein the parameter is searched
Rope model has optimal weight parameter.
Optionally, evaluation module 902 is specifically configured to: exporting parameter search model by the task of each task model
The product of the corresponding confidence level of feature and the weight parameter of the corresponding parameter search model of each task model is as every
The confidence weight of a task model.
Optionally, evaluation module 902 is specifically configured to: determining optimal set according to the confidence weight of each task model
Believe the corresponding task model of weight, appoints using the task of the corresponding task model of optimal confidence weight output feature as what is recommended
Business output feature.
A kind of above-mentioned model for the present embodiment exports the exemplary scheme of recommendation apparatus.It should be noted that the model
The technical solution of the technical solution and above-mentioned model output recommended method that export recommendation apparatus belongs to same design, model output
The detail content that the technical solution of recommendation apparatus is not described in detail may refer to the technical side of above-mentioned model output recommended method
The description of case.
One embodiment of the application also provides a kind of calculating equipment, including memory, processor and storage are on a memory simultaneously
The computer instruction that can be run on a processor, the processor realize that model output as described above pushes away when executing described instruction
Recommend the training method or model output recommended method of system.
One embodiment of the application also provides a kind of computer readable storage medium, is stored with computer instruction, the instruction
The training method of model output recommender system as described above or the step of model output recommended method are realized when being executed by processor
Suddenly.
A kind of exemplary scheme of above-mentioned computer readable storage medium for the present embodiment.It should be noted that this is deposited
The technical side of the training method or model output recommended method of the technical solution of storage media and above-mentioned model output recommender system
Case belongs to same design, and the detail content that the technical solution of storage medium is not described in detail may refer to above-mentioned model output
The description of the technical solution of training method or model the output recommended method of recommender system.
The computer instruction includes computer program code, the computer program code can for source code form,
Object identification code form, executable file or certain intermediate forms etc..The computer-readable medium may include: that can carry institute
State any entity or device, recording medium, USB flash disk, mobile hard disk, magnetic disk, CD, the computer storage of computer program code
Device, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory),
Electric carrier signal, telecommunication signal and software distribution medium etc..It should be noted that the computer-readable medium include it is interior
Increase and decrease appropriate can be carried out according to the requirement made laws in jurisdiction with patent practice by holding, such as in certain jurisdictions of courts
Area does not include electric carrier signal and telecommunication signal according to legislation and patent practice, computer-readable medium.
The embodiment of the present application also discloses a kind of chip, is stored with computer instruction, which is performed realization such as
The step of training method or model output recommended method of the upper model output recommender system.
It should be noted that for the various method embodiments described above, describing for simplicity, therefore, it is stated as a series of
Combination of actions, but those skilled in the art should understand that, the application is not limited by the described action sequence because
According to the application, certain steps can use other sequences or carry out simultaneously.Secondly, those skilled in the art should also know
It knows, the embodiments described in the specification are all preferred embodiments, and related actions and modules might not all be this Shen
It please be necessary.
In the above-described embodiments, it all emphasizes particularly on different fields to the description of each embodiment, there is no the portion being described in detail in some embodiment
Point, it may refer to the associated description of other embodiments.
The application preferred embodiment disclosed above is only intended to help to illustrate the application.There is no detailed for alternative embodiment
All details are described, are not limited the invention to the specific embodiments described.Obviously, according to the content of this specification,
It can make many modifications and variations.These embodiments are chosen and specifically described to this specification, is in order to preferably explain the application
Principle and practical application, so that skilled artisan be enable to better understand and utilize the application.The application is only
It is limited by claims and its full scope and equivalent.
Claims (14)
1. a kind of model exports recommended method, which is characterized in that the described method includes:
Task input feature vector is separately input into few two task models, obtains task output feature and the institute of each task model
State the corresponding confidence level of task output feature；
The task of corresponding at least two task model of the task input feature vector is exported into feature and the task exports feature
Corresponding confidence level inputs parameter search model, so that the parameter search model exports spy according to the task of each task model
The weight parameter for levying corresponding confidence level and the parameter search model obtains the confidence weight of each task model, and root
Feature is exported according to the task that the confidence weight of each task model is recommended, wherein the parameter search model has optimal
Weight parameter.
2. the method as described in claim 1, which is characterized in that the parameter search model is according to the task of each task model
The weight parameter for exporting the corresponding confidence level of feature and the parameter search model, obtains the confidence weighting of each task model
Weight, comprising:
The parameter search model exports the corresponding confidence level of feature and each task model according to the task of each task model
The product of the weight parameter of the corresponding parameter search model, obtains the confidence weight of each task model.
3. the method as described in claim 1, which is characterized in that appointed according to what the confidence weight of each task model was recommended
Business output feature, comprising:
The corresponding task model of optimal confidence weight is determined according to the confidence weight of each task model；
Feature is exported using the task of the corresponding task model of optimal confidence weight output feature as recommending for task.
4. a kind of training method of model output recommender system, which is characterized in that the described method includes:
Task input feature vector is separately input into few two task models, obtain the task output feature of each task model and is appointed
The corresponding confidence level of business output feature；
The task of corresponding at least two task model of the task input feature vector is exported into feature and task output feature is corresponding
Confidence level as training sample, using the task input feature vector corresponding task verifying feature as training label input parameter
Search model is trained parameter search model, so that the training sample is associated with the trained label, and makes described
Parameter search model has optimal weight parameter.
5. method as claimed in claim 4, which is characterized in that by the corresponding at least two tasks mould of the task input feature vector
The task output feature and the corresponding confidence level of task output feature of type are corresponded to as training sample, by the task input feature vector
Task verifying feature as training label input parameter search model, parameter search model is trained, comprising:
The corresponding confidence level of feature and the corresponding parameter of each task model are exported according to the task of each task model
The weight parameter of search model obtains the confidence weight of each task model；
Feature is exported according to the task of at least two task models and the corresponding task of the task input feature vector verifies feature, is obtained
To the corresponding evaluation coefficient of each task model；
The weight parameter for adjusting the corresponding parameter search model of each task model, makes the task model that confidence weight is optimal
For the task model that evaluation coefficient is optimal.
6. method as claimed in claim 4, which is characterized in that adjust the corresponding parameter search model of each task model
Weight parameter, comprising:
Determine the search parameter of the parameter search model；
By Gaussian function stochastical sampling, the weight parameter of the corresponding parameter search model of each task model is generated,
In, the Gaussian function is equal with the weight parameter of the corresponding parameter search model before the adjustment of the task model
Value, using search parameter as variance.
7. a kind of model exports recommendation apparatus, which is characterized in that described device includes:
First input module is configured as task input feature vector being separately input into few two task models, obtains each task
The task output feature and the corresponding confidence level of task output feature of model；
Evaluation module, be configured as the task of corresponding at least two task model of task input feature vector output feature and
Task exports the corresponding confidence level of feature and inputs parameter search model, so that the parameter search model is according to each task model
Task output the corresponding confidence level of feature and the parameter search model weight parameter, obtain setting for each task model
Believe weight, and feature is exported according to the task that the confidence weight of each task model is recommended, wherein the parameter search mould
Type has optimal weight parameter.
8. device as claimed in claim 7, which is characterized in that the evaluation module is specifically configured to: searching the parameter
Rope model exports the corresponding confidence level of feature and the corresponding parameter of each task model according to the task of each task model
The product of the weight parameter of search model obtains the confidence weight of each task model.
9. device as claimed in claim 7, which is characterized in that the evaluation module is specifically configured to:
The corresponding task model of optimal confidence weight is determined according to the confidence weight of each task model；
Feature is exported using the task of the corresponding task model of optimal confidence weight output feature as recommending for task.
10. a kind of training device of model output recommender system, which is characterized in that described device includes:
Second input module is configured as task input feature vector being separately input into few two task models, obtains each task
The task output feature and the corresponding confidence level of task output feature of model；
Training module, be configured as the task of corresponding at least two task model of task input feature vector output feature and
Task export the corresponding confidence level of feature verified as training sample, using the corresponding task of the task input feature vector feature as
Training label inputs parameter search model, is trained to parameter search model, so that the training sample and the training are marked
Label are associated, and the parameter search model is made to have optimal weight parameter.
11. a kind of model exports recommender system characterized by comprising
At least two task models, each task model receive the task input feature vector of input, obtain each task model
Task output feature and the corresponding confidence level of task output feature；
Parameter search model, the task output for receiving corresponding at least two task model of the task input feature vector of input are special
Task of seeking peace exports the corresponding confidence level of feature, so that the parameter search model exports spy according to the task of each task model
The weight parameter for levying corresponding confidence level and the parameter search model obtains the confidence weight of each task model, and root
Feature is exported according to the task that the confidence weight of each task model is recommended, wherein the parameter search model has optimal
Weight parameter.
12. a kind of calculating equipment including memory, processor and stores the calculating that can be run on a memory and on a processor
Machine instruction, which is characterized in that the processor is realized described in claim 1-3 or 4-6 any one when executing described instruction
The step of method.
13. a kind of computer readable storage medium, is stored with computer instruction, which is characterized in that the instruction is held by processor
The step of claim 1-3 or 4-6 any one the method are realized when row.
14. a kind of chip, is stored with computer instruction, which is characterized in that the instruction be performed realize claim 1-3 or
The step of person's 4-6 any one the method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910105297.0A CN109800441B (en) | 2019-02-01 | 2019-02-01 | Model output recommendation method and device and model output recommendation system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910105297.0A CN109800441B (en) | 2019-02-01 | 2019-02-01 | Model output recommendation method and device and model output recommendation system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109800441A true CN109800441A (en) | 2019-05-24 |
| CN109800441B CN109800441B (en) | 2023-03-24 |
Family
ID=66561973
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910105297.0A Active CN109800441B (en) | 2019-02-01 | 2019-02-01 | Model output recommendation method and device and model output recommendation system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109800441B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112804074A (en) * | 2019-11-14 | 2021-05-14 | 华为技术有限公司 | Network parameter configuration method and device |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060204107A1 (en) * | 2005-03-04 | 2006-09-14 | Lockheed Martin Corporation | Object recognition system using dynamic length genetic training |
| JP2015191397A (en) * | 2014-03-28 | 2015-11-02 | 日本電気株式会社 | Design parameter search device, design parameter search method, and program |
| CN107240395A (en) * | 2017-06-16 | 2017-10-10 | 百度在线网络技术（北京）有限公司 | A kind of acoustic training model method and apparatus, computer equipment, storage medium |
| CN107657237A (en) * | 2017-09-28 | 2018-02-02 | 东南大学 | Car crass detection method and system based on deep learning |
| CN108875963A (en) * | 2018-06-28 | 2018-11-23 | 北京字节跳动网络技术有限公司 | Optimization method, device, terminal device and the storage medium of machine learning model |
| US20180357538A1 (en) * | 2017-06-09 | 2018-12-13 | Korea Advanced Institute Of Science And Technology | Electronic apparatus and method for generating trained model |
| CN109002810A (en) * | 2018-08-01 | 2018-12-14 | 西南交通大学 | Model evaluation method, Radar Signal Recognition method and corresponding intrument |
| CN109165249A (en) * | 2018-08-07 | 2019-01-08 | 阿里巴巴集团控股有限公司 | Data processing model construction method, device, server and user terminal |
| CN109243172A (en) * | 2018-07-25 | 2019-01-18 | 华南理工大学 | Traffic flow forecasting method based on genetic algorithm optimization LSTM neural network |
-
2019
- 2019-02-01 CN CN201910105297.0A patent/CN109800441B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060204107A1 (en) * | 2005-03-04 | 2006-09-14 | Lockheed Martin Corporation | Object recognition system using dynamic length genetic training |
| JP2015191397A (en) * | 2014-03-28 | 2015-11-02 | 日本電気株式会社 | Design parameter search device, design parameter search method, and program |
| US20180357538A1 (en) * | 2017-06-09 | 2018-12-13 | Korea Advanced Institute Of Science And Technology | Electronic apparatus and method for generating trained model |
| CN107240395A (en) * | 2017-06-16 | 2017-10-10 | 百度在线网络技术（北京）有限公司 | A kind of acoustic training model method and apparatus, computer equipment, storage medium |
| CN107657237A (en) * | 2017-09-28 | 2018-02-02 | 东南大学 | Car crass detection method and system based on deep learning |
| CN108875963A (en) * | 2018-06-28 | 2018-11-23 | 北京字节跳动网络技术有限公司 | Optimization method, device, terminal device and the storage medium of machine learning model |
| CN109243172A (en) * | 2018-07-25 | 2019-01-18 | 华南理工大学 | Traffic flow forecasting method based on genetic algorithm optimization LSTM neural network |
| CN109002810A (en) * | 2018-08-01 | 2018-12-14 | 西南交通大学 | Model evaluation method, Radar Signal Recognition method and corresponding intrument |
| CN109165249A (en) * | 2018-08-07 | 2019-01-08 | 阿里巴巴集团控股有限公司 | Data processing model construction method, device, server and user terminal |
Non-Patent Citations (1)
| Title |
|---|
| 郑晓康: "面向汉英专利文献的神经网络翻译模型的集外词翻译研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112804074A (en) * | 2019-11-14 | 2021-05-14 | 华为技术有限公司 | Network parameter configuration method and device |
| CN112804074B (en) * | 2019-11-14 | 2022-08-09 | 华为技术有限公司 | Network parameter configuration method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109800441B (en) | 2023-03-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109800807B (en) | Training method and classification method and device of classification network, and electronic equipment | |
| TWI689871B (en) | Gradient lifting decision tree (GBDT) model feature interpretation method and device | |
| US20220036178A1 (en) | Dynamic gradient aggregation for training neural networks | |
| CN109829550A (en) | Model evaluation method and apparatus, model evaluation system and its training method and device | |
| CN113989519B (en) | Long-tail target detection method and system | |
| CN110134967A (en) | Text handling method, calculates equipment and computer readable storage medium at device | |
| CN114625838B (en) | Optimization method and device of search system, storage medium and computer equipment | |
| CN111564179A (en) | Species biology classification method and system based on triple neural network | |
| CN110163250A (en) | Image desensitization process system, method and device based on distributed scheduling | |
| CN109002810A (en) | Model evaluation method, Radar Signal Recognition method and corresponding intrument | |
| EP4338395A1 (en) | Artificial intelligence based cognitive test script generation | |
| CN110276587A (en) | The method, apparatus of project examination calculates equipment and computer readable storage medium | |
| EP4193261A1 (en) | Test script generation from test specifications using natural language processing | |
| CN106326904A (en) | Device and method of acquiring feature ranking model and feature ranking method | |
| CN111626338B (en) | Cloud environment matching method, device, equipment and medium based on fusion classification model | |
| CN115937660A (en) | Verification code identification method and device | |
| CN110147414B (en) | Entity characterization method and device of knowledge graph | |
| CN109800441A (en) | A kind of model output recommended method and device, model export recommender system | |
| CN115034300A (en) | Classification model training method and device | |
| CN116561562B (en) | An Optimal Acquisition Method of Sound Source Depth Based on Waveguide Singular Point | |
| CN114299119B (en) | Target tracking method and device | |
| CN117216550A (en) | Classification model training method, device, equipment, medium and program product | |
| CN117371511A (en) | Training method, device, equipment and storage medium for image classification model | |
| CN109871946A (en) | A kind of application method and device, training method and device of neural network model | |
| CN114358313A (en) | Data processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |