CN116011542A - Intelligent questionnaire interview model training method, intelligent questionnaire interview method and device - Google Patents

Abstract

The application discloses an intelligent questionnaire interview model training method, an intelligent questionnaire interview device and an intelligent questionnaire interview storage medium based on man-machine conversation, which relate to the field of computer technology, in particular to the field of artificial intelligence and natural language processing. The specific implementation scheme of the intelligent questionnaire interview model training method is as follows: acquiring a voice sample and tag information of the voice sample; inputting the voice sample into an intelligent questionnaire interview model; obtaining semantic feature vectors and voice feature vectors of a voice sample text based on the voice samples; and predicting question-answer option contents matched with the voice samples by adopting a classification prediction unit according to the semantic feature vector and the voice feature vector to obtain prediction information output by the classification prediction unit, and performing model training on the intelligent questionnaire interview model according to the prediction information and the label information. Semantic understanding and generalization capabilities of the intelligent questionnaire interview model can be improved.

Description

Intelligent questionnaire interview model training method, intelligent questionnaire interview method and device

technical field

The present application relates to the field of computer technology, in particular to the field of artificial intelligence and natural language processing, and in particular to an intelligent questionnaire interview model training method, intelligent questionnaire interview method and device.

Background technique

Questionnaire interview system is a kind of implementation scenario of task-based dialogue system. After getting the user's voice input, the natural language understanding module in the system needs to process the voice accordingly to recognize the user's intention and complete the slot filling.

In related technologies, natural language understanding is performed based on text obtained through speech recognition of a user's speech input. However, in the voice dialogue questionnaire interview scenario, due to factors such as environmental sounds, dialects, and colloquialism, the voice recognition results are not accurate, and reliable semantic information cannot be extracted during the natural language understanding process. In addition, the rule-based discrimination method requires the design of the rule template. If the content of the user's answer exceeds the design scope of the interview template, the questionnaire interview model will not be able to correctly discriminate, and there is a problem of poor generalization.

Contents of the invention

The present application provides an intelligent questionnaire interview model training method, an intelligent questionnaire interview method based on man-machine dialogue, a device, a device and a storage medium.

According to the first aspect of the present application, an intelligent questionnaire interview model training method is provided, including: acquiring voice samples and label information of the voice samples; inputting the voice samples into the intelligent questionnaire interview model; wherein, the intelligent The questionnaire interview model includes a speech recognition unit, a text semantic feature extraction unit, a speech feature extraction unit and a classification prediction unit; the speech sample is recognized based on the speech recognition unit to obtain a corresponding speech sample text, and based on the text semantics The feature extraction unit encodes the speech sample text to obtain the semantic feature vector of the speech sample text; based on the speech feature extraction unit, the speech sample is subjected to feature extraction to obtain a corresponding speech feature vector; according to the semantic The feature vector and the speech feature vector, using the classification prediction unit to predict the content of the question and answer option matching the speech sample, obtaining the prediction information output by the classification prediction unit, and according to the prediction information and the label information. The intelligent questionnaire interview model performs model training.

In an implementation manner, the intelligent questionnaire interview model further includes a feature fusion unit; according to the semantic feature vector and the voice feature vector, the classification prediction unit is used to predict the question and answer option matching the voice sample The content is to obtain the prediction information output by the classification prediction unit, including: performing feature fusion processing on the semantic feature vector and the speech feature vector based on the feature fusion unit to obtain a fusion feature vector; inputting the fusion feature vector to the classification prediction unit to predict the content of the question and answer option matching the speech sample, and obtain the prediction information output by the classification prediction unit.

In an optional implementation manner, performing feature fusion processing on the semantic feature vector and the speech feature vector based on the feature fusion unit to obtain a fusion feature vector includes: combining the feature vector based on the feature fusion unit performing splicing processing on the semantic feature vector and the speech feature vector, and performing dimensionality reduction processing on the vector obtained after the splicing processing, so as to obtain the fusion feature vector.

In an implementation manner, the performing model training on the intelligent questionnaire interview model according to the prediction information and the label information includes: calculating a model loss according to the prediction information and the label information; The loss calculates the gradient and performs backpropagation to update the model parameters of the intelligent questionnaire interview model by using the gradient descent method.

According to the second aspect of the present application, an intelligent questionnaire interview method based on man-machine dialogue is provided, including: obtaining the voice input by the user in the current dialogue interview round; converting the voice into corresponding voice text, and analyzing the Encoding the speech text to obtain the semantic feature vector of the speech text; performing feature extraction on the speech to obtain a corresponding speech feature vector; generating a fusion feature vector according to the semantic feature vector and the speech feature vector; The fused feature vector is used to obtain the content of the question and answer option matched with the voice.

In an implementation manner, the converting the speech into a corresponding speech text, and encoding the speech text to obtain the semantic feature vector of the speech text includes: inputting the speech into an intelligent questionnaire interview model; wherein, the intelligent questionnaire interview model is a model obtained by using the training method described in any one of the first aspect; based on the speech recognition unit, the speech is recognized to obtain the corresponding speech text; based on the The text semantic feature extraction unit encodes the speech text to obtain a semantic feature vector of the speech text.

In an optional implementation manner, the performing feature extraction on the speech to obtain a corresponding speech feature vector includes: performing feature extraction on the speech based on the speech feature extraction unit to obtain a corresponding speech feature vector .

In an optional implementation manner, the obtaining the content of the question and answer option matching the voice according to the fusion feature vector includes: inputting the fusion feature vector into the classification prediction unit for prediction, and obtaining the The prediction information output by the classification prediction unit; according to the prediction information, the content of the question and answer option matching the voice is determined.

In an implementation manner, the method further includes: based on the preset questionnaire rules and the content of the question and answer options, triggering the corresponding action of the next dialogue interview round, and obtaining the question text of the next dialogue interview round ; performing text-to-speech TTS speech synthesis on the question text to obtain a question voice corresponding to the question text; playing the question voice to the user through a voice playback module.

According to the third aspect of the present application, an intelligent questionnaire interview model training device is provided, including: an acquisition module, which acquires voice samples and label information of the voice samples; an input module, which is used to input the voice samples into the intelligent questionnaire Interview model; wherein, the intelligent questionnaire interview model includes a speech recognition unit, a text semantic feature extraction unit, a speech feature extraction unit and a classification prediction unit; a first processing module, for performing the speech sample on the basis of the speech recognition unit Recognize, obtain the corresponding speech sample text, and encode the speech sample text based on the text semantic feature extraction unit to obtain the semantic feature vector of the speech sample text; the second processing module is used for based on the speech feature The extraction unit performs feature extraction on the speech sample to obtain the corresponding speech feature vector; the third processing module is used to use the classification prediction unit to predict the speech sample according to the semantic feature vector and the speech feature vector The matching question and answer option content is used to obtain the prediction information output by the classification prediction unit, and perform model training on the intelligent questionnaire interview model according to the prediction information and the label information.

In an implementation manner, the intelligent questionnaire interview model further includes a feature fusion unit; the third processing module is specifically configured to: perform feature extraction of the semantic feature vector and the speech feature vector based on the feature fusion unit Fusion processing to obtain a fusion feature vector; input the fusion feature vector into the classification prediction unit to predict the content of the question and answer option matching the speech sample, and obtain the prediction information output by the classification prediction unit.

In an optional implementation manner, the third processing module is specifically configured to: perform splicing processing on the semantic feature vector and the speech feature vector based on the feature fusion unit, and perform splicing processing on the obtained Vectors are subjected to dimensionality reduction processing to obtain the fused feature vectors.

In an optional implementation manner, the third processing module is specifically configured to: calculate a model loss according to the prediction information and the label information; calculate a gradient according to the model loss, and perform backpropagation to utilize The gradient descent method updates the model parameters of the intelligent questionnaire interview model.

According to the fourth aspect of the present application, there is provided an intelligent questionnaire interview device based on man-machine dialogue, including: an acquisition module for acquiring the voice input by the user in the current dialogue interview round; an encoding module for converting the voice into Corresponding voice text, and encode the voice text to obtain the semantic feature vector of the voice text; the feature extraction module is used to perform feature extraction on the voice to obtain the corresponding voice feature vector; the first processing module, A fusion feature vector is generated according to the semantic feature vector and the voice feature vector; a second processing module is used to obtain the content of the question and answer option matching the voice according to the fusion feature vector.

In an implementation manner, the encoding module is specifically configured to: input the voice into an intelligent questionnaire interview model; wherein, the intelligent questionnaire interview model is a model trained by using the training method as described in the first aspect; Recognizing the speech based on the speech recognition unit to obtain a corresponding speech text; encoding the speech text based on the text semantic feature extraction unit to obtain a semantic feature vector of the speech text.

In an optional implementation manner, the encoding module is specifically configured to: perform feature extraction on the speech based on the speech feature extraction unit to obtain a corresponding speech feature vector.

In an optional implementation manner, the encoding module is specifically configured to: input the fused feature vector into the classification prediction unit for prediction, and obtain prediction information output by the classification prediction unit; according to the prediction The information identifies the content of the question and answer options that match the voice.

In one implementation, the device further includes: a third processing module, configured to trigger corresponding actions in the next round of dialogue and interview based on the preset questionnaire rules and the content of the question and answer options, and obtain the next The question text of dialogue interview round; Speech synthesis module, is used for carrying out from text to speech TTS speech synthesis to described question text, obtains the question speech corresponding to described question text; Said question speech is played to by speech playing module said user.

According to a fifth aspect of the present application, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores information executable by the at least one processor. instructions, the instructions are executed by the at least one processor, so that the at least one processor can execute the method described in the first aspect, or execute the method described in the second aspect.

According to a sixth aspect of the present application, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method described in the first aspect, or to execute The method described in the second aspect.

According to a seventh aspect of the present application, there is provided a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method according to the first aspect, or implements the steps of the method according to the second aspect. steps of the method.

According to the technology of the present application, the speech feature vector and the speech feature vector can be obtained based on the acquired speech samples, and combined with the prediction of the classification prediction unit, the prediction information representing the content of the question and answer options matching the speech samples can be obtained, so that the intelligent The questionnaire interview model is trained to improve the semantic understanding and generalization capabilities of the intelligent questionnaire interview model, reduce the interview rounds of the questionnaire dialogue, improve user satisfaction during the questionnaire interview process, and increase the questionnaire recovery rate.

It should be understood that what is described in this section is not intended to identify key or important features of the embodiments of the application, nor is it intended to limit the scope of the application. Other features of the present application will be easily understood from the following description.

Description of drawings

The accompanying drawings are used to better understand the solution, and do not constitute a limitation to the application. in:

Fig. 1 is a schematic diagram according to the first embodiment of the present application;

Fig. 2 is a schematic diagram according to the second embodiment of the present application;

Fig. 3 is a schematic diagram according to a third embodiment of the present application;

Fig. 4 is a schematic diagram according to a fourth embodiment of the present application;

Fig. 5 is a schematic diagram according to a fifth embodiment of the present application;

FIG. 6 is a schematic diagram of an intelligent questionnaire interview scheme based on multimodal deep learning provided by an embodiment of the present application;

FIG. 7 is an application flow chart in an intelligent questionnaire interview system provided by an embodiment of the present application;

Fig. 8 is a schematic diagram of an intelligent questionnaire interview model training device provided by an embodiment of the present application;

Fig. 9 is a schematic structural diagram of an intelligent questionnaire interview device based on man-machine dialogue provided by an embodiment of the present application;

Fig. 10 is a schematic structural diagram of another intelligent questionnaire interview device for man-machine dialogue provided by the embodiment of the present application;

Fig. 11 is a block diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

Exemplary embodiments of the present application are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present application to facilitate understanding, and they should be regarded as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Among them, in the description of this application, unless otherwise specified, "/" means or means, for example, A/B can mean A or B; "and/or" in this article is only a kind of association describing associated objects A relationship means that there may be three kinds of relationships, for example, A and/or B means: A exists alone, A and B exist simultaneously, and B exists alone. The first, second, and other numbers involved in the present application are only for convenience of description, and are not used to limit the scope of the embodiments of the present application, nor do they indicate a sequence.

It should be noted that in the technical solution of this application, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved are all in compliance with relevant laws and regulations, and do not violate public order and good customs.

Please refer to FIG. 1 , which is a schematic diagram according to a first embodiment of the present application. As shown in Figure 1, the intelligent questionnaire interview model training method of this embodiment may include but not limited to the following steps:

Step S101: Obtain a voice sample and label information of the voice sample.

For example, the speech input by the user in the questionnaire interview dialogue scene is obtained as a speech sample, and the tag information of the content category to which the content of the speech sample belongs is obtained. For example, during the interview, the user may be provided with options such as A, B, and C. Assuming that the voice sample is option A, or other content similar to the content of option A, or part of the content of option A, the voice sample’s Labels can be category labels representing option A.

Step S102: Input the voice sample into the intelligent questionnaire interview model.

Wherein, in the embodiment of the present application, the intelligent questionnaire interview model includes a speech recognition unit, a text semantic feature extraction unit, a speech feature extraction unit and a classification prediction unit.

Step S103: Recognize the speech samples based on the speech recognition unit to obtain corresponding speech sample texts, and encode the speech sample texts based on the text semantic feature extraction unit to obtain semantic feature vectors of the speech sample texts.

For example, perform speech recognition on the speech sample based on the speech recognition unit to obtain the speech sample text corresponding to the speech sample, and perform feature encoding on the speech sample text based on the text semantic feature extraction unit to obtain the semantic feature vector of the speech sample text .

Wherein, in the embodiment of the present application, the text semantic feature extraction unit may be a BERT (BidirectionalEncoder Representations from Transformers) model.

In some embodiments of the present application, the pre-training Pre-training BERT model can be initialized, and then the parameters of the BERT model can be fine-tuned based on the interview content data with label information, and the processed BERT model can be used Get the semantic feature vector of the speech sample text.

In some embodiments of the present application, ASR (Automatic Speech Recognition, automatic speech recognition) enhancement technology can be used, combined with speech features and scene features, to enhance the speech text recognition function in a specific questionnaire interview scene to obtain the enhanced speech sample text , and then encode the enhanced speech sample text to obtain the semantic feature vector of the speech sample text.

Step S104: Perform feature extraction on the speech sample based on the speech feature extraction unit to obtain a corresponding speech feature vector.

For example, the acquired speech samples are input into the speech feature extraction unit, and the MFCC (Mel-Frequency Cepstral Coefficients, Mel-Frequency Cepstral Coefficients) method is used to sequentially perform pre-emphasis, framing, windowing and FFT (Fast Fourier Coefficients) on the speech samples. transform, Fast Fourier Transform) processing, input the processed data into the Mel filter bank to obtain output data, perform logarithmic operation on the output data, and then perform DCT (Discrete Cosine Transform, Discrete Cosine Transform) to complete feature extraction, and obtain as The speech feature vector represented by the physical information of the speech sample.

Step S105: According to the semantic feature vector and the voice feature vector, use the classification prediction unit to predict the content of the question and answer options that match the voice sample, obtain the prediction information output by the classification prediction unit, and perform model training on the intelligent questionnaire interview model according to the prediction information and label information .

For example, the semantic feature vector and the speech feature vector are used as input data, input into the classification prediction unit, and the prediction information output by the classification prediction unit is obtained, and the prediction information is a question and answer matching the speech sample corresponding to the above semantic feature vector and speech feature vector option content, and perform model training on the intelligent questionnaire interview model according to the prediction information and the label information corresponding to the voice sample.

By implementing the embodiment of the present application, the speech feature vector and the speech feature vector can be obtained based on the acquired speech sample, and combined with the prediction of the classification prediction unit, the prediction information representing the content of the question and answer option matching the speech sample can be obtained, so that the prediction information and label information can be used for The intelligent questionnaire interview model performs model training to improve the semantic understanding and generalization capabilities of the intelligent questionnaire interview model.

In an implementation manner, the semantic feature vector and the speech feature vector may be fused, so as to obtain prediction information based on the fused feature vector and classification prediction unit obtained through fusion. As an example, please refer to FIG. 2 , which is a schematic diagram according to a second embodiment of the present application. As shown in FIG. 2 , the intelligent questionnaire interview model training method of this embodiment may include but not limited to the following steps.

Step S201: Acquire voice samples and label information of the voice samples.

In the embodiments of the present application, step S201 may be implemented in any one of the embodiments of the present application, which is not limited in the embodiments of the present application, and will not be repeated here.

Step S202: Input the voice sample into the intelligent questionnaire interview model.

In the embodiment of the present application, step S202 may be implemented in any one of the embodiments of the present application, which is not limited in the embodiment of the present application, and will not be repeated here.

Among them, the intelligent questionnaire interview model includes a speech recognition unit, a text semantic feature extraction unit, a speech feature extraction unit and a classification prediction unit.

Step S203: Recognize the speech samples based on the speech recognition unit to obtain corresponding speech sample texts, and encode the speech sample texts based on the text semantic feature extraction unit to obtain semantic feature vectors of the speech sample texts.

In the embodiments of the present application, step S203 may be implemented in any one of the embodiments of the present application, which is not limited in the embodiments of the present application, and will not be repeated here.

Step S204: Perform feature extraction on the speech sample based on the speech feature extraction unit to obtain a corresponding speech feature vector.

In the embodiment of the present application, step S204 may be implemented in any one of the embodiments of the present application, which is not limited in the embodiment of the present application, and will not be repeated here.

Step S205: Perform feature fusion processing on the semantic feature vector and the speech feature vector based on the feature fusion unit to obtain a fusion feature vector.

In an optional implementation, the feature fusion unit performs feature fusion processing on the semantic feature vector and the speech feature vector to obtain the fusion feature vector, including: splicing the semantic feature vector and the speech feature vector based on the feature fusion unit , and perform dimensionality reduction on the vectors obtained after splicing to obtain the fusion feature vector.

For example, the semantic feature vector and the phonetic feature vector are input into the feature fusion unit, so that the semantic feature vector and the phonetic feature vector are directly vector-spliced, and the vector obtained after the splicing process is subjected to PCA (Principal Component Analysis, principal component analysis) reduction. dimension processing.

Wherein, the above-mentioned splicing process of the semantic feature vector and the phonetic feature vector can be expressed as:

v=[v ₁ ,v ₂ ]∈R ^n+m

Among them, v is the vector obtained after splicing, v ₁ is the semantic feature vector v ₁ ∈ R ⁿ , v ₂ is the phonetic feature vector, v ₂ ∈ R ^m , m and n are positive integers.

In some embodiments of the present application, a linear mapping method may be used to convert the vector obtained after splicing into the form of v=Wv∈R ⁿ .

Step S206: Input the fused feature vector into the classification prediction unit to predict the content of the question and answer option matching the voice sample, and obtain the prediction information output by the classification prediction unit.

For example, the fused feature vector is used as input data to the classification prediction unit to obtain the prediction information output by the classification prediction unit. The prediction information is the content of the question and answer options predicted by the classification prediction unit and matched with the speech sample.

Step S207: Perform model training on the intelligent questionnaire interview model according to the prediction information and label information.

For example, the current loss value of the model is obtained according to the prediction information and label information, and the intelligent questionnaire interview model parameters are updated according to the current loss value to reduce the loss value of the model, thereby training the model.

By implementing the embodiment of the present application, the speech feature vector and the speech feature vector can be obtained based on the acquired speech samples, and the semantic feature vector and the speech feature vector can be fused together, so that the representative and The prediction information of the content of the question and answer option matched by the voice sample is used to perform model training on the intelligent questionnaire interview model according to the prediction information and label information. It can improve the semantic understanding ability and generalization ability of the intelligent questionnaire interview model.

In an implementation manner, the model loss may be calculated according to the prediction information and the label information, so as to update the model parameters according to the model loss. As an example, please refer to FIG. 3 , which is a schematic diagram according to a third embodiment of the present application. As shown in FIG. 3 , the intelligent questionnaire interview model training method of this embodiment may include but not limited to the following steps.

Step S301: Obtain a voice sample and label information of the voice sample.

In the embodiment of the present application, step S301 may be implemented in any one of the embodiments of the present application, which is not limited in the embodiment of the present application, and will not be repeated here.

Step S302: Input the voice sample into the intelligent questionnaire interview model.

In the embodiment of the present application, step S302 may be implemented in any one of the embodiments of the present application, which is not limited in the embodiment of the present application, and will not be repeated here.

Step S303: Recognize the speech samples based on the speech recognition unit to obtain corresponding speech sample texts, and encode the speech sample texts based on the text semantic feature extraction unit to obtain semantic feature vectors of the speech sample texts.

In the embodiment of the present application, step S303 may be implemented in any one of the embodiments of the present application, which is not limited in the embodiment of the present application, and will not be repeated here.

Step S304: Perform feature extraction on the speech sample based on the speech feature extraction unit to obtain a corresponding speech feature vector.

In the embodiment of the present application, step S304 may be implemented in any one of the embodiments of the present application, which is not limited in the embodiment of the present application, and will not be repeated here.

Step S305: According to the semantic feature vector and the speech feature vector, the classification prediction unit is used to predict the content of the question and answer option matching the speech sample, and the prediction information output by the classification prediction unit is obtained.

In the embodiment of the present application, step S305 may be implemented in any one of the embodiments of the present application, which is not limited in the embodiment of the present application, and will not be repeated here.

Step S306: Calculate model loss according to prediction information and label information.

For example, according to the prediction information and label information, a preset loss function is used to calculate the model loss.

Step S307: Calculate the gradient according to the model loss, and perform backpropagation to update the model parameters of the intelligent questionnaire interview model by gradient descent.

For example, the gradient is calculated according to the model loss, and in the backpropagation process, the gradient descent method is used to iteratively search for the minimum value of the model loss function to update the model parameters of the intelligent questionnaire interview model.

By implementing the embodiment of the present application, the speech feature vector and the speech feature vector can be obtained based on the acquired speech samples, and combined with the prediction of the classification prediction unit to obtain the prediction information representing the content of the question and answer option matching the speech sample, so as to calculate according to the prediction information and label information Model loss, to update the model parameters according to the model loss, realize the model training of the intelligent questionnaire interview model, and improve the semantic understanding ability and generalization ability of the intelligent questionnaire interview model.

Please refer to FIG. 4 , which is a schematic diagram according to a fourth embodiment of the present application. As shown in FIG. 4 , the intelligent questionnaire interview method based on man-machine dialogue in this embodiment may include but not limited to the following steps.

Step S401: Obtain the voice input by the user in the current dialogue interview round.

For example, to obtain the voice information input by the user in the current interview round of the ongoing dialogue.

Step S402: converting the speech into corresponding speech text, and encoding the speech text to obtain a semantic feature vector of the speech text.

For example, speech recognition is performed on the speech to convert the speech into corresponding speech text, and the speech text is input into the BERT model for encoding to obtain the semantic feature vector of the speech text.

Step S403: Perform feature extraction on the speech to obtain a corresponding speech feature vector.

For example, the MFCC method is used to perform feature extraction on the speech to obtain the corresponding speech feature vector.

Step S404: Generate a fusion feature vector according to the semantic feature vector and the phonetic feature vector.

For example, the semantic feature vector and the phonetic feature vector are vector concatenated to generate a fusion feature vector.

Step S405: Obtain the content of the question and answer option matched with the voice according to the fused feature vector.

For example, semantic understanding processing is performed on the fused feature vector to understand dialogue semantics, so as to identify corresponding question and answer option content.

By implementing the embodiment of the present application, the semantic feature vector and the speech feature vector obtained based on the voice input by the user can be fused to generate a fused feature vector, so as to obtain the question and answer option content matched with the voice according to the fused feature vector. It can improve the semantic understanding ability of the user's voice in the man-machine dialogue scene, thereby improving the user satisfaction in the questionnaire interview process and improving the questionnaire recovery rate.

In an implementation manner, based on the intelligent questionnaire interview model provided by the embodiment of the present application, the speech may be converted into corresponding speech text, and the speech text may be encoded to obtain a semantic feature vector of the speech text. As an example, please refer to FIG. 5 , which is a schematic diagram according to a fifth embodiment of the present application. As shown in FIG. 5 , the intelligent questionnaire interview method based on man-machine dialogue in this embodiment may include but not limited to the following steps.

Step S501: Obtain the voice input by the user in the current dialogue interview round.

In the embodiment of the present application, step S501 may be implemented in any one of the embodiments of the present application, which is not limited in the embodiment of the present application, and will not be repeated here.

Step S502: Input voice into the intelligent questionnaire interview model.

Wherein, in the embodiment of the present application, the intelligent questionnaire interview model is a model trained by the training method provided in any embodiment of the present application.

Step S503: Recognize the voice based on the voice recognition unit to obtain the corresponding voice text.

For example, based on the speech recognition unit in the intelligent questionnaire interview model, the speech input by the user is recognized to obtain the corresponding speech text.

Step S504: Encoding the speech text based on the text semantic feature extraction unit to obtain a semantic feature vector of the speech text.

For example, based on the text semantic feature extraction unit in the intelligent questionnaire interview model, feature encoding is performed on the speech text to obtain the corresponding semantic feature vector of the speech text.

Step S505: Perform feature extraction on the speech based on the speech feature extraction unit to obtain a corresponding speech feature vector.

For example, the speech feature extraction unit in the intelligent questionnaire interview model performs feature extraction on the speech input by the user to obtain a corresponding speech feature vector.

Step S506: Input the fused feature vector into the classification prediction unit for prediction, and obtain the prediction information output by the classification prediction unit.

For example, the fusion feature vector is input into the classification prediction unit as input data, and the prediction information output by the classification prediction unit is obtained.

Step S507: Determine the content of the question and answer option that matches the voice according to the prediction information.

For example, different preset question and answer option contents are correspondingly set in advance for a variety of different preset prediction information, and the corresponding target preset prediction information is determined according to the prediction information output by the classification prediction unit, and the target preset prediction information is corresponding to The preset question-and-answer option content of is determined as the voice-matched question-and-answer option content.

By implementing the embodiment of the present application, the voice input by the user can be input into the intelligent questionnaire interview model obtained through pre-training, which can be processed to obtain prediction information, so as to determine the content of the question and answer option matching the voice according to the prediction information. It can improve the semantic understanding ability of the user's voice in the man-machine dialogue scene, thereby improving the user satisfaction in the questionnaire interview process and improving the questionnaire recovery rate.

In some embodiments of the present application, the above-mentioned intelligent questionnaire interview method based on man-machine dialogue may also include: based on the preset questionnaire rules and question and answer option content, triggering the corresponding action of the next dialogue interview round, and obtaining the next dialogue The question text of the interview round; TTS (Text To Speech, from text to speech) speech synthesis is performed on the question text to obtain the question voice corresponding to the question text; the question voice is played to the user through the voice playback module.

As an example, assuming that the voice input by the user is female, based on the logic of the preset questionnaire rules, in the next dialogue interview round, the user will not be asked questions about men, and the questions corresponding to women will be directly obtained text, and perform TTS speech synthesis on the question text to obtain the question voice corresponding to the question text; play the question voice to the user through the voice playback module. This reduces the number of dialogue interview rounds.

Please refer to FIG. 6 . FIG. 6 is a schematic diagram of an intelligent questionnaire interview scheme based on multimodal deep learning provided by an embodiment of the present application. As shown in Figure 6, the technical solution of the present application first obtains the voice of the user, performs feature encoding on the voice based on the MFCC method to obtain the voice feature vector, converts the voice into text, obtains the semantic feature vector based on the BERT model, and combines the voice feature vector and Semantic feature vectors are fused, and classification prediction is performed based on the fused vectors, and prediction information representing the content of the question and answer options matched with the voice sample is obtained.

Please refer to FIG. 7 . FIG. 7 is an application flowchart of an intelligent questionnaire interview system provided by an embodiment of the present application. As shown in Figure 7, the intelligent questionnaire interview system of the present application is limited to obtain the speech input by the user, and then perform MFCC feature extraction on the acquired speech data based on the data processing module to obtain the speech feature vector, and use ASR and BERT to convert the obtained speech Text features are constructed to obtain text features (ie, semantic feature vectors), and then based on the feature fusion module, speech features and text features are feature-fused, and the fused feature vectors are input into the classification prediction module for classification prediction, and the prediction results are obtained. The results are sent to the questionnaire system, and the questionnaire system stores the prediction results and can judge and process dialogue actions based on the prediction results.

Please refer to FIG. 8 . FIG. 8 is a schematic diagram of an intelligent questionnaire interview model training device provided by an embodiment of the present application. As shown in Figure 8, the device 800 includes: an acquisition module 801, which is used to acquire voice samples and label information of the voice samples; an input module 802, which is used to input voice samples into the intelligent questionnaire interview model; wherein, the intelligent questionnaire interview model Including a speech recognition unit, a text semantic feature extraction unit, a speech feature extraction unit and a classification prediction unit; the first processing module 803 is used to identify the speech sample based on the speech recognition unit, obtain the corresponding speech sample text, and based on the text semantic feature The extraction unit encodes the speech sample text to obtain the semantic feature vector of the speech sample text; the second processing module 804 is used to perform feature extraction on the speech sample based on the speech feature extraction unit to obtain a corresponding speech feature vector; the third processing module 805 , which is used to predict the content of the question and answer options that match the voice sample by using the classification prediction unit according to the semantic feature vector and the speech feature vector, and obtain the prediction information output by the classification prediction unit, and perform model training on the intelligent questionnaire interview model according to the prediction information and label information .

In one implementation, the intelligent questionnaire interview model also includes a feature fusion unit; the third processing module 805 is specifically used to: perform feature fusion processing on the semantic feature vector and the speech feature vector based on the feature fusion unit to obtain the fusion feature vector; The feature vector is input into the classification prediction unit to predict the content of the question and answer option matched with the voice sample, and the prediction information output by the classification prediction unit is obtained.

In an optional implementation manner, the third processing module 805 is specifically configured to: perform splicing processing on the semantic feature vector and the speech feature vector based on the feature fusion unit, and perform dimensionality reduction processing on the vector obtained after splicing processing, so as to obtain Merge feature vectors.

In an optional implementation, the third processing module 805 is specifically configured to: calculate the model loss according to the prediction information and the label information; calculate the gradient according to the model loss, and perform backpropagation to use the gradient descent method to interview the intelligent questionnaire The model parameters of the model are updated.

The device of the embodiment of the present application can obtain the speech feature vector and the speech feature vector based on the acquired speech sample, and combine the prediction of the classification prediction unit to obtain the prediction information representing the content of the question and answer option that matches the speech sample, so that according to the prediction information and label information The intelligent questionnaire interview model performs model training to improve the semantic understanding and generalization capabilities of the intelligent questionnaire interview model.

Please refer to FIG. 9 . FIG. 9 is a schematic structural diagram of an intelligent questionnaire interview device based on man-machine dialogue provided by an embodiment of the present application. As shown in FIG. 9 , the device 900 includes: an acquisition module 901 for acquiring the voice input by the user in the current dialogue interview round; an encoding module 902 for converting the voice into corresponding voice text, and encoding the voice text to Obtain the semantic feature vector of speech text; Feature extraction module 903 is used for feature extraction to speech, obtains corresponding speech feature vector; The first processing module 904 is used for generating fusion feature vector according to semantic feature vector and speech feature vector; The second processing module 905 is configured to obtain the content of the question and answer option matched with the voice according to the fusion feature vector.

In one implementation, the encoding module 902 is specifically configured to: input the voice into the intelligent questionnaire interview model; wherein, the intelligent questionnaire interview model is a model obtained by using the training method as in the first aspect; perform voice recognition based on the voice recognition unit Recognition to obtain the corresponding speech text; based on the text semantic feature extraction unit, the speech text is encoded to obtain the semantic feature vector of the speech text.

In an optional implementation manner, the encoding module 902 is specifically configured to: perform feature extraction on the speech based on the speech feature extraction unit to obtain a corresponding speech feature vector.

In an optional implementation, the encoding module 902 is specifically configured to: input the fused feature vector into the classification prediction unit for prediction, and obtain the prediction information output by the classification prediction unit; determine the content of the question and answer option that matches the voice according to the prediction information .

Through the device of the embodiment of the present application, the semantic feature vector and the speech feature vector obtained based on the voice input by the user can be fused to generate a fused feature vector, so as to obtain the question and answer option content matching the voice according to the fused feature vector. It can improve the semantic understanding ability of the user's voice in the man-machine dialogue scene, thereby improving the user satisfaction in the questionnaire interview process and improving the questionnaire recovery rate.

In an implementation manner, the foregoing apparatus further includes a third processing module. As an example, please refer to FIG. 10 . FIG. 10 is a schematic structural diagram of another intelligent questionnaire and interview device for man-machine dialogue provided by the embodiment of the present application. As shown in Figure 10, the device 1000 also includes a third processing module 1006, configured to trigger the corresponding actions of the next round of dialogue and interview based on the preset questionnaire rules and question and answer option content, and obtain the results of the next round of dialogue and interview. Question text; speech synthesis module, used for text-to-speech TTS speech synthesis to the question text to obtain question speech corresponding to the question text; play the question speech to the user through the speech playback module. Wherein, 1001-1005 in FIG. 10 and 901-905 in FIG. 9 have the same function and structure.

Regarding the apparatus in the foregoing embodiments, the specific manner in which each module executes operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

According to the embodiments of the present application, the present application also provides an electronic device and a readable storage medium.

As shown in FIG. 11 , it is a block diagram of an electronic device provided by an embodiment of the present application. The electronic device is a block diagram of an electronic device capable of executing the intelligent questionnaire interview model training method according to the embodiment of the present application, or the intelligent questionnaire interview method based on man-machine dialogue. Electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are by way of example only, and are not intended to limit implementations of the applications described and/or claimed herein.

As shown in FIG. 11 , the electronic device includes: one or more processors 1101 , a memory 1102 , and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be mounted on a common motherboard or otherwise as desired. The processor may process instructions executed within the electronic device, including instructions stored in or on the memory, to display graphical information of a GUI on an external input/output device such as a display device coupled to an interface. In other implementations, multiple processors and/or multiple buses may be used with multiple memories and multiple memories, if desired. Likewise, multiple electronic devices may be connected, with each device providing some of the necessary operations (eg, as a server array, a set of blade servers, or a multi-processor system). In FIG. 11, a processor 1101 is taken as an example.

The memory 1102 is a non-transitory computer-readable storage medium provided in this application. Wherein, the memory stores instructions executable by at least one processor, so that the at least one processor executes the intelligent questionnaire interview model training method provided in this application, or the intelligent questionnaire interview method based on human-computer dialogue. The non-transitory computer-readable storage medium of the present application stores computer instructions, and the computer instructions are used to make the computer execute the intelligent questionnaire interview model training method provided in the present application, or the intelligent questionnaire interview method based on man-machine dialogue.

The memory 1102, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs and modules, such as program instructions/modules corresponding to the intelligent questionnaire interview model training method in the embodiment of the present application ( For example, the acquisition module 801, input module 802, first processing module 803, second processing module 804, and third processing module 805 shown in Figure 8) or program instructions/modules corresponding to the intelligent questionnaire interview method based on man-machine dialogue (For example, acquisition module 901, encoding module 902, feature extraction module 903, first processing module 904 and second processing module 905 shown in FIG. 9, and third processing module 1006 shown in FIG. 10). The processor 1101 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1102, that is, to realize the intelligent questionnaire interview model training method in the above method embodiment, or based on human Intelligent questionnaire interview method of machine dialogue.

The memory 1102 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application program required by at least one function; Data created by the use of electronic equipment using the smart questionnaire interview method, etc. In addition, the memory 1102 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory 1102 may optionally include a memory set remotely relative to the processor 1101, and these remote memories may be connected to electronic devices of the intelligent questionnaire interview model training method or the intelligent questionnaire interview method based on man-machine dialogue through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The electronic equipment of the intelligent questionnaire interview model training method, or the intelligent questionnaire interview method based on man-machine dialogue may further include: an input device 1103 and an output device 1104 . The processor 1101, the memory 1102, the input device 1103, and the output device 1104 may be connected through a bus or in other ways, and connection through a bus is taken as an example in FIG. 11 .

The input device 1103 can receive input digital or character information, and generate key signal input related to the intelligent questionnaire interview model training method, or the user setting and function control of electronic equipment based on the intelligent questionnaire interview method of man-machine dialogue, such as touch screen, Input devices such as keypads, mice, trackpads, touchpads, pointing sticks, one or more mouse buttons, trackballs, joysticks, etc. The output device 1104 may include a display device, an auxiliary lighting device (eg, LED), a tactile feedback device (eg, a vibration motor), and the like. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some implementations, the display device may be a touch screen.

Various implementations of the systems and techniques described herein can be implemented in digital electronic circuitry, integrated circuit systems, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpreted on a programmable system including at least one programmable processor, the programmable processor Can be special-purpose or general-purpose programmable processor, can receive data and instruction from storage system, at least one input device, and at least one output device, and transmit data and instruction to this storage system, this at least one input device, and this at least one output device an output device.

These computing programs (also referred to as programs, software, software applications, or codes) include machine instructions for a programmable processor and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine language calculation program. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or means for providing machine instructions and/or data to a programmable processor ( For example, magnetic disks, optical disks, memories, programmable logic devices (PLDs), including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide for interaction with the user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user. ); and a keyboard and pointing device (eg, a mouse or a trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and can be in any form (including Acoustic input, speech input or, tactile input) to receive input from the user.

The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., as a a user computer having a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system can be interconnected by any form or medium of digital data communication, eg, a communication network. Examples of communication networks include: local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

A computer system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the problem of traditional physical host and VPS service ("Virtual Private Server", or "VPS") Among them, there are defects such as difficult management and weak business scalability. The server can also be a server of a distributed system, or a server combined with a blockchain.

According to the technical solution of the embodiment of the present application, the speech feature vector and the speech feature vector can be obtained based on the acquired speech samples, and combined with the prediction of the classification prediction unit, the prediction information representing the content of the question and answer option matching the speech sample can be obtained, so that according to the prediction information and the label Information conducts model training on the intelligent questionnaire interview model to improve the semantic understanding and generalization capabilities of the intelligent questionnaire interview model, thereby improving user satisfaction during the questionnaire interview process and improving the questionnaire recovery rate.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the present application may be executed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in the present application can be achieved, no limitation is imposed herein.

The above specific implementation methods are not intended to limit the protection scope of the present application. It should be apparent to those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims (21)

1. A method for training an intelligent questionnaire interview model, comprising: Obtaining a voice sample and label information of the voice sample; The speech sample is input to the intelligent questionnaire interview model; wherein, the intelligent questionnaire interview model includes a speech recognition unit, a text semantic feature extraction unit, a speech feature extraction unit and a classification prediction unit; Recognizing the speech sample based on the speech recognition unit to obtain a corresponding speech sample text, and encoding the speech sample text based on the text semantic feature extraction unit to obtain a semantic feature vector of the speech sample text; Carrying out feature extraction to the speech sample based on the speech feature extraction unit to obtain a corresponding speech feature vector; According to the semantic feature vector and the speech feature vector, use the classification prediction unit to predict the content of the question and answer option matching the speech sample, obtain the prediction information output by the classification prediction unit, and according to the prediction information and the obtained The label information is used to perform model training on the intelligent questionnaire interview model. 2. method as claimed in claim 1, wherein, described intelligent questionnaire interview model also comprises feature fusion unit; Described according to described semantic feature vector and described speech feature vector, adopt described classification prediction unit to predict and described The content of the question and answer option matched by the voice sample is obtained to obtain the prediction information output by the classification prediction unit, including: performing feature fusion processing on the semantic feature vector and the speech feature vector based on the feature fusion unit to obtain a fusion feature vector; The fused feature vector is input into the classification prediction unit to predict the content of the question and answer option matching the speech sample, and the prediction information output by the classification prediction unit is obtained. 3. The method according to claim 2, wherein said feature fusion unit performs feature fusion processing on said semantic feature vector and said speech feature vector to obtain a fusion feature vector, comprising: Splicing the semantic feature vector and the speech feature vector based on the feature fusion unit, and performing dimensionality reduction processing on the vector obtained after the splicing process, so as to obtain the fusion feature vector. 4. The method according to claim 1, wherein said model training is carried out to said intelligent questionnaire interview model according to said prediction information and said label information, comprising: calculating a model loss according to the prediction information and the label information; The gradient is calculated according to the model loss, and backpropagation is performed to update the model parameters of the intelligent questionnaire interview model in a gradient descent manner. 5. An intelligent questionnaire interview method based on man-machine dialogue, comprising: Obtain the voice input by the user in the current dialogue interview round; Converting the speech into a corresponding phonetic text, and encoding the phonetic text to obtain a semantic feature vector of the phonetic text; Carry out feature extraction to described speech, obtain corresponding speech feature vector; Generate a fusion feature vector according to the semantic feature vector and the speech feature vector; The content of the question and answer option matching the voice is obtained according to the fused feature vector. 6. The method as claimed in claim 5, wherein said converting said speech into corresponding phonetic text, and encoding said phonetic text to obtain the semantic feature vector of said phonetic text, comprising: Said speech is input to intelligent questionnaire interview model; Wherein, said intelligent questionnaire interview model is the model that adopts training method training as described in any one of claims 1 to 4; Recognizing the voice based on the voice recognition unit to obtain a corresponding voice text; Encoding the phonetic text based on the text semantic feature extraction unit to obtain a semantic feature vector of the phonetic text. 7. method as claimed in claim 6, wherein, described speech is carried out feature extraction, obtains corresponding speech feature vector, comprises: Based on the speech feature extraction unit, feature extraction is performed on the speech to obtain a corresponding speech feature vector. 8. The method according to claim 6, wherein said obtaining the question-and-answer option content matched with said voice according to said fusion feature vector comprises: inputting the fused feature vector into the classification prediction unit for prediction, and obtaining prediction information output by the classification prediction unit; The content of the question and answer option matching the voice is determined according to the prediction information. 9. The method of any one of claims 5 to 8, further comprising: Based on the preset questionnaire rules and the content of the question and answer options, trigger the corresponding action of the next dialogue interview round, and obtain the question text of the next dialogue interview round; Carrying out TTS speech synthesis from text to speech to described question text, obtains the question speech corresponding to described question text; The voice of the question is played to the user through the voice playing module. 10. An intelligent questionnaire interview model training device, comprising: An acquisition module, voice acquisition of voice samples and tag information of the voice samples; The input module is used to input the speech sample into the intelligent questionnaire interview model; wherein, the intelligent questionnaire interview model includes a speech recognition unit, a text semantic feature extraction unit, a speech feature extraction unit and a classification prediction unit; A first processing module, configured to recognize the speech sample based on the speech recognition unit to obtain a corresponding speech sample text, and encode the speech sample text based on the text semantic feature extraction unit to obtain the speech The semantic feature vector of the sample text; The second processing module is used to perform feature extraction on the speech sample based on the speech feature extraction unit to obtain a corresponding speech feature vector; A third processing module, configured to use the classification prediction unit to predict the content of the question and answer option matching the speech sample according to the semantic feature vector and the speech feature vector, obtain the prediction information output by the classification prediction unit, and Model training is performed on the intelligent questionnaire interview model according to the prediction information and the label information. 11. The device according to claim 10, wherein the intelligent questionnaire interview model also includes a feature fusion unit; the third processing module is specifically used for: performing feature fusion processing on the semantic feature vector and the speech feature vector based on the feature fusion unit to obtain a fusion feature vector; The fused feature vector is input into the classification prediction unit to predict the content of the question and answer option matching the speech sample, and the prediction information output by the classification prediction unit is obtained. 12. The device according to claim 11, wherein the third processing module is specifically configured to: Splicing the semantic feature vector and the speech feature vector based on the feature fusion unit, and performing dimensionality reduction processing on the vector obtained after the splicing process, so as to obtain the fusion feature vector. 13. The device according to claim 10, wherein the third processing module is specifically configured to: calculating a model loss according to the prediction information and the label information; The gradient is calculated according to the model loss, and backpropagation is performed to update the model parameters of the intelligent questionnaire interview model in a gradient descent manner. 14. An intelligent questionnaire interview device based on man-machine dialogue, comprising: An acquisition module, configured to acquire the voice input by the user in the current dialogue interview round; An encoding module, configured to convert the speech into a corresponding phonetic text, and encode the phonetic text to obtain a semantic feature vector of the phonetic text; A feature extraction module, configured to perform feature extraction on the speech to obtain a corresponding speech feature vector; A first processing module, configured to generate a fusion feature vector according to the semantic feature vector and the speech feature vector; The second processing module is configured to acquire the content of the question and answer option matched with the voice according to the fused feature vector. 15. The device according to claim 14, wherein the encoding module is specifically used for: Said speech is input to intelligent questionnaire interview model; Wherein, said intelligent questionnaire interview model is the model that adopts training method training as described in any one of claims 1 to 4; Recognizing the voice based on the voice recognition unit to obtain a corresponding voice text; Encoding the phonetic text based on the text semantic feature extraction unit to obtain a semantic feature vector of the phonetic text. 16. The device according to claim 15, wherein the encoding module is specifically used for: Based on the speech feature extraction unit, feature extraction is performed on the speech to obtain a corresponding speech feature vector. 17. The device according to claim 15, wherein the encoding module is specifically used for: inputting the fused feature vector into the classification prediction unit for prediction, and obtaining prediction information output by the classification prediction unit; The content of the question and answer option matching the voice is determined according to the prediction information. 18. The apparatus of any one of claims 14 to 17, further comprising: The third processing module is used to trigger the corresponding action of the next dialogue interview round based on the preset questionnaire rules and the content of the question and answer options, and obtain the question text of the next dialogue interview round; Speech synthesis module, for carrying out from text to speech TTS speech synthesis to described question text, obtains the question speech corresponding to described question text; The voice of the question is played to the user through the voice playing module. 19. An electronic device, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions are executed by the at least one processor, so that the at least one processor can perform any one of claims 1 to 4 The method, or, carry out the method described in any one in claim 5 to 9. 20. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 4, or to execute the The method described in any one of 5 to 9 is required. 21. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 4, or implements the steps according to any one of claims 5 to 9 The step of any described method.

Images