CN110110145A - Document creation method and device are described - Google Patents

Abstract

The application discloses a description text generating method and device, belonging to the field of information processing. The method includes: extracting at least one visual feature vector from a target object, where the target object is a video or a picture; acquiring a semantic feature vector corresponding to each of the visual feature vectors; for the at least one visual feature vector, And each semantic feature vector corresponding to the visual feature vector is processed to obtain the description text of the target object. The description accuracy and flexibility of the method provided by this application are relatively high.

Description

Description text generation method and device

technical field

The invention relates to the field of information processing, in particular to a description text generation method and device.

Background technique

The description text generation method is a method for generating text for describing video content using natural language. After the description text of the video is generated by this method, it is convenient for the user to quickly retrieve the desired video through the text, and it can help the visually impaired to understand the content of the video through text or voice.

In related technologies, when generating video description text, a pre-trained classifier can be used to identify visual objects (such as objects, scenes and actions, etc.) in the video, and then a predetermined language template can be used to analyze the identified visual objects. The text corresponding to the object is organized to obtain the description text of the video. Wherein, the language template may be obtained by mining a large amount of text data in advance.

However, since the methods in the related art all use fixed language templates to organize texts when generating description texts for different videos, the flexibility and accuracy of the descriptions are relatively low.

Contents of the invention

Embodiments of the present invention provide a description text generation method and device, which can solve the problem of low flexibility and accuracy in the description text generation method in the related art. Described technical scheme is as follows:

In one aspect, a method for generating description text is provided, the method comprising:

Extracting at least one visual feature vector from a target object, where the target object is a video or a picture;

obtaining a semantic feature vector corresponding to each of the visual feature vectors;

The at least one visual feature vector and the semantic feature vector corresponding to each visual feature vector are processed to obtain the description text of the target object.

In another aspect, an apparatus for generating description text is provided, the apparatus comprising:

An extraction module, configured to extract at least one visual feature vector from a target object, where the target object is a video or a picture;

An acquisition module, configured to acquire a semantic feature vector corresponding to each of the visual feature vectors;

The processing module is configured to process the at least one visual feature vector and the semantic feature vector corresponding to each visual feature vector to obtain the description text of the target object.

In yet another aspect, a terminal is provided, the terminal includes a processor and a memory, at least one instruction, at least one program, code set or instruction set are stored in the memory, the at least one instruction, the at least one program . The code set or instruction set is loaded and executed by the processor to implement the description text generation method provided in the above aspect.

In yet another aspect, a computer-readable storage medium is provided, wherein at least one instruction, at least one section of program, code set or instruction set is stored in the storage medium, and the at least one instruction, the at least one section of program, the code The set or instruction set is loaded and executed by the processor to implement the method for generating description text as provided in the above aspect.

The beneficial effects brought by the technical solution provided by the embodiments of the present invention are:

The embodiment of the present invention provides a description text generation method and device, which can extract at least one visual feature vector from the target object, and can obtain the semantic feature vector corresponding to each visual feature vector, and then can based on the at least one visual feature vector The feature vectors and the semantic feature vectors of each visual feature vector generate a descriptive text for the target object. Since the semantic feature vector corresponding to each visual feature vector can reflect the semantic feature of the target object, the accuracy and flexibility of the description can be improved by assisting the generation of description text through the semantic feature vector.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained based on these drawings without creative effort.

FIG. 1 is a schematic diagram of a device used to describe a text generation method provided by an embodiment of the present invention;

Fig. 2 is a flow chart of a description text generation method provided by an embodiment of the present invention;

Fig. 3 is an algorithm block diagram of a description text generation method provided by an embodiment of the present invention;

FIG. 4 is a flowchart of another description text generation method provided by an embodiment of the present invention;

5 is a flowchart of a method for obtaining a semantic feature vector corresponding to a first visual feature vector according to an embodiment of the present invention;

Fig. 6 is a flow chart of a method for training a memory model provided by an embodiment of the present invention;

FIG. 7 is an algorithm block diagram of a training memory model provided by an embodiment of the present invention;

Fig. 8 is a schematic structural diagram of a description text generation device provided by an embodiment of the present invention;

Fig. 9 is a schematic structural diagram of an acquisition module provided by an embodiment of the present invention;

Fig. 10 is a schematic structural diagram of another descriptive text generation device provided by an embodiment of the present invention;

Fig. 11 is a schematic structural diagram of an extraction module provided by an embodiment of the present invention;

Fig. 12 is a schematic structural diagram of a terminal block provided by an embodiment of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the implementation manner of the present invention will be further described in detail below in conjunction with the accompanying drawings.

In related technologies, in addition to the method based on the language template, the method based on machine translation is generally used to describe the video content. The principle of machine translation is to translate a text S described in the source language into a text T described in the target language. The translation process generally needs to be completed through multiple subtasks (such as word translation, word alignment and reordering, etc.). Among them, when translating words, it can be realized by maximizing the conditional probability p(T|S). In recent years, with the rise of deep learning technology, machine translation methods have generally begun to use recurrent neural network-based encoding and decoding models. In the encoding and decoding model, a cyclic neural network for encoding can convert the input text described in the source language into a feature vector, and then another cyclic neural network for decoding generates the target language text based on the feature vector. The text of the description. Wherein, the recurrent neural network used in the encoding and decoding model may be a Long Short-Term Memory (Long Short-Term Memory, LSTM) network.

The descriptive text generation method based on machine translation can also adopt a similar encoding and decoding model, which can include an encoder and a decoder. The encoder mainly includes a convolutional neural network and a recurrent neural network. The convolutional neural network can be respectively Each frame image in the video is processed separately to extract the feature vector of each frame image, and then the cyclic neural network can encode the feature vector of each frame image into a visual feature vector and input it to the decoder. The decoder is a decoder based on a cyclic neural network, and the decoder can decode the input visual feature vector into a text composed of multiple words, thereby realizing the description of the video content.

However, because the method in the related art can only extract a single visual feature vector in each frame of image, it cannot effectively mine the relationship between various visual objects in the video, and its description effect is poor. Moreover, this method mainly obtains the codec model between the video and the description text by training a large number of marked training samples (each training sample includes a video and a description text corresponding to the video), and the codec model is for the marked The dependence of the training samples is large, and due to the complex characteristics of the video itself, the effect of the codec model still needs to be improved.

Fig. 1 is a schematic diagram of a device applied to a description text generation method provided by an embodiment of the present invention. Referring to FIG. 1 , the description text generation method can be applied to a description text generation device. The descriptive text generating device may include any one of a smart phone, a computer, a tablet computer, a wearable device, a vehicle-mounted device, or a server, and the embodiment of the present invention does not limit the type of the descriptive text generating device.

When the descriptive text generation method is applied to terminal devices such as smart phones or computers, the terminal device can generate the descriptive text of the video or image selected by the user based on this method; when the descriptive text generation method is applied to the server, the server can Based on this method, a large number of videos or images in a material library (such as a retrieval database) are described in text, so as to improve the efficiency and accuracy of text retrieval of videos or images.

Please refer to FIG. 2 , which shows a flowchart of a method for generating description text provided by an embodiment of the present invention. This embodiment is described by taking the descriptive text generating method applied to the descriptive text generating device shown in FIG. 1 as an example. Referring to Figure 2, the method may include:

Step 101. Extract at least one visual feature vector from the target object.

In this embodiment of the present invention, the target object may be a video or an image. When extracting features, if the target object is an image, the description text generation device can directly extract a plurality of basic visual feature vectors from the image; if the target object is a video, the description text generation device can extract each A plurality of basic visual feature vectors are respectively extracted from the frame image.

Further, the descriptive text generation device may encode the multiple extracted basic visual feature vectors into one visual feature vector. Alternatively, the descriptive text generation device may also obtain at least one set of predetermined attention coefficients, and then use each set of attention coefficients to weight the multiple basic visual feature vectors to obtain the corresponding to the at least one set of attention coefficients At least one visual feature vector of .

Step 102, acquiring a semantic feature vector corresponding to each visual feature vector.

Further, the device for generating description text may acquire at least one set of sample data associated with each visual feature vector, and generate the semantic feature vector based on the at least one set of sample data. Wherein, each set of sample data may include: a sample image and annotation text corresponding to the sample image, the annotation text may be an attribute annotation text or a relationship annotation text, and the attribute annotation text may be used to indicate the attribute of the visual object in the sample image feature, the relationship annotation text can be used to indicate the relationship feature between the visual objects in the sample picture. In this embodiment of the present invention, the device for generating description text may use a k-nearest neighbor (k-Nearest Neighbor, KNN) algorithm to obtain at least one set of sample data associated with each visual feature vector.

Step 103: Process the at least one visual feature vector and the semantic feature vector corresponding to each visual feature vector to obtain the description text of the target object.

In this embodiment of the present invention, the descriptive text generating device may use a language model to generate the descriptive text of the target object. The language model may be a model based on a cyclic neural network (also called a decoder), and the cyclic neural network may be an LSTM network. Also, the language model may include at least one iteration unit, and each iteration unit may be used to generate a word. After the at least one visual feature vector and the semantic feature vector corresponding to each visual feature vector are input to the language model, each iteration unit can generate a word according to the input visual feature vector and semantic feature vector, and the at least one iteration unit The generated at least one word can form a description text for describing the target object.

It should be noted that a word in this embodiment of the present invention may refer to a text unit used to generate a description text, and the type of the word may be different for description texts in different languages. For example, if the language used in the description text is Chinese, the word may refer to a word or a single Chinese character; if the language used in the description text is an Indo-European language such as English, the word may refer to a single character composed of several letters.

To sum up, the embodiment of the present invention provides a description text generation method, which can extract at least one visual feature vector from the target object, and can obtain the semantic feature vector corresponding to each visual feature vector, and then can A description text of the target object is generated based on the at least one visual feature vector and the semantic feature vector of each visual feature vector. Since the semantic feature vector corresponding to each visual feature vector can reflect the semantic feature of the target object, the accuracy and flexibility of the description can be improved by assisting the generation of description text through the semantic feature vector.

Fig. 3 is an algorithm block diagram of a description text generation method provided by the embodiment of the present invention. Referring to Fig. 3, it can be seen that the description text generation method provided by the embodiment of the present invention mainly uses the feature extraction model 01, the memory model 02 and the language model 03. The language model 03 may include at least one iteration unit 031, and each iteration unit 031 is used to generate a word. Wherein, the feature extraction model 01 is used to extract at least one visual feature vector corresponding to the at least one iteration unit 031 from the target object, and the memory model 02 is used to obtain the semantic feature vector corresponding to each visual feature vector , the language model 03 is used to generate a description text of the target object based on the at least one visual feature vector and the semantic feature vector corresponding to each visual feature vector.

FIG. 4 is a flowchart of another method for describing text generation provided by an embodiment of the present invention. The method can be applied to the implementation environment shown in FIG. 1 and can be implemented based on the algorithm shown in FIG. 3 . Referring to Fig. 4, the description text generation method may specifically include:

Step 201, using a feature extraction model to extract at least one basic visual feature vector of a target object.

When the target object is an image, the at least one basic visual feature vector can be a feature vector extracted from different local positions in the image; when the target object is a video including multiple frames of images, the at least one basic visual feature vector can be A feature vector extracted from each frame of image is included, and the feature vector extracted from each frame of image may include visual feature vectors extracted from different local positions in the frame of image.

Since in visual tasks such as image classification or object detection, the high-level features of the convolutional neural network can effectively reflect the semantic information related to the object, so in the embodiment of the present invention, the model based on the convolutional neural network can be used as the feature extraction Model. For example, the feature extraction model can use a pre-trained residual network to extract at least one basic visual feature vector of the target object, that is, the target object can be input into the residual network, and then the last At least one feature vector output by the convolutional layer is used as at least one basic visual feature vector of the target object.

Exemplary, as shown in Figure 3, assuming that the target object is a video that includes several frames of images, then the M (M can be a positive integer, such as a positive integer greater than 1) extracted from the video basic visual features The vectors can be expressed as {x ₁ , x ₂ , . . . , x _M }, and the M basic visual feature vectors can include feature vectors extracted from different positions of each frame of the video.

Step 202, determine the at least one basic visual feature vector, each basic visual feature vector corresponds to the attention coefficient of each iteration unit.

Referring to Fig. 3, it can be seen that the language model in the video description algorithm may include at least one iterative unit, in order to improve the accuracy of the words generated by each iterative unit, the feature extraction model may first determine that each basic visual feature vector corresponds to each The attention coefficient of the iteration unit, and then based on the at least one basic visual feature vector and the corresponding attention coefficient, a visual feature vector corresponding to each iteration unit can be generated; correspondingly, when the language model is used to generate the description text, Each visual feature vector may be input to a corresponding iterative unit for the iterative unit to generate a corresponding word.

Among them, the attention coefficient of each basic visual feature vector corresponding to a certain iteration unit can be used to indicate the importance of the basic visual feature vector when the iteration unit generates words, and the size of the attention coefficient is positively correlated with the importance .

In the embodiment of the present invention, the language model may be a model based on a recurrent neural network, assuming that the number of iteration units included in the language model is T, and the number of basic visual feature vectors extracted from the target object is M( T and M are both positive integers), then in determining the at least one basic visual feature vector, when each basic visual feature vector corresponds to the attention coefficient of the tth iterative unit, the t-1th iterative unit can be obtained first The feature vector h _t-1 of the hidden layer in , where t is a positive integer not greater than T, for the first iteration unit, the feature extraction model can directly obtain the preset initial feature vector h ₀ , the initial feature vector h ₀ can be a zero vector. Further, based on the feature vector h _t-1 of the hidden layer, the attention coefficient of each basic visual feature vector corresponding to the t-th iteration unit can be determined. Among them, the m-th basic visual feature vector x _m corresponds to the attention coefficient of the t-th iteration unit It can be expressed as:

Wherein, f _att is a preset linear transformation function, for example, f _att can be a multi-layer perceptron, S is a preset normalization function, for example, S can be a Softmax function, and m is a positive integer not greater than M. The attention factor The larger the value, the higher the importance of the m-th basic visual feature vector x _m is when generating words in the t-th iteration unit.

Finally, the feature extraction model can determine T groups of attention coefficients corresponding to the T iteration units, wherein each group of attention coefficients can include M attention coefficients corresponding to the M basic visual feature vectors one-to-one.

Step 203 : Obtain at least one visual feature vector based on the at least one basic visual feature vector and the attention coefficient of each basic visual feature vector corresponding to each iteration unit.

In the embodiment of the present invention, for any iterative unit in at least one iterative unit in the language model, based on the attention coefficient of each basic visual feature vector corresponding to any iterative unit, the at least one basic visual feature vector Perform weighted summation to obtain the visual feature vector corresponding to any iterative unit. The visual feature vector V _t corresponding to the tth iteration unit in the at least one iteration unit may satisfy:

Correspondingly, the at least one visual feature vector extracted by the feature extraction model and corresponding to the at least one iteration unit may be {V ₁ , V ₂ , . . . , V _T }.

In the embodiment of the present invention, the feature extraction model based on convolutional neural network can adaptively extract from the target object that can reflect The visual feature vector of the most important visual features of the target object.

Step 204, acquiring at least one set of sample data associated with each visual feature vector.

In the embodiment of the present invention, a k-nearest neighbor (k-Nearest Neighbor, KNN) algorithm may be used to acquire at least one set of sample data associated with each visual feature vector from a preset sample database. Each set of sample data may include: a sample image and annotation text corresponding to the sample image, and the annotation text may be manually annotated attribute annotation text or relation annotation text. Among them, the attribute annotation text can be used to describe the semantic attributes of the visual objects (such as objects or object behaviors) in the picture, and the relationship annotation text can be used to describe the relationship between the visual objects in the picture.

Optionally, when acquiring at least one set of sample data associated with the first visual feature vector, the memory model may be used to respectively extract the reference feature vectors of each set of sample data in the sample database, for example, the memory model may be used The image processing model obtains the reference feature vectors of the sample pictures in each set of sample data, and then calculates the vector distances between the first visual feature vector and the reference feature vectors of each set of sample data to obtain multiple vector distances, and finally At least one set of sample data whose vector distance is not greater than a preset distance threshold may be acquired as the sample data associated with the first visual feature vector.

Wherein, the first visual feature vector may be any visual feature vector in the at least one visual feature vector. The preset distance threshold can be the average value of the multiple vector distances; or, the multiple vector distances can be sorted from small to large, and then among the sorted multiple vector distances, the Kth vector distance As the preset distance threshold, correspondingly, at least one group of sample data that is not greater than the preset distance threshold is K groups of sample data that have a shorter vector distance from the first visual feature vector in the sample database . The vector distance between two vectors may refer to the Euclidean distance between two vectors.

It should be noted that the sample database used in the embodiment of the present invention to obtain the sample data can be a manually labeled visual knowledge map dataset Visual Genome, which contains 100,000 pictures and 5 million picture region descriptions , 1 million visual questions and answers, 3 million objects, 2 million attribute annotations for the visual objects in the 100,000 pictures, and 2 million relationship annotations for the visual objects in the 100,000 pictures. The method provided by the embodiment of the present invention mainly uses the pictures marked with attributes and pictures marked with relations in the data set.

In addition, although the method provided by the embodiment of the present invention also needs to obtain pre-marked sample data, since the marked data in the sample data obtained by the method of the present application is the attribute annotation text or the relationship annotation text of the sample image, and the related art The tag data required by the method is the description text of the video. Therefore, compared with the method in the related art, the sample data required by the method of the present application is easier to obtain, and there are more databases available.

Step 205, using a memory model to process at least one set of sample data associated with each visual feature vector to obtain a semantic feature vector corresponding to each visual feature vector.

In the embodiment of the present invention, the memory model may be a pre-trained convolutional neural network-based model, and the memory model may include a picture processing model and a text processing model F _s . When using the memory model to process the at least one set of sample data associated with any visual feature vector, the image processing model _{Fv can} be used to extract the visual feature vectors of the sample pictures in each set of sample data respectively, and The text processing model F _s can be used to extract the semantic feature vectors of the labeled text in each group of sample data, and then based on the visual feature vector of the sample picture and the semantic feature vector of the labeled text, the corresponding visual feature vector can be obtained Semantic feature vectors.

Since the annotation text in at least one set of sample data can indicate the attribute features of visual objects in the sample picture or the relationship features between visual objects, and the at least one set of sample data is the sample data associated with the visual feature vector in the target object , so in the process of generating description text, at least one set of sample data associated with each visual feature vector can assist in the extraction of attribute features or relational features in the target object, which can effectively improve the accuracy of generated description text. That is, the memory module can adaptively select attribute features and relationship features related to the target object to assist in word generation.

Fig. 5 is a flow chart of a method for obtaining a semantic feature vector corresponding to a first visual feature vector according to an embodiment of the present invention. Referring to Fig. 5, a memory model is used to process at least one set of sample data associated with the first visual feature vector , the process of obtaining the semantic feature vector corresponding to the first visual feature vector may specifically include:

Step 2051: Use the image processing model in the memory model to process the sample pictures in each set of sample data to obtain the visual feature vectors of the sample pictures in each set of sample data.

The image processing model F _v may include a multi-layer convolutional network and a fully connected network. Assuming that the sample data associated with the first visual feature vector includes K groups, the image processing model F _v is used to sequentially analyze the K groups of sample data After the sample pictures in are processed, the visual feature vectors of a total of K sample pictures can be obtained. Assuming that the image processing model F _v is used to process the sample pictures in the i-th group of sample data, the visual feature vector of the sample pictures in the i-th group of sample data obtained is denoted as p _i , then the visual features of the K sample pictures The feature vectors can be expressed as a set {p _i } _i=1,...,K .

Step 2052: Use the text processing model in the memory model to process the labeled text in each set of sample data to obtain semantic feature vectors of the labeled text in each set of sample data.

The text processing model F _s can include a word vector model (such as a word2vector model) and a pooling layer. During processing, the word vector model can be used to obtain the semantic vector of each word in the labeled text in each group of sample data, At least one semantic vector is obtained, and then a pooling operation is performed on the at least one semantic vector by the pooling layer to obtain the semantic feature vector of the labeled text in each set of sample data.

After using the text processing model F _s to sequentially process the labeled texts in K groups of sample data, a total of K semantic feature vectors of labeled texts can be obtained. Assuming that the text processing model F _s is used to process the annotation text in the i-th group of sample data, the semantic feature vector of the annotation text in the i-th group of sample data obtained is expressed as q _i , then the semantics of the K annotation texts The feature vectors can be expressed as a set {q _i } _i=1,...,K .

In addition, in the embodiment of the present invention, as shown in FIG. 3 , the visual feature vectors {p _i } _i=1,...,K of the K sample pictures corresponding to each visual feature vector can also be called keys ( Key) vectors, correspondingly, the semantic feature vectors {q _i } _i=1, .

Step 2053, according to the visual feature vectors of the sample pictures in each group of sample data, determine the weight of the labeled text in each group of sample data.

Wherein, the weight of the labeled text in each set of sample data is positively correlated with the size of the visual feature vector of the sample picture. That is, the larger the visual feature vector of a certain sample picture is, the greater the weight of the labeled text corresponding to the sample picture will be. For the K groups of sample data associated with the first visual feature vector V _t , the weight c _i of the labeled text in the i-th group of sample data can satisfy:

in, Indicates the transposition of V _t , p _i is the visual feature vector of the sample picture in the i-th group of sample data, p _j is the visual feature vector of the sample picture in the j-th group of sample data, and both i and j are not greater than K positive integer.

Step 2054: Based on the weight of the labeled text in each set of sample data, perform weighted summation of the semantic feature vectors of the labeled text in at least one set of sample data to obtain the semantic feature vector corresponding to the first visual feature vector.

Based on the K weights determined in the above step 2053, the semantic feature vectors {q _i } _i=1,...,K of the K labeled texts are weighted and summed, and the first visual feature vector V _t obtained is The corresponding semantic feature vector can be expressed as:

Step 206, using each iteration unit in at least one iteration unit in the language model to process the corresponding visual feature vector and semantic feature vector to obtain at least one word.

The language model may be a model based on a recurrent neural network, for example, a model based on an LSTM network. Referring to FIG. 3, it can be seen that the language model may include at least one iteration unit, and each iteration unit may be used to generate a word. When generating the description text, each visual feature vector and the corresponding semantic feature vector can be input to the corresponding iteration unit, and the iteration unit can generate a word according to the input feature vector.

As shown in FIG. 3 , each iteration unit 031 may include a first linear processing unit L1 and a second linear processing unit L2 . Each visual feature vector can be input to the first linear processing unit L1 in the corresponding iteration unit, and the semantic feature vector corresponding to each visual feature vector can be input to the second linear processing unit L2 in the corresponding iteration unit.

Wherein, the first linear processing unit L1 in the first iteration unit 031 can generate an output vector based on the input visual feature vector V ₁ , the preset initial feature vector h ₀ and the preset initialization word (such as "start"), And the output vector can be input to the second linear processing unit L2 and the first linear processing unit L1 of the next iteration unit; the first linear processing unit L1 in each of the remaining iteration units 031 can be based on the input visual feature vector, above The output vector of the first linear processing unit L1 in an iterative unit (that is, the feature vector of the hidden layer of the last iterative unit), and the word generated by the last iterative unit generate an output vector, and the output vector can be input to the first The second linear processing unit L2 and the first linear processing unit L1 of the next iteration unit.

The second linear processing unit L2 in each iteration unit can generate a word based on the input semantic feature vector and the output vector of the first linear processing unit L1, and can input the word to the first linear processing unit L1 of the next iteration unit . Wherein, after each second linear processing unit L2 performs linear processing on the input vector, it needs to use a preset normalization function S to further normalize the linearly processed vector to generate words. The preset normalization function S may be a Softmax function. In addition, for the last iteration unit, the word generated by the iteration unit based on the input feature vector may be a preset terminator word (for example, "end"), and when the description text generation device detects the terminator word, it may be determined to use The words used to generate the description text have been generated.

Exemplarily, the first linear processing unit L2 in the second iteration unit may be based on the input visual feature vector V ₂ , the output vector of the first linear processing unit L1 in the first iteration unit and the output vector generated by the first iteration unit The word "one" generates an output vector and inputs the output vector to the second linear processing unit L2 and the first linear processing unit L1 in the third iteration unit; the second linear processing unit L2 in the second iteration unit The word "person" may be generated based _on the input semantic feature vector R2 and the output vector of the first linear processing unit L1, and the word may be input to the first processing unit L1 in the third iteration unit.

Step 207, use the text composed of at least one word as the description text of the target object.

Finally, the language model can compose the words generated by each iteration unit into text, which is used as the description text of the target object. For example, assuming that the target object is a video, and the words generated by the multiple iterative units included in the language model are "one", "person", "in" and "running", then the description text of the video generated by the language model Can be "A man is running".

It should be noted that each model used in the description text generation method provided by the embodiment of the present invention can be implemented by using a deep learning framework such as Pytorch, Caffe or Tensorflow.

To sum up, the embodiment of the present invention provides a description text generation method, which can obtain at least one set of sample data associated with each visual feature vector of the target object from the sample database, and can be based on at least A visual feature vector and a semantic feature vector obtained from the at least one set of sample data generate a description text of the target object. Since each at least one set of sample data is associated with a visual feature vector in the target object and is pre-labeled data, the at least one set of sample data can assist in extracting the features in the target object, effectively improving the video description. accuracy and flexibility. Moreover, since in the method provided by the embodiment of the present invention, the annotation text in each set of sample data associated with the visual feature vector can be attribute annotation text or relation annotation text, so this method can effectively utilize the attribute characteristics of the visual objects in the sample picture and relational features to guide the description text generation of the target object. This method takes into account the semantic features, attribute features and relational features of the visual objects in the target object, so it can effectively improve the accuracy of description.

As mentioned above, the memory model used in the descriptive text generation method provided by the embodiment of the present invention is a pre-trained model. FIG. 6 is a flow chart of a method for training the memory model provided by the embodiment of the present invention. Refer to FIG. 6. The training method may include:

Step 301. Acquire at least one set of training data.

In the embodiment of the present invention, the at least one set of training data may be obtained from a preset sample database, and the sample database used to obtain training data and the sample database used to obtain sample data may be the same database, or They are different databases, which is not limited in this embodiment of the present invention. Each set of acquired training data may also include: a training picture and a training annotation text corresponding to the training picture, and each set of training data may also be called a training sample pair. The training annotation text may include attribute annotation text or relation annotation text.

It should be noted that, in order to ensure the effect of the memory model obtained through training, when selecting the at least one set of training data, it should be ensured that the training annotation text in the at least one set of training data includes both attribute annotation text and relation annotation text . For example, assuming that it is necessary to obtain N sets of training data (the N is a positive integer), the training label text of each set of training data in the selected N/2 sets of training data can be the attribute label text, and the remaining N/2 sets of training data The training annotation text of each set of training data in is the relation annotation text.

Step 302, using the image processing model to process the training pictures in each set of training data to obtain visual feature vectors of the training pictures in each set of training data.

Fig. 7 is an algorithm framework diagram of a training memory model provided by an embodiment of the present invention. Referring to Fig. 7, the image processing model may be a model based on a convolutional neural network, for example, the image processing model may include a multi-layer convolutional network and A fully connected network. If N groups of training data have been obtained in the above step 301, after the image processing model F _v is used to process the training image v _n in the nth group of training data, the obtained feature vector of the training image v _n can be expressed as F _v (v _n ), where n is a positive integer not greater than N.

Step 303 , using a text processing model to process the training marked text in each set of training data to obtain a semantic feature vector of the training marked text in each set of training data.

The text processing model may include a pre-trained word vector model (such as a word2vector model), and a pooling layer. When the text processing model is used to process the training annotation text, the semantic vector of each word in the training annotation text can be obtained through the word vector model first, and at least one semantic vector can be obtained, and then the at least one semantic vector can be obtained through the pooling layer. A semantic vector is pooled to obtain the semantic feature vector of the training labeled text in each set of training data. Wherein, after using the text processing model F _s to process the training labeled text s _n in the nth set of training data, the obtained feature vector of the training labeled text s _n can be expressed as F _s (s _n ).

For example, as shown in Figure 7, assuming that the training annotation text of a certain training picture includes six words such as "hut", "window" and "tree", when the text processing model is used to process the training annotation text, it can be First obtain the semantic vector of each word through the word vector model to obtain six semantic vectors; then the six semantic vectors can be pooled through the pooling layer to obtain the semantic feature vector of the training annotation text.

Step 304: Construct a loss function based on the visual feature vector of the training picture and the semantic feature vector of the training labeled text.

When constructing the loss function, the distance between the visual feature vector of the training picture and the semantic feature vector of the training labeled text in each set of training data can be calculated separately to obtain N distances. The distance dn between the feature vector F _v (v _n ) of the training picture v _n in the nth set of training data and the feature vector F _s (s _n ) of the training label text s _n can be expressed as: d _n =|| F _v (v _n )-F _s (s _n )||.

Further, the loss function can be constructed according to the N distances, and the loss function L can satisfy:

In the above formula (1), τ is the preset hyperparameter, max(τ-d _i , 0) means to take the larger value between (τ-d _i ) and 0, w is the parameter in the memory model, Ω (w) represents the two-norm of w (also known as L2 regularization), λ is the weight decay factor (weightdecay), l _n represents the paired marker corresponding to the nth group of sample data, and the value of l _n is 0 or 1. Among them, the value of l _n is 1, which means that the training picture in the nth group of sample data is related to the corresponding training label text, that is, the training label text is marked for the training picture; the value of l _n is 0 means that the training picture in the nth group of sample data is not related to the corresponding training label text, that is, the training picture and the training label text are randomly generated training sample pairs.

It should be noted that, in the embodiment of the present invention, the loss function may also be called a contrastive loss function, or a contrastive constraint function, etc., which is not limited in the embodiment of the present invention.

Step 305, using the loss function to train the memory model to obtain the image processing model and the text processing model.

Using the loss function shown in the above formula (1) to train the memory model, the parameter w in the memory model can be regularized, so as to obtain the image processing model and the text processing model. The training process can be expressed as: That is to solve the value of the independent variable w when the value of the loss function is minimized. The trained memory model can effectively fit the at least one set of training data, and can learn the vector transformation when mapping the picture and the text to the semantic feature space shared by both. Wherein, during the training process, the parameter w in the memory model can be updated by means of backward transfer until the loss function L converges.

To sum up, the embodiment of the present invention provides a memory model training method, the memory model trained by this method can adaptively select the attribute features and relationship features related to the target object during the generation process of the description text. Auxiliary word generation.

It should be noted that the order of the steps of the description text generation method and the memory model training method provided by the embodiment of the present invention can be adjusted appropriately, and the steps can also be increased or decreased according to the situation. For example, step 2052 may be executed simultaneously with step 2051 , and step 303 may be executed concurrently with step 302 . Any person skilled in the art within the technical scope disclosed in the present invention can easily think of changing methods, which should be covered within the scope of protection of the present invention, and thus will not be repeated here.

Fig. 8 is a schematic structural diagram of a descriptive text generation device provided by an embodiment of the present invention. The device can be configured in the descriptive text generation device shown in Fig. 1. Referring to Fig. 8, the device may include:

The extraction module 401 is configured to extract at least one visual feature vector from a target object, where the target object is a video or a picture.

The obtaining module 402 is configured to obtain a semantic feature vector corresponding to each visual feature vector.

The processing module 403 is configured to process the at least one visual feature vector and each semantic feature vector corresponding to the visual feature vector to obtain a description text of the target object.

Optionally, referring to FIG. 9, the acquiring module 402 may include:

The acquisition sub-module 4021 is configured to acquire at least one set of sample data associated with each visual feature vector, each set of sample data includes: a sample picture and an annotation text corresponding to the sample image, the annotation text includes attribute annotation text or Relationship label text.

The processing sub-module 4022 is configured to use a memory model to process at least one set of sample data associated with each visual feature vector to obtain a semantic feature vector corresponding to each visual feature vector.

Optionally, the memory model may include a picture processing model and a text processing model. Referring to FIG. 10, the device may also include:

The data acquisition module 404 is configured to acquire at least one set of training data, each set of training data includes: a training picture and a training labeled text corresponding to the training picture.

The image processing module 405 is configured to use the image processing model to process the training images in each set of training data to obtain visual feature vectors of the training images in each set of training data.

The text processing module 406 is configured to use the text processing model to process the training marked text in each set of training data to obtain the semantic feature vector of the training marked text in each set of training data.

A construction module 407, configured to construct a loss function based on the visual feature vector of the training picture and the semantic feature vector of the training labeled text.

The training module 408 is configured to use the loss function to train the memory model to obtain the image processing model and the text processing model.

Optionally, the number of groups of training data obtained by the data acquisition module 404 is N, where N is a positive integer, and the construction module 407 can be used for:

In each set of training data, the distance between the visual feature vector of the training picture and the semantic feature vector of the training annotation text is calculated separately, and N distances are obtained, wherein the visual feature vector F of the training picture v _n in the nth set of training data The distance d _n between _v (v _n ) and the semantic feature vector F _s (s _n ) of the training annotation text s _n satisfies: d _n =||F _v (v _n )-F _s (s _n )||, n is a positive integer not greater than N;

According to the N distances, a loss function is constructed, and the loss function L satisfies:

Among them, l _n represents the paired label corresponding to the nth group of sample data, and the value of l _n is 0 or 1, τ is a preset hyperparameter, max(τ-d _i , 0) means to take (τ-d _i ) and 0, w is a parameter in the memory model, Ω(w) represents the two-norm of w, and λ is a weight attenuation factor.

Optionally, the memory model includes an image processing model and a text processing model, and the processing submodule 4022 uses the memory model to process at least one set of sample data associated with the first visual feature vector to obtain the semantics corresponding to the first visual feature vector The process of eigenvectors can include:

The image processing model is used to process the sample images in each set of sample data to obtain visual feature vectors of the sample images in each set of sample data.

The text processing model is used to process the labeled text in each group of sample data, and the semantic feature vector of the labeled text in each group of sample data is obtained.

According to the visual feature vector of the sample picture in each set of sample data, determine the weight of the labeled text in each set of sample data, wherein the weight of the labeled text in each set of sample data is positively correlated with the size of the visual feature vector of the sample picture .

Based on the weights of the labeled text in each set of sample data, weighted summation is performed on the semantic feature vectors of the labeled text in the at least one set of sample data to obtain a semantic feature vector corresponding to the first visual feature vector.

Wherein, the processing sub-module 4022 determines the weight of the labeled text in each set of sample data according to the visual feature vector of the sample picture in each set of sample data may include:

According to the first visual feature vector V _t and the visual feature vector of the sample picture in each set of sample data, determine the weight of the marked text in each set of sample data, and the weight c _i of the marked text in the i-th set of sample data satisfies :

Among them, K is the group number of sample data associated with each visual feature vector, Indicates the transposition of V _t , p _i is the visual feature vector of the sample picture in the i-th group of sample data, p _j is the visual feature vector of the sample picture in the j-th group of sample data, and both i and j are not greater than K positive integer.

Optionally, the acquiring submodule 4021 can be used for:

Using the memory model to extract the reference feature vectors of each group of sample data in the sample database;

Calculate the vector distance between the first visual feature vector and the reference feature vector of each group of sample data;

At least one set of sample data whose vector distance is not greater than a preset distance threshold is acquired as sample data associated with the first visual feature vector.

Optionally, the memory model includes a picture processing model and a text processing model; the process of the acquisition sub-module 4021 using the memory model to respectively extract the reference feature vectors of each group of sample data in the sample database may include:

The image processing model is used to extract the reference feature vectors of the sample pictures in each group of sample data in the sample database.

Optionally, the description text of the target object is generated by a language model, and the language model may include at least one iteration unit, each of which is used to generate a word;

The extraction module 401 can be used for:

At least one visual feature vector corresponding to the at least one iteration unit is extracted from the target object.

Correspondingly, the processing module 403 can be used for:

Using each iteration unit in the at least one iteration unit in turn to process the corresponding visual feature vector and semantic feature vector to obtain at least one word;

The text composed of at least one word is used as the description text of the target object.

Optionally, referring to FIG. 11, the extracting module 401 may include:

The extraction sub-module 4011 is configured to extract at least one basic visual feature vector of the target object, the target object includes multiple frame images, and the at least one basic visual feature vector includes a visual feature vector extracted from each frame of image.

The determination sub-module 4012 is configured to determine the attention coefficient of each basic visual feature vector corresponding to each iteration unit among the at least one basic visual feature vector.

Weighting sub-module 4013, for any iterative unit, based on the attention coefficient of each basic visual feature vector corresponding to the any iterative unit, carry out weighted summation on the at least one basic visual feature vector, and obtain the The visual feature vector corresponding to the iteration unit.

Optionally, the language model is a model based on a recurrent neural network, and the determining submodule 4012 can be used for:

Obtain the feature vector h _t-1 of the hidden layer in the t-1th iteration unit, where t is a positive integer not greater than T, and T is the number of iteration units included in the language model;

Based on the feature vector h _t-1 of the hidden layer, determine the attention coefficient of each basic visual feature vector corresponding to the t-th iteration unit, where the m-th basic visual feature vector x _m corresponds to the t-th iteration unit Attention coefficient of iterative unit Satisfy:

Wherein, f _att is a preset linear transformation function, S is a preset normalization function, m is a positive integer not greater than M, and M is the number of basic feature vectors extracted from the target object.

Optionally, the language model is a model based on a recurrent neural network;

The processing module 403 uses the first iteration unit in the at least one iteration unit to process the corresponding visual feature vector and semantic feature vector, and the process of obtaining a word may include:

The first iterative unit is used to process the corresponding visual feature vector, semantic feature vector, preset initial feature vector and preset initialization word to obtain a word.

The processing module 403 uses any iteration unit other than the first iteration unit to process the corresponding visual feature vector and semantic feature vector, and the process of obtaining a word may include:

Any iteration unit is used to process the corresponding visual feature vector, semantic feature vector, feature vector of the hidden layer of the previous iteration unit and the word generated by the previous iteration unit to obtain a word.

To sum up, the embodiment of the present invention provides a device for generating description text, which can extract at least one visual feature vector from the target object, and can obtain the semantic feature vector corresponding to each visual feature vector, and then based on the At least one visual feature vector and a semantic feature vector of each visual feature vector generate a description text of the target object. Since the semantic feature vector corresponding to each visual feature vector can reflect the semantic feature of the target object, the accuracy and flexibility of the description can be improved by assisting the generation of description text through the semantic feature vector.

Regarding the apparatus in the foregoing embodiments, the specific manner in which each module executes operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

Fig. 12 shows a structural block diagram of a terminal 1200 provided by an exemplary embodiment of the present invention. The terminal 1200 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, moving picture experts compress standard audio layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, moving picture experts compress standard audio layer 4) player, laptop or desktop computer. The terminal 1200 may also be called user equipment, portable terminal, laptop terminal, desktop terminal and other names.

Generally, the terminal 1200 includes: a processor 1201 and a memory 1202 .

The processor 1201 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1201 can adopt at least one hardware form among DSP (Digital Signal Processing, digital signal processing), FPGA (Field-Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array, programmable logic array) accomplish. The processor 1201 may also include a main processor and a coprocessor, the main processor is a processor for processing data in the wake-up state, and is also called a CPU (Central Processing Unit, central processing unit); the coprocessor is used to Low-power processor for processing data in standby state. In some embodiments, the processor 1201 may be integrated with a GPU (Graphics Processing Unit, image processor), and the GPU is used for rendering and drawing content that needs to be displayed on the display screen. In some embodiments, the processor 1201 may further include an AI (Artificial Intelligence, artificial intelligence) processor, where the AI processor is configured to process computing operations related to machine learning.

Memory 1202 may include one or more computer-readable storage media, which may be non-transitory. The memory 1202 may also include high-speed random access memory and non-volatile memory, such as one or more magnetic disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1202 is used to store at least one instruction, and the at least one instruction is used to be executed by the processor 1201 to implement the description text provided by the method embodiments in this application generate method.

In some embodiments, the terminal 1200 may optionally further include: a peripheral device interface 1203 and at least one peripheral device. The processor 1201, the memory 1202, and the peripheral device interface 1203 may be connected through buses or signal lines. Each peripheral device can be connected to the peripheral device interface 1203 through a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1204 , a touch screen 1205 , a camera 1206 , an audio circuit 1207 , a positioning component 1208 and a power supply 1209 .

The peripheral device interface 1203 may be used to connect at least one peripheral device related to I/O (Input/Output, input/output) to the processor 1201 and the memory 1202 . In some embodiments, the processor 1201, memory 1202 and peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one of the processor 1201, memory 1202 and peripheral device interface 1203 or The two can be implemented on a separate chip or circuit board, which is not limited in this embodiment.

The radio frequency circuit 1204 is used for receiving and transmitting RF (Radio Frequency, radio frequency) signals, also called electromagnetic signals. The radio frequency circuit 1204 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1204 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1204 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 1204 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and/or a WiFi (Wireless Fidelity, wireless fidelity) network. In some embodiments, the radio frequency circuit 1204 may also include circuits related to NFC (NearField Communication, short-range wireless communication), which is not limited in this application.

The display screen 1205 is used to display a UI (User Interface, user interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1205 is a touch display screen, the display screen 1205 also has the ability to collect touch signals on or above the surface of the display screen 1205 . The touch signal can be input to the processor 1201 as a control signal for processing. At this time, the display screen 1205 can also be used to provide virtual buttons and/or virtual keyboards, also called soft buttons and/or soft keyboards. In some embodiments, there may be one display screen 1205, which is provided on the front panel of the terminal 1200; in other embodiments, there may be at least two display screens 1205, which are respectively provided on different surfaces of the terminal 1200 or in a folding design; In some other embodiments, the display screen 1205 may be a flexible display screen, which is arranged on the curved surface or the folded surface of the terminal 1200 . Even, the display screen 1205 can also be set as a non-rectangular irregular figure, that is, a special-shaped screen. The display screen 1205 may be made of LCD (Liquid Crystal Display, liquid crystal display), OLED (Organic Light-Emitting Diode, organic light-emitting diode) and other materials.

The camera assembly 1206 is used to capture images or videos. Optionally, the camera component 1206 includes a front camera and a rear camera. Usually, the front camera is set on the front panel of the terminal, and the rear camera is set on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of the main camera, depth-of-field camera, wide-angle camera, and telephoto camera, so as to realize the fusion of the main camera and the depth-of-field camera to realize the background blur function. Combined with the wide-angle camera to realize panoramic shooting and VR (Virtual Reality, virtual reality) shooting functions or other fusion shooting functions. In some embodiments, camera assembly 1206 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. Dual color temperature flash refers to the combination of warm light flash and cold light flash, which can be used for light compensation under different color temperatures.

Audio circuitry 1207 may include a microphone and speakers. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into electrical signals and input them to the processor 1201 for processing, or input them to the radio frequency circuit 1204 to realize voice communication. For the purpose of stereo sound collection or noise reduction, there may be multiple microphones, which are respectively set at different parts of the terminal 1200 . The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signal from the processor 1201 or the radio frequency circuit 1204 into sound waves. The loudspeaker can be a conventional membrane loudspeaker or a piezoelectric ceramic loudspeaker. When the speaker is a piezoelectric ceramic speaker, it is possible not only to convert electrical signals into sound waves audible to humans, but also to convert electrical signals into sound waves inaudible to humans for purposes such as distance measurement. In some embodiments, audio circuitry 1207 may also include a headphone jack.

The positioning component 1208 is used to locate the current geographic location of the terminal 1200 to implement navigation or LBS (Location Based Service, location-based service). The positioning component 1208 may be a positioning component based on the GPS (Global Positioning System, Global Positioning System) of the United States, the Beidou system of China, the Greinus system of Russia, or the Galileo system of the European Union.

The power supply 1209 is used to supply power to various components in the terminal 1200 . The power source 1209 can be alternating current, direct current, disposable batteries, or rechargeable batteries. When the power source 1209 includes a rechargeable battery, the rechargeable battery may support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

In some embodiments, the terminal 1200 further includes one or more sensors 1210 . The one or more sensors 1210 include, but are not limited to: an acceleration sensor 1211 , a gyroscope sensor 1212 , a pressure sensor 1213 , a fingerprint sensor 1214 , an optical sensor 1215 and a proximity sensor 1216 .

The acceleration sensor 1211 can detect the acceleration on the three coordinate axes of the coordinate system established by the terminal 1200 . For example, the acceleration sensor 1211 can be used to detect the components of the acceleration of gravity on the three coordinate axes. The processor 1201 may control the touch display screen 1205 to display a user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 1211 . The acceleration sensor 1211 can also be used for collecting game or user's motion data.

The gyro sensor 1212 can detect the body direction and rotation angle of the terminal 1200 , and the gyro sensor 1212 can cooperate with the acceleration sensor 1211 to collect 3D actions of the user on the terminal 1200 . According to the data collected by the gyroscope sensor 1212, the processor 1201 can realize the following functions: motion sensing (such as changing the UI according to the tilt operation of the user), image stabilization during shooting, game control and inertial navigation.

The pressure sensor 1213 may be disposed on a side frame of the terminal 1200 and/or a lower layer of the touch display screen 1205 . When the pressure sensor 1213 is installed on the side frame of the terminal 1200 , it can detect the user's grip signal on the terminal 1200 , and the processor 1201 performs left and right hand recognition or shortcut operation according to the grip signal collected by the pressure sensor 1213 . When the pressure sensor 1213 is arranged on the lower layer of the touch screen 1205, the processor 1201 controls the operable controls on the UI interface according to the user's pressure operation on the touch screen 1205. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

The fingerprint sensor 1214 is used to collect the user's fingerprint, and the processor 1201 recognizes the identity of the user according to the fingerprint collected by the fingerprint sensor 1214, or, the fingerprint sensor 1214 recognizes the user's identity according to the collected fingerprint. When the identity of the user is identified as a trusted identity, the processor 1201 authorizes the user to perform related sensitive operations, such sensitive operations include unlocking the screen, viewing encrypted information, downloading software, making payment, and changing settings. The fingerprint sensor 1214 may be provided on the front, rear or side of the terminal 1200 . When the terminal 1200 is provided with a physical button or a manufacturer's Logo, the fingerprint sensor 1214 may be integrated with the physical button or the manufacturer's Logo.

The optical sensor 1215 is used to collect ambient light intensity. In one embodiment, the processor 1201 can control the display brightness of the touch screen 1205 according to the ambient light intensity collected by the optical sensor 1215 . Specifically, when the ambient light intensity is high, the display brightness of the touch screen 1205 is increased; when the ambient light intensity is low, the display brightness of the touch screen 1205 is decreased. In another embodiment, the processor 1201 may also dynamically adjust shooting parameters of the camera assembly 1206 according to the ambient light intensity collected by the optical sensor 1215 .

The proximity sensor 1216 , also called a distance sensor, is usually arranged on the front panel of the terminal 1200 . The proximity sensor 1216 is used to collect the distance between the user and the front of the terminal 1200 . In one embodiment, when the proximity sensor 1216 detects that the distance between the user and the front of the terminal 1200 gradually decreases, the processor 1201 controls the touch display screen 1205 to switch from the bright screen state to the off-screen state; when the proximity sensor 1216 detects When the distance between the user and the front of the terminal 1200 gradually increases, the processor 1201 controls the touch display screen 1205 to switch from the off-screen state to the on-screen state.

Those skilled in the art can understand that the structure shown in FIG. 12 does not constitute a limitation on the terminal 1200, and may include more or less components than shown in the figure, or combine certain components, or adopt a different component arrangement.

The embodiment of the present invention also provides a computer-readable storage medium, at least one instruction, at least one program, code set or instruction set is stored in the storage medium, the at least one instruction, the at least one program, the code set or instruction set The set is loaded and executed by the processor to implement the description text generation method provided by the above-mentioned embodiment.

Those of ordinary skill in the art can understand that all or part of the steps for implementing the above embodiments can be completed by hardware, and can also be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium. The above-mentioned The storage medium mentioned may be a read-only memory, a magnetic disk or an optical disk, and the like.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within range.

Claims (15)

1. A description text generation method is characterized in that, the method comprises: Extracting at least one visual feature vector from a target object, where the target object is a video or a picture; obtaining a semantic feature vector corresponding to each of the visual feature vectors; The at least one visual feature vector and the semantic feature vector corresponding to each visual feature vector are processed to obtain the description text of the target object. 2. The method according to claim 1, wherein said obtaining the semantic feature vector corresponding to each said visual feature vector comprises: Obtaining at least one set of sample data associated with each of the visual feature vectors, each set of sample data includes: a sample picture and annotation text corresponding to the sample image, the annotation text includes attribute annotation text or relationship annotation text ; At least one set of sample data associated with each of the visual feature vectors is processed by using a memory model to obtain a semantic feature vector corresponding to each of the visual feature vectors. 3. The method according to claim 2, wherein the memory model comprises a picture processing model and a text processing model, and the method further comprises: Obtain at least one set of training data, each set of training data includes: a training picture and a training label text corresponding to the training picture, and the training label text includes attribute label text or relationship label text; Using the image processing model to process the training pictures in each group of training data to obtain the visual feature vectors of the training pictures in each group of training data; Using the text processing model to process the training marked text in each group of training data to obtain the semantic feature vector of the training marked text in each group of training data; Constructing a loss function based on the visual feature vector of the training picture and the semantic feature vector of the training labeled text; The memory model is trained by using the loss function to obtain the image processing model and the text processing model. 4. The method according to claim 3, wherein the number of groups of the training data obtained is N, the visual feature vector based on the training picture and the semantic feature vector of the training label text, Construct a loss function, including: In each set of training data, the distance between the visual feature vector of the training picture and the semantic feature vector of the training annotation text is calculated separately, and N distances are obtained, wherein the visual feature vector F of the training picture v _n in the nth set of training data The distance d _n between _v (v _n ) and the semantic feature vector F _s (s _n ) of the training annotation text s _n satisfies: d _n =||F _v (v _n )-F _s (s _n )||, n is a positive integer not greater than N; According to the N distances, a loss function is constructed, and the loss function L satisfies: Among them, l _n represents the paired label corresponding to the nth group of sample data, and the value of l _n is 0 or 1, τ is a preset hyperparameter, max(τ-d _i , 0) means to take (τ-d _i ) and 0, w is a parameter in the memory model, Ω(w) represents the two-norm of w, and λ is a weight attenuation factor. 5. The method according to any one of claims 2 to 4, wherein the memory model includes a picture processing model and a text processing model, and at least a group of samples associated with the first visual feature vector are used in the memory model The data is processed to obtain the semantic feature vector corresponding to the first visual feature vector, including: Processing the sample pictures in each group of sample data by using the picture processing model to obtain the visual feature vectors of the sample pictures in each group of sample data; Processing the labeled text in each set of sample data by using the text processing model to obtain a semantic feature vector of the labeled text in each set of sample data; According to the visual feature vector of the sample picture in each set of sample data, determine the weight of the labeled text in each set of sample data, wherein the weight of the labeled text in each set of sample data is positively correlated with the size of the visual feature vector of the sample picture ; Based on the weights of the labeled text in each set of sample data, weighted summation is performed on the semantic feature vectors of the labeled text in the at least one set of sample data to obtain a semantic feature vector corresponding to the first visual feature vector. 6. The method according to claim 5, wherein said determining the weight of the label text in each group of sample data according to the visual feature vector of the sample picture in each group of sample data includes: According to the first visual feature vector V _t and the visual feature vector of the sample picture in each set of sample data, determine the weight of the labeled text in each set of sample data, and the weight c _i of the labeled text in the i-th set of sample data Satisfy: Wherein, K is the group number of sample data associated with each described visual feature vector, V _t ^T represents the transposition of V _t , p _i is the visual feature vector of the sample picture in the i-th group of sample data, and p _j is The visual feature vectors of the sample pictures in the jth group of sample data, i and j are both positive integers not greater than K. 7. The method according to any one of claims 2 to 4, wherein obtaining at least one set of sample data associated with the first visual feature vector comprises: Using the memory model to extract the reference feature vectors of each group of sample data in the sample database; respectively calculating the vector distance between the first visual feature vector and the reference feature vector of each set of sample data; At least one set of sample data whose vector distance is not greater than a preset distance threshold is acquired as the sample data associated with the first visual feature vector. 8. The method according to claim 7, wherein the memory model comprises a picture processing model and a text processing model; The use of the memory model to respectively extract the reference feature vectors of each group of sample data in the sample database includes: The reference feature vectors of the sample pictures in each group of sample data in the sample database are respectively extracted by using the picture processing model. 9. The method according to any one of claims 1 to 4, wherein the description text of the target object is generated by a language model, and the language model includes at least one iterative unit, each of which is used to generate a word; The extracting at least one visual feature vector from the target object includes: Extracting at least one visual feature vector corresponding to the at least one iteration unit one-to-one from the target object; The processing of the at least one visual feature vector and the semantic feature vector corresponding to each of the visual feature vectors to obtain the description text of the target object includes: Using each iteration unit in the at least one iteration unit in turn to process the corresponding visual feature vector and semantic feature vector to obtain at least one word; The text composed of the at least one word is used as the description text of the target object. 10. The method according to claim 9, wherein said extracting at least one visual feature vector from the target object comprises: Extracting at least one basic visual feature vector of the target object, where the target object includes multiple frame images, and the at least one basic visual feature vector includes a visual feature vector extracted from each frame of image; Determining the at least one basic visual feature vector, each basic visual feature vector corresponds to the attention coefficient of each iteration unit; For any iterative unit, based on each basic visual feature vector corresponding to the attention coefficient of any iterative unit, the at least one basic visual feature vector is weighted and summed to obtain the corresponding to any iterative unit Visual feature vector. 11. The method according to claim 10, wherein the language model is a model based on a recurrent neural network, and in determining the at least one basic visual feature vector, each basic visual feature vector corresponds to the tth iteration The attention coefficient of the unit, including: Obtain the feature vector h _t-1 of the hidden layer in the t-1th iteration unit, where t is a positive integer not greater than T, and T is the number of iteration units included in the language model; Based on the feature vector h _t-1 of the hidden layer, determine the attention coefficient of each basic visual feature vector corresponding to the t-th iteration unit, wherein the m-th basic visual feature vector x _m corresponds to the Attention coefficient of the tth iteration unit Satisfy: Wherein, f _att is a preset linear transformation function, S is a preset normalization function, m is a positive integer not greater than M, and M is the number of basic feature vectors extracted from the target object. 12. The method according to claim 9, wherein the language model is a model based on a recurrent neural network; Using the first iteration unit in the at least one iteration unit to process the corresponding visual feature vector and semantic feature vector to obtain a word, including: Using the first iteration unit to process the corresponding visual feature vector, semantic feature vector, preset initial feature vector and preset initialization word to obtain a word; Use any iteration unit except the first iteration unit to process the corresponding visual feature vector and semantic feature vector to obtain a word, including: Using any of the iteration units to process the corresponding visual feature vector, semantic feature vector, feature vector of the hidden layer of the previous iteration unit, and the word generated by the previous iteration unit to obtain a word. 13. A device for generating description text, characterized in that the device comprises: An extraction module, configured to extract at least one visual feature vector from a target object, where the target object is a video or a picture; An acquisition module, configured to acquire a semantic feature vector corresponding to each of the visual feature vectors; The processing module is configured to process the at least one visual feature vector and the semantic feature vector corresponding to each visual feature vector to obtain the description text of the target object. 14. A terminal, characterized in that the terminal includes a processor and a memory, at least one instruction, at least one program, code set or instruction set are stored in the memory, and the at least one instruction, the at least one program , the code set or instruction set is loaded and executed by the processor to implement the description text generation method according to any one of claims 1 to 12. 15. A computer-readable storage medium, characterized in that at least one instruction, at least one section of program, code set or instruction set is stored in said storage medium, said at least one instruction, said at least one section of program, said code The set or instruction set is loaded and executed by the processor to implement the method for generating description text as claimed in any one of claims 1 to 12.