CN117390213A - Training method of image and text retrieval model based on OSCAR and method of implementing image and text retrieval - Google Patents

Abstract

The present invention provides a training method for an OSCAR-based image and text retrieval model and a method for realizing image and text retrieval. The training method includes: obtaining a training set; inputting multiple image-text sample pairs in the training set into pre-training for visual language tasks. In the model OSCAR, feature extraction is performed to obtain image feature representation and text feature representation; each sample in the training set is used as an anchor sample, and multiple negative samples of different difficulties corresponding to the anchor sample are generated based on the image feature representation and text feature representation. ; Calculate the positive similarity between the image and text in the positive sample pair, the negative similarity between the image and text in the negative sample pair and the generated negative sample pair; calculate the loss function based on the positive similarity and negative similarity, and use the loss function to The pre-trained model OSCAR is fine-tuned to obtain the fully trained OSCAR image and text retrieval model. In this case, it can improve the generalization ability of the model and improve the accuracy and efficiency of model image and text retrieval.

Description

Training method of image and text retrieval model based on OSCAR and implementation of image and text retrieval method

Technical field

The invention relates to the technical field of information retrieval, and in particular to a training method of an OSCAR-based image and text retrieval model and a method for realizing image and text retrieval.

Background technique

The purpose of image and text retrieval is to associate a given picture with the corresponding text description to achieve matching between images and text. Image and text retrieval plays a key role in many important cross-modal tasks, such as semantic image retrieval, image description, visual quality assurance, etc. However, image-text matching faces some important challenges, mainly including heterogeneous differences and semantic differences. Heterogeneous differences refer to the inconsistent feature representation of image and text data from different modalities, while semantic differences refer to the inconsistency in capturing Misalignment problems in cross-modal correspondence between images and text.

Currently, many studies extract image and text features by using pre-trained modules such as convolutional neural networks and recurrent neural networks to bridge heterogeneous differences. However, the feature extractors in these pre-training modules have not been specially trained or networked on image-text pair data, so they cannot achieve better image or text embedding effects. Another common image-text matching method is to use a triplet loss to encourage the model such that the similarity score of positive image-text pairs is higher than the similarity score of negative image-text pairs. However, the existing cost function does not fully consider the difficulty of negative samples, which is one of the main reasons for inaccurate image-text matching in the model. Some studies have shown that increasing the batch size to obtain more negative samples leads to a sharp increase in computational complexity with gradually decreasing returns in performance gains.

Currently, the performance of the OSCAR model is very powerful in visual language tasks. It has been pre-trained on millions of image-text pairs, and jointly processes images and text to obtain meaningful feature representations that can capture Intricate associations between text and images and learn more discriminative image-text embeddings. The OSCAR model has good learning and understanding capabilities for feature representations of images and articles, but the model's generalization ability is still weak.

Therefore, this application builds a new image and text retrieval model based on the OSCAR model to improve the generalization ability of the model and improve the accuracy and efficiency of the model's image and text retrieval.

Contents of the invention

The purpose of the present invention is to provide a training method for an OSCAR-based image and text retrieval model and a method for implementing image and text retrieval, which can improve the generalization ability of the model and improve the accuracy and efficiency of the model's image and text retrieval.

In order to achieve the above objects, the present invention provides the following technical solutions:

In a first aspect, the present invention provides a training method for an OSCAR-based image and text retrieval model, which method includes:

Obtain a training set, where the training set includes a plurality of image-text sample pairs;

Input multiple image-text sample pairs in the training set into the pre-trained model OSCAR for visual language tasks, and perform feature extraction to generate image feature representations and text feature representations;

Each sample in the training set is used as an anchor sample, and based on the image feature representation and the text feature representation, multiple negative samples of different difficulties corresponding to the anchor sample are generated; the generated negative samples are consistent with the Anchor point samples form the generated negative sample pairs;

Calculate the positive similarity between the image and the text in the positive sample pair, the negative similarity between the negative sample pair and the generated negative sample pair between the image and the text;

A loss function is calculated based on the positive similarity and negative similarity, and the pre-trained model OSCAR is fine-tuned through the loss function to obtain a trained OSCAR image and text retrieval model.

Further, each sample in the training set is used as an anchor sample, and based on the image feature representation and the text feature representation, a plurality of negative samples of different difficulties corresponding to the anchor point sample are generated, including:

Select a sample as the anchor point sample q, where the sample is an image sample or a text sample;

Based on the anchor sample q, global semantic clustering is performed on each sample in the training set to obtain a negative sample cluster set G={g ₁ , g ₂ ,..., g _M }, where _gi ={ x _i1 , x _i2 ,…, x _iN } represents a negative sample set of N negative samples with similar semantics, x _ij represents the j-th negative sample in the negative sample set g _i , i takes any integer from 1 to M , j takes any integer from 1 to N;

Based on the kernel function, the similarity and the corresponding weight between each negative sample and the anchor sample q are calculated, and a weighted average is performed to obtain multiple negative samples of different difficulties.

Further, the similarity and the corresponding weight between each negative sample and the anchor sample q are calculated based on the kernel function, and a weighted average is performed to obtain multiple negative samples of different difficulties, including:

Calculate the similarity between each negative sample and the anchor sample based on the Gaussian radial basis function:

Among them, k represents the similarity between the anchor sample q and the negative sample x _jn , ||·|| represents the paradigm distance, and σ is the width parameter;

Calculate the weight W _n corresponding to the similarity between each negative sample and the anchor sample according to the following formula:

J(W)＝min|XW _n ||

Among them, J(W) is the cost function representing the error in the least squares method, W is the weight matrix to be optimized; X represents the input data matrix, each row represents a negative sample, and each column represents a feature; W _n is the weight matrix W Weight value; ||·|| represents calculation error;

The generated negative samples are calculated by weighted average:

in, Represents the generated negative sample corresponding to the anchor sample.

Further, the loss function can be expressed as:

Among them, v represents the image feature representation, c represents the text feature representation; s ^vC+ represents the positive similarity when the anchor sample is an image sample, s ^cv+ represents the positive similarity when the anchor sample is a text sample; ^wvc represents the anchor sample. When the image sample is a set of positive similarities and negative similarities, S ^cv represents the set of positive similarities and negative similarities when the anchor sample is a text sample; and/> represents the penalty term; τ is a hyperparameter; ||·||| represents the set size.

Further, multiple image-text sample pairs in the training set are input into the pre-trained OSCAR image and text retrieval model, and feature extraction is performed to generate image feature representations and text feature representations, including:

Obtain image samples in the training set, extract regional visual features and regional position features of the image samples, linearly combine the regional visual features and the regional position features to obtain image embeddings; the image samples include n object area;

Obtain the text samples in the training set, use word segmentation technology to divide the text samples into multiple tags, and obtain the text embedding corresponding to each tag based on the OSCAR-base model;

Based on the image embedding and the text embedding, an attention mechanism is used to generate a joint feature representation, and the image feature representation and the text feature representation are generated through average pooling.

In a second aspect, the present invention also provides a method for realizing image and text retrieval using the OSCAR image and text retrieval model, which is trained by the training method described in any one of claims 1 to 5, so The methods include:

Obtain the target text and target image to be retrieved;

Perform feature extraction on the target text based on the text encoder in the image-text retrieval model to obtain text feature representation;

Perform feature extraction on the target image based on the image encoder in the image-text retrieval model to obtain image feature representation;

Based on the text feature representation and the image feature representation, an image retrieval result of the target text in the target image is determined, and/or a text retrieval result of the target image in the target text is determined.

In a third aspect, the present invention also provides an OSCAR-based image and text retrieval model training device. The device includes:

A data acquisition module, used to acquire a training set, where the training set includes multiple image-text sample pairs;

A feature extraction module, used to input multiple image-text sample pairs in the training set into the pre-training model OSCAR for visual language tasks, and perform feature extraction to generate image feature representations and text feature representations;

A negative sample synthesis module, configured to use each sample in the training set as an anchor sample, and generate multiple negative samples of different difficulties corresponding to the anchor sample based on the image feature representation and the text feature representation; The generated negative sample and the anchor point sample form a generated negative sample pair;

A similarity calculation module, used to calculate the positive similarity between the image and the text in the positive sample pair, the negative similarity between the negative sample pair and the generated negative sample pair between the image and the text;

A comparison loss calculation module is used to calculate a loss function based on the positive similarity and negative similarity, and fine-tune the pre-trained model OSCAR through the loss function to obtain a trained OSCAR image and text retrieval model.

In a fourth aspect, the present invention also provides a computer device. The computer device includes a processor and a memory; the memory stores at least one instruction, and the at least one instruction is used to be executed by the processor to implement any of the above. The method described in 1.

In a fifth aspect, the present invention also provides a computer-readable storage medium that stores at least one instruction, and the at least one instruction is used to be executed by a processor to implement any of the above methods.

The beneficial effects of the present invention are: the embodiment of the present invention provides a training method for an image and text retrieval model based on OSCAR, which uses the visual language pre-training model OSCAR to extract features from image samples and text samples, and generates non-linear images through a negative sample synthesis module. Challenging negative samples increase the difficulty between images and texts. The loss function is designed using the positive similarity between the image and the article in the positive sample pair, the negative similarity between the negative sample pair and the generated negative sample pair between the image and the text. , the target OSCAR model is obtained based on the new loss function training, which improves the generalization ability of the image and text retrieval model, thereby improving the efficiency and accuracy of the model for image and text retrieval.

The above description is only an overview of the technical solutions of the present invention. In order to have a clearer understanding of the technical means of the present invention and implement them according to the contents of the description, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings.

Description of the drawings

Figure 1 is a schematic flow chart of an OSCAR-based image and text retrieval model training method provided by an embodiment of the present invention;

Figure 2 is a schematic flowchart of a method for implementing image and text retrieval provided by an embodiment of the present invention;

Figure 3 is a structural block diagram of an OSCAR-based image and text retrieval model training device provided by an embodiment of the present invention;

Figure 4 is a schematic structural diagram of a computer device provided by an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

In addition, "and/or" in this article is just an association relationship that describes related objects, indicating that there can be three relationships, for example, A and/or B, which can mean: A exists alone, and A and B exist at the same time. There are three cases of B alone. In addition, the character "/" in this article generally indicates that the related objects are an "or" relationship.

Embodiments of the present application provide an OSCAR-based image and text retrieval model training method. The execution subjects of the training method include but are not limited to servers, terminals, and other creative devices that can be configured to execute the method provided by the embodiments of the present application. A sort of.

Please refer to FIG. 1 , which is a schematic flow chart of an OSCAR-based image and text retrieval model training method provided by an embodiment of the present invention. In this embodiment, the training method includes:

Step S101: Obtain a training set, which includes multiple image-text sample pairs.

In the embodiment of the present invention, a data set can be obtained from a designated open source natural language learning model corpus, or a large number of image-text pairs can be obtained from a designated website using a Python script with data capture capabilities. The embodiment of the present invention does not provide training The method of obtaining the set is specifically limited.

Step S102: Input multiple image-text sample pairs in the training set into the pre-trained model OSCAR pre-trained for visual language tasks, and perform feature extraction to obtain image feature representation and text feature representation.

It is understandable that the pre-trained OSCAR visual language model has been pre-trained on millions of image-text pairs, and can jointly process images and text to obtain meaningful feature representations to capture text and images. and learn more discriminative image-text embeddings. In other words, the OSCAR model has good learning and understanding capabilities for the feature representation of images and texts, and can extract richer feature information from images and texts.

Specifically, the steps to generate image feature representation and text feature representation based on the pre-trained OSCAR model include:

1) Obtain the image samples in the training set, extract the regional visual features and regional position features of the image samples, and linearly combine the regional visual features and regional position features to obtain the image embedding.

Among them, the image sample is divided into n object areas.

In one example, the Faster R-CNN model pre-trained on the Visual Genome data set is used to extract the regional visual features and regional position features of the image, and the regional visual features and regional position features are linearly combined through linear projection to obtain each The image embedding corresponding to the image sample.

2) Obtain the text samples in the training set, use word segmentation technology to divide the text samples into multiple tags, and obtain the text embedding corresponding to each tag based on the OSCAR-base model.

In the embodiment of the present invention, for a given text sample c, first use word segmentation technology to divide it into z tags, that is, c = {o ₁ , o ₂ ,..., o _z }, and then obtain each tag through the OSCAR-base model. The text corresponding to the mark is embedded in E _tok , then there are:

in, Represents the i-th token of the text sample.

Therefore, the text embedding corresponding to the text sample is expressed as:

3) Based on image embedding and text embedding, the attention mechanism is used to generate joint feature representation, and the image feature representation and text feature representation are generated through average pooling.

In this embodiment, the obtained image embedding and text embedding are input to a single Transformer model in the OSCAR visual language model to obtain the joint feature representation of the image and text, and then the local features of the image and the local features of the text are mapped as Global features with smaller dimensions and retain the average information of features to generate image feature representation and text feature representation. Among them, the Transformer model is based on the attention mechanism, captures the complex relationship between image and text elements, and obtains the joint feature representation of the image-text pair based on the mutual relationship between the image and text.

Step S103: Use each sample in the training set as an anchor sample, and generate multiple negative samples of different difficulties corresponding to the anchor sample based on the image feature representation and the text feature representation. Among them, the generated negative sample and the anchor point sample constitute the generated negative sample.

In the embodiment of the present invention, considering that during the training process of the image and text retrieval model, the diversity of samples will affect the retrieval effect of the model, a negative sample synthesis module is designed to generate negative samples of different difficulties to achieve challenging results. Training the model with negative samples can improve the generalization ability of the model.

Among them, the steps of generating multiple negative samples of different difficulties corresponding to the anchor point sample include:

1) Select a sample as the anchor sample q, which is an image sample or a text sample.

In the following embodiments of this article, the anchor point sample q is an image sample as an example for detailed explanation.

2) Based on the anchor sample q, perform global semantic clustering on each sample in the training set to obtain a negative sample cluster set G = {g ₁ , g ₂ ,..., g _M }, where _gi = {x _i1 , x _i2 ,..., x _iN } represents a set of negative samples with similar semantics of N negative samples, x _ij represents the j-th negative sample in the negative sample set g _i , i is any number from 1 to M Integer, j takes any integer from 1 to N.

Specifically, negative samples that do not match the anchor samples are selected in the mini-batch of the training set, the k-means algorithm is performed on these negative samples, and these negative samples are divided into multiple different negative sample sets according to semantics. These negative sample sets The final negative sample cluster set G = {g ₁ , g ₂ ,..., g _M } is formed. Each element in the cluster set G represents a group of semantically similar negative samples. Among them, the number of negative sample sets is determined by the parameter k, which is usually specified before executing the algorithm.

3) Calculate the similarity and corresponding weight between each negative sample and the anchor sample q based on the kernel function, and perform a weighted average to obtain multiple negative samples of different difficulties.

In the embodiment of the present invention, the kernel function used is a Gaussian radial basis function. Specifically, the steps of calculating the similarity between each negative sample and the anchor sample q and the corresponding weight based on the kernel function, and performing a weighted average to obtain multiple negative samples of different difficulties include:

1) Calculate the similarity between each negative sample and the anchor sample based on the Gaussian radial basis function:

Among them, k represents the similarity between the anchor sample q and the negative sample x _in , ||·|| represents the paradigm distance, and σ is the width parameter;

2) Calculate the weight w _n corresponding to the similarity between each negative sample and the anchor sample according to the following formula:

J(W)=min||X-W||

Among them, J(w) represents the cost function representing the error in the least squares method, w represents the weight matrix to be optimized; X represents the input data matrix, each row represents a sample, and each column represents a feature; ||·|| represents the paradigm distance;

The embodiment of this application uses the least squares method to optimize the weight matrix. The goal of the least squares method is to adjust the parameter matrix

3) The generated negative samples are calculated through weighted average calculation:

in, Represents the generated negative sample corresponding to the anchor sample.

It can be understood that if the anchor sample is an image sample v, the generated negative sample is a text negative sample If the anchor sample is text sample c, the generated negative sample is image sample/>

Step S104: Calculate the positive similarity between the image and the text in the positive sample pair, and the negative similarity between the image and the text in the negative sample pair and the generated negative sample pair.

The positive sample that matches the anchor sample is a positive sample. The positive sample and the anchor sample form a positive sample pair, and the positive similarity between the image and the text in the positive sample pair is calculated. The negative samples that do not match the anchor point are negative samples. The negative samples and the anchor point samples form a negative sample pair. The negative similarity between the image and the text in the negative sample pair and the generated sample pair is calculated. Positive and negative similarities are combined to form a third degree of similarity.

Step S105: Calculate a loss function based on positive similarity and negative similarity, fine-tune the pre-trained model OSCAR through the loss function, and obtain the OSCAR image and text retrieval model that has completed training.

The embodiment of the present invention proposes a brand new loss function based on InfoCMR for comparing positive and negative samples from different sources. The loss function can be expressed by the formula:

Among them, v represents the image feature representation, c represents the text feature representation; s ^vc+ represents the positive similarity when the anchor sample is an image sample, s ^cv+ represents the positive similarity when the anchor sample is a text sample; S ^vc represents the anchor sample. is the set of positive similarity and negative similarity when the anchor sample is an image sample, S ^cv represents the set of positive similarity and negative similarity when the anchor sample is a text sample; and/> represents the penalty term; τ is a hyperparameter; ||•|| represents the set size.

Among them, in order to reduce the risk of model overfitting, an additional penalty term is introduced. Randomly sample Z Gaussian noise vectors from a Gaussian distribution, each vector having the same dimensions as the anchor vector corresponding to the anchor sample in the embedding space. These Gaussian noise vectors form a high-confidence negative relationship with each sample in the batch. Sample pairs,help to smooth the representation space. It should be noted that these Gaussian noise vectors will not participate in the formation of positive sample pairs.

The loss function designed by this invention integrates the information between positive samples, negative samples and generated negative samples to further reduce the heterogeneity difference in image-text matching. At the same time, an additional penalty term is added to the loss function, and a high-confidence negative sample pair is formed by randomly sampling Gaussian noise vectors, which reduces the risk of overfitting, helps to smooth the representation of controls and improves the generalization ability of the model.

An embodiment of the present invention provides a training method for an image and text retrieval model based on OSCAR, which uses the visual language pre-training model OSCAR to extract features from image samples and text samples, and generates unchallenging negative samples through a negative sample synthesis module. It increases the difficulty between the image and the text, and uses the positive similarity between the image and the article in the positive sample pair, the negative similarity between the negative sample pair and the generated negative sample pair to design the loss function, and trains based on the new loss function The target OSCAR model is obtained, which improves the generalization ability of the image and text retrieval model, thereby improving the efficiency and accuracy of the model for image and text retrieval.

Please refer to Figure 2, which is a schematic flow chart of an image-text retrieval method implemented using an image-text retrieval model trained as described above, provided by an embodiment of the present invention. The method includes:

Obtain the target text and target image to be retrieved;

Based on the text encoder in the image-text retrieval model, feature extraction is performed on the target text to obtain text feature representation;

Based on the image encoder in the image-text retrieval model, feature extraction is performed on the target image to obtain image feature representation;

Based on the text feature representation and the image feature representation, determine the image retrieval result of the target text in the target image, and/or determine the text retrieval result of the target image in the target text.

Using the image and text retrieval method provided by the embodiment of the present invention can improve the efficiency and accuracy of image and text retrieval.

Please refer to Figure 3, which is a structural block diagram of a training device for an OSCAR-based image and text retrieval model provided by an embodiment of the present invention. The device includes:

The data acquisition module 310 is used to acquire a training set, which includes multiple image-text sample pairs;

The feature extraction module 320 is used to input multiple image-text sample pairs in the training set into the pre-training model OSCAR for visual language tasks, and perform feature extraction to generate image feature representations and text feature representations;

The negative sample synthesis module 330 is used to use each sample in the training set as an anchor sample, and generate multiple negative samples of different difficulties corresponding to the anchor sample based on the image feature representation and the text feature representation; the generated negative sample and the anchor point Sample composition generates negative sample pairs;

The similarity calculation module 340 is used to calculate the positive similarity between the image and the text in the positive sample pair, and the negative similarity between the image and the text in the negative sample pair and the generated negative sample pair;

The comparison loss calculation module 350 is used to calculate a loss function based on positive similarity and negative similarity, and fine-tune the pre-trained model OSCAR through the loss function to obtain a trained OSCAR image and text retrieval model.

Please refer to Figure 4, which is a schematic structural diagram of a computer device according to an embodiment of the present invention. The computer device may include a processor 20, a memory 21 and a bus, and may also include a program stored in the memory 21 and capable of running on the processor 20. Computer program.

The memory 21 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card memory (such as SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 21 may be an internal storage unit of the computer device, such as a mobile hard disk of the computer device. In other embodiments, the memory 21 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), or a secure digital (Secure Digital, SD) card equipped on a computer device. , Flash Card, etc. Further, the memory 21 may also include both an internal storage unit of the computer device and an external storage device. The memory 21 can not only be used to store application software and various types of data installed on the computer device, but can also be used to temporarily store data that has been output or is to be output.

In some embodiments, the processor 20 may be composed of an integrated circuit, for example, it may be composed of a single packaged integrated circuit, or it may be composed of multiple integrated circuits packaged with the same function or different functions, including one or more central processing units. (Central Processing unit, CPU), microprocessor, digital processing chip, graphics processor and various control chip combinations, etc. The processor 20 is the control core (Control Unit) of the computer device. It uses various interfaces and lines to connect various components of the entire electronic device, runs or executes programs or modules stored in the memory 21, and calls programs stored in the memory 21. Data to perform the various functions of the computer equipment and to process the data.

The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into address bus, data bus, control bus, etc. The bus is provided to enable connection communication between the memory 21 and at least one processor 20 and the like.

Figure 4 only shows a computer device with components. Those skilled in the art can understand that the structure shown in Figure 4 does not constitute a limitation on the computer device, and may include fewer or more components than shown in the figure. Or combining certain parts, or different parts arrangements.

For example, although not shown, the computer device may also include a power supply (such as a battery) that supplies power to various components. Preferably, the power supply may be logically connected to at least one processor 20 through a power management device, thereby realizing charging management through the power management device. Discharge management, power consumption management and other functions. The power supply may also include one or more DC or AC power supplies, recharging devices, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components. The computer equipment may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

Further, the computer device may also include a network interface. Optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which are usually used between the computer device and other computer devices. Establish a communication connection between them.

Optionally, the computer device may also include a user interface, which may be a display (Display) or an input unit (such as a keyboard). Optionally, the user interface may also be a standard wired interface or a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, or the like. The display may also be appropriately referred to as a display screen or a display unit, and is used for displaying information processed in the computer device and for displaying a visualized user interface.

It should be understood that the above embodiments are for illustration only, and the scope of the patent application is not limited by this structure.

Computer equipment integrated modules/units can be stored in a computer-readable storage medium if they are implemented in the form of software functional units and sold or used as independent products. Computer-readable storage media may be volatile or non-volatile. For example, computer-readable media may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, mobile hard drives, magnetic disks, optical disks, computer memory, and read-only memory (ROM, Read-Only Memory).

The present invention also provides a computer-readable storage medium. The readable storage medium stores a computer program, and the computer program is executed by a processor of an electronic device.

In addition, each functional module in various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of hardware plus software function modules.

It is obvious to those skilled in the art that the present invention is not limited to the details of the above-described exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

Therefore, the embodiments should be regarded as illustrative and non-restrictive from any point of view, and the scope of the present invention is defined by the appended claims rather than the above description, and it is therefore intended that all claims falling within the claims All changes within the meaning and scope of equivalent elements are included in the present invention. Any accompanying reference signs in the claims shall not be construed as limiting the claim in question.

The embodiments of this application can obtain and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. .

Furthermore, it is clear that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claims may also be implemented by one unit or device by software or hardware. Second-class words are used to denote names, not any particular order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified. Modifications or equivalent substitutions may be made without departing from the spirit and scope of the technical solution of the present invention.

Claims (9)

1. A training method for an OSCAR-based image and text retrieval model, characterized in that the method includes: Obtain a training set, where the training set includes a plurality of image-text sample pairs; Input multiple image-text sample pairs in the training set into the pre-trained model OSCAR for visual language tasks, and perform feature extraction to obtain image feature representation and text feature representation; Each sample in the training set is used as an anchor sample, and based on the image feature representation and the text feature representation, multiple negative samples of different difficulties corresponding to the anchor sample are generated; the generated negative samples are consistent with the Anchor point samples form the generated negative sample pairs; Calculate the positive similarity between the image and the text in the positive sample pair, the negative similarity between the negative sample pair and the generated negative sample pair between the image and the text; A loss function is calculated based on the positive similarity and negative similarity, and the visual language pre-training model OSCAR is fine-tuned through the loss function to obtain the OSCAR image and text retrieval model that has completed training. 2. The training method according to claim 1, characterized in that each sample in the training set is used as an anchor point sample, and the anchor point is generated based on the image feature representation and the text feature representation. Multiple negative samples of different difficulties corresponding to the sample, including: Select a sample as the anchor point sample q, where the sample is an image sample or a text sample; Based on the anchor sample q, global semantic clustering is performed on each sample in the training set to obtain a negative sample cluster set G = {g ₁ , g ₂ ,..., g _M }, where _gi = { x _i1 ,x _i2 ,…,x _iN } represents a negative sample set of N negative samples with similar semantics, x _ij represents the j-th negative sample in the negative sample set g _i , i takes any integer from 1 to M , j takes any integer from 1 to N; Based on the kernel function, the similarity and the corresponding weight between each negative sample and the anchor sample q are calculated, and a weighted average is performed to obtain multiple negative samples of different difficulties. 3. The training method according to claim 2, characterized in that the similarity and corresponding weight between each negative sample and the anchor sample q are calculated based on the kernel function, and a weighted average is performed to obtain a plurality of different Difficulty negative samples include: Calculate the similarity between each negative sample and the anchor sample based on the Gaussian radial basis function: Among them, k represents the similarity between the anchor sample q and the negative sample x _jn , ||·|| represents the paradigm distance, and σ is the width parameter; Calculate the weight W _n corresponding to the similarity between each negative sample and the anchor sample according to the following formula: J(W)=min||XW _n || Among them, J(W) is the cost function representing the error in the least squares method, W is the weight matrix to be optimized; X represents the input data matrix, each row represents a negative sample, and each column represents a feature; W _n is the weight matrix W Weight value; ||·|| represents calculation error; The generated negative samples are calculated by weighted average: in, Represents the generated negative sample corresponding to the anchor sample. 4. The training method according to claim 1, characterized in that the loss function can be expressed as: Among them, v represents the image feature representation, c represents the text feature representation; s ^vc+ represents the positive similarity when the anchor sample is an image sample, s ^cv+ represents the positive similarity when the anchor sample is a text sample; S ^vc represents the anchor sample. is the set of positive similarity and negative similarity when the anchor sample is an image sample, S ^cv represents the set of positive similarity and negative similarity when the anchor sample is a text sample; and/> represents the penalty term; τ is a hyperparameter; ||·|| represents the set size. 5. The training method according to claim 1, characterized in that the plurality of image-text sample pairs in the training set are input into a pre-trained OSCAR image and text retrieval model, and feature extraction is performed to generate an image feature representation and Text feature representation, including: Obtain image samples in the training set, extract regional visual features and regional position features of the image samples, linearly combine the regional visual features and the regional position features to obtain image embeddings; the image samples include n object area; Obtain the text samples in the training set, use word segmentation technology to divide the text samples into multiple tags, and obtain the text embedding corresponding to each tag based on the OSCAR-base model; Based on the image embedding and the text embedding, an attention mechanism is used to generate a joint feature representation, and the image feature representation and the text feature representation are generated through average pooling. 6. A method for realizing image and text retrieval using the OSCAR image and text retrieval model. The OSCAR image and text retrieval model is trained by the training method according to any one of claims 1 to 5, characterized in that, the method include: Obtain the target text and target image to be retrieved; Perform feature extraction on the target text based on the text encoder in the image-text retrieval model to obtain text feature representation; Perform feature extraction on the target image based on the image encoder in the image-text retrieval model to obtain image feature representation; Based on the text feature representation and the image feature representation, an image retrieval result of the target text in the target image is determined, and/or a text retrieval result of the target image in the target text is determined. 7. An image and text retrieval model training device based on OSCAR, characterized in that the device includes: A data acquisition module, used to acquire a training set, where the training set includes multiple image-text sample pairs; A feature extraction module, used to input multiple image-text sample pairs in the training set into the pre-training model OSCAR for visual language tasks, and perform feature extraction to obtain image feature representation and text feature representation; A negative sample synthesis module, configured to use each sample in the training set as an anchor sample, and generate multiple negative samples of different difficulties corresponding to the anchor sample based on the image feature representation and the text feature representation; The generated negative sample and the anchor point sample form a generated negative sample pair; A similarity calculation module, used to calculate the positive similarity between the image and the text in the positive sample pair, the negative similarity between the negative sample pair and the generated negative sample pair between the image and the text; A comparison loss calculation module is used to calculate a loss function based on the positive similarity and negative similarity, and fine-tune the pre-trained model OSCAR through the loss function to obtain a trained OSCAR image and text retrieval model. 8. A computer device, characterized in that the computer device includes a processor and a memory; the memory stores at least one instruction, and the at least one instruction is used to be executed by the processor to implement claims 1 to 6 any of the methods described. 9. A computer-readable storage medium, characterized in that the storage medium stores at least one instruction, and the at least one instruction is used to be executed by a processor to implement the method according to any one of claims 1 to 6.