CN118552985B - Label identification method, object processing method, computing device, storage medium, and program product - Google Patents

Abstract

The embodiment of the application provides a tag identification method, an object processing method, computing equipment, a storage medium and a program product. The method comprises the steps of obtaining a plurality of labels, grouping the labels according to a hierarchical relationship to obtain a plurality of groups of label combinations, extracting label characteristics corresponding to the labels in the plurality of groups of label combinations, wherein the plurality of groups of label combinations are used for matching with an object to be identified according to the sequence from high to low of the hierarchy based on the label characteristics and the image characteristic object characteristics of the object to be identified to obtain at least one target label of the object to be identified. The technical scheme provided by the embodiment of the application improves the accuracy of label identification.

Description

Tag identification method, object processing method, computing device, storage medium and program Product

Technical Field

The embodiments of the present application relate to the field of data processing technology, and in particular to a tag recognition method, an object processing method, a computing device, a storage medium, and a program product.

Background Art

With the rapid development of artificial intelligence technology, object recognition and automatic labeling technology are increasingly becoming key components in the field of information processing and data analysis. For example, in recent years, image recognition algorithms based on artificial intelligence (AI) have become mature and widely used in various industries, becoming a bridge between the physical world and the digital world. Many service providers at home and abroad are competing to launch object recognition services, which can provide users with convenient label generation capabilities. Users provide objects to be identified, such as images, and can use the service provider's object recognition model to generate labels corresponding to the objects to be identified.

However, although the existing object recognition service provides a rich tag library that can meet most general needs, its mechanism for matching tags corresponding to the object to be recognized is still to filter out matching tags from the tag library, which cannot meet the user's refined tag needs. Therefore, how to accurately identify the tags that match the object to be recognized has become a technical problem that technicians in this field need to solve.

Summary of the invention

The embodiments of the present application provide a tag recognition method, an object processing method, a computing device, a storage medium, and a program product to solve the technical problem of inaccurate tag recognition results in the prior art.

In a first aspect, an embodiment of the present application provides a label processing method, including:

Get multiple tags;

Grouping the multiple tags according to a hierarchical relationship to obtain multiple groups of tag combinations;

Extracting label features corresponding to the labels in the plurality of label combinations;

The multiple groups of label combinations are used to match the object to be identified in descending order of hierarchy based on the label features and the object features of the object to be identified to obtain at least one target label of the object to be identified.

Optionally, grouping the multiple tags according to a hierarchical relationship to obtain multiple groups of tag combinations includes:

Identifying the hierarchy to which the multiple tags belong;

Combine labels at the same level to obtain at least one set of label combinations.

Optionally, the identifying the levels to which the multiple tags belong includes:

The hypernyms of the multiple tags are searched in a vocabulary database until a root node is reached, and the level to which the tag belongs is determined according to the path length from the tag to the root node.

Optionally, searching for hypernyms of the plurality of tags in a vocabulary database until a root node is reached comprises:

determining a synonym set of the plurality of tags in the vocabulary database;

The hypernyms of the synonym set in the vocabulary database are searched level by level until a root node is reached.

Optionally, after acquiring the plurality of tags, the method further includes:

Tags that meet a filtering condition are filtered out from the plurality of tags to update the plurality of tags.

Optionally, acquiring multiple tags includes:

Get a tag list including multiple tags provided by the user;

The step of extracting label features corresponding to the labels in the plurality of label combinations comprises:

A multimodal model is used to extract label features corresponding to the labels in the multiple groups of label combinations; the multimodal model is a pre-trained model.

In a second aspect, an embodiment of the present application provides an object processing method, including:

Get the object to be identified;

Extracting object features of the object to be identified;

Determine the label features corresponding to the labels in the extracted multiple label combinations; the multiple label combinations are obtained by grouping the acquired multiple labels according to a hierarchical relationship;

Based on the object features and the label features, the multiple groups of label combinations are matched with the objects to be identified respectively to obtain at least one target label corresponding to the objects to be identified.

Optionally, matching the plurality of groups of label combinations with the objects to be identified respectively based on the object features and the label features to obtain at least one target label corresponding to the object features includes:

Based on the label features and the object features, matching the object to be identified with the labels in the highest-level label combination to obtain a matching successful label;

Starting from the next-level label combination of the highest-level label combination, determining at least one label in the current-level label combination that has a superior-fewer-superior relationship with the successfully matched label in the previous-level label combination;

Matching the at least one tag with the object to be identified to obtain a successful matching tag;

At least one successfully matched tag corresponding to the multiple tag combinations is used as at least one target tag corresponding to the object to be identified.

Optionally, matching the plurality of groups of label combinations with the objects to be identified respectively based on the object features and the label features to obtain at least one target label corresponding to the objects to be identified comprises:

Calculating feature similarities between label features corresponding to labels in the plurality of label combinations and object features of the object to be identified;

At least one target tag with similar features that satisfies a similarity condition is determined.

Optionally, extracting the object features of the object to be identified includes:

Extracting object features of the object to be identified using a multimodal model;

The step of determining the label features corresponding to the labels in the extracted multiple sets of label combinations includes:

Determine the tag features corresponding to the tags in the multiple tag combinations extracted using the multimodal model; the multiple tags obtained include a tag list including multiple tags provided by a user.

Optionally, the object to be identified is an image to be identified, and the method further includes:

Determining an image category of the image to be identified based on the at least one target tag;

or;

Based on the at least one target tag, searching for a target image matching the image to be identified;

or;

Based on the at least one target tag, verify whether the image to be identified meets image requirements.

In a third aspect, an embodiment of the present application provides a computing device, comprising a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the label processing method described in the first aspect above, or the object processing method described in the second aspect above.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processing component, the label processing method described in the first aspect or the object processing method described in the second aspect is implemented.

In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program/instruction, which, when executed by a processing component, implements the label processing method described in the first aspect above, or the object processing method described in the second aspect above.

In an embodiment of the present application, a plurality of labels are obtained; the plurality of labels are grouped according to a hierarchical relationship to obtain a plurality of label combinations; the label features corresponding to the labels in the plurality of label combinations are extracted; wherein the plurality of label combinations are used to match the object to be identified in a descending order of hierarchy based on the label features and the object features of the object to be identified to obtain at least one target label of the object to be identified. By grouping a plurality of labels according to a hierarchical relationship, the embodiment of the present application can better understand and utilize the context and semantic associations between labels to ensure that the subsequent label recognition process is more detailed and accurate. Matching the target label of the object to be identified based on the label text features and the visual features of the image to be identified helps to improve the accuracy of recognition. Matching in a descending order of the label hierarchy to obtain at least one target label of the object to be identified can further reduce the probability of misidentification, improve label coverage, and further ensure label recognition accuracy.

These and other aspects of the present application will become more clearly understood in the description of the following embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

FIG1 shows a flow chart of an embodiment of a label processing method provided by the present application;

FIG2 shows a flow chart of an embodiment of an object processing method provided by the present application;

FIG3 is a schematic diagram showing a scene interaction in a practical application of the technical solution of the embodiment of the present application;

FIG4 shows a schematic structural diagram of an embodiment of a label processing device provided by the present application;

FIG5 is a schematic structural diagram of an embodiment of an object processing device provided by the present application;

FIG6 shows a schematic diagram of the structure of a computing device provided by the present application.

DETAILED DESCRIPTION

In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

The technical solution of the embodiment of the present application can be applied to object recognition scenarios, such as commodity classification on e-commerce platforms, disease diagnosis of medical image analysis, object recognition in smart homes, and customized content recommendation systems. In a practical application, it can be applied to image recognition scenarios, such as e-commerce scenarios, where it is necessary to accurately identify clothing pictures uploaded by users to match the corresponding commodity categories.

As described in the background technology, the traditional method usually relies on artificial intelligence (AI) recognition models generated by some service providers to perform object recognition. The AI model is usually obtained based on pre-label training. For example, in the clothing image recognition scenario, due to the large number of clothing styles and rapid updates, the preset labels are difficult to cover all details and emerging trend elements, so the recognition results may not meet user needs.

In order to overcome the limitations of traditional preset labels and generate more accurate labels, the inventors thought that users can retrain the model by customizing labels, but this solution requires users to collect sample images and customize labels by themselves, and then use the model provided by the service provider for customized training, and then deploy it to the production environment to realize personalized label recognition services. This method has a cumbersome and complicated process, which not only involves a lot of data preparation work and professional algorithm knowledge, but also needs to cross the technical threshold of model training and deployment, and if multi-label recognition is involved, the complexity will be higher. Therefore, the inventors have proposed the technical solution of the embodiment of the present application after a series of in-depth studies. By obtaining multiple expressions and grouping the labels in multiple labels according to the hierarchical relationship, multiple groups of label combinations are obtained to obtain multiple labels, that is, the system automatically performs intelligent grouping according to the hierarchical relationship of the labels, so as to better understand and utilize the context and semantic associations between the labels to ensure that the subsequent label recognition process is more detailed and accurate. Matching the target label of the object to be identified based on the text features of the label and the visual features of the image to be identified helps to improve the accuracy of recognition. Matching is performed in descending order of the label levels to obtain at least one target label of the object to be identified, which can not only reduce the probability of misidentification and improve label coverage, but also ensure label recognition accuracy.

The technical solution of the embodiment of the present application not only simplifies the processing flow of tag recognition and lowers the technical threshold, so that users without AI expertise can easily apply it, but also significantly improves the flexibility and accuracy of object recognition, enhances user experience, and broadens the possibility of personalized services, bringing revolutionary changes to the tag recognition process in various industries.

It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described in this article within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

It should be noted that the technical solution of the embodiment of the present application is applicable to a network virtual environment. The users described generally refer to "virtual users". Real users can register a user account in the server through registration to obtain a user identity in the network environment.

The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

FIG1 is a flow chart of an embodiment of a tag processing method provided by an embodiment of the present application. The technical solution of this embodiment can be executed by a server. In an actual application, the technical solution of the embodiment of the present application can be applied to a system architecture composed of a user end and a server. The user end can interact with the server through a network to receive or send messages, etc.

The user end can be a browser, an APP (Application), or a web application such as an H5 (HyperTextMarkup Language5, Hypertext Markup Language Version 5) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application, etc. The user end can be deployed in an electronic device and needs to rely on the device to run or some apps in the device to run, etc. The electronic device can, for example, have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, a desktop computer, a smart speaker, a smart watch, etc.

The server side may include servers that provide various services, such as servers for tag processing, and servers for object recognition. It should be noted that the server side can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

The method of the embodiment shown in FIG1 may include the following steps:

101: Get multiple tags.

The multiple tags obtained may be tags directly provided by the user, or tags selected or configured by the user from system-provided tags according to needs. The specific forms of the multiple tags obtained may include but are not limited to: a tag list, a tag set.

"Getting multiple tags" is the starting point of the entire tag processing process, and its core is to collect multiple tags configured by the user for the object to be identified. "Tag" can refer to a keyword or phrase that can be used to describe or classify the object to be identified. The object to be identified described in this article can be an image, text, audio or video, etc., and this application does not limit this. The tags provided by the user can reflect their understanding and classification intention of the object to be identified, which increases the retrievability of the object to be identified.

In this step, obtaining multiple tags includes: obtaining a tag list provided by the user including multiple tags. This step emphasizes that the user can play an active role in the annotation process. The server allows the user to customize the tags according to personal understanding or specific needs, thereby improving the personalization and pertinence of the tags. In other words, this application can solve the problem of poor flexibility in tag identification caused by matching the tags corresponding to the objects to be identified through the general tag library in the existing technology by obtaining the tags provided by the user. It can realize the identification of customized tags, meet the personalized needs of users, and improve the user experience.

Among them, the multiple tags provided by the user are not limited by the preset tag library, and can cover a wider range of topics and details, which is particularly suitable for scenarios with specific industry backgrounds or personalized needs. The multiple tags provided by the user can be very diverse, including but not limited to keywords, phrases, numbers, and even self-created tags. In addition, the user can provide multiple tags through the user terminal, for example, an input bar with tags is displayed in the display interface of the user terminal for the user to enter tags. Taking the acquisition of multiple tags as an example, which can be a tag list submitted by the user, the tag list can be in the form of a string.

In an embodiment of the present application, the scenario in which the present application obtains multiple tags provided by the user can be a scenario in which a picture is posted on social media. For example, when a user uploads a photo on social media, he wants to add tags such as "travel", "food", "pets" and the like according to his needs, so that these pictures are easier to be found by friends who are interested, and it is also convenient for him to review and search in the future. It can also be a scenario in which merchants on an e-commerce platform need to add trendy tags to new fashions. For example, when merchants put new clothing on the shelves, they can manually add tags such as "dopamine", "retro style", "sustainable materials" and the like according to fashion trends and product features to help consumers quickly locate their favorite products and increase product exposure. In addition, other scenarios can also be included. The purpose of the present application obtaining multiple tags provided by the user is to enable the subsequently obtained objects to be identified to be marked with at least one target tag in the tag list according to user needs, rather than using the system default tags for marking.

Optionally, after acquiring the multiple tags, the method may further include: filtering out tags that meet a filtering condition from the multiple tags to update the multiple tags.

In order to further improve the accuracy and practicality of the tags, after obtaining multiple tags provided by the user, the multiple tags can be verified and cleaned, such as removing some meaningless tags such as garbled tags, etc. The screening condition can be a definition rule for meaningless tags, such as a regular expression, so that tags that meet the definition rules can be screened.

In addition, a classification model can also be used to identify meaningless labels from multiple labels, and the screening condition is that the classification result of the classification model is a meaningless label. The classification model can be pre-trained, for example, it can be trained based on sample labels and training labels of whether the sample labels are meaningless labels (for example, meaningless labels are 0, and other cases are 1).

The named entities in multiple tags can also be identified through the NER (Name Entity Recognition) algorithm, so that the tags of non-named entities can be removed.

By performing tag verification and cleaning, we can use intelligent means to remove tags that may be inaccurate, repeated, or have a weak correlation with the object to be identified, ensuring that the multiple tags used for matching are more refined and highly relevant. This helps improve the efficiency and accuracy of subsequent tag matching.

102: Group multiple tags according to a hierarchical relationship to obtain multiple tag combinations.

Step 102 involves classifying and integrating the tags among the multiple tags provided by the user according to the hierarchical relationship or association between the tags. The principle of tag grouping is usually based on the universality, specificity and semantic inclusion relationship of the tags, forming a semantic structure of one or more levels, each level representing a more specific or more generalized concept category. Through this grouping, a clearly hierarchical tag system that is easy to understand and retrieve can be constructed.

It can be understood that the hierarchical relationship is a semantic structure determined according to the hierarchical relationship between tags in multiple tags.

Among them, the construction of the hierarchical relationship is based on the intrinsic logic of the label, wherein the intrinsic logic of the label refers to the hierarchical relationship determined according to the structure and/or semantics. The construction of the hierarchical relationship of the present application is essentially to organize the logical information of the label into a hierarchical structure. This process mainly includes: first, determine the broadest and most abstract top-level label as the root node of the hierarchical structure; second, starting from the top-level label, gradually identify and define more specific and detailed subordinate concepts to form a hierarchical progressive relationship. Assuming that "animal" is taken as the top-level label as an example, it can contain the subordinate label "mammal", and "mammal" can be further subdivided into "feline", "canine" and so on.

It should be noted that label grouping is not limited to a single dimension, but can also be divided into multiple levels based on different dimensions, such as by function, by subject, by time, etc., to form a multi-dimensional label tree structure.

The following embodiments are all exemplified by obtaining a plurality of tags to provide a tag list to a user. However, it should be noted that the present application is not limited to the plurality of tags obtained being provided by the user, nor is it limited to the plurality of tags being in the form of a tag list.

For example, taking the tag list provided by the user including: "animal", "person", "cat", "dog", "woman", "man" as an example, according to the requirements of step 102, the tags in the tag list input by the user need to be grouped according to the hierarchical relationship. For a given tag list ["animal", "person", "woman", "man", "cat", "dog"], the following grouping structure can be constructed to reflect the hierarchical relationship between them:

The first level includes: animals and people, among which animals and people belong to different dimensions;

The second level includes: "cat", "dog", "man", "woman" (among which "cat" and "dog" are sub-tags of "animal", and "man" and "woman" are sub-tags of "person").

In the above grouping, "animals" and "people" are the broadest categories, while "cats", "dogs", "women" and "men" are more specific subcategories. Among them, "cats" and "dogs" are subcategories of "animals", while "women" and "men" are subcategories of "people", reflecting the inclusion relationship between labels.

Through such grouping, tags can be effectively organized and managed, so that in the subsequent process of determining the target tag corresponding to the object to be identified, the matching target tag can be more accurately located, thereby improving the efficiency of tag processing and the accuracy of tag identification.

Furthermore, in step 102, after the tags in the plurality of tags are grouped according to the hierarchical relationship, a plurality of tag combinations can be obtained based on the grouping results.

103: Extract label features corresponding to labels in multiple groups of label combinations.

Among them, the above-mentioned multiple groups of label combinations are used to match the object to be identified in order from high to low levels based on label features corresponding to the labels in the multiple groups of label combinations and object features of the object to be identified to obtain at least one target label of the object to be identified.

In this step, the method of extracting the label features corresponding to the labels in the multiple groups of label combinations includes: using a multimodal model to extract the label features corresponding to the labels in the multiple groups of label combinations, wherein the multimodal model is a machine learning model that can process and integrate multiple types of data (such as images, text, audio, video, sensor data, etc.). Among them, the multimodal model is a pre-trained model, which can be a pre-trained large model. The pre-trained model solves the problem of not needing data calibration and model training, and reduces the recognition complexity. In this case, the pre-trained multimodal model can extract useful features more efficiently, without the need to train from scratch, greatly shortening the training time and improving the recognition accuracy. In this step, the multimodal model can be used to extract the label features corresponding to the labels in the multiple groups of label combinations, thereby ensuring the accuracy of subsequent label recognition. The multimodal model has zero-shot learning capabilities, which enables the multimodal model to classify or identify new categories that have never been seen without retraining. In an embodiment of the present application, the multimodal model can be a multimodal algorithm model that realizes text and object matching, which can understand and process data in two different forms of text and objects. For example, in the image recognition scenario, the multimodal model can be CLIP (Contrastive Language-Image Pre-training, a cross-modal pre-training model based on text-image contrast learning), which is pre-trained on large-scale unlabeled image-text pairs and can learn the joint embedding representation of images and texts. The label features extracted by the multimodal model can include the alignment relationship with the object, making the label features more accurate.

Optionally, the object features of the object to be identified can be extracted using the multimodal model.

The multimodal model can combine the extracted label features with the object features of the object to be identified and match them at different levels. From the highest level (more abstract labels) to the lower levels (specific and detailed labels), the matching range is gradually narrowed until at least one target label that best matches the object to be identified is found. This process utilizes a hierarchical combination of labels to ensure the accuracy and efficiency of recognition.

Optionally, label features corresponding to the labels in multiple label combinations can be saved according to a hierarchical relationship, so that when there are objects to be identified, label features corresponding to the labels in multiple label combinations can be obtained from the saved data, and matched with the labels in each label combination in order from high to low levels.

In this embodiment, multiple labels are obtained; multiple labels are grouped according to a hierarchical relationship to obtain multiple groups of label combinations; label features corresponding to the labels in the multiple groups of label combinations are extracted; wherein the multiple groups of label combinations are used to match the object to be identified in a descending order of hierarchy based on the label features and the object features of the object to be identified to obtain at least one target label of the object to be identified. The embodiment of the present application obtains multiple labels to meet personalized needs. By grouping multiple labels according to a hierarchical relationship, the context and semantic associations between labels can be better understood and utilized to ensure that the subsequent label recognition process is more detailed and accurate. Matching the target label of the object to be identified based on the label text features and the visual features of the image to be identified helps to improve the accuracy of recognition. Matching in a descending order of the label hierarchy to obtain at least one target label of the object to be identified can not only reduce the probability of misidentification and improve label coverage, but also ensure label recognition accuracy.

In some embodiments, grouping the tags among the multiple tags according to a hierarchical relationship to obtain multiple groups of tag combinations may include: identifying the hierarchies to which the tags among the multiple tags belong; and combining tags at the same level to obtain at least one group of tag combinations.

First, clarify the level to which each tag belongs in the semantic structure. The hierarchical relationship of tags defines the inclusion and inclusion relationship between them. Subsequently, tags belonging to the same level can be grouped together to form at least one group of tag combinations. Such operations help distinguish tag clusters at different abstraction levels, which is convenient for subsequent tag processing.

As a possible implementation method, the labels at the same level can be combined to obtain a set of label combinations. It can be understood that there is only one set of label combinations for one layer of labels, and the corresponding number of label combinations can be obtained according to the number of levels.

For example, taking the second layer in the above example including: "cat", "dog", "man", "woman" as an example, since "cat", "dog", "man", and "woman" all belong to the same level, "cat", "dog", "man", and "woman" can be combined to obtain a set of label combinations of "cat+dog+man+woman".

As another possible implementation method, labels of the same level and the same dimension may be combined to obtain one or more groups of label combinations.

Specifically, when combining labels at the same level, labels at the same level and the same dimension can be combined to obtain multiple sets of label combinations, which can greatly expand the flexibility and creativity of label application. This means that when combining labels at the same level, it is also necessary to consider the label combinations within a single dimension at the same level, so as to capture more complex and delicate associations and user needs. The following is a supplementary explanation and specific examples of this concept:

The free arrangement and combination of labels of the same level and dimension refers to the free combination of labels belonging to the same dimension within the same level. For example, in the above example, the second level includes: "cat", "dog", "man", and "woman". "Cat" and "dog" both belong to the dimension of "animal". "Cat" and "dog" can be combined to obtain a set of "cat + dog" label combination, while "man" and "woman" both belong to the dimension of "character". "Man" and "woman" can be combined to obtain a set of "man + woman" label combination. At this time, two sets of label combinations can be obtained, namely the label combination of "cat + dog" and the label combination of "man + woman".

For the above-mentioned solution of "combining labels of the same level and the same dimension to obtain multiple sets of label combinations", a detailed implementation example is provided:

For example, taking a tag list including A, B, C, D, E, F, G, H, I, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0 as an example, by identifying the levels to which the tags in the tag list belong and grouping them according to the hierarchical relationship, the following three levels can be obtained:

The first layer: A, B, C; A, B, C belong to different dimensions;

The second layer: D, E, F, G, H, I (D and E belong to the lower-level labels of A, F and G belong to the lower-level labels of B, and H and I belong to the lower-level labels of C); DE, FG, and HI belong to different dimensions;

The third layer: 1, 2, 3, 4, 5, 6, 7, 8, 9, 0 (1 and 2 belong to the lower-level labels of D, 3 and 4 belong to the lower-level labels of E, 5 and 6 belong to the lower-level labels of F, 7 and 8 belong to the lower-level labels of G, 9 belongs to the lower-level labels of H, and 0 belongs to the lower-level labels of I); 12, 34, 56, 78, 9, and 0 belong to different dimensions;

Furthermore, once the level of each tag is identified, the next step is to merge tags at the same level together to form different tag combinations. The advantage of this is that the tag set at the same level can be quickly located and operated, which is convenient for subsequent processing and application.

By combining the labels of the same level and dimension, we can get the following sets of label combinations:

The first layer of multiple sets of label combinations:

Combination 11: {A};

Combination 12: {B};

Combination 13: {C};

The second layer of multiple label combinations:

Combination 21: {D, E};

Combination 22: {F, G};

Combination 23: {H, I};

The third layer of multiple label combinations:

Combination 31: {1, 2};

Combination 32: {3, 4}

Combination 33: {5, 6};

Combination 34: {7, 8};

Combination 35: {9};

Combination 36: {0};

In the above grouping, it is reflected that the way to combine tags at the same level can be to combine tags at the same level and the same dimension. Through such grouping, the hierarchical structure and attribution relationship between tags can be clearly seen, which provides a convenient basis for subsequent tag management and content processing.

There are many implementation methods for identifying the level to which the tags among the multiple tags belong.

In one possible implementation, identifying the level to which a tag among the multiple tags belongs may include: searching a vocabulary database for hypernyms of the tags among the multiple tags until a root node is reached, and determining the level to which the tag belongs based on a path length from the tag to the root node;

Among them, the above hierarchical relationship is a semantic structure determined according to the hierarchical relationship between tags in multiple tags. The above root node can be the highest-level vocabulary corresponding to the tag in the vocabulary database, which is usually an entity or event, which is a relatively broad category.

In this embodiment, the purpose of searching for hypernyms of tags among multiple tags until the root node is reached is to determine the hierarchical position of the tags provided by the user in the semantic structure. First, the hypernyms of these tags must be found until the highest level of the entire semantic structure, the root node, is reached. The root node is the starting point of concepts of different dimensions (i.e., there can be multiple root nodes). Through this process, the hypernym relationship between tags can be established.

Among them, the vocabulary database is a dictionary that organizes words according to semantic relationships to form a semantic network, and the semantic relationship includes the hypernym relationship. For a tag among multiple tags, in the vocabulary database, the hypernym of the tag can be searched step by step according to the hypernym relationship until the root node is found. From the root node, tracing down to the tag, the level of the tag can be determined.

For example, the tag list input by the user contains two tags "A, B". At this time, it is not clear to which level "A" and "B" belong, so it is necessary to query through the vocabulary database. Assuming that "A" is found in the vocabulary database as the root node, and "B" finds "C" based on its hypernym, and then further finds the root node "D" based on "C", it can be determined that "A" belongs to the first level (the highest level), which can also be called a first-level tag. "B" belongs to the third level (that is, D>C>B), which can also be called a third-level tag.

In which, in the vocabulary database, words of each part of speech are organized into synonym sets, each synonym set contains words with similar meanings, and the synonym sets are linked to each other through semantic relationships to form a semantic network. Therefore, in some embodiments, querying the vocabulary database for hypernyms of tags in the tag list until the root node is reached may include: determining the synonym sets of tags among multiple tags in the vocabulary database; searching for hypernyms of the synonym sets in the vocabulary database until the root node is reached.

That is, for a tag among multiple tags, the synonym set that matches can be searched from the vocabulary database. The synonym set may include the tag or its synonyms, and then the hypernym set of the synonym set is searched level by level until the root node is reached.

Optionally, the vocabulary database may be, for example, WordNet (a dictionary based on cognitive linguistics), wherein WordNet contains a large number of vocabulary entries and semantic relationships therebetween, including synonym sets, hypernyms, hyponyms, etc. For each tag input by the user, it is possible to search in WordNet to find its corresponding hypernym, and the hypernym can then continue to search for its hypernym, and this process is repeated until the corresponding root node is reached.

In this embodiment, it is considered that if the label input by the user does not exist in the vocabulary database, this will hinder the determination of the hierarchical relationship. In order to solve this problem, by determining the synonym set of the label among multiple labels in the vocabulary database; searching for the hypernym of the synonym set in the vocabulary database step by step until the root node is reached. Among them, the synonym set contains words that are similar or identical in meaning to the input label. These words have a high probability of existing in the database and can represent the meaning of the input label. By searching the vocabulary database for a set of words that are similar in meaning to the input label, even if the input label itself is not directly recorded, its hypernym relationship can be indirectly determined through the record of its synonyms in the vocabulary database.

For example, the tag list input by the user includes the tag "sports car". When the server assigns an appropriate classification level to the tag, the specific tag "sports car" is not directly recorded in the vocabulary database (such as WordNet or other professional databases).

At this point, the server first identifies possible synonyms or near-synonyms for "sports car". These synonyms may include "racing car", "high-performance car", "super sports car", etc. These words have corresponding entries in the database and can represent the basic meaning of "sports car".

Next, the server searches for hypernyms of these synonym sets in the vocabulary database. For example, a possible hypernym for "racing car" is "car", and "high-performance car" and "supercar" may also point to "car". This process continues until a root node is found, such as "transportation tool".

Through the above steps, the server can construct a hierarchical relationship of "sports car". Although "sports car" itself does not appear directly in the database, through its synonym set, the system can indirectly determine the hierarchy of "sports car". For example, the final semantic structure is: "transportation" > "car" > "sports car", which can indirectly determine that the hierarchy of "sports car" is the third level, which can also be called a third-level label.

Furthermore, by finding the hypernyms of the tag or its synonym set up to the root node, the level of the tag can be determined by the path length from the tag to the root node. The farther the tag is from the root node, the lower its level in the semantic structure, that is, the more specific it is; otherwise, it is more abstract or generalized.

Starting from the root node, the server traces down along the hypernym relationship and records the path length from each label to the root node. Based on this length, the server can assign labels to corresponding levels to form a hierarchical label structure. Specifically, the server needs to determine one or more broad categories as root nodes, which cover all possible subordinate concepts. For example, in a product classification system, "commodity" can be used as a root node. Further, the server uses existing vocabulary databases (such as WordNet, etc.) to query the hypernyms of each label until the root node is reached. For example, for the label "laptop", its hypernyms may be "computer", "electronic equipment" and even "commodity". During the query process, the server records the path length from each label to the root node, that is, the number of hypernyms, and determines the corresponding level according to the number of hypernyms from each label to the root node. For example, the label "laptop", its hypernyms include "computer", "electronic equipment" and even the root node "commodity". When "commodity" is used as the first layer, the hypernym data of "laptop" can determine that "laptop" is located in the fourth layer.

Through the above steps, even if the tags input by the user are not directly recorded in the vocabulary database, the system can determine the semantic level of these tags by searching the synonym set and tracing back to the root node, and build a multi-level, orderly tag system. This not only improves the flexibility and adaptability of tag processing, but also enhances the accuracy and efficiency of tag applications based on semantic understanding, such as content classification, information retrieval, and personalized recommendation.

Optionally, the path length corresponding to each layer can be set according to demand. For example, when the path length of each layer is set to 1, for "short-sleeved T-shirt", the hypernym is found to be "T-shirt" in the vocabulary database, and further traced back to "top", and finally reached the root node "clothing". The path is: "clothing" > "top" > "T-shirt" > "short-sleeved T-shirt", then the path length of "short-sleeved T-shirt" is determined to be 4 (starting from the root node "clothing", through "top", "T-shirt" to "short-sleeved T-shirt"), then the level of short-sleeved T-shirt can be determined to be the fourth level, and so on, the path length of "T-shirt" is 3, then the level of "T-shirt" can be determined to be the third level, where the path length reflects the abstractness of the label, which can be understood as the larger the value of the path length, the more specific the label.

In yet another possible implementation, identifying the level to which a tag among the multiple tags belongs may include: using a large model to identify the level to which different tags among the multiple tags belong.

That is, the labels in multiple labels can be layered according to the hierarchical relationship through the large model to determine the level to which each label belongs and the labels belonging to the same level.

Optionally, the large model can be a large language model (LLM), which refers to a natural language processing model with a very high number of parameters and computing power. It can be used for various natural language processing tasks such as text generation, machine translation, question-answering systems, etc. The large language model is driven by AI algorithms based on deep learning technology and uses a large data set to evaluate, standardize and generate relevant content, as well as make accurate predictions. For example, the large language model can be GPT-3 (Generative Pre-Trained Transformer -3, the third generation of generative pre-trained model), GPT-4 (Generative Pre-Trained Transforme-4, the fourth generation of generative pre-trained model), BERT (Bidirectional Encoder Representation from Transformers, a bidirectional encoder model based on Transformers), Turing NLG (Turing Natural language Generation), etc., which are not limited in this application.

In some embodiments, the method may further include: obtaining an object to be identified; extracting image feature object features of the object to be identified; and matching multiple groups of label combinations with the object to be identified based on the image feature object features and label features to obtain at least one target label corresponding to the object to be identified.

That is, the embodiment of the present application can also obtain multiple tags and objects to be identified in real time, and obtain at least one target tag that successfully matches the object to be identified from the multiple tags.

In some embodiments, a multimodal model may be used to extract label features corresponding to labels in multiple label combinations and image features and object features of the object to be identified. The multimodal model is usually a large model. To facilitate model identification, the process of extracting label features corresponding to labels in multiple label combinations using the multimodal model may include:

Generate prompt information for any set of labels and auxiliary text information in the label combination; the auxiliary text information is used to assist the multimodal model in understanding the label so that the multimodal model can extract label features corresponding to the label;

The multimodal model is used to extract label features corresponding to labels based on prompt information.

In this embodiment, in addition to the label, auxiliary text information is also added. This auxiliary text information is intended to provide context for the multimodal model to help the model better grasp the context and meaning of the label. For example, for the label "mountain", the auxiliary text information may be "a natural terrain feature towering into the clouds with a sharp top."

Furthermore, based on the labels and auxiliary text information, a "prompt" or "query" form of information is generated. This step is like preparing a precise question for the multimodal model: for example, "Please extract the features related to 'mountain' based on the content of this picture and the description of 'natural terrain features towering into the clouds with sharp tops'." Such prompt information contains both visual elements (through labels) and in-depth descriptions of language (auxiliary text information), providing rich input for the processing of the multimodal model.

A multimodal model is an AI model that can process and integrate multiple types of data (such as images, text, etc.) at the same time. After receiving input containing image information and prompt information, it will try to fuse these different modal data to understand and parse the image content at a deeper level. The model will analyze the image and specifically identify and extract features that match the labels and descriptions in the prompt information. For example, for the prompt of "mountain", the model may look for features such as edge contours and light and shadow changes in the image to confirm and accurately locate the position and shape of the mountain. Such feature extraction is not limited to intuitive visual features, but may also include abstract semantic understanding to ensure that the extracted features are consistent with the deep meaning of the label.

In summary, this method of generating prompt information and extracting label features using a multimodal model improves the accuracy and depth of image analysis, enabling the model to better grasp the subtle differences and contextual information in complex scenes, thereby playing an important role in fields such as image classification, content generation, and retrieval.

Of course, as can be seen from the foregoing description, the label features extracted by the multimodal model can also be pre-grouped and saved, so that for any object to be identified, the object and the label can be matched based on the saved label features without re-grouping and calculation. In the object processing method shown in FIG2, the technical solution of the embodiment shown in FIG2 can be executed by the server, and the method can include the following steps:

201: Get the object to be identified;

First, obtain the object whose label needs to be identified. The object to be identified can be a picture, a video, an audio, or a text or other multimedia content. For example, on an e-commerce platform, the object to be identified may be a picture of a product.

202: Extracting object features of the object to be identified;

Use multimodal models to analyze and extract key features of the object to be identified. Multimodal models can process information in different modes (such as vision, hearing, text, etc.) at the same time, and can extract corresponding key features for information in different modes. For example, for visual content (such as pictures), multimodal models extract visual features such as color, shape, texture, and object category. Specifically, for a picture of clothing, the model may extract features such as "blue", "long sleeves", and "round neck".

203: Determine tag features corresponding to tags in the extracted multiple tag combinations; the multiple tag combinations are obtained by grouping the tags in the acquired multiple tags according to a hierarchical relationship;

According to the description of the embodiment of FIG. 1 , the obtained multiple tags are organized into multiple groups of tag combinations with a hierarchical relationship. These tags are grouped according to their hierarchical relationship to generate multiple groups of tag combinations. For example, under "clothing" there is "top", and under "top" there are "T-shirt" and "shirt". Furthermore, the model will generate corresponding tag features for the tags in the multiple groups of tag combinations to facilitate matching with object features.

204: Based on the object features and label features, multiple label combinations are matched with the object to be identified respectively to obtain at least one target label corresponding to the object to be identified. In this step, based on the extracted object features and label features, the server matches each of the multiple label combinations to find the label combination that best matches the object to be identified and the labels in the label combination. The matching algorithm considers the similarity between the label features of the successfully matched labels and the object features, and selects the target label that best describes the object content.

In some embodiments, based on the tag feature and the object feature of the object to be identified, the process of matching with the object to be identified to obtain at least one target tag of the object to be identified may include:

Based on the label features and the object features, the object to be identified is matched with the labels in the highest-level label combination to obtain a successfully matched label; starting from the label combination of the next level of the highest-level label combination, at least one label in the current level label combination that has a hierarchical relationship with the successfully matched label in the previous level label combination is determined; the at least one label is matched with the object to be identified to obtain a successfully matched label; and at least one successfully matched label corresponding to the multiple groups of label combinations is used as at least one target label corresponding to the object to be identified.

This embodiment describes a hierarchical label matching method for automatically assigning appropriate labels to objects to be identified from a series of predefined label sets. This method improves the matching accuracy of labels by gradually refining from the broadest categories to more specific details.

The following takes the object to be identified as an image to be identified as an example to explain the process in detail:

Top-level matching: First, the server uses a multimodal model to extract object features from the object to be identified. Second, these object features are compared with the label features in the top-level label combination. The top-level labels are usually the most extensive and abstract categories, such as "animals", "plants", "transportation tools", etc. The matching algorithm will identify the top-level label that best matches the object features as the successful match label.

Gradually refined matching: Once a successfully matched tag is found in the highest-level tag combination, the server will go down to the next level to find tags that have a hierarchical relationship with the successfully matched tags in the previous level. For example, if "animal" is matched at the highest level, the next step may be to consider more specific categories such as "mammals" and "birds". This step is based on the logic of hierarchical relationships, ensuring the hierarchy and accuracy of the tags.

Continue the matching process: For each next level, the server will repeat the matching process, looking for tags that have a hierarchical relationship with the successfully matched tags in the previous level in the current level. In this way, the system gradually refines from macro classification to more specific details, and each step ensures the matching degree of the selected tags with the object characteristics.

Determine the target label: Ultimately, all labels that are successfully matched in this step-by-step matching process, no matter which level they come from, will be summarized as "at least one target label" for the image to be identified. These labels comprehensively reflect the main content and details of the image and can be used for a variety of purposes such as image classification, search, and recommendation.

In the embodiment of the present application, taking the case where the object to be identified includes an image to be identified as an example, a simple image recognition scenario is envisioned, the purpose of which is to assign a suitable label to a landscape picture containing "mountains and sunsets".

First, you need to group the input label list (such as the label list includes: A, B, C, D, E, F, G, H, I, J, K, L, M, 1, 2, 3, 4, 5, 6, 7, 8, 9, etc.) according to the hierarchical relationship to obtain the following groups:

The first level: A, B, C, D, where C stands for "mountains" and D stands for "sky";

The second level: E, F, G, H, I, J, K, L, M. F and G represent the next level labels of the mountain range C, F represents "peak", H and I represent the next level labels of the sky D, and I represents "sunset";

The third level: 1, 2, 3, 4, 5, 6, 7, 8, 9, among which 8 and 9 represent the next layer of labels of I sunset, and 9 represents "sunset glow".

The matching process is as follows:

First-level matching: Analyze the image to be identified and find that the image to be identified contains the image features "mountains" and "sky", and the image features successfully match the labels C mountains and D sky in the first level. Therefore, the matching successful labels are obtained: C, D. It should be noted that if the match is successful, it is necessary to continue the next level of matching until the most accurate target label is found.

Second-level matching: Based on the matching of mountain C, we find labels with a hierarchical relationship with mountain C from the next level, that is, we find labels related to mountain C in F and G, and find that the peak features in the image match the peak F. Similarly, based on the matching of sky D, we find labels H and I with a hierarchical relationship with sky D from the next level, and recognize that the sunset in the image matches sunset I. Therefore, the successful matching labels of the second level are F and I.

Third-level matching: Based on the matching of peak F, we find labels with a superior-subordinate relationship with peak F from the next level, but no label is successfully matched; based on the matching of sunset I, we find labels 8 and 9 with a superior-subordinate relationship with sunset I from the next level, and find that the sunset glow feature above the peak at sunset in the image matches 9 (sunset glow). Therefore, the label with a successful match in the third level is 9.

Through the above process, at least one target label sequence finally assigned to the image to be identified is CD->FI->9, namely "mountain range", "peak and sunset" and "sunset glow", which reflects the multi-level characteristics of the image content, from the macro natural landscape to the specific visual elements. By refining the labels step by step, not only the coverage of the identified labels is improved, but also the accuracy and richness of the description are improved.

In the above embodiment, the label combination generated by the first level (A, B, C, D) includes: {A, B, C, D}. Through the multimodal model, according to the label features and the object features of the object to be identified, it can be determined that the matching labels are C and D, so that when the object to be identified includes multiple matching labels, more matching labels can be covered by feature matching, thereby improving the coverage of the identified labels. In addition, when performing the next layer of matching labels, the present application only needs to consider the labels that have a contextual relationship with labels C and D, and there is no need to traverse whether the label features of other irrelevant labels match the image features of the image, thereby reducing the amount of calculation in the label recognition process and improving the efficiency of label recognition.

In some embodiments, matching label features corresponding to labels in the multiple groups of label combinations with object features of the object to be identified to obtain at least one target label corresponding to the object to be identified includes: calculating feature similarities between label features corresponding to labels in the multiple groups of label combinations and object features of the object to be identified; and determining at least one target label with similar features that satisfies a similarity condition.

The object features extracted from the object to be identified and the label features extracted from the labels in the label combination need to be converted into numerical forms that can be understood by the computer, usually in vector form. The feature extraction method can be implemented by methods such as word embedding, deep learning feature extraction, etc. Further, by calculating the feature similarity of the label features and the object features, the matching successful label is determined. The similarity condition can be, for example, the highest feature similarity or the feature similarity is greater than the similarity threshold, and the label with the highest feature similarity or the feature similarity greater than the similarity threshold can be determined as the matching successful label. Among them, the feature similarity can be obtained by calculating, for example, cosine similarity, Euclidean distance, etc., and this application does not specifically limit this. The higher the similarity value, the better the match between the label feature and the object feature. Finally, according to the preset similarity threshold or ranking rule, the label with a similarity higher than the similarity threshold is selected as the matching successful label. This may include directly selecting the label with the highest feature similarity, or selecting all or part of the labels on the premise of meeting certain similarity standards.

In an embodiment of the present application, assume that there is an image recognition task, and the goal is to label an image containing "mountain bike". The label list provided by the user includes: "outdoor sports", "rock climbing", "transportation", and "mountain bike". Through server-side preprocessing, it is determined that the label combination corresponding to the label list is: first layer: outdoor sports, transportation; second layer: mountain bike, rock climbing, among which mountain bike belongs to the next layer of labels under transportation, and rock climbing belongs to the next layer of labels under outdoor sports.

After obtaining the image features and label features, the feature similarity of "outdoor sports" and "transportation" in the first layer is calculated in turn. Assume that for the "outdoor sports" label feature, the similarity between it and the image feature vector is calculated to be 0.2. For the "transportation" label feature, the similarity is 0.7, so it is determined that the label "transportation" is successfully matched, so it is necessary to further calculate the similarity between the next layer label "mountain bike" of "transportation" and the image features. For example, for the "mountain bike" label, since this label is more specific and highly related to the image content, the calculated similarity is 0.9, so "transportation-mountain bike" is used as the target label of the image.

Through this process, the system not only identifies the highly relevant label "mountain bike", ensuring the accuracy and pertinence of the label, but also only selects the next-layer label "mountain bike" of "transportation" that meets the similarity requirements for subsequent matching. There is no need to calculate the match between the next-layer label "rock climbing" of "outdoor sports" and the image features, thereby reducing the amount of calculation in the label recognition process and improving the efficiency of label recognition.

In a practical application, the object to be identified may be an image to be identified, and there may be multiple application scenarios for at least one target label obtained for the image to be identified.

As an optional manner, the method further includes: determining the image category of the image to be identified based on at least one target label of the image to be identified.

The purpose of determining the image category is to classify it into one or more predefined image categories based on at least one target label on the image to be identified. For example, if the target labels are "dog" and "outdoor", the system may classify this picture into the "Pet Outdoor Activities" category. Achieving this function usually requires a trained image classification model that can output one or more most likely categories based on the content of the image and the associated label information.

As another optional manner, the method further includes: searching for a target image matching the image to be identified based on the at least one target tag.

The purpose of searching for matching target images is to use at least one target tag as a query condition to find other images from an image database that are similar or related to the content of the image to be identified. This is very useful in scenarios such as image retrieval, content recommendation, and copyright detection. For example, if the target tag is "Eiffel Tower", the system will search for other images containing the Eiffel Tower. Implementing this function usually involves image feature extraction and similarity calculation, as well as efficient image indexing and retrieval algorithms.

As another optional manner, the method further includes: verifying whether the image to be identified meets image requirements based on the at least one target tag.

The purpose of verifying whether an image meets the image requirements is to check whether the image to be identified meets specific standards or requirements. These standards may be quality (such as resolution, clarity), content (such as must contain specific elements), or compliance (such as no sensitive content). Depending on the target label, the system may define a set of verification rules or use a machine learning model to evaluate whether the image meets the standards. For example, if the target label is "product display", the system will verify whether the image clearly shows the full picture of the product without occlusion or blur.

In summary, through these three operations (classification, search, and verification) based on target labels, this method can effectively manage and process image data, which not only improves the degree of automation of image processing, but also enhances the intelligence level of applications, such as playing an important role in content management, search engines, intelligent monitoring and other fields.

The following describes the technical solution of the embodiment of the present application by taking the image to be identified as an example of an object to be identified and combining it with the scene interaction schematic diagram shown in FIG3 .

In this scenario, the user end 301 and the server end 302 achieve accurate label matching of the "image to be identified" through a series of interactions. The following is a detailed description of the specific interaction process:

Step 30: The user inputs multiple tags in the user terminal 301.

Step 31: The client 301 provides the plurality of labels to the server 302. The user of the client 301 (which may be a mobile application, a web interface or other client application) manually inputs or selects a series of labels, such as "animal", "canine", "golden retriever", according to his needs or observed object features. These labels represent the user's basic understanding or expected classification of the image to be identified. In technical implementation, the user interface can be set to allow the user to enter free text in the user interface.

Step 32: The server 302 may group the tags among the obtained multiple tags according to the hierarchical relationship to obtain multiple tag combinations. After receiving the multiple tags submitted by the user 301, the server 302 groups them according to the hierarchical relationship. For example, "animal" is the first layer, "canine" is the second layer, and "golden retriever" is the third layer.

Step 33: The server 302 extracts the label features corresponding to the labels in the plurality of label combinations. The server 302 processes the grouped label combinations and extracts the feature vector of each label. This step forms a deep understanding of the label by analyzing the text description, related images or other modal data of the label.

Step 34: The server 302 is also used to obtain the image to be identified and extract the image features of the image to be identified. The server 302 also extracts the feature vector of the image to be identified from these data, and this feature vector reflects the visual or other modal information of the image.

The source of the image to be identified may at least include: provided by the user through the user terminal 301, automatically captured by the server 302, integrated by a third-party API, or called by a historical database, etc.

Specifically, the image provided by the user through the user terminal 301 may be an image uploaded by the user through a user terminal application (such as a mobile phone application, a web page form, etc.). The automatic capture by the server 302 may be that the server 302 automatically captures content on the Internet, such as pictures and video sharing on social media platforms, media content on news websites, etc., and automatically identifies and processes them to obtain the image to be identified. The third-party API integration may be that the server 302 can obtain the image to be identified by integrating the API of a third-party service, such as downloading a file authorized to be shared by the user from a cloud storage service, or receiving a data stream from a content provider. The historical database call may be that the server 302 selects data from its own historical database as the image to be identified. These data may have been uploaded by the user before, or collected through other channels, and this application does not limit this.

Step 35: Based on the image features and the label features, the server 302 matches the multiple sets of label combinations with the image to be identified respectively to obtain at least one target label corresponding to the image to be identified. In this process, the server 302 uses the previously extracted label features and the features of the image to be identified to perform a matching operation. This process may include calculating the similarity between feature vectors and finding the label set that best matches the image features from the multiple sets of label combinations. Based on the matching results, the server 302 determines at least one target label that is most relevant to the image to be identified, such as "Golden Retriever", and feeds this information back to the user end 301 (such as displaying "Golden Retriever" on the display interface of the user end 301). In this way, the user can obtain accurate classification information about the image to be identified to meet his identification, search or classification needs.

Step 36: In the case where the image to be identified is provided by the user through the user terminal 301 , the server terminal 302 may also feed back at least one target tag corresponding to the image to be identified to the user terminal 301 .

It should be noted that the user who provides the image to be identified and the user who provides the multiple tags may be the same user and share the same user account, or they may be different users.

Of course, the server 302 may also perform other processing on the image to be identified in combination with at least one target tag, such as determining the image category of the image to be identified, searching for a target image that matches the image to be identified, or verifying whether the image to be identified meets image requirements, etc., and may then feed back the processing results to relevant personnel.

FIG4 is a schematic diagram of a structure of an embodiment of a label processing device provided by the present application. As shown in FIG4 , the device includes:

The first acquisition module 41 is used to acquire multiple tags; group the tags in the multiple tags according to the hierarchical relationship to obtain multiple groups of tag combinations;

A first extraction module 42 is used to extract label features corresponding to the labels in the plurality of label combinations;

The multiple groups of label combinations are used to match the object to be identified in descending order of hierarchy based on the label features and the object features of the object to be identified to obtain at least one target label of the object to be identified.

Optionally, in an embodiment of the present application, the first acquisition module 41 is specifically used to identify the level to which the tags among the multiple tags belong; and combine tags at the same level to obtain at least one group of tag combinations.

Optionally, in an embodiment of the present application, the first acquisition module 41 is also used to query the hypernyms of the label in the multiple labels in the vocabulary database until the root node is reached, and determine the level to which the label belongs based on the path length from the label to the root node.

Optionally, in the embodiment of the present application, the first acquisition module 41 is further used to determine a synonym set of a tag among the multiple tags in the vocabulary database; and search for hypernyms of the synonym set in the vocabulary database step by step until a root node is reached.

Optionally, in the embodiment of the present application, the device further includes: a first processing module 43;

The first processing module 43 is used to filter out tags that meet a filter condition from the multiple tags to update the multiple tags.

Optionally, in the embodiment of the present application, the first acquisition module 41 is specifically used to acquire a tag list including multiple tags provided by a user;

The first extraction module 42 is specifically used to extract label features corresponding to the labels in the multiple groups of label combinations by using a multimodal model; the multimodal model is a pre-trained model.

The label processing device shown in FIG4 can execute the label processing method described in the embodiment shown in FIG1, and its implementation principle and technical effect are not described in detail. The specific manner in which each module and unit performs operations in the label processing device in the above embodiment has been described in detail in the embodiment of the method, and will not be described in detail here.

FIG5 is a schematic diagram of the structure of an embodiment of an object processing device provided by the present application. As shown in FIG5 , the device includes:

A second acquisition module 51 is used to acquire an object to be identified;

A second extraction module 52, used to extract object features of the object to be identified;

A second determination module 53 is used to determine the label features corresponding to the labels in the extracted multiple label combinations; the multiple label combinations are obtained by grouping the labels in the acquired multiple labels according to a hierarchical relationship;

The second matching module 54 is used to match the multiple groups of label combinations with the objects to be identified respectively based on the object features and the label features, so as to obtain at least one target label corresponding to the objects to be identified.

Optionally, in an embodiment of the present application, the second matching module 54 is specifically used to match the object to be identified with the label in the highest-level label combination based on the label features and the object features to obtain a successfully matched label; starting from the next-level label combination of the highest-level label combination, determine at least one label in the current-level label combination that has a hierarchical relationship with the successfully matched label in the previous-level label combination; match the at least one label with the object to be identified to obtain a successfully matched label; and use at least one successfully matched label corresponding to the multiple groups of label combinations as at least one target label corresponding to the object to be identified.

Optionally, in an embodiment of the present application, the second matching module 54 is specifically used to calculate the feature similarity between the label features corresponding to the labels in the multiple groups of label combinations and the object features of the object to be identified; and determine at least one target label with similar features that meets the similarity condition.

Optionally, in the embodiment of the present application, the second extraction module 52 is specifically used to extract the object features of the object to be identified by using a multimodal model;

The second determination module 53 is specifically used to determine the label features corresponding to the labels in the multiple groups of label combinations extracted using the multimodal model; the multiple labels obtained include a label list including multiple labels provided by the user.

Optionally, in the embodiment of the present application, the object to be identified is an image to be identified, and the device further includes: a second search module 55 and a second verification module 56;

The second determination module 53 is also used to determine the image category of the image to be identified based on the at least one target tag; or, the second search module 55 is used to search for a target image matching the image to be identified based on the at least one target tag; or; the second verification module 56 is used to verify whether the image to be identified meets the image requirements based on the at least one target tag.

The object processing device shown in FIG5 can execute the object processing method shown in the embodiment shown in FIG2, and its implementation principle and technical effect are not described in detail. The specific manner in which each module and unit performs operations in the object processing device in the above embodiment has been described in detail in the embodiment of the method, and will not be described in detail here.

The embodiment of the present application further provides a computing device, as shown in FIG6 , the computing device may include a storage component 61 and a processing component 62;

The storage component 61 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component to obtain multiple labels; group the labels in the multiple labels according to a hierarchical relationship to obtain multiple label combinations; extract label features corresponding to the labels in the multiple label combinations; wherein the multiple label combinations are used to match the object to be identified in a descending order of hierarchy based on the label features and the object features of the object to be identified to obtain at least one target label of the object to be identified.

Of course, the computing device may also include other components, such as input/output interfaces, display components, communication components, etc.

The input/output interface provides an interface between the processing component 62 and a peripheral interface module, which may be an output device, an input device, etc. The communication component is configured to facilitate wired or wireless communication between the computing device and other devices.

The processing component 62 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

The storage component is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

The display component may be an electroluminescent (EL) element, a liquid crystal display or a micro display having a similar structure, or a retinal direct display or a similar laser scanning display.

It should be noted that when the computing device implements the tag processing method shown in FIG1 or the object processing method shown in FIG2, it can be a physical device or an elastic computing host provided by a cloud computing platform, etc. It can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device.

The embodiment of the present application also provides a computer-readable storage medium storing a computer program, which can implement the tag processing method shown in FIG1 or the object processing method shown in FIG2 when executed by a computer. The computer-readable medium can be included in the computing device described in the above embodiment; or it can exist independently without being assembled into the computing device.

The embodiment of the present application also provides a computer program product, which includes a computer program carried on a computer-readable storage medium, and when the computer program is executed by a computer, it can implement the label processing method shown in Figure 1 or the object processing method shown in Figure 2. In such an embodiment, the computer program can be downloaded and installed from a network, and/or installed from a removable medium. When the computer program is executed by a processor, various functions defined in the system of the present application are executed.

It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described in this article within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims (13)

1. A tag processing method, comprising:

Acquiring a plurality of labels provided by a user;

Grouping the labels in the labels according to a hierarchical relationship to obtain a plurality of groups of label combinations, wherein the labels in the same group of label combinations belong to the same hierarchy;

Extracting label characteristics corresponding to labels in the plurality of groups of label combinations respectively;

grouping and storing the tag characteristics;

the multi-group label combination is used for matching with the object to be identified based on the label characteristics and the object characteristics of the object to be identified according to the order of the hierarchy from high to low so as to obtain at least one target label of the object to be identified;

the plurality of groups of label combinations are matched with the object to be identified specifically in the following mode:

Based on the tag characteristics and the object characteristics, matching the object to be identified with the tag in the highest-level tag combination to obtain a successfully matched tag;

determining at least one label which has an upper-lower relationship with a successfully matched label in a previous-level label combination in the current-level label combination from the next-level label combination of the highest-level label combination;

matching the at least one tag with the object to be identified to obtain a successfully matched tag;

And using at least one successfully matched tag corresponding to the plurality of groups of tag combinations as at least one target tag corresponding to the object to be identified.

2. The method of claim 1, wherein grouping the tags of the plurality of tags according to a hierarchical relationship, obtaining a plurality of groups of tag combinations comprises:

identifying a hierarchy to which a tag of the plurality of tags belongs;

And combining the labels of the same level to obtain at least one group of label combinations.

3. The method of claim 2, wherein the identifying the belonging hierarchy of tags in the plurality of tags comprises:

And inquiring the upper words of the labels in the vocabulary database until reaching a root node, and determining the belonging level of the labels according to the path length from the labels to the root node.

4. The method of claim 3, wherein querying the vocabulary database for the hypernym of the tag in the plurality of tags until the root node is reached comprises:

Determining a synonym set of tags in the vocabulary database;

And searching the upper words of the synonym set in the vocabulary database step by step until reaching a root node.

5. The method of claim 1, wherein after the obtaining the plurality of tags, the method further comprises:

And screening out the labels meeting the screening condition from the labels so as to update the labels.

6. The method of claim 1, wherein the obtaining a plurality of tags comprises:

Acquiring a tag list comprising a plurality of tags provided by a user;

The extracting the label characteristics corresponding to the labels in the plurality of groups of label combinations respectively comprises the following steps:

And extracting label characteristics corresponding to labels in the plurality of groups of label combinations by using a multi-modal model, wherein the multi-modal model is a pre-training model.

7. An object processing method, comprising:

Acquiring an object to be identified;

Extracting object characteristics of the object to be identified;

determining the label characteristics corresponding to the labels in the extracted multiple groups of label combinations, wherein the multiple groups of label combinations are obtained by grouping the labels in the acquired multiple labels according to a hierarchical relationship, and the labels in the same group of label combinations belong to the same hierarchy;

Based on the tag characteristics and the object characteristics, matching the object to be identified with the tag in the highest-level tag combination to obtain a successfully matched tag;

determining at least one label which has an upper-lower relationship with a successfully matched label in a previous-level label combination in the current-level label combination from the next-level label combination of the highest-level label combination;

matching the at least one tag with the object to be identified to obtain a successfully matched tag;

And using at least one successfully matched tag corresponding to the plurality of groups of tag combinations as at least one target tag corresponding to the object to be identified.

8. The method of claim 7, wherein the matching the plurality of sets of tag combinations with the object to be identified based on the object features and the tag features, respectively, to obtain at least one target tag corresponding to the object to be identified comprises:

Calculating feature similarity between the tag features corresponding to the tags in the plurality of groups of tag combinations and the object features of the object to be identified respectively;

At least one target tag satisfying a similarity condition is determined that features are similar.

9. The method of claim 7, wherein the extracting object features of the object to be identified comprises:

extracting object characteristics of the object to be identified by utilizing a multi-modal model;

the determining the tag characteristics corresponding to the tags in the extracted plurality of groups of tag combinations comprises the following steps:

And determining the tag characteristics corresponding to the tags in the multi-group tag combination extracted by using the multi-mode model, wherein the acquired plurality of tags comprise a tag list provided by a user and comprising a plurality of tags.

10. The method of claim 7, wherein the object to be identified is an image to be identified, the method further comprising:

determining an image category of the image to be identified based on the at least one target tag;

Or alternatively;

Searching for a target image matching the image to be identified based on the at least one target tag;

Or alternatively;

and verifying whether the image to be identified meets the image requirement or not based on the at least one target label.

11. A computing device comprising a processing component and a storage component, the storage component storing one or more computer instructions for execution by the processing component to implement the tag processing method of any of claims 1-6 or the object processing method of any of claims 7-10.

12. A computer-readable storage medium, on which a computer program is stored which, when executed by a processing component, implements the tag processing method of any one of claims 1 to 6 or the object processing method of any one of claims 7 to 10.

13. A computer program product comprising computer programs/instructions which when executed by a processing component implement the method of tag processing according to any one of claims 1 to 6 or the method of object processing according to any one of claims 7 to 10.