CN113516735B

CN113516735B - Image processing method, device, computer readable medium and electronic device

Info

Publication number: CN113516735B
Application number: CN202110035971.XA
Authority: CN
Inventors: 贺思颖; 李敏睿; 古丽敏; 谭杰; 朱禹宏; 涂金林; 李松南
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-01-12
Filing date: 2021-01-12
Publication date: 2025-01-28
Anticipated expiration: 2041-01-12
Also published as: CN113516735A

Abstract

The embodiments of the present application provide an image processing method, device, computer-readable medium and electronic device. The image processing method includes: performing target detection on the image to be processed; if it is detected that the image to be processed contains a target object, generating a heat map according to the feature map of the image to be processed; determining the target pixel in the heat map according to the thermal value of each pixel in the heat map, and determining the display area corresponding to the position of the target pixel in the image to be processed according to the position of the target pixel; adding a first multimedia element in the display area. The technical solution of the embodiment of the present application effectively improves the efficiency of image processing.

Description

Image processing method, device, computer readable medium and electronic equipment

Technical Field

The present application relates to the field of image processing technologies, and in particular, to an image processing method, an image processing device, a computer readable medium, and an electronic apparatus.

Background

With the advent of the 5G era, the live broadcast technology is increasingly popular in the industries of entertainment, education and the like, and a plurality of novel and interesting new playing methods, such as live broadcast interaction, personalized stickers and the like, are developed, and lively and interesting stickers are attached to user head portraits in real time, so that the user experience can be enhanced.

However, the existing image processing method related to the application of the sticker has the defects of overlong algorithm flow, overlarge calculated amount, overlarge redundant calculation and the like.

Disclosure of Invention

The embodiment of the application provides an image processing method, an image processing device, a computer readable medium and electronic equipment, and further can effectively improve the image processing efficiency at least to a certain extent.

Other features and advantages of the application will be apparent from the following detailed description, or may be learned by the practice of the application.

According to one aspect of the embodiment of the application, an image processing method is provided, and the method comprises the steps of carrying out target detection on an image to be processed, generating a thermodynamic diagram according to a feature diagram of the image to be processed if the image to be processed contains a target object, determining target pixel points in the thermodynamic diagram according to the magnitude of thermodynamic values of all pixel points in the thermodynamic diagram, determining a display area corresponding to the position of the target pixel points in the image to be processed according to the position of the target pixel points, and adding a first multimedia element into the display area.

According to one aspect of the embodiment of the application, an image processing device is provided, and the image processing device comprises a detection unit, a generation unit, a first determination unit and a first adding unit, wherein the detection unit is used for carrying out target detection on an image to be processed, the generation unit is used for generating a thermodynamic diagram according to a feature diagram of the image to be processed if the image to be processed contains a target object, the first determination unit is used for determining target pixel points in the thermodynamic diagram according to the magnitude of thermodynamic values of all pixel points in the thermodynamic diagram, and determining a display area corresponding to the position of the target pixel points in the image to be processed according to the position of the target pixel points, and the first adding unit is used for adding a first multimedia element in the display area.

In some embodiments of the present application, based on the foregoing solution, the first determining unit includes a first determining subunit configured to determine, according to a magnitude of a thermal value of each pixel in the thermodynamic diagram, a pixel having a maximum thermal value in the thermodynamic diagram, and a second determining subunit configured to use, as the target pixel, the pixel having the maximum thermal value in the thermodynamic diagram.

In some embodiments of the present application, based on the foregoing, the second determining subunit is configured to use the pixel with the largest thermal value as the target pixel if the pixel with the largest thermal value is in the designated area of the thermodynamic diagram, and determine that the target pixel is not present in the thermodynamic diagram if the pixel with the largest thermal value is not in the designated area.

In some embodiments of the application, based on the foregoing, the apparatus further comprises a second adding unit configured to add a second multimedia element within a predetermined area in the image to be processed if it is determined that the target pixel point does not exist in the thermodynamic diagram.

In some embodiments of the application, based on the foregoing, the generating unit is configured to generate the thermodynamic diagram based on feature values of a partial region specified in a feature map of the image to be processed.

In some embodiments of the present application, based on the foregoing solutions, the apparatus further includes an input unit configured to input the image to be processed into a pre-trained feature extraction model, where the feature extraction model includes a plurality of feature extraction layers, and a selection unit configured to select a feature map output by any one of the feature extraction layers in the feature extraction model as the feature map of the image to be processed, or generate the feature map of the image to be processed according to the feature maps respectively output by the plurality of feature extraction layers in the feature extraction model.

In some embodiments of the present application, based on the foregoing scheme, the apparatus further includes an acquisition unit configured to acquire a first probability that the target object is included in the image to be processed and a second probability that the target object is not included in the image to be processed, a calculation unit configured to calculate a first ratio of the first probability to the second probability and a second ratio of the second probability to the first probability, and a second determination unit configured to determine whether the target object is included in the image to be processed based on a relationship between the first ratio and the second ratio.

In some embodiments of the present application, based on the foregoing solutions, the obtaining unit includes an extracting subunit configured to perform feature extraction on the image to be processed to obtain a feature vector of the image to be processed, and a classifying subunit configured to classify the image to be processed based on the feature vector to obtain a first probability that the image to be processed includes the target object and a second probability that the image to be processed does not include the target object.

In some embodiments of the present application, based on the foregoing scheme, the extracting subunit is configured to perform feature extraction on the image to be processed by using a pre-trained feature extraction model, to obtain features output by each feature extraction layer of the feature extraction model, and generate a feature vector of the image to be processed based on the features output by each feature extraction layer.

In some embodiments of the present application, based on the foregoing scheme, the classifying subunit is configured to input the feature vector of the image to be processed into a predetermined classifier, and obtain a classification result output by the classifier, where the classification result includes a first probability that the image to be processed includes the target object and a second probability that the image to be processed does not include the target object.

In some embodiments of the present application, based on the foregoing solution, the first determining unit is configured to map, according to a mapping relationship between the thermodynamic diagram and the image to be processed, a position of a target pixel point in the thermodynamic diagram into the image to be processed, to obtain a target position corresponding to the position of the target pixel point in the image to be processed, and to use a region including a preset size of the target position as a display region corresponding to the position of the target pixel point.

In some embodiments of the application, based on the foregoing, the apparatus further comprises a third adding unit configured to add a second multimedia element within a predetermined area in the image to be processed if it is determined that the target object is not included in the image to be processed.

According to an aspect of the embodiments of the present application, there is provided a computer-readable medium having stored thereon a computer program which, when executed by a processor, implements the image processing method as described in the above embodiments.

According to an aspect of an embodiment of the present application, there is provided an electronic device including one or more processors, and storage means for storing one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the image processing method as described in the above embodiment.

According to an aspect of embodiments of the present application, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions so that the computer device performs the image processing methods provided in the above-described various alternative embodiments.

In the technical solutions provided in some embodiments of the present application, when a target object is detected to be included in an image to be processed, a thermodynamic diagram is generated according to a feature diagram of the image to be processed, then, according to the magnitude of a thermodynamic value of each pixel point in the thermodynamic diagram, a target pixel point in the thermodynamic diagram is determined, and according to the position of the target pixel point, a display area corresponding to the position of the target pixel point is determined in the image to be processed, so that the purpose of adding a first multimedia element in the display area is achieved. According to the technical scheme provided by the embodiment of the application, the thermodynamic diagram is generated through the characteristic diagram of the image to be processed, and then the display area is determined according to the position of the target pixel point in the thermodynamic diagram, so that the first multimedia element is added in the display area, the problems of large calculation amount, long algorithm flow and the like in the image processing process in the prior art are overcome, the labor cost is saved, the image processing efficiency is effectively improved, and the method is applicable to any image file and has wide applicability.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application as claimed.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application. It is evident that the drawings in the following description are only some embodiments of the present application and that other drawings may be obtained from these drawings without inventive effort for a person of ordinary skill in the art. In the drawings:

FIG. 1 shows a schematic diagram of an exemplary system architecture to which embodiments of the present application may be applied;

FIG. 2 shows a flow chart of an image processing method according to one embodiment of the application;

FIG. 3 shows a flow chart of an image processing method according to one embodiment of the application;

FIG. 4 shows a flow chart of an image processing method according to one embodiment of the application;

FIG. 5 shows a flow chart of an image processing method according to one embodiment of the application;

FIG. 6 shows a flow chart of an image processing method according to one embodiment of the application;

7A-7B illustrate a thermodynamic diagram and a schematic diagram of an image to be processed according to an embodiment of the present application;

FIG. 8 shows a logic diagram of an image processing method according to one embodiment of the application;

Fig. 9 shows a block diagram of an image processing apparatus according to an embodiment of the present application;

Fig. 10 shows a schematic diagram of a computer system suitable for use in implementing an embodiment of the application.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments can be embodied in many different forms and should not be construed as limited to the examples set forth herein, but rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the application may be practiced without one or more of the specific details, or with other methods, components, devices, steps, etc. In other instances, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the application.

It should be noted that the terms used in the description of the present application and the claims and the above-mentioned drawings are only used for describing the embodiments, and are not intended to limit the scope of the present application. It will be understood that the terms "comprises," "comprising," "includes," "including" and/or "having," when used herein, specify the presence of stated features, integers, steps, operations, elements, components, and/or groups thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It will be further understood that, although the terms "first," "second," "third," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first element could be termed a second element without departing from the scope of the present invention. Similarly, the second element may be referred to as a first element. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

The block diagrams depicted in the figures are merely functional entities and do not necessarily correspond to physically separate entities. That is, the functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.

The flow diagrams depicted in the figures are exemplary only, and do not necessarily include all of the elements and operations/steps, nor must they be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the order of actual execution may be changed according to actual situations.

It should be noted that the term "plurality" as used herein means two or more. "and/or" describes the association relationship of the association object, and indicates that there may be three relationships, for example, a and/or B may indicate that there are three cases of a alone, a and B together, and B alone. The character "/" generally indicates that the context-dependent object is an "or" relationship.

Fig. 1 shows a schematic diagram of an exemplary system architecture to which the technical solution of an embodiment of the present application may be applied.

As shown in fig. 1, the system architecture 100 may include a terminal device 102 and a server 104, where each of the terminal device 102 and the server 104 may be separately configured to perform the image processing method provided in the embodiment of the present application. The terminal device 102 and the server 104 may also cooperatively execute the image processing method provided in the embodiment of the present application through interaction.

In the present application, the terminal device 102 may be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal device 102 and the server 104 may be directly or indirectly connected through wired or wireless communication, and the present application is not limited herein.

The server 104 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, a content delivery network (Content Delivery Network, CDN), basic cloud computing services such as big data and an artificial intelligent platform.

In the following, the image processing method according to the embodiment of the present application will be described with reference to the terminal device 102 as an execution body, in one embodiment of the present application, the terminal device 102 may perform target detection on an image to be processed, where the image to be processed may be an image obtained by the terminal device 102 from a local or other database, if the terminal device 102 detects that the image to be processed includes a target object, a thermodynamic diagram may be further generated according to a feature map of the image to be processed, then the terminal device 102 may determine a target pixel in the thermodynamic diagram according to a magnitude of a thermodynamic value of each pixel in the thermodynamic diagram, after determining the target pixel, the terminal device 102 may determine a display area corresponding to the location of the target pixel in the image to be processed according to the location of the target pixel, and finally, the terminal device 102 may add the first multimedia element into the display area. The first multimedia element may be a still image, a moving image (such as a dynamic GIF file), or a flash, a video, a three-dimensional animation file, etc.

The image processing method provided by the embodiment of the application relates to the technologies of artificial intelligence, such as computer vision technology, machine learning and the like. Wherein:

Artificial intelligence (ARTIFICIAL INTELLIGENCE, AI) is the theory, method, technique, and application system that simulates, extends, and extends human intelligence using a digital computer or a machine controlled by a digital computer, perceives the environment, obtains knowledge, and uses the knowledge to obtain optimal results. Artificial intelligence infrastructure technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and other directions.

Computer Vision (CV) is a science of how to make a machine "look at", and more specifically, to replace human eyes with a camera and a Computer to perform machine Vision such as recognition, tracking and measurement on a target, and further perform graphic processing, so that the Computer processes the target into an image more suitable for human eyes to observe or transmit to an instrument to detect. As a scientific discipline, computer vision research-related theory and technology has attempted to build artificial intelligence systems that can acquire information from images or multidimensional data. Computer vision techniques typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, three-dimensional object reconstruction, 3D techniques, virtual reality, augmented reality, synchronous positioning, and map construction, among others, as well as common biometric recognition techniques such as face recognition, fingerprint recognition, and others.

Machine learning (MACHINE LEARNING, ML) is a multi-domain interdisciplinary, involving multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and the like. It is specially studied how a computer simulates or implements learning behavior of a human to acquire new knowledge or skills, and reorganizes existing knowledge structures to continuously improve own performance. Machine learning is the core of artificial intelligence, a fundamental approach to letting computers have intelligence, which is applied throughout various areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, confidence networks, reinforcement learning, transfer learning, induction learning, teaching learning, and the like.

The present application will be described with reference to the following examples in terms of computer vision techniques, machine learning techniques, and the like.

In one embodiment, as shown in fig. 2, an image processing method is provided, which is exemplified as the method applied to a computer device. The computer device may be in particular the terminal device 102 or the server 104 in fig. 1. The image processing method includes the steps of:

Step S210, performing target detection on an image to be processed;

Step S220, if the fact that the image to be processed contains the target object is detected, generating a thermodynamic diagram according to the feature diagram of the image to be processed;

step S230, determining a target pixel point in the thermodynamic diagram according to the thermodynamic value of each pixel point in the thermodynamic diagram, and determining a display area corresponding to the position of the target pixel point in the image to be processed according to the position of the target pixel point;

step S240, adding a first multimedia element in the display area.

These steps are described in detail below.

In step S210, object detection is performed on an image to be processed.

In this embodiment, the target detection may be performed on the image to be processed first, where the target detection is used to detect whether a target object exists in the image to be processed, the target object may include, but is not limited to, a face, an animal, a building, a virtual object (game role, animation role, etc.), or other entity objects, and the target object may be one or more different objects, specifically may depend on an actual application scenario, for example, when the image to be processed is an image related to a face, the target object may be a face in the image to be processed, when the image to be processed is an image related to a building, the target object may be a building in the image to be processed, and when the image to be processed is an image related to a classroom, the target object may include a blackboard, a chalk, a desk, or the like, and may also include a person.

The image to be processed may be obtained in real time, for example, an image obtained by capturing in real time through a camera of the terminal device, or may be an image obtained from a database, a blockchain or a distributed file system, which is not limited in the embodiment of the present application. The image to be processed can be a single image or a frame in video.

The target detection of the image to be processed can be realized by adopting the existing target detection algorithm, and the target detection algorithm is mainly divided into two main types, namely a single-step (one-stage) method and a two-step (two-stage) method. The single-step method mainly outputs the coordinates (i.e. the predicted frame including the target object) and the confidence of the target object directly through a plurality of feature maps, and the main representative algorithms of the single-step method include Yolo algorithm and single-step multi-frame detector (Single Shot MultiBox Detector, SSD), wherein YoLo algorithm is a one-stage rapid target detection algorithm. YoLo dividing the whole graph into S x S grids, wherein each grid is responsible for target detection in the grid, and predicting coordinate values, confidence degrees and all category probability vectors of target objects contained in all grids at one time is adopted to solve the problem at one time. The SSD algorithm is a one-stage target detection algorithm, which uses convolutional neural networks to detect directly, rather than after the full connection layer as in Yolo.

The two-step method generally comprises two steps, wherein the first step is to process the image to obtain a plurality of candidate regions (i.e. Proposals), and then to finely classify the candidate regions and to perform coordinate regression to obtain the final result. However, because of the huge amount of candidate areas, in order to eliminate redundant candidate areas, find the optimal object detection position and accelerate the speed of target detection and recognition, a method such as a Non-maximum suppression (Non-maximum suppression, NMS) algorithm is generally selected to remove redundant candidate areas. The main representative algorithms of the two-step method are R-CNN system algorithms (R-CNN, fast R-CNN, face R-CNN, etc.).

In step S220, if it is detected that the image to be processed includes the target object, a thermodynamic diagram is generated according to the feature map of the image to be processed.

If the target object is detected to be contained in the image to be processed, a thermodynamic diagram can be further generated according to the feature diagram of the image to be processed. The generated thermodynamic diagram is a diagram for displaying image characteristic information of the image to be processed in a preset display form, wherein the preset display form can comprise a special highlighting form, a designated display color form and the like, and the image characteristic information of the image to be processed comprises characteristic information of a target object in the image to be processed.

The thermodynamic diagram can be generated according to the feature map of the image to be processed, and the thermodynamic diagram can be generated by firstly marking the feature points of the feature map of the image to be processed, wherein the feature points can be feature points on a target object in the image to be processed and generate feature point coordinates, then determining a thermodynamic area corresponding to the feature point coordinates by taking the feature point coordinates as a center and taking a preset distance threshold as a side length, further calculating the thermodynamic value of each pixel point in the thermodynamic area according to the thermodynamic weight corresponding to each pixel point in the thermodynamic area and the feature value corresponding to each pixel point, and displaying colors corresponding to the thermodynamic value of each pixel point in the thermodynamic area according to the thermodynamic value of each pixel point in the thermodynamic area. The thermal weight corresponding to each pixel point in the thermal area can be determined according to the distance between each pixel point in the thermal area and the center point of the thermal area, and the farther the pixel point is from the center point, the smaller the corresponding thermal value is, and the thermal value of the pixel point at the boundary of the thermal area is decreased to 0.

The thermal area may be a square area, or may be a triangle, a circle, a regular polygon or an irregular polygon, and the specific shape of the thermal area may be determined according to actual needs, which is not limited herein.

In some embodiments, after determining the thermal area, an initial thermal value may be set for the feature point coordinate, and the initial thermal value is used as a mean value, then the mean value and a preset variance are input into a thermal value calculation function, the thermal value of each pixel point in the thermal area is calculated, and finally, according to the thermal value of each pixel point in the thermal area, a color corresponding to the thermal value of each pixel point is presented in the thermal area, so as to generate a thermodynamic diagram. In particular, the thermodynamic value computation function may be a gaussian function.

In step S230, a target pixel point in the thermodynamic diagram is determined according to the magnitude of the thermodynamic value of each pixel point in the thermodynamic diagram, and a display area corresponding to the position of the target pixel point is determined in the image to be processed according to the position of the target pixel point.

After generating the thermodynamic diagram, the target pixel point in the thermodynamic diagram can be further determined according to the thermodynamic value of each pixel point in the thermodynamic diagram, and a display area corresponding to the position of the target pixel point is determined in the image to be processed according to the position of the target pixel point.

It can be understood that the thermodynamic diagram is a diagram of displaying image feature information of an image to be processed in a preset display form, the magnitude of a thermodynamic value can reflect the feature significance of each pixel point, and the computer device can directly determine a target pixel point according to the magnitude of the thermodynamic value of each pixel point in the thermodynamic diagram, in other words, the computer device can position the target pixel point to the pixel point with significant features in the image to be processed by selecting the target pixel point, so that a display area corresponding to the position of the target pixel point, namely, a region with significant features in the image to be processed is determined in the image to be processed.

The meaning of feature saliency can be understood as that the difference between the pixel value of the pixel point and the pixel values of a plurality of surrounding pixel points is larger than the preset pixel difference, and the feature saliency of each pixel point of the target object in the image to be processed is stronger than the feature saliency of each pixel point except the target object.

In step S240, a first multimedia element is added within the display area.

The first multimedia element can be used for being added into a display area in the image to be processed, the expression form of the image can be enriched by adding the first multimedia element into the image to be processed, and fresh and interesting feeling is brought to a user.

The first multimedia element may be a static image file, a dynamic image file (such as a dynamic GIF file), or a flash, video, three-dimensional animation file, or may be apparel or a scene from the content, for example, the first multimedia element may be a decorative element of a human face, such as a hat, a fan, handkerchiefs, glasses, etc., a constituent element of a human face, such as a specially-made mouth, eyebrows, beards, eyes, etc., and a background element, such as a landscape, a smiling background, etc.

After the display area is determined, a first multimedia element may be added within the display area. In a specific implementation, the first multimedia element may be added in the display area according to a preset manner, where the preset manner may be a manner of aligning a center of the first multimedia element with a center of the display area, or may be a manner of executing addition of the corresponding first multimedia element according to a target object in the display area, for example, when the target object is a person, an eyeglass element serving as the first multimedia element may be added to an eye portion of the target object in the display area, or may be other manners, which are not limited herein specifically.

Based on the technical scheme of the embodiment, the thermodynamic diagram is generated through the feature diagram of the image to be processed, and then the display area is determined according to the position of the target pixel point in the thermodynamic diagram, so that the first multimedia element is added in the display area, the problems of large calculation amount, long algorithm flow and the like in the image processing process in the prior art are overcome, the labor cost is saved, the image processing efficiency is effectively improved, and the method is applicable to any image file and has wide applicability.

In one embodiment of the present application, if the target object is not included in the image to be processed, an operation of adding the second multimedia element within a predetermined area in the image to be processed may be performed. In this embodiment, unlike the case where the target object is included in the image to be processed, in the case where the target object is included in the image to be processed, the display area may be further determined in the image to be processed, and in the case where the target object is not included in the image to be processed, the second multimedia element may be directly added in the predetermined area in the image to be processed.

The predetermined area may be a fixed area preset in the image to be processed, and the second multimedia element may be any multimedia element that may be used for display, including but not limited to a still image, a moving image, a flash, a video, a three-dimensional animation file, and the like.

It should be noted that the second multimedia element may be used to prompt that the image to be processed does not include the target object, for example, when the target object is a human face, the image with the hollowed-out portion may be added as the second multimedia element to the predetermined area, where the hollowed-out portion forms a shape of the human face, so that it may be prompted that the image to be processed does not include the target object.

In one embodiment of the present application, as shown in fig. 3, determining the target pixel point in the thermodynamic diagram may specifically include steps S310-S320:

Step S310, determining the pixel point with the largest thermodynamic value in the thermodynamic diagram according to the thermodynamic value of each pixel point in the thermodynamic diagram;

In step S320, the pixel with the largest thermodynamic diagram is taken as the target pixel.

In this embodiment, the pixel with the largest thermodynamic value in the thermodynamic diagram may be determined according to the thermodynamic value of each pixel in the thermodynamic diagram, and then the pixel with the largest thermodynamic value in the thermodynamic diagram may be used as the target pixel.

In one embodiment of the present application, after determining the pixel with the maximum thermodynamic value in the thermodynamic diagram, it may be further determined whether the pixel with the maximum thermodynamic value is in the specified area of the thermodynamic diagram, and if the pixel with the maximum thermodynamic value is in the specified area of the thermodynamic diagram, the pixel with the maximum thermodynamic value may be used as the target pixel.

In this embodiment, since the meaning of adding the first multimedia element in the edge area is not great in consideration of the situation that the target object may be in the edge area of the image to be processed, only the pixel with the largest thermal value is in the designated area of the thermodynamic diagram, the pixel with the largest thermal value is taken as the target pixel, and further the display area is determined according to the target pixel, and the first multimedia element is added in the display area. Otherwise, if the pixel point with the largest thermodynamic value is not in the designated area of the thermodynamic diagram, it may be determined that the target pixel point does not exist in the thermodynamic diagram.

In one embodiment of the application, if it is determined that the target pixel point is not present in the thermodynamic diagram, and thus the display area corresponding to the position of the target pixel point in the image to be processed does not need to be determined according to the target pixel point, then in this case the second multimedia element may be added directly in the predetermined area in the image to be processed.

In one embodiment of the present application, since the probability that the target object is in the edge region of the image to be processed is small in practical application, when generating the thermodynamic diagram according to the feature map of the image to be processed, the thermodynamic diagram may be generated only for the partial region specified in the feature map, so that the calculation amount in the image processing process may be further reduced.

Specifically, in this embodiment, the computer device may first obtain the feature value of the specified partial area in the feature map of the image to be processed, where the feature value is a value corresponding to each feature point in the specified partial area, and the computer device may calculate the feature value of each feature point through a preset algorithm. The designated partial region may be determined according to actual conditions, and is not particularly limited herein.

It will be understood that the designated partial area is a thermal area, so after the feature value of the designated partial area is obtained, the thermal value of each pixel point may be calculated according to the feature value corresponding to each pixel point in the designated partial area, and then, the color corresponding to the thermal value of each pixel point is presented in the designated partial area according to the thermal value of each pixel point, so as to generate a thermodynamic diagram.

In one embodiment of the present application, the feature map of the image to be processed may be obtained by inputting the image to be processed into a pre-trained feature extraction model to perform feature extraction. Specifically, the feature extraction model may include a plurality of feature extraction layers, where each feature extraction layer is configured to perform feature processing on an input image to be processed, and output a feature map.

The feature extraction model is obtained by training a model formed by a neural network, and the neural network model can be a convolutional neural network model (Convolutional Neural Network, CNN), a deep neural network model (Deep Neural Network, DNN), a cyclic neural network model (Recurrent Neural Network, RNN) and the like, and the embodiment of the application is not particularly limited herein.

Wherein the convolutional neural network comprises a convolutional layer and a pooling layer. The deep neural network comprises an input layer, an implicit layer and an output layer, wherein the layers are in full-connection relation. A recurrent neural network is a neural network modeling sequence data, i.e. a sequence's current output is also related to the previous output. The specific expression is that the network will memorize the previous information and apply it to the calculation of the current output, i.e. the nodes between the hidden layers are no longer connectionless but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment.

In this embodiment, the feature extraction is performed by inputting the image to be processed into a pre-trained feature extraction model to obtain the feature map, which may be the feature map output by any one of the feature extraction layers in the feature extraction model, or may be the feature map of the image to be processed, or may be generated according to the feature maps respectively output by a plurality of the feature extraction layers in the feature extraction model, for example, the feature maps respectively input by the plurality of feature extraction layers are fused to obtain the feature map of the image to be processed.

In one embodiment of the present application, as shown in fig. 4, whether the image to be processed contains the target object may be determined by a first probability that the image to be processed contains the target object and a second probability that the image to be processed does not contain the target object, and in this embodiment, the method specifically includes steps S410 to S430, which are described in detail below:

step S410, a first probability that the image to be processed contains the target object and a second probability that the image to be processed does not contain the target object are obtained.

In order to determine whether the image to be processed contains the target object, the computer device may obtain in advance a first probability that the image to be processed contains the target object and a second probability that the image to be processed does not contain the target object. It will be appreciated that the image to be processed contains either the target object or no target object, and therefore the sum of the first probability and the second probability is 1.

The method for obtaining the first probability that the image to be processed contains the target object and the second probability that the image to be processed does not contain the target object may be to predict through a probability prediction model, for example, by using a machine learning method, training based on a training image sample set to obtain a probability prediction model, and then obtaining the corresponding probability based on the probability prediction model.

Step S420, a first ratio of the first probability to the second probability and a second ratio of the second probability to the first probability are calculated.

After the first probability and the first probability are obtained, a first ratio of the first probability to the second probability and a second ratio of the second probability to the first probability may be calculated according to the first probability and the second probability.

Step S430, determining whether the image to be processed contains the target object based on the relation between the first ratio and the second ratio.

In this embodiment, after the first ratio and the second ratio are calculated, it may be determined whether the image to be processed includes the target object based on the relationship between the first ratio and the second ratio.

In some embodiments, since the probability problem may be converted into the chance ratio problem of predicting occurrence of an event, after the first ratio and the second ratio are calculated, the log value of the first ratio and the log value of the second ratio may be further calculated, if the log value of the first ratio is greater than or equal to the log value of the second ratio, it may be determined that the target object is included in the image to be processed, and conversely, if the log value of the first ratio is less than the log value of the second ratio, it may be determined that the target object is not included in the image to be processed.

In an embodiment of the present application, the method for obtaining the first probability that the image to be processed contains the target object and the second probability that the image to be processed does not contain the target object may also be obtained by classifying the image to be processed, and in this embodiment, as shown in fig. 5, the method may specifically include step S510 to step S520, which are specifically described as follows:

In step S510, feature extraction is performed on the image to be processed, so as to obtain feature vectors of the image to be processed.

In this embodiment, feature extraction may be performed on an image to be processed first to obtain feature vectors of the image to be processed. The feature vector is a vectorized representation of features of the image, and the features are used for describing characteristics of the image, and can be extracted according to related information of the image to be processed, for example, according to at least one of information such as content contained in the image, and the like, and also can be extracted according to attribute information corresponding to the image to be processed, for example, an author of the image to be processed or a label corresponding to the image to be processed, and the like, so as to obtain corresponding features.

In one embodiment, the feature extraction of the image to be processed may be performed by using a pre-trained feature extraction model to perform feature extraction on the image to be processed, so as to obtain features output by each feature extraction layer of the feature extraction model, and generate feature vectors of the image to be processed based on the features output by each feature extraction layer. The feature vector of the image to be processed consists of multiple layers of features of the image to be processed extracted from the feature extraction model. In the feature extraction model, an image to be processed is input into the lowest layer of the feature extraction model, one feature is extracted in each layer, and the extracted features are sequentially transmitted to the next layer, namely, the extracted features of the previous layer are input of the next layer. The multi-layer characteristics of the image to be processed can be extracted from the whole characteristic extraction model, and the characteristic vector of the image to be processed can be formed according to the multi-layer characteristics.

In step S520, the image to be processed is classified based on the feature vector, so as to obtain a first probability that the image to be processed contains the target object and a second probability that the image to be processed does not contain the target object.

After extracting the feature vector of the image to be processed, the computer device may classify the image to be processed based on the feature vector of the image to be processed, and then obtain a first probability that the image to be processed contains the target object and a second probability that the image to be processed does not contain the target object based on the classification data.

In specific implementation, one implementation manner is that the feature vector of the image to be processed is input into a preset classifier, and the classification result is output by the preset classifier. The classifier is a machine learning model with the capability of classifying images, and the machine learning model can be a model obtained through training of a convolutional neural network model, a cyclic neural network model, a support vector machine model and the like.

The classification result may include probability values corresponding to respective preset classification categories, and the classification result may be a vector, where each vector value in the vector represents a probability corresponding to the respective preset classification category. In this embodiment, the preset classification categories may include two categories that the image to be processed includes the target object and the image to be processed does not include the target object, so after the feature vector of the image to be processed is input into the predetermined classifier, the first probability that the image to be processed includes the target object and the second probability that the image to be processed does not include the target object may be obtained through the classification result output by the classifier.

For example, if the classification result output by the classifier is (0.8.2), the first probability that the image to be processed contains the target object is 0.8, and the second probability that the image to be processed does not contain the target object is 0.2.

In another implementation manner, for each preset classification category, the probability that the image to be processed belongs to each preset classification category may be determined according to the distance between the feature vector and the category feature vector of each preset classification category and the weight of the category feature vector of each preset classification category. The class feature vector may be determined based on training feature vectors of training samples under corresponding preset classification classes.

In one embodiment of the present application, as shown in fig. 6, the method for determining the display area corresponding to the position of the target pixel point may include steps S610 to S620, which are described in detail as follows:

step S610, mapping the position of the target pixel point in the thermodynamic diagram to the image to be processed according to the mapping relation between the thermodynamic diagram and the image to be processed, so as to obtain the target position corresponding to the position of the target pixel point in the image to be processed.

After determining the target pixel point in the thermodynamic diagram, the position of the target pixel point in the thermodynamic diagram can be mapped into the image to be processed directly according to the mapping relation between the thermodynamic diagram and the image to be processed, so as to obtain the target position corresponding to the position of the target pixel point in the image to be processed.

Specifically, the thermodynamic diagram may be regarded as a first matrix composed of a plurality of rows by a plurality of columns of pixels, the image to be processed may be regarded as a second matrix composed of a plurality of rows by a plurality of columns of pixels, a proportional relationship between the thermodynamic diagram and the image to be processed may be determined according to the length and width of the first matrix and the length and width of the second matrix, and a mapping relationship between the thermodynamic diagram and the image to be processed may be obtained based on the proportional relationship.

Therefore, in the embodiment of the present application, if the position of the target pixel is known, the position of the target pixel may be mapped to the image to be processed, so as to obtain the target position corresponding to the position of the target pixel in the image to be processed.

Referring to fig. 7A and 7B, fig. 7A-7B are schematic diagrams of a thermodynamic diagram and an image to be processed according to an embodiment of the present application, where each rectangular block in the thermodynamic diagram represents a pixel, the thermodynamic diagram is a5×5 matrix, a target pixel exists in the thermodynamic diagram, the target pixel is denoted by an letter "C", the target pixel is located at the very center of the first matrix, and each rectangular block in the image to be processed also represents a pixel, and the size of the image to be processed is a 20×20 matrix, as shown in fig. 7B. Therefore, according to the length-width relation of the two matrixes, one pixel point in the thermodynamic diagram can correspond to four pixel points in the image to be processed, and based on the mapping relation, the position of the target pixel point C can be mapped into the image to be processed, so that the target position corresponding to the position of the target pixel point in the image to be processed, namely the black area C', is obtained.

Step S620, using the area of the preset size including the target position as the display area corresponding to the position of the target pixel.

In particular, in consideration of the case where the display area is for adding the first multimedia element and the size of the target position may not be identical to the size of the first multimedia element, in this embodiment, an area including a preset size of the target position may be used as the display area corresponding to the position of the target pixel point.

The region of a predetermined size including the target position may be a region of a predetermined size centered around the target position, or may be a region of a predetermined size not centered around the target position. The preset size may be the same as or larger than the first multimedia element.

Fig. 8 shows a logic diagram of an image processing method according to an embodiment of the present application, which may specifically include the following steps, as shown in fig. 8:

s1, carrying out target detection on an image to be processed, if the image to be processed does not contain a target object, executing a step S2, and if the image to be processed contains the target object, executing a step S3.

S2, adding a second multimedia element in a preset area in the image to be processed.

Wherein the second multimedia element may be used to prompt that the image to be processed does not contain the target object, for example, as shown in an image shown by reference numeral (1) in fig. 8, when the target object is a human face, an image with a hollowed-out portion formed in a shape of the human face may be added as the second multimedia element to a predetermined area, so that it may be prompted that the image to be processed does not contain the target object-human.

S3, generating a thermodynamic diagram according to the feature diagram of the image to be processed, and determining a target pixel point in the thermodynamic diagram according to the thermodynamic value of each pixel point in the thermodynamic diagram.

Referring to fig. 8, an image corresponding to the reference numeral (2) is shown, wherein the letter "C" indicates the target pixel point in the determined thermodynamic diagram.

S4, determining a display area corresponding to the position of the target pixel point in the image to be processed according to the position of the target pixel point. As shown in the image corresponding to the reference numeral (3) in fig. 8, a display area determined from the position of the target pixel point C in the image corresponding to the reference numeral (2) in fig. 8 is shown, and is also denoted by the letter "C".

S5, adding the first multimedia element in the display area.

As shown in the image corresponding to the reference numeral (4) in fig. 8, an image obtained after adding a first multimedia element in the display area is shown, where the image to be processed includes a target object, i.e., a face, and the first multimedia element is an image including a "GOOD" word.

The following describes an embodiment of the apparatus of the present application, which can be used to perform the image processing method in the above-described embodiment of the present application. For details not disclosed in the embodiments of the apparatus of the present application, please refer to the embodiments of the image processing method of the present application.

Fig. 9 shows a block diagram of an image processing apparatus according to an embodiment of the present application, and referring to fig. 9, an image processing apparatus 900 according to an embodiment of the present application includes a detection unit 902, a generation unit 904, a first determination unit 906, and a first addition unit 908.

The device comprises a detection unit 902 configured to detect a target in an image to be processed, a generation unit 904 configured to generate a thermodynamic diagram according to a feature diagram of the image to be processed if the image to be processed contains the target object, a first determination unit 906 configured to determine a target pixel point in the thermodynamic diagram according to a thermodynamic value of each pixel point in the thermodynamic diagram, and determine a display area corresponding to the position of the target pixel point in the image to be processed according to the position of the target pixel point, and a first adding unit 908 configured to add a first multimedia element in the display area.

In some embodiments of the present application, the first determining unit 906 includes a first determining subunit configured to determine, according to the magnitude of the thermal value of each pixel in the thermodynamic diagram, a pixel with a maximum thermal value in the thermodynamic diagram, and a second determining subunit configured to use the pixel with the maximum thermal value in the thermodynamic diagram as the target pixel.

In some embodiments of the present application, the second determining subunit is configured to use the pixel with the largest thermal value as the target pixel if the pixel with the largest thermal value is in the designated area of the thermodynamic diagram, and determine that the target pixel is not present in the thermodynamic diagram if the pixel with the largest thermal value is not in the designated area.

In some embodiments of the application the apparatus further comprises a second adding unit configured to add a second multimedia element within a predetermined area in the image to be processed if it is determined that the target pixel point is not present in the thermodynamic diagram.

In some embodiments of the application the generating unit 904 is configured to generate the thermodynamic diagram based on feature values of a partial region specified in a feature map of the image to be processed.

In some embodiments of the present application, the apparatus further includes an input unit configured to input the image to be processed into a pre-trained feature extraction model, where the feature extraction model includes a plurality of feature extraction layers, and a selection unit configured to select a feature map output by any one of the feature extraction layers in the feature extraction model as a feature map of the image to be processed, or generate a feature map of the image to be processed according to feature maps respectively output by the plurality of feature extraction layers in the feature extraction model.

In some embodiments of the application, the device further comprises an acquisition unit configured to acquire a first probability that the target object is contained in the image to be processed and a second probability that the target object is not contained in the image to be processed, a calculation unit configured to calculate a first ratio of the first probability to the second probability and a second ratio of the second probability to the first probability, and a second determination unit configured to determine whether the target object is contained in the image to be processed based on a relation between the first ratio and the second ratio.

In some embodiments of the present application, the obtaining unit includes an extracting subunit configured to perform feature extraction on the image to be processed to obtain a feature vector of the image to be processed, and a classifying subunit configured to classify the image to be processed based on the feature vector to obtain a first probability that the image to be processed includes the target object and a second probability that the image to be processed does not include the target object.

In some embodiments of the present application, the extraction subunit is configured to perform feature extraction on the image to be processed by using a pre-trained feature extraction model, obtain features output by each feature extraction layer of the feature extraction model, and generate a feature vector of the image to be processed based on the features output by each feature extraction layer.

In some embodiments of the present application, the classifying subunit is configured to input the feature vector of the image to be processed into a predetermined classifier, and obtain a classification result output by the classifier, where the classification result includes a first probability that the image to be processed includes the target object and a second probability that the image to be processed does not include the target object.

In some embodiments of the present application, the first determining unit 906 is configured to map, according to a mapping relationship between the thermodynamic diagram and the image to be processed, a position of a target pixel point in the thermodynamic diagram to the image to be processed, to obtain a target position corresponding to the position of the target pixel point in the image to be processed, and to use a region with a preset size including the target position as a display region corresponding to the position of the target pixel point.

In some embodiments of the application the apparatus further comprises a third adding unit configured to add a second multimedia element within a predetermined area in the image to be processed if it is determined that the target object is not included in the image to be processed.

It should be noted that, the computer system 1000 of the electronic device shown in fig. 10 is only an example, and should not impose any limitation on the functions and the application scope of the embodiments of the present application.

As shown in fig. 10, the computer system 1000 includes a central processing unit (Central Processing Unit, CPU) 1001 that can perform various appropriate actions and processes, such as performing the method described in the above embodiment, according to a program stored in a Read-Only Memory (ROM) 1002 or a program loaded from a storage portion 1008 into a random access Memory (Random Access Memory, RAM) 1003. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, ROM 1002, and RAM 1003 are connected to each other by a bus 1004. An Input/Output (I/O) interface 1005 is also connected to bus 1004.

Connected to the I/O interface 1005 are an input section 1006 including a keyboard, a mouse, and the like, an output section 1007 including a Cathode Ray Tube (CRT), a Liquid crystal display (Liquid CRYSTAL DISPLAY, LCD), and the like, and a speaker, a storage section 1008 including a hard disk, and the like, and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, and the like. The communication section 1009 performs communication processing via a network such as the internet. The drive 1010 is also connected to the I/O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like, is installed as needed in the drive 1010, so that a computer program read out therefrom is installed as needed in the storage section 1008.

In particular, according to embodiments of the present application, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising a computer program for performing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 1009, and/or installed from the removable medium 1011. When executed by a Central Processing Unit (CPU) 1001, the computer program performs various functions defined in the system of the present application.

It should be noted that, the computer readable medium shown in the embodiments of the present application may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of a computer-readable storage medium may include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-Only Memory (ROM), an erasable programmable read-Only Memory (Erasable Programmable Read Only Memory, EPROM), a flash Memory, an optical fiber, a portable compact disc read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present application, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with a computer-readable computer program embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, etc., or any suitable combination of the foregoing.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. Where each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units involved in the embodiments of the present application may be implemented by software, or may be implemented by hardware, and the described units may also be provided in a processor. Wherein the names of the units do not constitute a limitation of the units themselves in some cases.

As another aspect, the present application also provides a computer-readable medium that may be included in the electronic device described in the above embodiment, or may exist alone without being incorporated into the electronic device. The computer-readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to implement the methods described in the above embodiments.

It should be noted that although in the above detailed description several modules or units of a device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functions of two or more modules or units described above may be embodied in one module or unit in accordance with embodiments of the application. Conversely, the features and functions of one module or unit described above may be further divided into a plurality of modules or units to be embodied.

From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present application may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, and includes several instructions to cause a computing device (may be a personal computer, a server, a touch terminal, or a network device, etc.) to perform the method according to the embodiments of the present application.

Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains.

It is to be understood that the application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims

1. An image processing method, the method comprising:

Performing target detection on the image to be processed;

if the image to be processed contains the target object, generating a thermodynamic diagram according to the feature diagram of the image to be processed;

Determining a pixel point with the largest thermodynamic value in the thermodynamic diagram according to the thermodynamic value of each pixel point in the thermodynamic diagram, taking the pixel point with the largest thermodynamic value as a target pixel point in the thermodynamic diagram if the pixel point with the largest thermodynamic value is in a designated area of the thermodynamic diagram, and determining a display area corresponding to the position of the target pixel point in the image to be processed according to the position of the target pixel point;

A first multimedia element is added within the display area.

2. The method according to claim 1, wherein the method further comprises:

And if the pixel point with the maximum thermodynamic value is not in the appointed area, determining that the target pixel point does not exist in the thermodynamic diagram.

3. The method according to claim 2, wherein the method further comprises:

and if the target pixel point does not exist in the thermodynamic diagram, adding a second multimedia element in a preset area in the image to be processed.

4. The method of claim 1, wherein generating a thermodynamic diagram from the feature map of the image to be processed comprises:

And generating the thermodynamic diagram based on the characteristic values of the designated partial areas in the characteristic diagram of the image to be processed.

5. The method according to claim 1, wherein the method further comprises:

inputting the image to be processed into a pre-trained feature extraction model, wherein the feature extraction model comprises a plurality of feature extraction layers;

And selecting a feature image output by any one feature extraction layer in the feature extraction model as the feature image of the image to be processed, or generating the feature image of the image to be processed according to the feature images respectively output by a plurality of feature extraction layers in the feature extraction model.

6. The method according to claim 1, wherein the method further comprises:

acquiring a first probability that the target object is contained in the image to be processed and a second probability that the target object is not contained in the image to be processed;

Calculating a first ratio of the first probability to the second probability and a second ratio of the second probability to the first probability;

And determining whether the image to be processed contains the target object or not based on the relation between the first ratio and the second ratio.

7. The method of claim 6, wherein obtaining a first probability that the target object is included in the image to be processed and a second probability that the target object is not included in the image to be processed comprises:

Extracting the characteristics of the image to be processed to obtain the characteristic vector of the image to be processed;

Classifying the image to be processed based on the feature vector to obtain a first probability that the image to be processed contains the target object and a second probability that the image to be processed does not contain the target object.

8. The method of claim 7, wherein extracting features of the image to be processed to obtain feature vectors of the image to be processed, comprises:

performing feature extraction on the image to be processed by using a pre-trained feature extraction model to obtain features output by each feature extraction layer of the feature extraction model;

and generating the feature vector of the image to be processed based on the features output by the feature extraction layers.

9. The method of claim 7, wherein classifying the image to be processed based on the feature vector to obtain a first probability that the image to be processed contains the target object and a second probability that the image to be processed does not contain the target object comprises:

inputting the feature vector of the image to be processed into a preset classifier;

and obtaining a classification result output by the classifier, wherein the classification result comprises a first probability that the image to be processed comprises the target object and a second probability that the image to be processed does not comprise the target object.

10. The method of claim 1, wherein determining a display area in the image to be processed corresponding to the location of the target pixel point based on the location of the target pixel point comprises:

According to the mapping relation between the thermodynamic diagram and the image to be processed, mapping the position of the target pixel point in the thermodynamic diagram into the image to be processed, and obtaining a target position corresponding to the position of the target pixel point in the image to be processed;

And taking the area with the preset size containing the target position as a display area corresponding to the position of the target pixel point.

11. The method according to claim 1, wherein the method further comprises:

and if the image to be processed does not contain the target object, adding a second multimedia element in a preset area in the image to be processed.

12. An image processing apparatus, characterized in that the apparatus comprises:

A detection unit configured to perform target detection on an image to be processed;

the generating unit is configured to generate a thermodynamic diagram according to the feature diagram of the image to be processed if the image to be processed contains the target object;

The first determining unit comprises a first determining subunit, a second determining subunit and a third determining subunit, wherein the first determining subunit is configured to determine the pixel point with the largest thermodynamic value in the thermodynamic diagram according to the thermodynamic value of each pixel point in the thermodynamic diagram; the second determining subunit is configured to take the pixel with the largest thermodynamic value as a target pixel in the thermodynamic diagram if the pixel with the largest thermodynamic value is in the appointed area of the thermodynamic diagram, and determine a display area corresponding to the position of the target pixel in the image to be processed according to the position of the target pixel;

and a first adding unit configured to add a first multimedia element in the display area.

13. A computer readable medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the image processing method according to any one of claims 1 to 11.

14. An electronic device, comprising:

One or more processors;

storage means for storing one or more programs which when executed by the one or more processors cause the one or more processors to implement the image processing method of any of claims 1 to 11.

15. A computer program product, characterized in that the computer program product comprises computer instructions stored in a computer-readable storage medium, from which computer-readable storage medium a processor of a computer device reads, which computer instructions are executed by a processor, such that the computer device performs the image processing method according to any of claims 1 to 11.