CN112016595A - Image classification method and device, electronic equipment and readable storage medium - Google Patents

Abstract

The disclosure relates to an image classification method and device, an electronic device and a readable storage medium. The method comprises the following steps: acquiring an initial image to be classified; acquiring a depth thermodynamic diagram of the initial image to be classified; the depth thermodynamic diagram refers to an image which represents depth-of-field depth information of an object in an initial image to be classified through different color gamuts; combining the RGB channel image of the initial image to be classified and the depth thermodynamic diagram to obtain a target image to be classified of 4 channels; and classifying the initial image to be classified according to the target image to be classified to obtain whether the image to be classified is an image with a large aperture type. In the embodiment, the accuracy of image classification can be improved by adding the depth thermodynamic diagram containing the depth information of the depth of field as one dimension characteristic of the image to be classified.

Description

Image classification method and apparatus, electronic device, and readable storage medium

technical field

The present disclosure relates to the technical field of image processing, and in particular, to an image classification method and apparatus, an electronic device, and a readable storage medium.

Background technique

At present, more and more users like to use cameras such as SLRs to shoot images and videos. During the shooting process, the photographer will use the Depth of Field (DOF) effect, at this time, high-definition, beautiful, and a certain stereo vision can be obtained. A wide-aperture image or a wide-aperture video of the effect.

In real life, users can also upload the above-mentioned large-aperture images or large-aperture videos to a video platform for sharing, such as Kuaishou. After a user uploads a video, existing video platforms may use a no-reference video quality assessment (NR-VQA) algorithm, such as the currently popular machine learning algorithm, to estimate the video quality score, so as to screen such large-aperture images or videos. It is used as the processing object of subsequent video recommendation algorithms, video super-resolution and other applications.

An important feature of large-aperture images and videos is that in the focal plane (that is, within the depth of field), the object to be photographed is clear, but before and after the focal plane, the incident light will be concentrated and diffused during shooting, resulting in blurred images. A circular area is formed, often referred to as a "diffuse circle". However, when the above machine learning algorithm is applied to large-aperture image or large-aperture video prediction, due to the existence of "circle of confusion", it cannot accurately distinguish whether the object to be predicted is a large-aperture image or a distorted image caused by compression. Make the screening results have a higher error rate.

SUMMARY OF THE INVENTION

The present disclosure provides an image classification method and apparatus, an electronic device, and a readable storage medium to at least solve the problems in the related art.

The technical solutions of the present disclosure are as follows:

According to a first aspect of the embodiments of the present disclosure, an image classification method is provided, including:

Get the initial image to be classified;

obtaining a depth heat map of the initial image to be classified; the depth heat map refers to an image representing depth of field depth information of an object in the initial image to be classified through different color gamuts;

Combining the RGB channel image of the initial to-be-classified image and the depth heatmap to obtain a 4-channel target to-be-classified image;

According to the target to-be-classified image, the initial to-be-classified image is classified to obtain whether the to-be-classified image is a large aperture type image.

Optionally, when the image to be classified is a frame in the video frame sequence of the video to be classified, the method further includes:

Based on the predicted classification of each initial to-be-classified image in the video frame sequence, a predicted classification indicating whether the to-be-classified video is of a large aperture type is obtained.

Optionally, the value of the predicted classification of the video to be classified is an average value of the predicted classification values of the initial images to be classified in the video frame sequence.

Optionally, classifying the initial to-be-classified image according to the target to-be-classified image to obtain whether the to-be-classified image is an image of a large aperture type, including:

Input the target classification image into the image classification prediction model, and determine whether the to-be-classified image is a large aperture type image according to the prediction classification information output by the image classification prediction model, wherein the image classification prediction model is based on the sample The depth heat map of the image and the RGB channel image of the sample image are obtained by training.

Optionally, the output layer of the image classification prediction model includes a Sigmoid function; the Sigmoid function is used to output a continuous value representing the predicted classification, and the closer the value is to 1, it indicates that the initial to-be-classified image is a large aperture type. the greater the probability.

According to a second aspect of the embodiments of the present disclosure, there is provided an image classification apparatus, the apparatus includes: an input module, an acquisition module, and a classification module;

The input module is configured to perform acquisition of an initial image to be classified, and send the initial image to be classified to the acquisition module and the classification module respectively;

The acquisition module is configured to perform acquisition of a depth heat map of the initial image to be classified; the depth heat map refers to an image representing depth of field depth information of an object in the initial image to be classified through different color gamuts;

The classification module is configured to perform combining the RGB channel image of the initial to-be-classified image and the depth heat map to obtain a 4-channel target to-be-classified image; and, according to the target to-be-classified image, perform a The classified images are classified to obtain whether the to-be-classified image is a large aperture type image.

Optionally, when the image to be classified is a frame in the video frame sequence of the video to be classified, the device further includes an output module;

The output module is configured to perform a predictive classification based on each initial to-be-classified image in the video frame sequence to obtain a predictive classification indicating whether the to-be-classified video is a large aperture type.

Optionally, the value of the predicted classification of the video to be classified is an average value of the predicted classification values of the initial images to be classified in the video frame sequence.

Optionally, the classification module includes:

an image input unit configured to perform inputting the target classification image into an image classification prediction model;

The classification determination unit is configured to execute the prediction classification information output according to the image classification prediction model, and determine whether the to-be-classified image is a large aperture type image, wherein the image classification prediction model is based on the depth heat map of the sample image and The RGB channel image of the sample image is obtained by training.

Optionally, the output layer of the image classification prediction model includes a Sigmoid function; the Sigmoid function is used to output a continuous value representing the predicted classification, and the closer the value is to 1, it indicates that the initial to-be-classified image is a large aperture type. the greater the probability.

According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, comprising:

processor;

A memory for storing an executable program of the processor; wherein the processor is configured to execute the executable program in the memory to implement the steps of the above method.

According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, when an executable program in the storage medium is executed, the steps of the above method can be performed.

According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer application program capable of executing the steps of the above-mentioned method when the application program is executed by a processor.

The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

In this embodiment, an initial image to be classified may be obtained; then a depth heat map of the initial image to be classified may be obtained; the depth heat map refers to an image representing depth of field depth information of objects in the initial image to be classified through different color gamuts; Then, combine the RGB channel image of the initial to-be-classified image and the depth heatmap to obtain a 4-channel target to-be-classified image; finally, according to the target to-be-classified image, classify the initial to-be-classified image to Obtain whether the image to be classified is a large aperture type image. In this way, in this embodiment, by adding a depth heat map including depth of field depth information as a dimensional feature of the image to be classified, the accuracy of image classification can be improved.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure and do not unduly limit the present disclosure.

FIG. 1 is a schematic diagram of an optical path for capturing a large aperture image shown in the related art.

FIG. 2 is an effect diagram of a large aperture image shown in the related art.

Fig. 3 is a flowchart of an image classification method according to an exemplary embodiment.

Fig. 4 is a schematic diagram of a deep thermal map according to an exemplary embodiment.

Fig. 5 is a block diagram of an image classification apparatus according to an exemplary embodiment.

Fig. 6 is a flow chart of acquiring a large aperture video data set according to an exemplary embodiment.

Fig. 7 is a block diagram of an image classification apparatus according to an exemplary embodiment.

Fig. 8 is a block diagram of an electronic device according to an exemplary embodiment.

Detailed ways

In order to make those skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

At present, more and more users like to use cameras such as SLRs to shoot images and videos. During the shooting process, the photographer will use the Depth of Field (DOF) effect, at this time, high-definition, beautiful, and a certain stereo vision can be obtained. A wide-aperture image or a wide-aperture video of the effect.

In real life, users can also upload the above-mentioned large-aperture images or large-aperture videos to a video platform for sharing, such as Kuaishou. After a user uploads a video, existing video platforms may use a no-reference video quality assessment (NR-VQA) algorithm, such as the currently popular machine learning algorithm, to estimate the video quality score, so as to screen such large-aperture images or videos. It is used as the processing object of subsequent video recommendation algorithms, video super-resolution and other applications.

An important feature of large-aperture images and videos is that in the focal plane (that is, within the depth of field), the object to be photographed is clear, but before and after the focal plane, the incident light will be concentrated and diffused during shooting, resulting in blurred images. A circular area is formed, often referred to as a "diffuse circle". However, when the above machine learning algorithm is applied to large-aperture image or large-aperture video prediction, due to the existence of "circle of confusion", it cannot accurately distinguish whether the object to be predicted is a large-aperture image or a distorted image caused by compression. Make the screening results have a higher error rate.

The inventor analyzes the large-aperture image based on the principle shown in Figure 1. Due to the depth of field effect, each scene of the large-aperture image will contain different depths. Referring to Figure 2, the object in the focal plane (in the white frame) will be relatively clear, while the focal plane will be relatively clear. Objects that are out of plane (inside the black frame) will be blurred in the image. That is to say, the large-aperture image contains certain depth information of the object, that is, the distance between the object corresponding to the clear area and the camera is consistent, while the relative position distance of the object corresponding to the blurred area from the camera is different.

To this end, an embodiment of the present disclosure provides an image classification method, the inventive concept of which is to acquire depth information of an image to be classified, and classify the image to be classified in combination with the depth information, thereby improving the accuracy of image classification. In practical applications, the above-mentioned depth information may be depth data, depth heat map, depth histogram, etc. of the image to be classified, and each embodiment will be described in the following by taking the depth information as a depth heat map as an example, wherein the depth heat map refers to different color gamuts. to represent the depth of field depth information of the object in the initial image to be classified.

Fig. 3 is a flowchart of an image classification method according to an exemplary embodiment, which can be adapted to electronic devices such as smart phones, servers, and personal computers. Referring to FIG. 3, an image classification method includes steps 31 to 34, wherein:

In step 31, an initial image to be classified is obtained.

In this embodiment, the electronic device may acquire a file to be classified, and the file to be classified may be an image or a video. When the input is an image, an image can be directly obtained as the image to be classified. When the input is video, the electronic device can collect 1 frame at a preset interval, such as every 10 frames, sample at least one video frame from the video stream, and use at least one video frame as an image to be classified for subsequent processing. The classification process of each video frame is the same as the classification process of an image, the difference is that the prediction classification of the video is obtained after processing the prediction classification of at least one video frame, which will be described in detail later, and will not be introduced here.

In step 32, a depth heat map of the initial image to be classified is obtained.

In this embodiment, the electronic device can obtain the depth heat map of the initial image to be classified, and the depth heat map refers to an image representing the depth of field depth information of the object in the initial image to be classified through different color gamuts, and the effect is shown in FIG. 4 . , the left image in Figure 4 represents the initial image to be classified, and the right image in Figure 4 represents the depth heat map. During specific implementation, the electronic device can obtain depth information such as depth data, depth heat map, depth histogram, etc. of the initial image to be classified. If the depth information can be obtained, the corresponding algorithm falls within the protection scope of the present disclosure. In an example, the electronic device may be implemented by using the DeepLens algorithm, and the DeepLens algorithm may include two parts: an encoder and a decoder. The encoder can adopt the pretrained ResNet50 architecture. The decoder is a series of up-sampling modules, while the decoder includes skip connections from the encoder. The resolution of the two layers corresponding to the skip connection is the same. The purpose is to reduce the feature loss of the downsampling operation during the forward calculation of the network and prevent the gradient disappearance during the network gradient propagation. The last layer of the decoder directly outputs to be classified. Predicted depth heatmap of the image.

In step 33, the RGB channel image of the initial to-be-classified image and the depth heat map are combined to obtain a 4-channel target to-be-classified image.

In this embodiment, the electronic device can combine the initial to-be-classified image and the depth heat map, that is, since the initial to-be-classified image is composed of RGB channel images, the electronic device can obtain the red channel image, the green channel image and the blue channel image, and use The color channel image is resized to the same size 224*224*1. Moreover, the electronic device can adjust the depth heatmap to 224*224*1. Then, the electronic device can add the depth heat map as the fourth channel of the initial image to be classified into the RGB channel image, thereby forming a target image to be classified including four dimensions of red, green, blue and depth, namely 224 *224*4.

It is understandable that when the initial image to be classified uses the RGB color system, it can also use the HSI color system, YUV color system, and YCbCr color system. The solution of acquiring the target image to be classified also falls within the protection scope of the present disclosure.

During specific implementation, the electronic device can also normalize each channel image in the target image to be classified. Taking the depth heat map as an example, the normalization method is as follows:

In formula (1), (x, y) represents the coordinates of each pixel in the depth heatmap, and the max() function is used to calculate the maximum value in the original depth heatmap d _i ; The normalized depth heatmap d _i ' is obtained. In this example, the depth thermal value of each pixel can be in the range of [0, 1] through normalization, which is convenient for calculation.

It should be noted that the normalization method of the images of other channels may refer to the normalization method of the depth heat map, which will not be repeated here. Finally, the electronic device can obtain the target image to be classified after each channel is normalized.

In step 34, the initial to-be-classified image is classified according to the target to-be-classified image to obtain whether the to-be-classified image is a large aperture type image.

In this embodiment, the electronic device can obtain the predicted classification of the target image to be classified. Since the target image to be classified substantially reflects the feature information of the initial image to be classified, the above predicted classification can also be used as the prediction of the initial image to be classified. Classification.

The value of the above prediction classification may be a continuous value, and the value range may be (0, 1), and the larger the value of the prediction classification, the higher the probability that the target image to be classified is of the large aperture type. In practical applications, a classification threshold (such as 0.4, which can be adjusted) can be preset. When the predicted classification is greater than or equal to the classification threshold, the predicted classification is a large aperture type, otherwise it is not a large aperture type. At this time, the predicted classification can include: large aperture type, not the wide aperture type. A technical person can select an appropriate solution according to a specific scenario, and the corresponding solution falls within the protection scope of the present disclosure.

In this example, the electronic device can use the ResNet18 algorithm to classify the target image to be classified. At this time, it is necessary to make appropriate adjustments to the input layer and output layer of the ResNet18 algorithm, as shown in Table 1.

In Table 1, Cov2d, 7×7 is a convolution operation with a kernel size of 7, Max pooling, 3×3 is a max pooling operation with a kernel size of 3, and Ave pooling, 3×3 is a kernel size A mean pooling operation of size 3, Bottleneck represents a residual module.

Referring to Table 1, the input layer of the ResNet18 algorithm in this example includes 4 input channels, that is, the input dimension of the original ResNet18 algorithm is modified from 224×224×3 to 224×224×4 to correspond to the RGB channels and RGB channels of the target image to be classified. depth channel.

Table 1 Comparison of ResNet18 algorithm before and after adjustment

Continuing to refer to Table 1, the output layer of the ResNet18 algorithm in this example includes 1 output channel, that is, the dimension 1000 of the fully connected layer of the original ResNet18 algorithm is changed to 1. Also add the Sigmoid function:

In formula (2), the output range of the sigmoid function is (0,1).

It should be noted that the predicted classification output by the ResNet18 algorithm is a continuous value. Another advantage is that during the training process, it is convenient to calculate the loss value of each training process.

In this example, the electronic device can also train the above-mentioned ResNet18 algorithm, and the loss function during training is realized by the cross-entropy loss function. The formula is as follows:

Loss=-y log Y-(1-y)log(1-Y); (3)

In formula (3), y represents the label of the training sample. When the value is 0, it indicates that the training sample is not of the large aperture type, and when the value is 1, it indicates that the training sample is of the large aperture type; Y represents the predicted classification of the classification module.

In this example, when the initial to-be-classified image is a frame in the video frame sequence of the to-be-classified video, the electronic device may also obtain an indication of whether the to-be-classified video is a large aperture type based on the predicted classification of each initial to-be-classified image in the video frame sequence predicted classification. Among them, the predicted classification of the video to be classified can be obtained by the following formula:

In formula (4), R represents the prediction score of whether the video to be classified is a large aperture, and its output range is (0, 1), and the closer to 1, the greater the probability of being a large aperture type, and N represents the video to be classified. The number of captured video frames, M(f _i ) represents the predicted classification of the ith video frame.

So far, in this embodiment, the initial image to be classified can be obtained; then the depth heat map of the initial image to be classified can be obtained; the depth heat map refers to the depth of field depth information of the object in the initial image to be classified through different color gamuts. image; then, combine the RGB channel image of the initial to-be-classified image and the depth heat map to obtain a 4-channel target to-be-classified image; finally, classify the initial to-be-classified image according to the target to-be-classified image , to obtain whether the image to be classified is a large aperture type image. In this way, in this embodiment, by adding a depth heat map including depth of field depth information as a dimensional feature of the image to be classified, the accuracy of image classification can be improved.

In combination with the above content, an embodiment of the present disclosure provides a network architecture of an image classification model to classify videos to be classified. Referring to FIG. 5 , the working process includes:

First, perform sampling frame processing on the video to be classified, sample 1 frame every 10 frames, and sample N frames to obtain a frame sequence S={f ₁ , f ₂ , . . . , f _N }, which can reduce the amount of calculation. For each video frame in the frame sequence, it is cut into a size of 224*224*3, where 224*224 represents the length*width, and 3 represents the three color channels of RGB, so as to meet the requirements of the subsequent ResNet18.

Then, input each frame of video frame f _i into the DeepLens algorithm in turn to obtain the depth heat map d _i of each video frame, whose dimension is 224*224*1, which is normalized according to formula (1):

In formula (1), (x, y) represents the coordinates of each pixel in the depth heatmap, and the max() function is used to calculate the maximum value in the original depth heatmap d _i ; The normalized depth heatmap d _i ' is obtained.

Similarly, each video frame is normalized by means of depth heatmap normalization.

After that, the normalized video frame f _i and the depth heat map d _i ' are combined according to the channel to obtain the target video frame to be classified, the dimension of which is 224*224*4, and the number of channels 4 represents the RGB three channels and thermal power of the video frame. Figure 1. Sum of channels.

After that, the target video frame to be classified is input into the ResNet18 algorithm shown in Table 1. The ResNet18 algorithm has a total of 18 convolutional layers, including skip connections and residual structures, which have excellent feature extraction performance. Thus, the ResNet18 algorithm outputs a predicted classification M(f _i ), which is a continuous value.

Finally, calculate the average value of the predicted classification values of the video frame sequence according to formula (4) to obtain the predicted classification R of the video to be classified. The value range of R is (0, 1), and the closer to 1, the more likely it is Aperture type video.

It is understandable that, in practical applications, after the image classification model shown in FIG. 5 is trained with a preset training set, the loss value output by the loss function is less than or equal to the preset loss threshold, the gradient value remains unchanged, or the training preset Stop training after the number of times, and at this time, you can apply the trained image classification model to the online classification of the video platform.

Considering that the training samples in the training set have a certain type distribution, for example, there are too many types of scenery, so that the image classification model after training will be more accurate for the classification of large-aperture images or videos of landscape type, while the large-aperture images or videos of portrait type will be more accurate. The classification is relatively poor, that is, the image classification model will cause its performance to degrade when migrating the dataset. To this end, in this example, the above image classification model is also used to obtain a training set, which is essentially a large aperture data set, see Figure 6, including:

(1) Obtain a training set composed of other types of large-aperture videos, the training set includes multiple training samples, and each training sample may be a large-aperture image or video in other fields (eg, landscape type).

(2) Use the above training set to train the image classification model, and stop the training when the loss function converges. The model M1 is obtained after the training is completed.

(3) A number of videos are randomly screened on the video platform to form a test set, and the test set contains more portrait-type videos. Use the model M1 to classify each video in the test set, and obtain at least two types of video sets, namely, _a large-aperture type video set and a non-large-aperture type video set. Since the accuracy of the prediction and classification of the video set of the large aperture type may be low, the video set of the large aperture type can be screened to screen out the large aperture images that meet the preset conditions. The preset conditions may include: the value of the predicted classification exceeds the predicted classification threshold (eg, 0.8), and manual screening.

(4) Add the selected large aperture images to the training set

(5) Steps (2) to (4) are iterated until the performance of the image classification model is stable, and a training set containing training samples of a large aperture type is obtained, that is, a large aperture video data set is obtained. The stable performance means that the number of training samples in the training set exceeds the preset number of samples threshold, or the proportion of the selected large-aperture images in the test set exceeds the preset proportion, or the large-aperture screen is selected from the test set with many types of scenery. The correct rate of the image is equal to the correct rate of the large-aperture image selected from the test set with more portrait types.

In this example, compared to manually screening large-aperture videos, labor costs can be greatly reduced; moreover, two tasks can be accomplished at low cost: improving the classification performance of the image classification model; and obtaining a large number of large-aperture datasets.

Fig. 7 is a flowchart of an image classification apparatus according to an exemplary embodiment, which can be adapted to electronic devices such as smart phones, servers, and personal computers. Referring to FIG. 7 , an image classification apparatus includes: an input module 71 , an acquisition module 72 and a classification module 73 . in,

The input module 71 is configured to execute the acquisition of the initial images to be classified, and send the initial images to be classified to the acquisition module 72 and the classification module 73 respectively.

The acquisition module 72 is configured to execute the acquisition of the depth heat map of the initial image to be classified, and send the depth heat map to the classification module 73; the depth heat map refers to representing objects in the initial image to be classified through different color gamuts the depth of field depth information of the image;

The classification module 73 is configured to perform combining the RGB channel image of the initial image to be classified and the depth heat map to obtain a 4-channel target image to be classified; and, according to the target image to be classified, classify the initial image to be classified The images are classified to obtain whether the image to be classified is a large aperture type image.

When the image to be classified is a frame in the video frame sequence of the video to be classified, the device further includes an output module;

The output module is configured to perform a predictive classification based on each initial to-be-classified image in the video frame sequence to obtain a predictive classification indicating whether the to-be-classified video is a large aperture type.

In one embodiment, the value of the predicted classification of the video to be classified is an average value of the predicted classification values of the initial images to be classified in the video frame sequence.

In one embodiment, the classification module includes:

an image input unit configured to perform inputting the target classification image into an image classification prediction model;

The classification determination unit is configured to execute the prediction classification information output according to the image classification prediction model, and determine whether the to-be-classified image is a large aperture type image, wherein the image classification prediction model is based on the depth heat map of the sample image and The RGB channel image of the sample image is obtained by training.

In one embodiment, the output layer of the image classification prediction model includes a Sigmoid function; the Sigmoid function is used to output a continuous value representing the predicted classification, and the closer the value is to 1, the larger the initial image to be classified is. The greater the probability of the aperture type.

Regarding the above apparatus embodiments, the working principle of the image classification method has been described in detail, and will not be described in detail here.

Fig. 8 is a block diagram of an electronic device according to an exemplary embodiment. For example, electronic device 800 may be a server, mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, and the like.

8, an electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, a sensor component 814 , and the communication component 816 .

The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to perform all or some of the steps of the methods described above. Additionally, processing component 802 may include one or more modules that facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

Memory 804 is configured to store various types of data to support operation at electronic device 800 . Examples of such data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. Memory 804 may be implemented by any type of volatile or nonvolatile storage device or combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic or Optical Disk.

Power supply assembly 806 provides power to various components of electronic device 800 . Power supply components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800 .

Multimedia component 808 includes a screen that provides an output interface between electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action. In some embodiments, the multimedia component 808 includes a front-facing camera and/or a rear-facing camera. When the electronic device 1000 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each of the front and rear cameras can be a fixed optical lens system or have focal length and optical zoom capability.

Audio component 810 is configured to output and/or input audio signals. For example, audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when electronic device 800 is in operating modes, such as calling mode, recording mode, and voice recognition mode. The received audio signal may be further stored in memory 804 or transmitted via communication component 816 . In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, a button, or the like. These buttons may include, but are not limited to: home button, volume buttons, start button, and lock button.

Sensor assembly 814 includes one or more sensors for providing status assessment of various aspects of electronic device 800 . For example, the sensor assembly 814 can detect the on/off state of the electronic device 800, the relative positioning of the components, such as the display and the keypad of the electronic device 800, the sensor assembly 814 can also detect the electronic device 800 or one of the electronic device 800 Changes in the position of components, presence or absence of user contact with the electronic device 800 , orientation or acceleration/deceleration of the electronic device 800 and changes in the temperature of the electronic device 800 . Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 may access wireless networks based on communication standards, such as WiFi, carrier networks (eg, 2G, 3G, 4G, or 5G), or a combination thereof. In one exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In one embodiment of the present disclosure, the electronic device 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field Programmable gate array (FPGA), controller, microcontroller, microprocessor or other electronic component implementation for carrying out the above method.

In an embodiment of the present disclosure, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the electronic device 800 to complete the steps of the above method. For example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

In an embodiment of the present disclosure, a computer application program is also provided, and when the computer application program is executed by a processor, the steps of the above method can be performed to obtain the same technical effect.

In an embodiment of the present disclosure, a computer program product is also provided, when the computer program product is executed by a processor of an electronic device, the electronic device can perform the steps of the above method to obtain the same technical effect.

It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

Each embodiment in this specification is described in a related manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. Especially, for the apparatus/electronic device/storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the partial description of the method embodiment.

Other embodiments of the present disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the above-described examples that follow the general principles of this disclosure and include common knowledge or practice in the art not disclosed by this disclosure means. The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (10)

1. an image classification method, is characterized in that, comprises: Get the initial image to be classified; obtaining a depth heat map of the initial image to be classified; the depth heat map refers to an image representing depth of field depth information of an object in the initial image to be classified through different color gamuts; Combining the RGB channel image of the initial to-be-classified image and the depth heatmap to obtain a 4-channel target to-be-classified image; According to the target to-be-classified image, the initial to-be-classified image is classified to obtain whether the to-be-classified image is a large aperture type image. 2. The image classification method according to claim 1, wherein when the image to be classified is a frame in the video frame sequence of the video to be classified, the method further comprises: Based on the predicted classification of each initial to-be-classified image in the video frame sequence, a predicted classification indicating whether the to-be-classified video is of a large aperture type is obtained. 3 . The image classification method according to claim 2 , wherein the value of the predicted classification of the video to be classified is an average value of the predicted classification values of each initial image to be classified in the video frame sequence. 4 . 4 . The image classification method according to claim 1 , wherein, according to the target image to be classified, the initial to-be-classified image is classified to obtain whether the to-be-classified image is of a large aperture type. 5 . images, including: Input the target classification image into the image classification prediction model, and determine whether the to-be-classified image is a large aperture type image according to the prediction classification information output by the image classification prediction model, wherein the image classification prediction model is based on the sample The depth heatmap of the image and the RGB channel image of the sample image are obtained by training. 5. The image classification method according to claim 4, wherein the output layer of the image classification prediction model comprises a Sigmoid function; The closer it is to 1, the greater the probability that the initial image to be classified is of the large aperture type. 6. An image classification device, characterized in that the device comprises: an input module, an acquisition module and a classification module; The input module is configured to perform acquisition of an initial image to be classified, and send the initial image to be classified to the acquisition module and the classification module respectively; The acquisition module is configured to perform acquisition of a depth heat map of the initial image to be classified; the depth heat map refers to an image representing depth of field depth information of an object in the initial image to be classified through different color gamuts; The classification module is configured to perform combining the RGB channel image of the initial to-be-classified image and the depth heat map to obtain a 4-channel target to-be-classified image; and, according to the target to-be-classified image, perform a The classified images are classified to obtain whether the to-be-classified image is a large aperture type image. 7. The image classification device according to claim 6, wherein when the image to be classified is a frame in the video frame sequence of the video to be classified, the device further comprises an output module; The output module is configured to perform a predictive classification based on each initial to-be-classified image in the video frame sequence to obtain a predictive classification indicating whether the to-be-classified video is a large aperture type. 8. An electronic device, characterized in that, comprising: processor; a memory for storing an executable program of the processor; wherein the processor is configured to execute the executable program in the memory to implement the steps of the method according to any one of claims 1-5. 9 . A non-transitory computer-readable storage medium, wherein when an executable program in the storage medium is executed, the steps of the method according to any one of claims 1 to 5 can be performed. 10 . 10 . A computer application program, characterized in that, when the computer application program is executed by a processor of a server, the server can implement the steps of the method according to any one of claims 1 to 5 .

Images