CN118736563A - A method, device and medium for detecting food maturity - Google Patents

Abstract

The application discloses a food material maturity detection method, equipment and a medium, wherein the method comprises the following steps: the HSV value of each pixel point in the image to be detected is adjusted, so that the color of the adjusted pixel point is not changed. And inputting the adjusted image to be detected into a maturity detection model for feature recognition to obtain a maturity detection result of the food material to be detected. The RGB of the pixel point and the HSV have a conversion relation corresponding to the interval, namely, the RGB conversion is carried out after the HSV value of the pixel point is adjusted, so that the same color as that before adjustment can be obtained. Because the color of the food material in the image is an index for judging the maturity of the food material, the HSV is finely adjusted on the premise that the color of the pixel point is not changed, the image characteristics of the food material can be enhanced on the basis that the maturity index judgment is not affected, and then the detection precision is improved. The food material maturity detection method disclosed by the application has the characteristics of instantaneity, reliability, generalization, controllability and reproducibility, and accords with the credibility.

Description

A method, device and medium for detecting food maturity

Technical Field

The present application relates to the field of image processing technology, and in particular to a method, device and medium for detecting the maturity of food.

Background Art

Most current smart ovens have a maturity detection function, which is used to detect ingredients during the baking process to determine whether the ingredients have been baked to the degree of maturity required by the user. For example, the user can set the steak to be baked to medium-rare. The smart oven will monitor the baking process of the ingredients through the built-in camera and prompt the user that the ingredients have been baked when it is determined that the ingredients have reached the medium-rare state.

Traditional maturity detection functions are mostly implemented based on detection network models. They use built-in models to detect video images of the interior of the oven to obtain evaluation indicators of the maturity of the ingredients (mainly the color and shape of the ingredients), and then determine whether the ingredients meet the maturity requirements.

However, the lighting environment inside the oven is relatively complex. Factors such as the light intensity of the infrared heating tube, the light reflection of the tray and tin foil, and external light sources will affect the imaging effect of the food during the image acquisition process. This will directly affect the accuracy of the model's image feature extraction of the food, thereby reducing the accuracy of the model's detection of the maturity of the food.

Summary of the invention

The embodiment of the present application provides a method, device and medium for detecting the maturity of food, which are used to enhance the image features of food without affecting the determination of the maturity index of the food itself, thereby improving the accuracy of food maturity detection.

In a first aspect, an embodiment of the present application provides a method for detecting the maturity of food materials, the method comprising:

In response to the detection instruction, acquiring an image to be tested; wherein the image to be tested includes the food to be tested;

Based on a preset pixel adjustment method, the color value HSV of the pixel point in the image to be tested is changed so that the color obtained by performing the color value RGB conversion of the pixel point before the HSV value is changed is the same as the color obtained by performing the RGB conversion of the pixel point after the HSV value is changed;

The processed image to be tested is input into the trained maturity detection model for feature recognition to obtain the maturity detection result of the food to be tested.

In some possible embodiments, the image to be tested is obtained by:

Receiving the type of food to be tested indicated by the user;

Capturing an image of a baking area of the smart oven to obtain a target image;

Inputting the target image into a trained target detection model for feature recognition to obtain food information of each food in the target image; wherein the food information includes the type of food and a detection frame representing the location of the food;

The image to be tested is determined according to a target detection frame in the target image; wherein the type of food corresponding to the target detection frame is the type of food to be tested.

In some possible embodiments, the image to be tested includes a first image and a second image; the first image includes all the food to be tested in the target image, and the second image includes any food to be tested in the first image;

The maturity test result is obtained in the following way:

Performing feature extraction on the first image and the second image respectively to obtain a first feature of the food to be tested in the first image and a second feature of the food to be tested in the second image;

Normalizing the first feature and the second feature, and fusing the normalized result with the first feature;

The maturity detection result is obtained by performing feature recognition on the result of feature fusion.

In some possible embodiments, determining the image to be detected according to the target detection frame in the target image includes:

Acquire vertex coordinate information of each target detection frame in the pixel coordinate system of the target image; wherein the vertex coordinate information includes the minimum horizontal coordinate and the minimum vertical coordinate of each first vertex on the target detection frame, and the maximum horizontal coordinate and the maximum vertical coordinate of each second vertex; the first vertex is the angle vertex of two specified borders in the target detection frame, and the line connecting the first vertex and the second vertex is the diagonal line of the target detection frame;

Determining the minimum circumscribed rectangular area of all target detection frames in the target image according to the vertex coordinate information of each target detection frame;

The first image is determined according to the minimum circumscribed rectangular area, and the second image is determined according to the target detection frame in the first image.

In some possible embodiments, before determining the first image according to the minimum circumscribed rectangular area, the method includes:

Adjusting the grayscale value of the designated area within the minimum circumscribed rectangular area to a preset value;

The designated area is determined according to a designated detection frame, and the designated detection frame is a detection frame corresponding to each non-to-be-tested food in the minimum circumscribed rectangular area.

In some possible embodiments, the food information further includes a detection confidence of the object detection model for each detection frame; and determining the second image according to the object detection frame in the first image includes:

The second image is determined according to the target detection box with the highest detection confidence in the first image.

In some possible embodiments, the changing the color value HSV of the pixel point in the image to be tested based on a preset pixel adjustment method includes:

Perform multiple rounds of iterative adjustments on the HSV value of the pixel point, and determine in each round of iteration whether the colors obtained by converting the color value RGB before and after the adjustment of the HSV value are the same; if they are not the same, enter the next round of iteration; if they are the same, use the HSV value obtained in this round as the adjusted HSV value of the pixel point;

Among them, any round of iterative adjustment includes:

Determine the current round correction value and current round adjustment method according to the current iteration round number; wherein the first round correction value is a preset value, and the non-first round correction value is a specified multiple of the first round correction value; the first round adjustment method is to increase or decrease the HSV value, and the non-first round adjustment method is opposite to the first round adjustment method;

The current round adjustment method is used to adjust the HSV value obtained in the previous round with the current round correction value.

In some possible embodiments, after obtaining the maturity test result of the food to be tested, the method further includes:

If the maturity detection result indicates that the food to be tested has reached a specified baking state, the intelligent oven is controlled to output a prompt message indicating that baking of the food to be tested is complete.

In a second aspect, an embodiment of the present application provides a baking device, including a data transmission unit and a processor; the data transmission unit is configured to: acquire an image to be tested in response to a detection indication; wherein the image to be tested includes food to be tested;

The processor is configured to: change the color value HSV of the pixel point in the image to be tested based on a preset pixel adjustment method, so that the color obtained by performing the color value RGB conversion of the pixel point before the HSV value is changed is the same as the color obtained by performing the RGB conversion of the pixel point after the HSV value is changed;

The processed image to be tested is input into the trained maturity detection model for feature recognition to obtain the maturity detection result of the food to be tested.

In some possible embodiments, the image to be tested is obtained by:

Receiving the type of food to be tested indicated by the user;

Capturing an image of a baking area of the smart oven to obtain a target image;

Inputting the target image into a trained target detection model for feature recognition to obtain food information of each food in the target image; wherein the food information includes the type of food and a detection frame representing the location of the food;

The image to be tested is determined according to a target detection frame in the target image; wherein the type of food corresponding to the target detection frame is the type of food to be tested.

In some possible embodiments, the image to be tested includes a first image and a second image; the first image includes all the food to be tested in the target image, and the second image includes any food to be tested in the first image;

The maturity test result is obtained in the following way:

Performing feature extraction on the first image and the second image respectively to obtain a first feature of the food to be tested in the first image and a second feature of the food to be tested in the second image;

Normalizing the first feature and the second feature, and fusing the normalized result with the first feature;

The maturity detection result is obtained by performing feature recognition on the result of feature fusion.

In some possible embodiments, to determine the image to be detected according to the target detection frame in the target image, the processor is configured to:

Acquire vertex coordinate information of each target detection frame in the pixel coordinate system of the target image; wherein the vertex coordinate information includes the minimum horizontal coordinate and the minimum vertical coordinate of each first vertex on the target detection frame, and the maximum horizontal coordinate and the maximum vertical coordinate of each second vertex; the first vertex is the angle vertex of two specified borders in the target detection frame, and the line connecting the first vertex and the second vertex is the diagonal line of the target detection frame;

Determining the minimum circumscribed rectangular area of all target detection frames in the target image according to the vertex coordinate information of each target detection frame;

The first image is determined according to the minimum circumscribed rectangular area, and the second image is determined according to the target detection frame in the first image.

In some possible embodiments, before executing the step of determining the first image according to the minimum circumscribed rectangular area, the processor is further configured to:

Adjusting the grayscale value of the designated area within the minimum circumscribed rectangular area to a preset value;

The designated area is determined according to a designated detection frame, and the designated detection frame is a detection frame corresponding to each non-to-be-tested food in the minimum circumscribed rectangular area.

In some possible embodiments, the food information further includes a detection confidence of the target detection model for each detection frame; and to perform determining the second image according to the target detection frame in the first image, the processor is configured to:

The second image is determined according to the target detection box with the highest detection confidence in the first image.

In some possible embodiments, the color value HSV of the pixel point in the image to be tested is changed based on a preset pixel adjustment method, and the processor is configured as follows:

Perform multiple rounds of iterative adjustments on the HSV value of the pixel point, and determine in each round of iteration whether the colors obtained by converting the color value RGB before and after the adjustment of the HSV value are the same; if they are not the same, enter the next round of iteration; if they are the same, use the HSV value obtained in this round as the adjusted HSV value of the pixel point;

Among them, any round of iterative adjustment includes:

Determine the current round correction value and current round adjustment method according to the current iteration round number; wherein the first round correction value is a preset value, and the non-first round correction value is a specified multiple of the first round correction value; the first round adjustment method is to increase or decrease the HSV value, and the non-first round adjustment method is opposite to the first round adjustment method;

The current round adjustment method is used to adjust the HSV value obtained in the previous round with the current round correction value.

In some possible embodiments, after obtaining the maturity test result of the food to be tested, the processor is further configured to:

If the maturity detection result indicates that the food to be tested has reached a specified baking state, the intelligent oven is controlled to output a prompt message indicating that baking of the food to be tested is complete.

In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any one of the methods in the first aspect is implemented.

In a fourth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; when a processor of a computer device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, so that the computer device implements any one of the methods in the first aspect.

In an embodiment of the present application, after obtaining an image to be tested containing the food to be tested, the HSV value of each pixel in the image to be tested is changed based on a preset pixel adjustment method, and the processed image to be tested is input into a trained maturity detection model for feature recognition to obtain a maturity detection result of the food to be tested.

Since the color of food in the image is one of the indicators for judging its maturity, and there is an interval-corresponding conversion relationship between RGB value and HSV value, that is, the same color can be obtained by converting the HSV value before and after adjustment to RGB. Therefore, by fine-tuning the HSV value while ensuring that the color of the pixel point remains unchanged, the image features of the food can be enhanced without affecting the determination of the maturity index of the food itself, thereby improving the accuracy of food maturity detection.

Other features and advantages of the present application will be described in the subsequent description, and partly become apparent from the description, or be understood by practicing the present disclosure. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of an application scenario provided by an embodiment of the present application;

FIG2 is an overall flow chart of a method for detecting the maturity of food provided in an embodiment of the present application;

FIG3 is a schematic diagram of a flow chart of how to obtain an image to be tested according to an embodiment of the present application;

FIG4 is a schematic diagram of the structure of a smart oven provided in an embodiment of the present application;

FIG5 is a schematic diagram of the structure of a target detection network provided in an embodiment of the present application;

FIG6 is a schematic diagram of a feature extraction module provided in an embodiment of the present application;

FIG. 7 is a schematic diagram of a food detection module provided in an embodiment of the present application;

FIG8 is a schematic diagram of the structure of an SPP module provided in an embodiment of the present application;

FIG9 is a schematic diagram of a detection head module provided in an embodiment of the present application;

FIG10 is a schematic diagram of the output of a target detection network provided in an embodiment of the present application;

FIG11 is a schematic diagram of a process of generating a first image and a second image according to an embodiment of the present application;

FIG12 is a schematic diagram of vertex coordinates of a target detection frame provided in an embodiment of the present application;

FIG13 is a schematic diagram of a minimum circumscribed rectangular area provided in an embodiment of the present application;

FIG14 is a schematic diagram of a first image and a second image provided in an embodiment of the present application;

FIG15 is a schematic diagram of grayscale value adjustment provided in an embodiment of the present application;

FIG16 is a schematic diagram of a second image acquisition process provided in an embodiment of the present application;

FIG17 is a schematic diagram of HSV conversion provided in an embodiment of the present application;

FIG18 is a schematic diagram of the structure of a maturity detection model provided in an embodiment of the present application;

FIG19 is a schematic diagram of feature fusion provided in an embodiment of the present application;

FIG. 20 is a schematic diagram of the structure of the baking equipment 160 provided in an embodiment of the present application.

DETAILED DESCRIPTION

In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other arbitrarily. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

As mentioned above, traditional maturity detection is mostly based on target detection technology, which uses built-in detection network models (such as Yolo series, Faster-Rnn and other network models) to perform target detection on video frames representing the internal images of the oven to obtain the evaluation index of the maturity of the food. The evaluation index mainly includes the changes in the color and shape of the food to be tested in the image. Then, it is determined whether the food meets the maturity requirements based on the evaluation index of the food maturity.

However, the lighting environment inside the oven is relatively complex. Factors such as the light intensity of the infrared heating tube, the light reflection of the tray and tin foil, and external light sources will affect the imaging effect of the food during the image acquisition process. This will directly affect the accuracy of the detection model in extracting food features in the image, thereby reducing the accuracy of the model in detecting the maturity of the food.

In order to solve the above problems, the inventive concept of the present application is as follows: after obtaining the image to be tested containing the food to be tested, the HSV value of each pixel in the image to be tested is changed based on a preset pixel adjustment method, and the processed image to be tested is input into the trained maturity detection model for feature recognition to obtain the maturity detection result of the food to be tested. Since the color of the food in the image is one of the indicators for judging its maturity, and there is an interval-corresponding conversion relationship between the RGB value and the HSV value, that is, the same color can be obtained by performing RGB conversion on the HSV values before and after the adjustment. Therefore, the HSV value is fine-tuned under the premise of ensuring that the color of the pixel point remains unchanged, and the image features of the food can be enhanced on the basis of not affecting the determination of the maturity index of the food itself, thereby improving the accuracy of the maturity detection of the food.

Next, as shown in FIG1 , FIG1 is a schematic diagram of an application scenario for food maturity detection provided by an embodiment of the present application. FIG1 shows a network 10 , a server 20 , a storage 30 and a baking device 40 .

The user can issue a maturity detection instruction to the baking device 40 through the network 10, for example, the user issues a detection instruction for the steak to be 70% done to the baking device. In response to the maturity detection instruction, the baking device 40 captures an image of the food inside the oven through a built-in camera, and transmits the captured image to the server 20 through the network 10.

The server 20 detects the target in the image through a built-in detection algorithm, determines the characteristic parameters representing the color and shape of the steak in the image, and compares the characteristic parameters with the preset parameters representing the medium-rare steak to determine whether the steak is baked to medium-rare. After reaching medium-rare, the server 20 controls the baking device 40 to send a prompt message to the user indicating that the steak has reached medium-rare.

It should be noted that, although only a single server or baking device is described in the above process, those skilled in the art should understand that the baking device 40, server 20 and memory 30 shown in FIG1 are intended to represent the operations of the baking device, server and memory involved in the technical solution of the present application. The single server and memory are described in detail at least for the convenience of explanation, and do not imply any limitation on the number, type or location of the baking devices and servers.

It should be noted that if additional modules are added to the illustrated environment or individual modules are removed therefrom, the underlying concepts of the exemplary embodiments of the present application will not be changed. In addition, although a bidirectional arrow from the storage 30 to the server 20 is shown in FIG. 1 for ease of explanation, it will be understood by those skilled in the art that the sending and receiving of the above data also needs to be implemented through the network 10.

It should be noted that the memory in the embodiment of the present application can be, for example, a cache system, or a hard disk storage, memory storage, etc., which stores information such as the type of smart oven, supported working modes, and evaluation criteria for different degrees of maturity of different ingredients.

Next, as shown in FIG. 2 , FIG. 2 shows the overall process of a method for detecting the maturity of food provided in an embodiment of the present application, including the following steps:

Step 201: In response to a detection instruction, acquiring an image to be tested; wherein the image to be tested includes food to be tested;

The image to be processed in the embodiment of the present application is an image containing the food to be tested, and the food to be tested is the food for maturity detection indicated by the user. The method for obtaining the image to be tested is specifically shown in FIG3 and includes the following steps:

Step 301: receiving the type of food to be tested indicated by the user; capturing an image of the baking area of the smart oven to obtain a target image;

The smart oven of the embodiment of the present application is provided with an image acquisition device, as specifically shown in FIG4 . The image acquisition device is disposed on the internal ceiling of the oven body, on the opposite side of the oven door, and captures the panoramic view inside the oven from a bird's-eye view, that is, the acquisition range of the image acquisition device includes each grill area in the oven. Therefore, no matter which grill area the user places the baking tray on, it will be completely captured by the image acquisition device.

When the smart oven receives the type of food to be tested indicated by the user, it controls the image acquisition device to acquire an image of the baking area shown in FIG. 4 to obtain a target image of the baking area.

It should be noted that the specific placement of the above-mentioned image acquisition device is only an example of the present application and does not limit the placement method. In practical applications, it is only necessary to ensure that the image acquisition device can completely capture each grill area.

Step 302: Input the target image into a trained target detection model for feature recognition to obtain food information of each food in the target image; wherein the food information includes the type of food and a detection frame representing the location of the food;

Since the purpose of maturity detection is to intelligently provide the food temperature required by the user, and large-scale intelligent ovens usually include multiple layers of grills, the heating conditions of the food in different grills are different at the same baking temperature. Therefore, the embodiment of the present application uses the number of grill layers where the food is located as one of the evaluation indicators of food maturity.

Based on this, the embodiment of the present application constructs the target detection model of the present application with a multi-task deep learning framework. As shown in Figure 5, the target detection model combines the two classification tasks of food category detection and shelf recognition. The information of the two tasks can be jointly utilized in training, and the model performance can be improved by sharing useful information between the two tasks, thereby shortening the time it takes for the optimal model to appear.

Among them, food detection mainly focuses on the characteristics of the food itself, and shelf classification mainly focuses on the background information of the food (i.e. the position of the baking tray, the position of the grill slide, etc.). By combining the two tasks, the feature extraction module can fully extract the information of the entire picture. In the training stage, the learning of the number of shelves helps the food detection model to locate the position range of the food more quickly, so that the strong discriminant features learned by classification learning can further improve the accuracy of target detection, thereby improving the performance and generalization ability of the algorithm.

As shown in Figure 5, the lower left of the model structure is the food detection module, and the lower right is the shelf classification module. By sharing the feature information of the feature extraction module and updating the parameters of the entire model through the back propagation of the loss function, the information of the shelf and the information of the food position will affect each other in this process, promoting the update speed of the model parameters, and the two are organically combined. After loading the data, transform data enhancement is introduced. This data enhancement method is beneficial to the target detection module. More feature information is extracted from the existing data through different data enhancement methods. However, some scaling, cropping, stretching and other operations of data enhancement will make the image size of the input model inconsistent, and fixed-size food features are required before the fully connected layer of the classification module. Therefore, adaptive average pooling is introduced in the shelf classification module. The food features of different inputs are obtained after adaptive average pooling. The food features of uniform size are obtained, which is convenient for the subsequent fully connected layer to extract classification features.

Specifically, the target detection grid consists of five parts: data loading, feature extraction, shelf classification, food detection, and detection head. During implementation, a data loading module is pre-built, through which the image information of the training samples is loaded and then the loaded data is transformed for data enhancement. The data enhancement process specifically includes image processing operations such as cropping, flipping, resizing, modifying brightness, contrast and saturation, mosaic data enhancement, and disrupting the order of pictures.

Next, a feature extraction module as shown in Figure 6 is constructed. Residual connection, dense connection and SE channel attention mechanism are introduced in the feature extraction module of this application. Among them, residual connection, as a model integration method, can effectively alleviate the gradient vanishing problem; dense connection can aggregate intermediate features with different receptive fields, and play a good role in information propagation and feature fusion stages; SE channel attention mechanism (Effective SE shown in Figure 6) can adaptively learn the dependencies and importance between different channels, and let the model purposefully enhance the feature learning on certain feature channels according to the importance. By combining the above residual connection and dense connection as a local CSPRes function processing module. And the SE channel attention mechanism module is introduced after each CSPRes function processing module. The above CSPRes function processing module is a basic module for extracting and aggregating the backbone and local features of the image, which can effectively improve the detection accuracy. The backbone is named. The feature aggregation module uses the PAN structure for feature aggregation.

After the feature extraction module, the two modules of shelf classification and food detection shown in the above-mentioned Figure 5 are connected. The shelf classification module includes convolution, normalization, relu activation function, adaptive average pooling, flatten function (used to flatten the extracted feature map to a one-dimensional vector), fully connected layer, and cross entropy loss function. Due to the aforementioned transform data enhancement process of the input sample image, the image will have size differences after processing, which will lead to inconsistent feature map sizes obtained by feature extraction. In addition, since in the model training task, the fully connected layer needs to set a fixed input size before parameter learning can be performed, the embodiment of the present application introduces adaptive average pooling in this module. Through adaptive average pooling, the spatial dimension of the feature map can be compressed and the mean of the corresponding dimension can be taken out at the same time, so as to obtain a feature map of fixed size, which is then passed to the fully connected layer for subsequent operations.

As shown in FIG. 7, the food detection module of the embodiment of the present application adds an SPP module and an improved PAN module for pyramid pooling on the basis of the traditional detection model of Yoloy3. The SPP module, as shown in FIG. 8, includes three maximum pooling layers, which are used to pass the input feature map through the maximum pooling layer with sliding window sizes of 1x1, 3x3, 5x5, and 9x9, and then perform Concat function processing on the feature maps of different scales. The SPP module in the embodiment of the present application replaces the conventional pooling layer after the convolution layer in the existing Yolov3 structure, which can increase the receptive field, obtain the fusion of multi-scale features, and the training speed is also faster. The introduction of the PAN module will strongly fuse the features of different levels. It adds a bottom-up feature pyramid structure on the basis of the FPN module, retains more shallow position features, and further improves the overall feature extraction capability. In addition, compared with the traditional Yolov3 structure, the present application replaces the fusion operation after the feature map extraction by the traditional add function processing with the Conca function processing, so that the feature map can be spliced based on the number of channels, thereby extracting richer image features.

FIG9 shows a detection head module of an embodiment of the present application, and the detection head module is used to connect with the food detection module shown in FIG7 . The detection head module of the embodiment of the present application is composed of detection heads of three sizes, and each detection head is used to scale the received feature map to 1/8, 1/16 and 1/32. Assuming that the size of the image input to the feature extraction module is 608x608, the size of the feature maps of the three detection heads output is 76*76, 38*38, and 19*19. The calculation formula for the number of channels of the feature map is: (4+1+classnum)*3. 4 represents the length, width, height and depth information of the detection frame, 1 represents whether there is a target to be detected at the current position, classnum is the number of targets to be detected, and 3 represents the above information of each of the three detection frames assigned to each detection head.

The overall loss function of the target detection model is obtained by weighting the sub-loss functions of the two modules, that is, Loss _total = α Loss ₁ + β Loss ₂ . Among them, α and β are the weight coefficients of the above two tasks (i.e., the food detection and shelf classification modules shown in Figure 5). In an optional alternative, the two sub-loss functions of Loss ₁ and Loss ₂ can be back-propagated alternately, and each updates the range of the sub-loss function that can be back-propagated. The two sub-loss functions are constrained by the norm (for example: the second norm) to avoid the difference between the two being too large and the impact on the model being uneven. If you want to emphasize the importance of a certain function, you can give the sub-loss function of the module a greater weight before back-propagation.

The embodiment of the present application improves the loss function of the food detection task (i.e., the above-mentioned Loss ₁ ), as shown in the following formula (1), which has a total of six lines; wherein, the first line of formula (1) defines the prediction error of the offset of the horizontal coordinate of the center point of the predicted value and the true value; the second line defines the prediction error of the offset of the vertical coordinate of the center point of the current detection frame; the third line defines the relative offset error of the width and height of the current detection frame and the label; the fourth and fifth lines respectively represent whether there is a target to be detected at the current position and the corresponding confidence; the last line is the classification loss for the target to be detected.

It should be noted that the part of calculating the classification loss in the last line of the above formula (1) uses the improved cross entropy loss function of this application. The original cross entropy loss function in Yolov3 is defined as shown in the following formula (2); and the improved cross entropy loss function of this application is defined as shown in the following formula (3):

Among them, L is the classification loss of the target (ingredient) to be detected, _Pi is the type of ingredient predicted by the model, _qi is the actual type of ingredient in the image, and _Wi is the reciprocal of the proportion of images of category _Pi in the sample image to the total number of sample images.

Step 303: determining the image to be tested according to the target detection frame in the target image; wherein the target detection frame is a detection frame of the food to be tested.

As shown in FIG5 above, the target detection model of the present application is a multi-task detection model structure, which not only has the traditional target detection capability, but also has the ability to identify the number of shelves where the ingredients are located. In the model use stage, by inputting the target image into the trained target detection model for feature recognition, the number of shelves where each ingredient is located and the detection frame Anchor corresponding to each ingredient can be obtained.

For example, the target image obtained in the aforementioned step 301 is shown in FIG10 , and the target image contains two cupcakes and one cream cake. By inputting the target image into the target detection model for feature recognition, the number of racks where each cupcake and cream cake are located, as well as the detection frame corresponding to each ingredient are obtained.

After the target image is acquired through the above process, the image to be tested is determined according to the target detection frame in the target image, as shown in FIG11, including the following steps:

Step 111: obtaining vertex coordinate information of each target detection frame in the pixel coordinate system of the target image;

Among them, the vertex coordinate information includes the minimum horizontal coordinate and the minimum vertical coordinate of each first vertex on the target detection frame, and the maximum horizontal coordinate and the maximum vertical coordinate of each second vertex; the first vertex is the angle vertex of two specified borders in the target detection frame, and the line connecting the first vertex and the second vertex is the diagonal line of the target detection frame.

The target detection frame of the embodiment of the present application is the detection frame including the food to be tested. Taking the aforementioned FIG. 10 as an example, assuming that the food to be tested indicated by the user is a cream cake, the detection frame Anchor2 of the cream cake shown in FIG. 10 is the target detection frame.

The first vertex and the second vertex in the above step 111 are both vertices of the target detection frame, and the line connecting the first vertex and the second vertex is the diagonal line of the target detection frame. Since the detection frame output by the detection model is a regular rectangle, the first vertex and the second vertex are the vertices of any two opposite corners of the rectangle.

Step 112: determining the minimum circumscribed rectangular area of all target detection frames in the target image according to the vertex coordinate information of each target detection frame;

The following is an example in which the first vertex is the upper left vertex of the target detection frame. Since the first vertex is the upper left vertex of the target detection frame, the second vertex is the lower right vertex of the target detection frame.

Assume that the food to be tested is a cupcake, and the target image shown in FIG12 contains n cupcakes. Let ( _xi , _yi ) be the coordinates of the center point of the target detection frame where the i-th cupcake is located in the target image, _Wi be the width of the target detection frame, and _Hi be the height of the target detection frame. Then the coordinates of the upper left corner of the target detection frame can be expressed as (xi _{- 0.5Wi} , yi _{- 0.5Hi} ), and the coordinates of the lower right corner can be expressed as ( _xi + _0.5Wi , _yi +0.5Hi ₎ .

Next, as shown in FIG13 , the smallest upper left corner horizontal coordinate Min( _{xi- 0.5Wi} ) and the smallest upper left corner vertical coordinate Min( _yi +0.5Hi ₎ are found from the upper left corner coordinates of each target detection frame. Point A indicated by the horizontal and vertical coordinates is used as the upper left corner coordinate of the minimum circumscribed rectangular area, and in the same way, point B indicated by the largest lower right corner horizontal coordinate Max( _xi +0.5Wi ₎ and the largest lower right corner vertical coordinate Max( _yi + _0.5Hi ) is used as the lower right corner coordinate of the minimum circumscribed rectangular area.

As shown in FIG13 , after determining the upper left corner vertex A and the lower right corner vertex B of the minimum circumscribed rectangular area in the above manner, the center point C of the minimum circumscribed rectangular area is determined according to the coordinates of points A and B. Then, the minimum circumscribed rectangular area containing all the ingredients to be tested can be obtained by using the minimum circumscribed rectangular calculation formula with the three known points.

Step 113: Determine the first image according to the minimum circumscribed rectangular area, and determine the second image according to the target detection frame in the first image.

Specifically, as shown in FIG14 , during implementation, the minimum bounding rectangular area can be expanded outward by a preset size, and then the image content corresponding to the minimum bounding rectangular area is used as the first image. Since the first image contains multiple ingredients to be tested (cup cakes), a target detection frame of any ingredient to be tested can be selected from the first image, and the area where the target detection frame is located is expanded to a certain size, and the image content corresponding to the expanded area is used as the second image.

In addition, considering that there are multiple ingredients placed in the same grill in actual applications, the cropped first image may include not only cupcakes but also cream cakes that are not ingredients to be tested, as shown in FIG15. In this case, before determining the first image according to the minimum bounding rectangular area, the grayscale value of the specified area in the minimum bounding rectangular area may be adjusted to a preset value.

In specific implementation, the grayscale value of the designated area can be adjusted to 0 as shown in FIG15, so that the designated area is displayed in full black. It should be noted that the designated area is determined according to the designated detection frame, which is the detection frame corresponding to each non-test food in the minimum circumscribed rectangular area, that is, the detection frame of the cream cake in FIG15. In this way, the feature interference of non-test food can be reduced during the subsequent feature recognition of the test food, thereby improving the detection accuracy of the maturity of the food.

Since the model output in the detection task will carry the confidence level for the output result, the food information in the embodiment of the present application may also include the confidence level of the target detection model for each detection frame.

Next, as shown in FIG16 , when determining the second image based on the target detection frame in the first image, the target detection frame with the highest confidence can be selected from the first image, and then the target detection frame is expanded to a certain size, and finally the second image is determined based on the image content corresponding to the expanded area. Thus, the first image containing all the ingredients to be tested and the second image containing one portion of the ingredients to be tested can be obtained through the above process.

Step 202: changing the color value HSV of the pixel point in the image to be tested based on a preset pixel adjustment method, so that the color obtained by performing the color value RGB conversion of the pixel point before the HSV value is changed is the same as the color obtained by performing the RGB conversion of the pixel point after the HSV value is changed;

Since the color of food is one of the evaluation indicators of its maturity, the lighting environment inside the oven is relatively complex. Factors such as the light intensity of the infrared heating tube, the light reflection of the tray and tin foil, and external light sources will affect the imaging effect of the food during the image acquisition process. This will directly affect the accuracy of the detection model in extracting food features in the image, thereby reducing the accuracy of the model in detecting the maturity of the food.

The HSV value (hue H, saturation S and brightness V) of the food and the color value RGB can be converted to each other, and the HSV value and the RGB value have an interval correspondence as shown in Figure 17. Specifically, the RGB value of a pixel can reflect the color of the pixel. After a small change in the HSV value of the pixel, the HSV to RGB operation is performed on the pixel. Although the pixel value obtained after the RGB conversion is different from that before the HSV change, the corresponding color value can be the same color. Taking Figure 17 as an example, the colors corresponding to hues H between 26 and 34 are all yellow. Assuming that the hue H of the current pixel is 26, it will be enhanced by 5% to 27.3. Although the pixel value after the RGB conversion is different from that before the adjustment, the corresponding color is still the same (still yellow).

Therefore, under the premise of ensuring that the color of the food remains unchanged, the HSV value of the pixel point can be fine-tuned to increase the difference in image characteristics between the pixels, thereby enhancing the overall characteristics of the food. In specific implementation, the HSV value of the pixel point can be adjusted for multiple rounds of iterations, and in each round of iterations, it is determined whether the color obtained by converting the HSV value to RGB before and after the adjustment is the same.

If they are not the same, the next round of iteration will be entered. If they are the same, the HSV value obtained in this round will be used as the HSV value after pixel adjustment. Among them, any round of iterative adjustment includes:

The correction value and adjustment method of this round are determined according to the current iteration round number; the correction value of the first round is a preset value, and the correction value of the non-first round is a specified multiple of the correction value of the first round; the adjustment method of the first round is to increase or decrease the HSV value, and the adjustment method of the non-first round is opposite to the adjustment method of the first round;

Specifically, some or all of the HSV values of the pixel points may be increased or decreased by 5%, assuming that the first adjustment is to increase by 5%, that is, the first round of correction value is 5%, and the first round of adjustment is to increase.

The current round adjustment method is used to adjust the HSV value obtained in the previous round with the current round correction value.

If the color obtained after the changed HSV value is converted to RGB changes, it is necessary to enter the second round of adjustment. At this time, the current adjustment round is 2, and the correction value of the current round is half of the correction value of the first round, that is, 2.5%. The adjustment method of the current round is opposite to that of the first round, that is, reduction.

At this time, the changed HSV value is adjusted in reverse. If the color obtained by RGB conversion of the adjusted HSV value is the same as the color before the iterative adjustment, the iterative process ends. Otherwise, the next round of iteration is entered. In this way, the image feature differences between pixels can be increased by fine-tuning the HSV value of the pixel while ensuring that the color of the food remains unchanged.

Step 203: input the processed image to be tested into the trained maturity detection model for feature recognition to obtain the maturity detection result of the food to be tested.

The structure of the maturity detection model of the embodiment of the present application is specifically shown in FIG18, which extracts features from the first image and the second image respectively to obtain the first feature of the food to be tested in the first image and the second feature of the food to be tested in the second image. Since the first image contains all the food to be tested, and the second image contains a portion of the food to be tested, the attention of the first feature is corrected by using the second feature to make the overall feature of the food richer. Finally, the extracted features are processed by conventional models such as convolution, upsampling, adaptive average pooling, flattening, and fully connected layers to perform feature recognition on the attention-corrected food features to obtain the maturity detection result of the food to be tested. Among them, the CBS shown in FIG18 is a feature extraction module, which is used to perform convolution, normalization, and SILU function activation on the input image to obtain the features of the food to be tested in the input image.

The attention modification process of the embodiment of the present application is shown in FIG19. After obtaining the first and second features of the same three dimensions (height H, width W, and number of channels C), they are processed into two-dimensional features of the same height and number of channels through the Reshape function. The processed two-dimensional first and second features are subjected to Softmax normalization, and the normalization result and the first feature are subjected to feature fusion with a preset weight coefficient, and then feature recognition is performed on the result of feature fusion to obtain the maturity detection result.

During implementation, the second image after HSV adjustment can be processed to generate multiple images. Specifically, after the HSV adjustment is performed on the second image, the adjusted second image can be processed to generate multiple images so that the number of second images is an even number. Then the even-numbered second images and the original image to be tested (1 image) are input into the model together for maturity detection, so that the maturity detection results for the odd-numbered images can be obtained, and then the largest number of the same maturity detection results are selected as the final maturity detection results of the food to be tested. For example, a maturity test is performed on 4 second images and 1 image to be tested, and 3 mature results and 2 immature results are obtained, then it is finally determined that the food to be tested has been baked to maturity. In addition, the confidence level of the maturity detection corresponding to each image can be weighted to formulate a reasonable decision-making plan, which is not limited in this application.

After obtaining the maturity test result of the food to be tested, if it is determined that the maturity test result indicates that the food to be tested has reached the specified baking state, the smart oven is controlled to output a prompt message indicating that the food is baked. In addition, when the baking tray contains only the same type of food, that is, there is only the food to be tested in the oven, the smart oven can be controlled to stop baking after determining that the food to be tested has reached the specified baking state.

In the above process, the HSV value of the pixel is fine-tuned by increasing or decreasing it, and during the adjustment process, it is ensured that the RGB conversion of the HSV value before and after the adjustment can obtain the same color. Since the color of the food is an evaluation index to measure its maturity, the image features of the food can be enhanced without affecting the determination of the maturity index of the food itself, thereby improving the accuracy of the food maturity detection.

Next, the trust characteristics of the application process of the technical solution of this application are analyzed. Through experimental measurements, the technical solution of this application is applied to baking equipment. After the image to be tested is collected by the built-in camera of the baking equipment and transmitted to the processor, the maturity test result can be obtained within 500 milliseconds, which meets the real-time characteristics of the trust characteristics. In addition, through statistical analysis of the test data, it is found that the accuracy of judging the type and maturity of the food in the image to be tested from the perspective of the whole screen can reach 100%, which meets the characteristics of reliability and generalizability in the trust characteristics.

In addition, after the processor feeds back the maturity test results to the front end (such as the display screen of the baking equipment), the user can choose to accept the intelligent suggestions or not, and the working process can be intervened by humans or other intelligent agents (users), which is in line with the controllability feature of the trustworthy characteristics. In addition, in the reproducible environment, the test found that placing the same ingredients in different positions of the baking equipment (such as different shelves of the oven), in different containers, with the increase of baking temperature and external light switches, did not interfere with the test results, which is in line with the reproducibility feature of the trustworthy characteristics.

Based on the same inventive concept, the embodiment of the present application provides a baking device 160 as shown in FIG. 20 , including: a data transmission unit 161 and a processor 162 ;

The data transmission unit 161 is configured to: acquire an image to be tested in response to a detection indication; wherein the image to be tested includes the food to be tested;

The processor 162 is configured to: change the color value HSV of the pixel point in the image to be tested based on a preset pixel adjustment method, so that the color obtained by performing the color value RGB conversion of the pixel point before the HSV value is changed is the same as the color obtained by performing the RGB conversion of the pixel point after the HSV value is changed;

The processed image to be tested is input into the trained maturity detection model for feature recognition to obtain the maturity detection result of the food to be tested.

In some possible embodiments, the image to be tested is obtained by:

Receiving the type of food to be tested indicated by the user;

Capturing an image of a baking area of the smart oven to obtain a target image;

Inputting the target image into a trained target detection model for feature recognition to obtain food information of each food in the target image; wherein the food information includes the type of food and a detection frame representing the location of the food;

The image to be tested is determined according to a target detection frame in the target image; wherein the type of food corresponding to the target detection frame is the type of food to be tested.

In some possible embodiments, the image to be tested includes a first image and a second image; the first image includes all the food to be tested in the target image, and the second image includes any food to be tested in the first image;

The maturity test result is obtained in the following way:

Performing feature extraction on the first image and the second image respectively to obtain a first feature of the food to be tested in the first image and a second feature of the food to be tested in the second image;

Normalizing the first feature and the second feature, and fusing the normalized result with the first feature;

The maturity detection result is obtained by performing feature recognition on the result of feature fusion.

In some possible embodiments, to determine the image to be detected according to the target detection frame in the target image, the processor 162 is configured to:

Acquire vertex coordinate information of each target detection frame in the pixel coordinate system of the target image; wherein the vertex coordinate information includes the minimum horizontal coordinate and the minimum vertical coordinate of each first vertex on the target detection frame, and the maximum horizontal coordinate and the maximum vertical coordinate of each second vertex; the first vertex is the angle vertex of two specified borders in the target detection frame, and the line connecting the first vertex and the second vertex is the diagonal line of the target detection frame;

Determining the minimum circumscribed rectangular area of all target detection frames in the target image according to the vertex coordinate information of each target detection frame;

The first image is determined according to the minimum circumscribed rectangular area, and the second image is determined according to the target detection frame in the first image.

In some possible embodiments, before determining the first image according to the minimum circumscribed rectangular area, the processor 162 is further configured to:

Adjusting the grayscale value of the designated area within the minimum circumscribed rectangular area to a preset value;

The designated area is determined according to a designated detection frame, and the designated detection frame is a detection frame corresponding to each non-to-be-tested food in the minimum circumscribed rectangular area.

In some possible embodiments, the food information further includes a detection confidence of the target detection model for each detection frame; and to perform determining the second image according to the target detection frame in the first image, the processor 162 is configured to:

The second image is determined according to the target detection box with the highest detection confidence in the first image.

In some possible embodiments, the color value HSV of the pixel point in the image to be tested is changed based on a preset pixel adjustment method, and the processor 162 is configured as follows:

Perform multiple rounds of iterative adjustments on the HSV value of the pixel point, and determine in each round of iteration whether the colors obtained by converting the color value RGB before and after the adjustment of the HSV value are the same; if they are not the same, enter the next round of iteration; if they are the same, use the HSV value obtained in this round as the adjusted HSV value of the pixel point;

Among them, any round of iterative adjustment includes:

Determine the current round correction value and current round adjustment method according to the current iteration round number; wherein the first round correction value is a preset value, and the non-first round correction value is a specified multiple of the first round correction value; the first round adjustment method is to increase or decrease the HSV value, and the non-first round adjustment method is opposite to the first round adjustment method;

The current round adjustment method is used to adjust the HSV value obtained in the previous round with the current round correction value.

In some possible embodiments, after obtaining the maturity test result of the food to be tested, the processor 162 is further configured to:

If the maturity detection result indicates that the food to be tested has reached a specified baking state, the intelligent oven is controlled to output a prompt message indicating that baking of the food to be tested is complete.

The baking device 160 can also communicate with one or more external devices (e.g., keyboards, pointing devices, etc.), and can also communicate with one or more devices that enable users to interact with the baking device 160, and/or communicate with any device (e.g., routers, modems, etc.) that enables the baking device 160 to communicate with one or more other baking devices. Such communication can be performed through an input/output (I/O) interface. In addition, the baking device 160 can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and/or public networks, such as the Internet) through a network adapter. The network adapter communicates with other modules for the baking device 160 through a bus. It should be understood that, although not shown in the figure, other hardware and/or software modules can be used in conjunction with the baking device 160, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the instructions can be executed by the processor 162 of the above-mentioned device to complete the above-mentioned method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

In an exemplary embodiment, a computer program product is also provided, including a computer program/instruction, which, when executed by the processor 162, implements any of the methods for detecting the maturity of food provided in the present application.

In an exemplary embodiment, various aspects of a method for detecting the maturity of food provided in the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a method for detecting the maturity of food according to various exemplary embodiments of the present application described above in this specification.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

The program product for detecting the maturity of food materials of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a baking device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also conventional procedural programming languages such as language or similar programming languages. The program code can be executed entirely on the user baking device, partially on the user device, as a separate software package, partially on the user baking device and partially on the remote baking device, or entirely on the remote baking device or server. In the case of a remote baking device, the remote baking device can be connected to the user baking device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external baking device (for example, using an Internet service provider to connect through the Internet).

It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and/or one step may be decomposed into multiple steps.

Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

The present application is described with reference to the flowcharts and/or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and/or box in the flowchart and/or block diagram, as well as the combination of the processes and/or boxes in the flowchart and/or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable image scaling device to produce a machine, so that the instructions executed by the processor of the computer or other programmable image scaling device produce a device for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable image scaling device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable image scaling device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows in the flowchart and/or one or more blocks in the block diagram.

Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims (10)

1. A method for detecting the maturity of food materials, comprising:

responding to the detection instruction, and acquiring an image to be detected; wherein the image to be measured comprises food materials to be measured;

Changing a color value HSV of a pixel point in the image to be detected based on a preset pixel adjustment mode, so that the color obtained by performing color value RGB conversion on the pixel point before the HSV value is changed is the same as the color obtained by performing RGB conversion on the pixel point after the HSV value is changed;

Inputting the processed image to be detected into a trained maturity detection model for feature recognition, and obtaining the maturity detection result of the food material to be detected.

2. The method according to claim 1, wherein the image to be measured is obtained by:

Receiving the type of the food material to be detected indicated by a user;

Acquiring an image of a baking area of the intelligent oven to obtain a target image;

Inputting the target image into a trained target detection model for feature recognition to obtain food material information of each food material in the target image; the food material information comprises the type of food materials and a detection frame for representing the positions of the food materials;

determining the image to be detected according to a target detection frame in the target image; the types of the food materials corresponding to the target detection frame are the types of the food materials to be detected.

3. The method of claim 2, wherein the image to be measured comprises a first image and a second image; the first image comprises all food materials to be detected in the target image, and the second image comprises any food material to be detected in the first image;

wherein, the maturity detection result is obtained by the following method:

Respectively extracting the characteristics of the first image and the second image to obtain a first characteristic of the food material to be detected in the first image and a second characteristic of the food material to be detected in the second image;

Normalizing the first feature and the second feature, and carrying out feature fusion on a normalization processing result and the first feature;

and obtaining the maturity detection result by carrying out feature recognition on the feature fusion result.

4. The method of claim 2, wherein the determining the image to be measured according to the target detection frame in the target image comprises:

Obtaining vertex coordinate information of each target detection frame in a pixel coordinate system of the target image; the vertex coordinate information comprises a minimum abscissa and a minimum ordinate of each first vertex on the target detection frame, and a maximum abscissa and a maximum ordinate of each second vertex; the first vertex is an included angle vertex of two specified frames in the target detection frame, and a connecting line of the first vertex and the second vertex is a diagonal line of the target detection frame;

Determining a minimum circumscribed rectangular area containing all target detection frames in the target image according to the vertex coordinate information of each target detection frame;

and determining the first image according to the minimum circumscribed rectangular area, and determining the second image according to a target detection frame in the first image.

5. A method according to claim 3, wherein before said determining said first image from said minimum bounding rectangular region, said method comprises:

the gray value of the appointed area in the minimum circumscribed rectangular area is adjusted to a preset value;

The specified area is determined according to a specified detection frame, and the specified detection frame is a detection frame corresponding to each non-food material to be detected in the minimum circumscribed rectangular area.

6. The method of claim 3, wherein the food material information further includes a detection confidence of the target detection model for each detection frame; the determining the second image according to the target detection frame in the first image includes:

and determining the second image according to the target detection frame with highest detection confidence in the first image.

7. The method according to claim 1, wherein the changing the color value HSV of the pixel point in the image to be measured based on the preset pixel adjustment method includes:

Performing multi-round iterative adjustment on the HSV value of the pixel point, and determining whether colors obtained by performing color value RGB conversion on the HSV value before and after adjustment are the same in each round of iterative process; if the HSV values are different, the next iteration is carried out, and if the HSV values are the same, the HSV values obtained in the round are used as the HSV values after the pixel point adjustment;

Wherein, any round of iterative adjustment includes:

Determining a correction value and an adjustment mode of the current iteration round according to the current iteration round number; the correction value of the first round is a preset value, and the correction value of the non-first round is a designated multiple of the correction value of the first round; the adjustment mode of the first wheel is to increase or decrease the HSV value, and the adjustment mode of the non-first wheel is opposite to the adjustment mode of the first wheel;

And the HSV value obtained in the previous round is adjusted by adopting the round adjustment mode through the round correction value.

8. The method according to any one of claims 1 to 7, wherein after obtaining the result of detecting the maturity of the food material to be measured, the method further comprises:

And if the maturity detection result indicates that the food material to be detected reaches the appointed baking state, controlling the intelligent oven to output prompt information indicating that baking of the food material to be detected is completed.

9. A baking apparatus comprising a data transmission unit and a processor;

The data transmission unit is configured to: responding to the detection instruction, and acquiring an image to be detected; wherein the image to be measured comprises food materials to be measured;

The processor is configured to: changing a color value HSV of a pixel point in the image to be detected based on a preset pixel adjustment mode, so that the color obtained by performing color value RGB conversion on the pixel point before the HSV value is changed is the same as the color obtained by performing RGB conversion on the pixel point after the HSV value is changed;

Inputting the processed image to be detected into a trained maturity detection model for feature recognition, and obtaining the maturity detection result of the food material to be detected.

10. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program comprising program instructions which, when executed by a computer, cause the computer to perform the method of any of claims 1-8.