CN115035237A - Control method, cooking equipment and storage medium - Google Patents

Abstract

The application discloses a control method, a cooking device and a storage medium. The control method comprises the following steps: responding to a starting signal, and acquiring a plurality of target food material images with different angles; constructing an initial three-dimensional model of the target food material according to the plurality of target food material images; and optimizing the initial three-dimensional model to obtain a final three-dimensional model of the target food material. The target food material images at different angles are obtained, subsequent stereoscopic vision calculation is facilitated, and a multi-azimuth image is provided for accurately constructing a final three-dimensional model of the target food material.

Description

Control method, cooking equipment and storage medium

technical field

The present application relates to the technical field of cooking equipment, and more particularly, to a control method, cooking equipment and non-volatile computer-readable storage medium.

Background technique

With the improvement of living standards, dietary health has received more and more attention from users. The information of ingredients obtained through cooking equipment can provide certain reference for cooking and healthy diet management, such as managing food intake and calorie intake. Among them, 3D modeling of ingredients is the basis for obtaining relevant information, and how to perform 3D modeling of ingredients based on the actual situation of cooking equipment has become an urgent problem to be solved.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a method for controlling a cooking device, the method comprising:

In response to the activation signal, acquiring a plurality of images of the target food material with different angles;

constructing an initial three-dimensional model of the target food material according to a plurality of images of the target food material;

The initial three-dimensional model is optimized to obtain the final three-dimensional model of the target food material.

In this way, after receiving the activation signal, it starts to acquire a plurality of images of the target food material with different angles. Then, based on the acquired image, an initial three-dimensional model of the target ingredient is constructed. Finally, the model is optimized to obtain the final three-dimensional model of the target ingredient. Obtaining images of target ingredients from different angles is conducive to subsequent stereo vision calculations, and provides multi-directional images for accurately constructing the final 3D model of the target ingredients.

In some embodiments, the control method further includes:

generating the activation signal in response to the detected cooking activation signal;

In response to the activation signal, before the cooking starts, a plurality of images of the target ingredient with different angles are acquired.

In this way, a start signal is generated after the detected cooking start signal, and after receiving the start signal, a plurality of images of target ingredients with different angles are acquired before cooking starts. Therefore, a three-dimensional model of the target ingredient can be obtained before cooking, which is convenient for subsequent cooking operations.

In some embodiments, the control method further includes:

In response to the detected door closing operation of the user, when the included angle between the door body and the heating chamber of the cooking device is less than a predetermined angle, generating the activation signal;

In response to the activation signal, during the closing process of the door of the cooking device, one image of the target food material is acquired at predetermined time intervals to acquire a plurality of images of the target food material with different angles.

In this way, it is detected that the user performs an operation of closing the door, and when the included angle formed by the door and the heating chamber of the cooking device is smaller than a preset angle, an activation signal is generated. By judging whether the activation signal is generated by the user closing the door and the included angle between the door and the heating chamber, the function of automatically acquiring multi-directional images of the target ingredients can be realized. During the process of closing the door body, an image of the target food material is acquired at predetermined time intervals, thereby acquiring a plurality of images of the target food material with different angles. The accuracy of the final three-dimensional model of the target food material can be controlled by setting the target food material images to be acquired at predetermined time intervals.

In some embodiments, the constructing an initial three-dimensional model of the target food material according to a plurality of images of the target food material includes:

Two-dimensional image segmentation is performed on a plurality of the target food material images;

Remove background information of non-target ingredients in multiple 2D images;

The initial three-dimensional model is constructed according to a plurality of images from which background information has been removed.

In this way, the method for constructing an initial three-dimensional model of a target food material by using a plurality of target food material images may include performing two-dimensional image segmentation on the plurality of target food food material images. The background information of the segmented image is removed, and the background information includes non-target ingredient information. Finally, an initial 3D model is constructed from multiple images with background information removed. Obtain a three-dimensional model constructed from all target food images with background information removed, so that the constructed initial three-dimensional model retains only the target food part.

In some embodiments, the constructing an initial three-dimensional model of the target food material according to a plurality of images of the target food material includes:

performing feature extraction on a plurality of the target food material images to perform feature matching among the plurality of target food material images;

generating a camera pose according to the feature matching result and preset information;

Obtain the depth value of the target food material according to a plurality of images of the target food material and the camera posture;

generating the initial three-dimensional model according to the depth value;

The background information of non-target ingredients in the initial three-dimensional model is removed.

In this way, the method for constructing an initial three-dimensional model of a target food material by using a plurality of target food material images may include extracting features of the plurality of target food food material images, and matching these features. Generate camera poses based on matching results and preset information. The depth value of the target food material is obtained by combining the camera posture after processing the multiple target food material images. Generate an initial 3D model based on the depth values. Finally, the background parts of the non-target ingredients in the initial 3D model are removed. A 3D model containing only the target ingredients can be obtained.

In some embodiments, the optimizing the initial three-dimensional model to obtain the final three-dimensional model of the target food material includes:

Filling processing is performed on the initial three-dimensional model after the background information is removed to obtain the final three-dimensional model of the target food material.

In this way, after the construction of the initial three-dimensional model is completed and the background part is removed, the initial three-dimensional model is optimized. The optimization process includes removing the background information of the initial three-dimensional model, the background information including non-target ingredients. Then, the initial three-dimensional model with the background information removed is filled in to obtain the final three-dimensional model of the target food material. By removing background information and filling the initial 3D model, the 3D model of the target ingredient can be constructed more accurately.

In some embodiments, the removing the background information of the non-target ingredients in the initial three-dimensional model includes:

Perform object segmentation on the initial three-dimensional model according to the position distribution information and surface texture information of the initial three-dimensional model;

The part of the three-dimensional model corresponding to the target food item is selected to remove the background information of the non-tested food item in the initial three-dimensional model.

In this way, the method of removing background information from the initial three-dimensional model may include segmenting the initial three-dimensional model according to the position distribution information and surface texture information of the initial three-dimensional model. The divided parts are compared with the target ingredients, and the parts corresponding to the target ingredients are selected for subsequent filling operations. In this way, the background information of the non-tested ingredients in the initial 3D model can be removed.

In some embodiments, the filling process on the initial three-dimensional model after removing the background information to obtain the final three-dimensional model of the target food material includes:

Detect the incomplete area in the 3D model after removing the background information;

The incomplete area is filled according to the surrounding information of the incomplete area to obtain the final three-dimensional model of the target food material.

In this way, the method of filling in the initial three-dimensional model may include detecting the three-dimensional model from which the background information has been removed, detecting its incomplete area, and filling the incomplete area according to the surrounding information of the incomplete area. After filling, the final 3D model of the target ingredient is obtained.

Embodiments of the present application also provide a cooking device, comprising:

a heating chamber for heating the target ingredient;

a door that can close the heating chamber;

a camera, which is used to obtain an image of the target ingredient;

The processor is configured to perform three-dimensional modeling of the target food material according to the image by using the above method.

As such, the cooking apparatus includes a heating chamber, a door, a camera, and a processor. The heating chamber can heat the target ingredient. The heating chamber is provided with a door, and the door can seal the heating chamber. The camera is installed in the cooking equipment, and the camera can obtain the image of the target ingredient. The processor can use the control method mentioned in this application to perform three-dimensional modeling of the target food according to the image obtained by the camera. Obtaining images of target ingredients from different angles is conducive to subsequent stereo vision calculations, and provides multi-directional images for accurately constructing the final 3D model of the target ingredients.

In some embodiments, the camera is mounted in the heating chamber and can be rotated at multiple angles.

In this way, before the cooking starts, the door of the cooking device may have been closed. Therefore, a camera that can move or rotate at multiple angles may be installed in the heating chamber at this time to acquire multiple images of the target food material at different angles.

In some embodiments, the camera is mounted on the door and faces the heating chamber.

In this way, it is detected that the user performs an operation of closing the door, and when the included angle formed by the door and the heating chamber of the cooking device is smaller than a preset angle, an activation signal is generated. By judging whether the activation signal is generated by the user closing the door and the angle between the door and the heating chamber, the function of automatically acquiring multi-directional images of the target ingredients can be realized. During the process of closing the door body, an image of the target food material is acquired at predetermined time intervals, thereby acquiring a plurality of images of the target food material with different angles. The accuracy of the final three-dimensional model of the target food material can be controlled by setting the target food material images to be acquired at predetermined time intervals.

Embodiments of the present application further provide a cooking device, including: a processor and a memory, where a computer program is stored in the memory, and when the computer program is executed by the processor, the above-mentioned control method is implemented.

Embodiments of the present application further provide a non-volatile computer-readable storage medium including a computer program, when the computer program is executed by a processor, the processor causes the processor to execute the above-mentioned control method.

The control method, the cooking device and the non-volatile computer-readable storage medium of the present application start the operation of closing the door of the cooking device after receiving the start signal. During the process of closing the door of the cooking device, multiple images of the target food material can be acquired. According to the acquired images of the target food ingredients, the initial three-dimensional model of the target food ingredients can be constructed. By optimizing the initial three-dimensional model, the final three-dimensional model of the target food material can be obtained. Obtaining images of target ingredients from different angles is conducive to subsequent stereo vision calculations, and provides multi-directional images for accurately constructing the final 3D model of the target ingredients.

Additional aspects and advantages of embodiments of the present application will be set forth, in part, in the following description, and in part will be apparent from the following description, or learned by practice of embodiments of the present application.

Description of drawings

The above and/or additional aspects and advantages of the present application will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

1 is a schematic flowchart of a control method of some embodiments of the present application;

2 is a schematic structural diagram of a cooking apparatus according to some embodiments of the present application;

3 is a schematic diagram of a scene of a control method according to some embodiments of the present application;

4 is a schematic flowchart of a control method of some embodiments of the present application;

5 is a schematic flowchart of a control method according to some embodiments of the present application;

6 is a schematic flowchart of a control method according to some embodiments of the present application;

7 is a schematic flowchart of a control method according to some embodiments of the present application;

8 is a schematic flowchart of a control method according to some embodiments of the present application;

9 is a schematic flowchart of a control method according to some embodiments of the present application;

10 is a schematic flowchart of a control method according to some embodiments of the present application;

FIG. 11 is a schematic diagram of a connection state between a non-volatile computer-readable storage medium and a processor according to some embodiments of the present application.

Detailed ways

Embodiments of the present application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, only used to explain the embodiments of the present application, and should not be construed as limitations on the embodiments of the present application.

Generally, with the popularity of smart furniture, smart cooking devices also appear. Smart cooking equipment can diversify the ingredients according to user needs. The control method provided by the present application can be used in an intelligent cooking device to perform three-dimensional modeling of the ingredients of the cooking device, and after the three-dimensional model is formed, the information of the three-dimensional model can be transmitted to the user or the intelligent cooking device. The user or the smart cooking device can clearly understand the content of the ingredients in the cooking device. For example, a three-dimensional model of an ingredient can be presented through a display device, so that a user can visually receive direct ingredient content information. And through the processing and analysis of the constructed three-dimensional model, the required information is extracted for subsequent processing. For example, smart cooking can provide users with suggestions for cooking parameters, such as cooking method, temperature and time, based on information such as the quantity, volume, and weight of ingredients. The control method provided in this application can provide a basis for the subsequent development and implementation of three-dimensional technology in the field of cooking equipment.

Please refer to FIG. 1, the control method of the embodiment of the present application includes the following steps:

01: In response to the start signal, acquire multiple images of the target food material with different angles;

02: Construct the initial 3D model of the target food according to the multiple target food images;

03: Optimizing the initial 3D model to obtain the final 3D model of the target ingredient.

The present application also provides a cooking apparatus including a memory and a processor. A computer program is stored in the memory, and the processor is used for acquiring a plurality of images of the target food material with different angles in response to the activation signal. and construct an initial three-dimensional model of the target food according to the multiple target food images. And the initial three-dimensional model is optimized to obtain the final three-dimensional model of the target food material.

Referring to FIG. 2 and FIG. 3 , an embodiment of the present application further provides a cooking device 300 . The cooking apparatus 300 includes a heating chamber 310 , a door 320 , a camera 330 and a processor 340 . The heating chamber 310 may heat the target ingredient. The heating chamber 310 is provided with a door 320 , and the door 320 can seal the heating chamber 310 . The camera 330 is installed in the cooking apparatus 300, and the camera 330 can acquire the image of the target ingredient. The processor 340 can use the control method mentioned in this application to perform three-dimensional modeling of the target food according to the image obtained by the camera 330 .

Specifically, the cooking device 300 may include micro-steaming devices such as microwave ovens, ovens, and steam ovens.

The start signal refers to a signal that instructs the device that acquires the image of the target food material to start acquiring the image. The manner of generating the activation signal includes generating the activation signal when it is detected that the door body 320 moves away from the original position or is located at a preset position. It may also be a signal generated after the user's cooking operation is detected.

For example, when it is detected that the angle between the door body 320 and the heating chamber 310 forms a predetermined angle or the distance from an end of the door body 320 to a certain end of the heating chamber 310 is a preset distance, a start signal is generated. The activation signal may also be generated according to the user's interaction with the control of the cooking apparatus 300 for closing the door 320 . The start signal may also be generated after the cooking apparatus 300 receives a user's voice request to close the door 320 . There is no restriction on how the start signal is generated.

The device for acquiring multiple images of the target food material with different angles may be a device for acquiring images, such as a digital or analog video camera, a digital camera, and a scanner. The number of devices that acquire the image can be one or more. Wherein, the camera may be a color camera. The camera 330 of the camera may be a movable camera, a rotatable camera, or the like. The camera 330 may be installed anywhere in the cooking apparatus 300 , such as in the heating chamber 310 or on the door 320 .

There is no specific limitation on the manner of acquiring multiple images of the target food material with different angles. For example, a camera that can move or rotate at multiple angles can be installed in the thermal chamber 310, and the camera 330 can be moved or changed in the cooking device 300 to acquire multiple images. Images of the target ingredients from different angles. A fixed or movable and rotatable non-fixed camera 330 may also be installed in the door body 320 facing the heating chamber 310, and a plurality of images of the target food material at different angles may be acquired during the movement of the door body 320.

The target ingredient image refers to an image including target ingredient information acquired during the closing process. A target ingredient is an ingredient whose information needs to be obtained.

The architecture and algorithm used for constructing the three-dimensional model are not specifically limited, for example, it may be a three-dimensional model constructing algorithm based on the combination of the Boosting architecture and Haar features. It can be a 3D model building algorithm based on machine learning and deep learning. It can be a 3D model building algorithm based on machine learning SVM or Bayes learning. It can be a 3D model building algorithm based on geometry analysis. It can also be a texture-based food feature point model building algorithm. Any suitable 3D model reconstruction architecture or algorithm can be selected according to specific implementation conditions.

The initial 3D model refers to the first construction of a 3D model that can reflect the target ingredients.

Optimization processing refers to the subsequent processing of the initial 3D model, so that the processed 3D model can more accurately reflect the target ingredients. The optimization processing method includes filling the part of the initial three-dimensional model that lacks the target food material to obtain a three-dimensional model that fully reflects the target food material.

The final 3D model refers to a 3D model that can fully reflect the target ingredients.

Referring to FIG. 2 and FIG. 3 , in an example, after the processor 340 receives the activation signal, it controls the camera 330 in the cooking device 300 to acquire a plurality of images including target ingredient information at different angles. The processor 340 constructs an initial three-dimensional model according to all the obtained images, and the three-dimensional model can embody the target ingredient. Then, the initial 3D model is optimized to obtain a 3D model that can completely and accurately reflect the target ingredients.

In this way, after receiving the activation signal, it starts to acquire a plurality of images of the target food material with different angles. Then, based on the acquired image, an initial three-dimensional model of the target ingredient is constructed. Finally, the model is optimized to obtain the final three-dimensional model of the target ingredient. Obtaining images of target ingredients from different angles is conducive to subsequent stereo vision calculations, and provides multi-directional images for accurately constructing the final 3D model of the target ingredients.

Referring to Figure 4, in some embodiments, the method further includes:

04: In response to the detected cooking start signal, a start signal is generated;

05: In response to the start signal, before cooking starts, acquire a plurality of images of the target ingredient with different angles.

The processor is used for generating a start signal in response to the detected cooking start signal, and for acquiring a plurality of target food material images with different angles in response to the start signal before cooking starts.

A movable or multi-angle rotatable camera is installed in the heating chamber 310, and the camera 330 moves or changes the angle in the cooking apparatus 300 to acquire multiple images of the target food at different angles.

Understandably, before the cooking starts, the door 320 of the cooking device 300 may have been closed. Therefore, a camera that can move or rotate at multiple angles can be installed in the heating chamber 310 at this time to acquire multiple targets with different angles. food images.

Specifically, the method of generating the cooking start signal includes, but is not limited to, the user interacting with the control of the cooking apparatus 300 to start cooking, the user sends a voice request to start cooking, and the cooking apparatus 300 generates the cooking start signal after receiving the voice request.

Understandably, the user can acquire the three-dimensional model of the target ingredient before preparing to cook. It is convenient for users to perform subsequent cooking operations according to the cooking parameter suggestions obtained from the 3D model.

In one example, the user puts ingredients into the cooking device 300, interacts with the control of the cooking device 300 to start cooking, and generates a cooking start signal, thereby generating the start signal. After receiving the start signal, the processor 340 controls the camera 330 to Before cooking starts, acquire multiple images of the target ingredient from different angles. For example, after the user puts ingredients into the cooking device 300, closes the door 320, and interacts with the control of the cooking device 300 to start cooking, the processor 340 controls the rotatable camera in the heating chamber 310 to take multi-angle shots before preparing for cooking , to obtain multiple images of the target ingredient with different angles.

In this way, a start signal is generated after the detected cooking start signal, and after receiving the start signal, a plurality of images of target ingredients with different angles are acquired before cooking starts. Therefore, a three-dimensional model of the target ingredient can be obtained before cooking, which is convenient for subsequent cooking operations.

Referring to Figure 5, in some embodiments, the method further includes:

06: In response to the detected door closing operation of the user, when the included angle between the door body and the heating chamber of the cooking device is less than a predetermined angle, a start signal is generated;

07: In response to the activation signal, during the closing process of the door of the cooking device, acquire an image of the target ingredient at predetermined time intervals to acquire a plurality of images of the target ingredient with different angles.

The processor is configured to generate an activation signal when the angle between the door body and the heating chamber of the cooking device is smaller than a predetermined angle in response to the detected door closing operation of the user. And in response to the start signal, during the closing process of the door of the cooking device, one image of the target ingredient is acquired at predetermined time intervals to acquire multiple images of the target ingredient with different angles.

Referring to FIG. 2 , a camera 330 facing the heating chamber 310 may be installed on the door body 320 .

It can be understood that, as the door body 320 moves, the angle at which the image is acquired also changes. Therefore, when a three-dimensional model is constructed from multi-angle images, the number of cameras 330 used can be reduced. For example, it is only necessary to install a fixed camera 330 facing the heating chamber 310 on the door body 320. With the movement of the door body 320, the angle at which the camera 330 obtains images also changes continuously, so that multiple multi-angle target ingredients can be obtained. image. Compared with installing the movable and rotatable camera 330 in the heating chamber 310 , or installing the fixed camera 330 in the heating chamber 310 in multiple directions. The fixed camera 330 is installed on the door body 320 , which can reduce the cost of the camera 330 while achieving the multi-angle image of the target food material.

Referring to FIG. 2 and FIG. 3 , in one example, the door 320 starts to close, and a start signal is generated. The processor 340 receives the start signal and controls the camera 330 on the door 320 facing the heating chamber 310 . The 320 continuously acquires pictures during the entire closing process, so as to obtain multi-angle images of the target ingredients. For example, when beef is placed in the heating chamber 310, the user starts to close the door 320, the processor receives the start signal, and controls the camera 330 on the door 320 facing the heating chamber 310, and the camera 330 on the door 320 is continuously captured during the closing process of the door 320. pictures, so as to obtain images including beef from multiple angles.

Specifically, the door closing operation of the user is detected, when the user closes the door body 320, the detection device detects the angle formed by the door body 320 and the heating chamber 310, and the processor 340 receives that the angle obtained by the detection device is smaller than the predetermined angle When the start signal is generated. The predetermined angle can be preset according to user requirements. It can also be set by the manufacturer according to the angle obtained by experiment or market research when it leaves the factory.

The detection device may be an angle sensor.

The closing process of the door body 320 is a process of closing the opening of the heating chamber 310 in the cooking apparatus 300 . The door body 320 may be covered from one side of the opening of the heating chamber 310 to the other side of the opening, so that the opening is in a closed state. For example, from the right to the left, or from the bottom to the top, the specific coverage direction is not limited. The door body 320 may also be divided into multiple parts, and each part is covered accordingly, so that the opening is in a closed state.

The predetermined time can be preset according to user needs. It can also be set by the manufacturer according to the time obtained by experiment or market research when it leaves the factory.

The device for acquiring the image of the target food material may acquire an image of the target food material at predetermined intervals. Each predetermined period of time may be the same or different.

In an example, when the user closes the door 320, the detection device detects the angle formed by the door 320 and the heating chamber 310, and the processor 340 generates a start signal when the angle obtained by the detection device is smaller than a predetermined angle. After the processor 340 receives the start signal, it controls the device for acquiring an image to acquire an image of a target food material at intervals of a predetermined time, so as to obtain a plurality of images of the target food material. For example, the preset angle is 90°, and the preset time is 1/7s. When the angle formed by the door body 320 of the cooking apparatus 300 and the heating chamber 310 is 80° during the closing process, which is smaller than the preset angle of 90°, a start signal is generated. After the processor 340 receives the start signal, it controls the camera 330 every time An image including beef is acquired at intervals of 1/7s, thereby acquiring multiple images including beef.

In this way, an operation of closing the door 320 by the user is detected, and when the included angle formed by the door 320 and the heating chamber 310 of the cooking apparatus 300 is smaller than a preset angle, an activation signal is generated. The user can determine whether to generate an activation signal by closing the door 320 and the included angle between the door 320 and the heating chamber 310 , so that the function of automatically acquiring multi-directional images of the target food material can be realized. During the process of closing the door body 320, an image of the target food material is acquired at predetermined time intervals, thereby acquiring a plurality of images of the target food material with different angles. The accuracy of the final three-dimensional model of the target food material can be controlled by setting the target food material images to be acquired at predetermined time intervals.

Referring to Figure 6, in some embodiments, step 02 includes:

020: Perform two-dimensional image segmentation on multiple target food images;

021: Remove the background information of the non-target ingredients in the multiple two-dimensional images;

022: Construct an initial three-dimensional model according to the multiple images from which background information has been removed.

The processor is configured to perform two-dimensional image segmentation on multiple images of target food materials, to remove background information of non-target ingredients in the multiple two-dimensional images, and to construct an initial three-dimensional model according to the multiple images from which background information has been removed.

Specifically, the two-dimensional image refers to a plane image without depth value information. There is no restriction on the segmentation method of the two-dimensional image. For example, the two-dimensional image is segmented by using network segmentation model algorithms such as fully convolutional network (FCN), SegNet, and Enet.

The background information refers to information that does not contain the target ingredients, such as the hardware parts of the cooking apparatus 300 such as the grill pan and the cavity wall of the heating chamber 310 .

In one example, after the device for acquiring images acquires a plurality of two-dimensional images of the target food material, the processor 340 divides these images into a part containing the target food material and a background part by using a related algorithm. Then, the background portion is removed, leaving a two-dimensional image containing only the target ingredient portion. Finally, an initial 3D model is constructed from a 2D image containing only the portion of the target ingredient. For example, beef is being placed in the cooking device 300. After the camera 330 acquires a plurality of two-dimensional images including the beef and the cooking device 300, the processor 340 divides these images into parts that only include beef and parts that only include beef. Contains cooking equipment 300 sections. Then, the portion containing only the cooking apparatus 300 is removed, leaving a two-dimensional image of the portion containing only the beef. Finally, the initial 3D model is constructed from the 2D image of the portion containing only the beef.

In this way, the method for constructing an initial three-dimensional model of a target food material by using a plurality of target food material images may include performing two-dimensional image segmentation on the plurality of target food food material images. The background information of the segmented image is removed, and the background information includes non-target ingredient information. Finally, an initial 3D model is constructed from multiple images with background information removed. Obtain a three-dimensional model constructed from all target food images with background information removed, so that the constructed initial three-dimensional model retains only the target food part.

Referring to Figure 7, in some embodiments, step 02 includes:

023: Perform feature extraction on the multiple target food material images to perform feature matching among the multiple target food material images;

024: Generate a camera pose according to the result of feature matching and preset information;

025: Acquire the depth value of the target food according to the multiple target food images and the camera posture;

026: Generate an initial 3D model according to the depth value;

027: Remove the background information of the non-target ingredients in the initial 3D model.

The processor is configured to perform feature extraction on the plurality of target food material images to perform feature matching among the plurality of target food material images. and is used to generate the camera pose based on the result of feature matching and preset information. and is used to obtain the depth value of the target food according to the multiple target food images and the camera pose. and for generating the initial 3D model based on the depth values. and background information for removing non-target ingredients in the initial 3D model.

Specifically, the feature may be a pixel or a set of pixels, and may include point-like features, line-like features, and regional features, and the like.

Feature matching refers to calculating the selected features after extraction, establishing the corresponding relationship between the features, and mapping the physical points in the same space to the mapping points in different images, so as to obtain the corresponding parallax image.

The preset information refers to the preset information for generating the camera pose from the auxiliary feature matching result. The information may include internal information such as optical or geometric parameters of the camera or external information such as the specific physical location of the camera's external environment. The way of obtaining the preset information can be obtained through experiments or calculations of the camera, and can also be obtained from the manufacturer.

The camera pose refers to the specific position of the camera in the world coordinate system.

The depth value refers to the distance of the pixel of the target ingredient from the camera in the world coordinate system.

There are many ways to generate the camera pose, and there are no specific restrictions. For example, the camera pose can be generated according to the preset information of the camera's internal parameters and the result of feature matching.

Information outside the camera, such as physical location information of the features of the hardware structure in the heating chamber 310 of the cooking apparatus 300, may also be used as the preset information. In the acquired image of the target food material, these features are extracted for retrieval and positioning, and the camera pose is generated according to the relationship between the preset physical location information of these features. The hardware structure may include a hot air outlet of the heating chamber 310, a customized baking pan, and the like. Features of the hardware structure may include shapes and patterns that are unique to each piece of hardware.

Understandably, at present, images can be obtained through a depth camera, and the depth camera can directly obtain the depth value of the target food, so as to reconstruct the three-dimensional model of the food. However, depth cameras have high cost and cannot be miniaturized. For example, depth cameras based on binocular vision cannot be miniaturized due to the limitation of camera baselines. And the emission beam is easily interfered by the strong light in the oven, resulting in the failure of reconstruction. For example, the emission beams of structured light depth cameras and LiDAR depth cameras are easily disturbed by the strong light in the oven, resulting in reconstruction failure. In the embodiments of the present application, the depth value of the target food material is obtained by calculating the image of the target food material from multiple angles, and a depth camera is not required.

In one example, after the device for acquiring images acquires images of the target food material from multiple angles, feature extraction is performed on the acquired images, and feature matching is performed between the images to obtain a matching result. The camera pose is generated according to the matching result and preset information. At the same time, the multiple acquired images are detected, and the epipolar geometric constraints are calculated for the overlapping object parts in the images, and the depth value of the target food is obtained according to the acquired images and the camera posture. Generate an initial three-dimensional model according to the depth value, and finally remove the background information that does not include the target ingredients in the initial three-dimensional model. For example, after the camera 330 acquires images including the beef and the cooking device 300 from multiple angles, the camera 330 performs feature extraction on the acquired images, and performs feature matching between the images to obtain a matching result. The camera pose is generated according to the matching result and preset information. At the same time, the acquired images are detected, and the epipolar geometric constraints are calculated for the overlapping object parts in the images, and the depth value of the object is obtained according to the acquired images and the camera pose. A first three-dimensional model is generated from the depth values, the three-dimensional model capable of representing the beef and cooking apparatus 300 portion. Finally, the interior of the cooking device 300 of the model is removed, leaving only the part that embodies the beef.

In this way, the method for constructing an initial three-dimensional model of a target food material by using a plurality of target food material images may include extracting features of the plurality of target food food material images, and matching these features. Generate camera poses based on matching results and preset information. The depth value of the target food material is obtained by combining the camera posture after processing the multiple target food material images. Generate an initial 3D model based on the depth values. Finally, the background parts of the non-target ingredients in the initial 3D model are removed. A 3D model containing only the target ingredients can be obtained.

Referring to Figure 8, in some embodiments, step 03 includes:

030: Perform filling processing on the initial three-dimensional model after removing the background information to obtain a final three-dimensional model of the target food material.

The processor is configured to perform filling processing on the initial three-dimensional model after removing the background information to obtain the final three-dimensional model of the target food material.

Specifically, the filling process is a means of repairing a three-dimensional model. The constructed 3D model may not fully reflect the target ingredients, that is, the 3D model has defective areas that cannot reflect the target ingredients. By filling in the defective area, a 3D model that can reflect the target ingredients can be obtained. The method of filling processing is not limited, such as using the copy method to copy the most edge pixels. Reflection method, copy the pixels in the image of interest on both sides or take the most edge pixel as the axis, and then copy and constant method to both sides, fill with output, etc.

In one example, after the initial three-dimensional model is constructed, the background information is removed. Then, the defect areas in the 3D model that cannot fully reflect the target ingredients are processed by means of filling. Get a 3D model containing only the complete target ingredients. For example, the initial three-dimensional model constructed includes a beef portion and a cooking apparatus 300 portion. The portion of the cooking apparatus 300 that is the background portion is removed, leaving a three-dimensional model containing the beef portion. Then, fill in the incomplete part of the beef to obtain a three-dimensional model containing the whole beef.

In this way, after the construction of the initial three-dimensional model is completed and the background part is removed, the initial three-dimensional model is optimized. The optimization process includes filling in the initial three-dimensional model with background information removed to obtain the final three-dimensional model of the target ingredient. Filling in the initial 3D model can more accurately construct the 3D model of the target ingredient.

Referring to Figure 9, in some embodiments, step 027 includes:

0270: Perform object segmentation on the initial 3D model according to the position distribution information and surface texture information of the initial 3D model;

0271: Select the part of the 3D model corresponding to the target ingredient to remove the background information of the non-tested ingredient in the initial 3D model.

The processor is configured to perform object segmentation on the initial three-dimensional model according to the position distribution information and surface texture information of the initial three-dimensional model. and is used to select the part of the 3D model corresponding to the target food item to remove the background information of the non-tested food item in the initial 3D model.

Specifically, the position distribution information refers to the information of the physical positions occupied by the three-dimensional model in the three-dimensional world. It can be represented by three-dimensional coordinates.

Texture information refers to the physical attribute information of the surface of the 3D model, which is a macroscopic manifestation of a certain local repetition pattern of the eigenvalue intensity in the image. Information such as grayscale and color can be included.

In one example, after the initial three-dimensional model is constructed, the three-dimensional model is segmented by a clustering algorithm according to the position distribution information and surface texture information of the three-dimensional model, the background part is removed, and the target food material part is left. For example, according to the position distribution information and surface texture information of the three-dimensional model, the part embodied in the cooking device 300 and the part of the beef are segmented by a clustering algorithm. Portions of cooking apparatus 300 are removed, leaving portions of beef.

In this way, the method of removing background information from the initial three-dimensional model may include segmenting the initial three-dimensional model according to the position distribution information and surface texture information of the initial three-dimensional model. The divided parts are compared with the target ingredients, and the parts corresponding to the target ingredients are selected for subsequent filling operations. In this way, the background information of the non-tested ingredients in the initial 3D model can be removed.

Referring to Figure 10, in some embodiments, step 030 includes:

0300: Detect the incomplete area in the 3D model after removing the background information;

0301: Fill in the incomplete area according to the surrounding information of the incomplete area to obtain the final three-dimensional model of the target ingredient.

The processor is used for detecting the incomplete area in the three-dimensional model after removing the background information, and for filling the incomplete area according to the peripheral information of the incomplete area to obtain the final three-dimensional model of the target food material.

The incomplete area refers to the area that cannot fully reflect the target ingredients.

Peripheral information refers to analyzing the edge of the incomplete area and the information around the edge, and analyzes the physical information including the three-dimensional shape, structure and slope, which can be used to fill the incomplete area.

There is no specific limitation on the method of filling the incomplete area according to the surrounding information of the incomplete area. For example, the incomplete area is divided into small areas along the edge of the area, and each small area is analyzed and selected for registration with regular geometry, and the small area of the 3D model is filled with the geometry with the smallest registration error. After all the small areas are filled, the incomplete areas are filled. And delete the redundant part of the geometry that is higher than the surface of the edge 3D model after filling. The missing area is filled. Regular geometry can be regular geometry such as spheres, cylinders, and cubes. In one example, it is detected that a certain area of the beef 3D model is incomplete, the incomplete area is divided into 5 small areas along the edge of the incomplete area, and each area is registered in turn, and filled with sphere, cylinder, sphere, Cylinders and cubes. After filling, the upper end of a cylinder is higher than the surface of the area, and the upper end of the cylinder that is higher than the surface of the area is deleted. The incomplete area of the beef 3D model is filled in.

For another example, along the edge of the incomplete region, the entire edge is divided into multiple edges, and the slope of each edge is analyzed. The slope of the edge is used to extend the surface to the center of the incomplete area, so that the 3D model forms a closed shape, thereby generating a complete 3D model. In one example, it is detected that a certain area of the beef three-dimensional model has defects, the slope of each edge of the area is analyzed, and the slope is used to extend the surface to the center of the defect area. The beef three-dimensional model is formed into a closed shape, thereby generating a complete beef three-dimensional model.

For another example, the structural information in the peripheral part of the edge of the incomplete area is analyzed, the structural information is fitted to a symmetry plane, and the incomplete area is filled with symmetry. In one example, a defect is detected in a certain area of the beef three-dimensional model, the structural information of the peripheral part of the edge of the area is analyzed, the structural information is fitted to a symmetry plane, and the defect area is filled with symmetry. Thereby generating a complete three-dimensional model of beef.

For another example, the incomplete 3D model is filled with the prior information of the neural network, the incomplete-complete 3D model pair is collected as the neural network training data, and the neural network is trained to learn how to extract the surrounding information from the incomplete 3D model. Use this information to fill in the incomplete area. In one example, it is detected that a certain area of the beef 3D model is incomplete, the incomplete 3D model is filled with the prior information of the neural network, the incomplete-complete 3D model pair is collected as the neural network training data, and the neural network is trained to make it learn How to extract the surrounding information of the incomplete area from the incomplete 3D model and use the information to fill the incomplete area. Thereby generating a complete three-dimensional model of beef.

In this way, the method of filling in the initial three-dimensional model may include detecting the three-dimensional model from which the background information has been removed, detecting its incomplete area, and filling the incomplete area according to the surrounding information of the incomplete area. After filling, the final 3D model of the target ingredient is obtained.

Referring to FIG. 11 , an embodiment of the present application further provides a non-volatile computer-readable storage medium 100 including a computer program 101 . When the computer program 101 is executed by one or more processors 200, the one or more processors 200 are caused to execute the control method of any one of the above-described embodiments.

Referring to FIG. 1, for example, when the computer program 101 is executed by one or more processors 200, the processor 200 executes the following steps:

01: In response to the start signal, acquire multiple images of the target food material with different angles;

02: Construct the initial 3D model of the target food according to the multiple target food images;

03: Optimizing the initial 3D model to obtain the final 3D model of the target ingredient.

In the description of this specification, reference to the description of the terms "certain embodiments," "in an example," "exemplarily," etc. means that a particular feature, structure, material, or characteristic described in connection with an embodiment or example is included in the present application at least one embodiment or example of . In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine the different implementations or examples described in this specification and the features of the different implementations or examples without conflicting each other.

Any description of a process or method in the flowcharts or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing a specified logical function or step of the process , and the scope of the preferred embodiments of the present application includes alternative implementations in which the functions may be performed out of the order shown or discussed, including performing the functions substantially concurrently or in the reverse order depending upon the functions involved, which should It is understood by those skilled in the art to which the embodiments of the present application belong.

Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limitations to the present application. Embodiments are subject to variations, modifications, substitutions and alterations.

Claims (13)

1. A method of controlling a cooking apparatus, the method comprising:

responding to the starting signal, and acquiring a plurality of target food material images with different angles;

constructing an initial three-dimensional model of the target food material according to the plurality of target food material images;

and optimizing the initial three-dimensional model to obtain a final three-dimensional model of the target food material.

2. The control method according to claim 1, characterized in that the method comprises:

generating the start signal in response to the detected cooking start signal;

and responding to the starting signal, and acquiring a plurality of target food material images with different angles before cooking is started.

3. The control method according to claim 1, characterized in that the method further comprises:

responding to the detected door closing operation of the user, and generating the starting signal when an included angle between the door body and a heating chamber of the cooking equipment is smaller than a preset angle;

responding to the starting signal, and in the closing process of the door body of the cooking equipment, acquiring one target food material image at preset time intervals so as to acquire a plurality of target food material images with different angles.

4. The control method according to claim 1, wherein the constructing of the initial three-dimensional model of the target food material according to the plurality of target food material images comprises:

performing two-dimensional image segmentation on a plurality of target food material images;

removing background information of non-target food materials in the plurality of two-dimensional images;

and constructing the initial three-dimensional model according to the plurality of images with the background information removed.

5. The control method according to claim 1, wherein the constructing of the initial three-dimensional model of the target food material according to the plurality of target food material images comprises:

performing feature extraction on the target food material images to perform feature matching among the target food material images;

generating a camera attitude according to the feature matching result and preset information;

acquiring the depth value of the target food material according to the plurality of target food material images and the camera postures;

generating the initial three-dimensional model according to the depth value;

and removing background information of non-target food materials in the initial three-dimensional model.

6. The control method according to claim 5, wherein the optimizing the initial three-dimensional model to obtain the final three-dimensional model of the target food material comprises:

and filling the initial three-dimensional model with the background information removed to obtain a final three-dimensional model of the target food material.

7. The control method according to claim 6, wherein the removing of the background information of the non-target food materials in the initial three-dimensional model comprises:

carrying out object segmentation on the initial three-dimensional model according to the position distribution information and the surface texture information of the initial three-dimensional model;

and selecting a three-dimensional model part corresponding to the target food material to remove the background information of the non-measured food material in the initial three-dimensional model.

8. The control method according to claim 6, wherein the filling the initial three-dimensional model from which the background information is removed to obtain the final three-dimensional model of the target food material comprises:

detecting a defective area in the three-dimensional model after background information is removed;

and filling the incomplete area according to the peripheral information of the incomplete area to obtain a final three-dimensional model of the target food material.

9. A cooking apparatus, characterized by comprising:

a heating chamber for heating a target food material;

a door body capable of sealing the heating chamber;

the camera is used for acquiring an image of the target food material;

a processor for three-dimensional modeling of the target food material according to the image using the method of any of claims 1-8.

10. The cooking apparatus of claim 9, wherein the camera is mounted within the heating chamber and is rotatable at multiple angles.

11. The cooking apparatus according to claim 9, wherein the camera is mounted on the door body and faces the heating chamber.

12. A cooking apparatus, characterized in that the cooking apparatus comprises a processor and a memory, in which a computer program is stored, which computer program, when being executed by the processor, realizes the control method according to any one of claims 1-8.

13. A non-transitory computer-readable storage medium comprising a computer program that, when executed by a processor, causes the processor to perform the control method of any one of claims 1-8.

Images