JP7641122B2 - Cooking device and trained model generation method - Google Patents

Description

The present invention relates to a cooking device and a method for generating a trained model.

The cooking device described in Patent Document 1 includes an operation input device as an input means, a heating means, an imaging device, and a central information processing device. The user uses the operation input device to set a desired heating menu from menus presented in advance. The imaging device images the food before heating. If the heating menu selected by the operation input device is grill-type heating, the central information processing device processes the captured image before heating into an image of the food after it has been finished so as to add a browned color to the surface of the food. The heating means then heats the food (object to be heated). The imaging device further captures an image of the food while it is being heated. The central information processing device determines whether the image of the processed food matches the image of the food captured while it is being heated, and if it determines that they match, stops the heating output.

Japanese Patent Application Publication No. 9-229373

However, in the cooking device described in Patent Document 1, the user must use the operation input device to select a desired heating menu from menus presented in advance. Therefore, the effort of setting the heating menu is a burden for the user.

The present invention has been made in consideration of the above problems, and aims to provide a cooking device and a learning model generation method that can reduce the effort required of a user to operate the cooking device.

According to one aspect of the present application, a cooking device includes a heating unit, an imaging unit, an estimation unit, and a heating control unit. The heating unit heats an object to be heated. The imaging unit images the object to be heated while the object is being heated. The estimation unit estimates the name of the object to be heated while the object is being heated based on the imaging result of the imaging unit. The heating control unit controls the heating unit according to the estimation result of the estimation unit.

According to another aspect of the present application, a trained model generation method includes the steps of acquiring training data and generating a trained model that estimates the name of a heated object by machine learning the training data. The training data includes a training target dish image showing a dish and name information showing the name of the dish.

The present invention reduces the effort required for users to operate a cooking device.

1 is a block diagram showing a cooking device according to a first embodiment of the present invention. 2 is a block diagram showing an outline of the operation of the cooking device according to the first embodiment. FIG. 1 is a block diagram showing a learning device according to a first embodiment; FIG. 4 is a diagram illustrating an example of learning data according to the first embodiment. FIG. 11 is a diagram illustrating another example of learning data according to the first embodiment. FIG. 11 is a diagram illustrating another example of learning data according to the first embodiment. FIG. 11 is a diagram illustrating another example of learning data according to the first embodiment. FIG. 11 is a diagram illustrating another example of learning data according to the first embodiment. FIG. 11 is a diagram illustrating another example of learning data according to the first embodiment. FIG. 2 is a block diagram showing input information and output information of a trained model according to the first embodiment. 1 is a flowchart showing a trained model generation method according to the first embodiment. 5A to 5C are diagrams illustrating an example of an imaging result of an imaging unit according to the first embodiment. 4 is a diagram illustrating a table stored in a storage unit according to the first embodiment. FIG. 5A to 5C are diagrams illustrating an example of an operation of a heating unit according to the first embodiment. 4 is a flowchart showing a process executed by the cooking device according to the first embodiment. FIG. 11 is a block diagram showing a heating cooker according to a second embodiment of the present invention. FIG. 11 is a diagram showing an example of a food image according to a second embodiment. FIG. 11 is a diagram showing another example of a food image according to the second embodiment. FIG. 11 is a diagram showing another example of a food image according to the second embodiment. FIG. 11 is a diagram showing another example of a food image according to the second embodiment. FIG. 11 is a diagram showing another example of a food image according to the second embodiment. FIG. 11 is a diagram showing another example of a food image according to the second embodiment. 13A and 13B are diagrams showing an imaging result of an imaging unit according to a second embodiment and an imaging result after processing. 10 is a flowchart showing a process executed by a cooking device according to a second embodiment. FIG. 13 is a diagram showing learning data according to a modified example of the first embodiment.

The following describes an embodiment of the present invention with reference to the drawings. Note that in the drawings, the same or equivalent parts are designated by the same reference symbols and will not be described repeatedly.

First, the cooking device 1 will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the cooking device 1.

The cooking device 1 is, for example, a microwave oven, an oven, a rice cooker, or an electric pot. Below, the present invention will be described using an example in which the cooking device 1 is a microwave oven. As shown in FIG. 1, in the first embodiment, the cooking device 1 includes a control unit 10, a storage unit 20, a heating unit 30, an imaging unit 40, an operation unit 50, and a main body unit (not shown).

The main body includes a heating chamber. An object to be heated is placed in the heating chamber. The object to be heated is, for example, food ingredients.

The heating unit 30 heats the object to be heated placed in the heating chamber. In the first embodiment, the heating unit 30 has, for example, a microwave heating function, a hot air circulation heating function, and a grill heating function. The microwave heating function is, for example, a function for cooking the object to be heated by radiating microwaves into the heating chamber. The hot air circulation heating function is, for example, a function for cooking the object to be heated by circulating hot air inside the heating chamber to make the temperature inside the heating chamber uniform, or by blowing hot air directly onto the top surface of the object to be heated. The grill heating function is a function for cooking the object to be heated mainly by exposing the object to radiant heat. That is, the heating unit 30 includes, for example, a microwave supply unit and a heater.

The imaging unit 40 is, for example, a camera. Specifically, the imaging unit 40 is, for example, a camera configured with a CCD (Charge Coupled Device) digital television camera or the like. The imaging unit 40 is, for example, installed outside the heating chamber so as to face an opening formed in a wall surface constituting the heating chamber. The imaging unit 40 images the heated object, for example, when the heated object is being heated by the heating unit 30. In the first embodiment, the imaging unit 40 images the heated object, for example, at predetermined time intervals. Specifically, the imaging unit 40 images the heated object, for example, every time a predetermined time elapses while the heated object is being heated by the heating unit 30. For example, the predetermined time is 2 seconds.

The storage unit 20 includes a storage device and stores computer programs such as software and various data. Specifically, the storage unit 20 includes a main storage device (e.g., a semiconductor memory) such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and an auxiliary storage device such as a solid state drive and/or a hard disk drive. The storage unit 20 may include removable media. The storage unit 20 is an example of a storage medium (e.g., a non-transitory computer-readable storage medium).

The memory unit 20 also stores, for example, a learned model LM and a table Tb.

In the first embodiment, the trained model LM is constructed by machine learning the training data. Hereinafter, in this specification, the training data may be referred to as training data TD. The training data TD will be described later.

In addition, in the first embodiment, in table Tb, each of the names of a plurality of dishes (pizza, toast, sponge cake, etc.) is associated with a heating time and a heating intensity corresponding to the dish. That is, the memory unit 20 stores the name of a dish in association with a heating time and a heating intensity corresponding to the dish. The heating intensity is, for example, the temperature or wattage when heating the object to be heated.

The operation unit 50 includes, for example, operation keys provided on the cooking device 1, or a touch panel provided on a display unit such as a liquid crystal display. Note that the cooking device 1 does not have to include a display unit. The operation keys also include input devices such as a mouse and/or a keyboard. The operation unit 50 is operated by a user to receive, for example, an instruction to start heating by the heating unit 30. The operation unit 50 is an example of a "reception unit."

The control unit 10 includes a processor such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). The memory unit 20 and the control unit 10 may be configured as an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). The processor is an example of a "computer."

The processor of the control unit 10 executes a computer program stored in the storage device of the storage unit 20 to control the storage unit 20, the heating unit 30, the imaging unit 40, and the operation unit 50. The processor of the control unit 10 also executes a computer program stored in the storage device of the storage unit 20 to function as the estimation unit 11, the heating control unit 12, and the measurement unit 13. In other words, the control unit 10 includes the estimation unit 11, the heating control unit 12, and the measurement unit 13.

The estimation unit 11 estimates the name of the object to be heated based on the imaging results of the imaging unit 40 while the object to be heated is being heated.

The heating control unit 12 controls the heating unit 30 according to the estimation result of the estimation unit 11. For example, the heating control unit 12 stops the heating of the object by the heating unit 30 after a certain time has elapsed according to the estimation result of the estimation unit 11. More specifically, the heating control unit 12 controls the heating unit 30 so that the object is heated for a heating time according to the name of the object estimated by the estimation unit 11.

The measuring unit 13 measures the time for which the object to be heated is heated. For example, the measuring unit 13 measures the time for which the object to be heated is heated from when the heating unit 30 starts heating the object to when the estimation unit 11 estimates the name of the object to be heated.

Next, an overview of the operation of the cooking device 1 will be described with reference to Figures 1 and 2. Figure 2 is a block diagram showing an overview of the operation of the cooking device 1.

As shown in FIG. 2, first, the measurement unit 13 starts measuring the time it takes for the object to be heated when the operation unit 50 receives an instruction to start heating by the heating unit 30.

Meanwhile, the estimation unit 11 inputs the food image HG, which is the image captured by the imaging unit 40, to the trained model LM when the heated object is being heated. Specifically, the estimation unit 11 inputs the food image HG to the trained model LM while the heated object is being heated. The food image HG shows the heated object imaged by the imaging unit 40.

Based on the input food image HG, the trained model LM outputs name information MN indicating the name of the heated object shown in the input food image HG. Specifically, the trained model LM outputs name information MN of the heated object shown in the food image HG while the heated object is being heated.

Then, the estimation unit 11 acquires the name information MN output by the trained model LM while the heated object is being heated. That is, the estimation unit 11 estimates the name of the heated object while the heated object is being heated based on the cooking image HG. Therefore, the estimation unit 11 can accurately estimate the name of the heated object. Then, the estimation unit 11 inputs the name information MN to the heating control unit 12.

The heating control unit 12 controls the heating unit 30 according to the name information MN. That is, the heating control unit 12 controls the heating unit 30 according to the estimation result of the estimation unit 11. Therefore, the object to be heated can be heated according to the name of the object to be heated without the user having to operate the operation unit 50 in advance to select a heating method (menu) for the object to be heated. As a result, the user's effort in operating the heating cooker 1 can be reduced.

Specifically, the heating control unit 12 determines the heating time and heating intensity according to the object to be heated, according to the estimation result of the estimation unit 11. Therefore, the object to be heated can be heated according to the name of the object, without the user having to operate the operation unit 50 in advance to select the heating time and heating intensity for the object to be heated. As a result, the object to be heated can be heated with a heating time and heating intensity according to the name of the object, while reducing the effort required for the user to operate the heating cooker 1. In other words, the object to be heated can be appropriately heated according to the name of the object to be heated.

Specifically, for example, when the object to be heated is toast, the estimation unit 11 estimates that the object to be heated is toast. Meanwhile, for example, in table Tb stored in the memory unit 20, toast is associated with a heating time indicating 11 minutes and a heating intensity indicating 180°C. Therefore, the heating control unit 12 determines the time for heating the object to be 11 minutes and the heating intensity for heating the object to be 180°C. Therefore, the heating control unit 12 controls the heating unit 30, for example, so that the object to be heated is heated at 180°C for 11 minutes. As a result, the toast is properly toasted.

More specifically, in the first embodiment, the heating control unit 12 determines the remaining heating time for the object to be heated based on the determined heating time and the measurement result of the measurement unit 13. Therefore, the heating control unit 12 controls the heating unit 30 so as to heat the object to be heated for the determined remaining heating time. As a result, it is possible to prevent the object to be heated from being excessively heated by the heating unit 30. In other words, it is possible to appropriately heat the object to be heated.

Next, with reference to FIG. 3, a learning device 100 that generates a trained model LM will be described. FIG. 3 is a block diagram showing the learning device 100. The learning device 100 is, for example, a computer. The learning device 100 generates the trained model LM by machine learning the training data TD. Then, the learning device 100 outputs the trained model LM. As described above, the trained model LM estimates the name of the heated object indicated by the input cooking image HG. The training data TD is the subject of machine learning by the learning device 100.

As shown in FIG. 3, the learning device 100 includes a processing unit 110 and a memory unit 120.

The processing unit 110 includes a processor such as a CPU and a GPU. The memory unit 120 includes a storage device and stores data and computer programs. The processor of the processing unit 110 executes computer programs stored in the storage device of the memory unit 120 to perform various processes. For example, the memory unit 120 may include a main storage device and an auxiliary storage device, and may also include removable media, similar to the memory unit 20 of the cooking appliance 1. The memory unit 120 is, for example, a non-transitory computer-readable storage medium.

The memory unit 120 stores a learning program LP. The learning program LP is a program for executing a machine learning algorithm to find certain rules from multiple pieces of learning data TD and generate a learned model LM that expresses the found rules.

The machine learning algorithm is not particularly limited as long as it is supervised learning, and may be, for example, a decision tree, a nearest neighbor method, a naive Bayes classifier, a support vector machine, or a neural network. Therefore, the trained model LM includes a decision tree, a nearest neighbor method, a naive Bayes classifier, a support vector machine, or a neural network. In the machine learning that generates the trained model LM, the backpropagation method may be used.

For example, the neural network includes an input layer, one or more intermediate layers, and an output layer. Specifically, the neural network is a deep neural network (DNN), a recurrent neural network (RNN), or a convolutional neural network (CNN), and performs deep learning. For example, the deep neural network includes an input layer, multiple intermediate layers, and an output layer.

The processing unit 110 performs machine learning on multiple pieces of training data TD based on the training program LP. As a result, certain rules are found from the multiple pieces of training data TD, and a trained model LM is generated. In other words, the trained model LM is constructed by machine learning the training data TD. The memory unit 120 stores the trained model LM.

Specifically, the processing unit 110 finds certain rules between the training target dish images LG and the name information MN contained in the training data TD, and generates a trained model LM. The training target dish images LG are explanatory variables, and the name information MN is a target variable. In embodiment 1, the training target dish images LG indicate dishes. The name information MN indicates the name of the dish shown in the training target dish images LG.

More specifically, the processing unit 110 calculates multiple learned parameters by performing machine learning on multiple pieces of training data TD based on the training program LP, and generates a trained model LM including one or more functions to which the multiple learned parameters are applied. The trained parameters are parameters (coefficients) obtained based on the results of machine learning using the multiple pieces of training data TD.

The trained model LM causes the computer to function so as to receive input information and output output information. In other words, the trained model LM receives input information and outputs output information. Specifically, the trained model LM causes the computer to function so as to estimate the name of the dish shown in the input food image HG. In other words, the trained model LM estimates the name of the dish shown in the input food image HG.

Next, the learning data TD will be described with reference to Figures 4A to 4F. Figure 4A shows an example of the learning data TD. Each of Figures 4B to 4F shows another example of the learning data TD. Hereinafter, the learning data TD shown in Figure 4A may be referred to as learning data TD1, the learning data TD shown in Figure 4B may be referred to as learning data TD2, the learning data TD shown in Figure 4C may be referred to as learning data TD3, the learning data TD shown in Figure 4D may be referred to as learning data TD4, the learning data TD shown in Figure 4E may be referred to as learning data TD5, and the learning data TD shown in Figure 4F may be referred to as learning data TD6.

In the first embodiment, the learning data TD includes a learning target dish image LG and name information MN. Also, in the first embodiment, the name of the dish shown in the learning target dish image LG is determined, for example, by a user. That is, the learning target dish image LG is labeled with name information MN indicating the name of the dish determined by the user. To facilitate understanding, the browning of the dish is indicated by hatching. Specifically, the wider the spacing between the hatching, the lighter the browning, and the narrower the spacing between the hatching, the darker the browning.

As shown in Figures 4A to 4F, the multiple learning data TD correspond to multiple dishes (e.g., pizza and toast, etc.). Specifically, the multiple learning data TD shown in Figures 4A to 4C correspond to pizza. Furthermore, the multiple learning data TD shown in Figures 4D to 4F correspond to toast. Furthermore, each of the learning target dish images LG of the multiple learning data TD shows a dish at a different time during the process of heating the same dish.

For example, each of the learning data TD1 to TD3 shown in Figures 4A to 4C represents a pizza at a different time during the process of heating the pizza.

As shown in FIG. 4A, the pizza shown in the training target dish image LG of training data TD1 is an earlier stage of pizza than the pizza shown in the training target dish image LG of training data TD2 and the training target dish image LG of training data TD3. In embodiment 1, the training target dish image LG of training data TD1 is labeled with name information MN indicating "pizza." In this specification, the name information MN indicating "pizza" may be referred to as name information MN1.

As shown in FIG. 4B, the pizza shown in the training subject dish image LG of training data TD2 represents a pizza at a stage between the pizza shown in the training subject dish image LG of training data TD1 and the pizza shown in the training subject dish image LG of training data TD3. In addition, in embodiment 1, the training subject dish image LG of training data TD2 is labeled with name information MN1 indicating "pizza."

Furthermore, as shown in FIG. 4C, the pizza shown in the training target dish image LG of training data TD3 represents a later stage of pizza than the pizza shown in the training target dish image LG of training data TD1 and the training target dish image LG of training data TD2. Also, in embodiment 1, the training target dish image LG of training data TD3 is labeled with name information MN1 indicating "pizza."

For example, each of the learning data TD4 to TD6 shown in Figures 4D to 4F shows toast at a different time during the heating process.

As shown in FIG. 4D, the toast shown in the training target dish image LG of training data TD4 is an earlier stage of toast than the toast shown in the training target dish image LG of training data TD5 and the toast shown in the training target dish image LG of training data TD6. In addition, in embodiment 1, the training target dish image LG of training data TD4 is labeled with name information MN indicating "toast." In this specification, the name information MN indicating "toast" may be referred to as name information MN2.

As shown in FIG. 4E, the toast shown in the training subject dish image LG of the training data TD5 represents a stage of toast between the toast shown in the training subject dish image LG of the training data TD4 and the toast shown in the training subject dish image LG of the training data TD6. In addition, in embodiment 1, the training subject dish image LG of the training data TD5 is labeled with name information MN2 indicating "toast."

Furthermore, as shown in FIG. 4F, the toast shown in the training subject dish image LG of training data TD6 shows a later stage of toast than the toast shown in the training subject dish image LG of training data TD4 and the toast shown in the training subject dish image LG of training data TD5. In embodiment 1, the training subject dish image LG of training data TD6 is labeled with name information MN2 indicating "toast."

The name information MN may be expressed as a character string such as "pizza" or "toast," or may be indicated by identification information (ID) corresponding to each of "pizza" and "toast."

Next, the input information and output information of the trained model LM will be described with reference to FIG. 5. FIG. 5 is a block diagram showing the input information and output information of the trained model LM.

As shown in FIG. 5, the input information of the trained model LM is a food image HG, which is the result of imaging by the imaging unit 40. Based on the input food image HG, the trained model LM outputs name information MN of the heated object shown in the food image HG. In the example shown in FIG. 5, the trained model LM estimates that the name of the heated object shown in the food image HG is "toast." Therefore, the trained model LM outputs name information MN indicating "toast."

Next, the learning method executed by the learning device 100 will be described with reference to Figs. 3 and 6. Fig. 6 is a flowchart showing the trained model generation method according to the first embodiment. As shown in Fig. 6, the trained model generation method includes steps S41 to S44.

As shown in FIG. 3 and FIG. 6, in step S41, the processing unit 110 of the learning device 100 acquires learning data TD. For example, the processing unit 110 acquires learning data TD input by a user.

Next, in step S42, the processing unit 110 performs machine learning on the learning data TD based on the learning program LP.

Next, in step S43, the processing unit 110 determines whether or not a learning end condition is satisfied. The learning end condition is a condition that is determined in advance for ending machine learning. For example, the learning end condition is that the number of iterations has reached a specified number.

If a negative determination (No) is made in step S43, processing proceeds to step S41. As a result, machine learning is repeated.

On the other hand, if the determination in step S43 is positive (Yes), processing proceeds to step S44.

In step S44, the processing unit 110 outputs a model (one or more functions) to which the latest multiple parameters (coefficients), i.e., multiple learned parameters (coefficients), are applied as the learned model LM. The memory unit 120 then stores the learned model LM.

As described above, the learning device 100 executes steps S41 to S44 to generate the learned model LM.

Next, the operation of the cooking device 1 will be described with reference to Figs. 7 to 9. First, with an object to be heated placed in the heating chamber, the operation unit 50 receives an instruction to start heating by the heating unit 30. Then, the heating control unit 12 controls the heating unit 30 to heat the object to be heated. As a result, the heating unit 30 starts heating the object to be heated. At this time, the heating unit 30 heats the object to be heated with, for example, a predetermined heating intensity. In addition, as the heating unit 30 starts heating the object to be heated, the measurement unit 13 starts measuring the time it takes for the object to be heated.

Figure 7 is a diagram showing a food image HG that is the result of imaging by the imaging unit 40. As shown in Figure 7, the estimation unit 11 inputs the food image HG that is the result of imaging by the imaging unit 40 to the trained model LM. The trained model LM outputs name information MN indicating "pizza" based on the input food image HG. Then, the estimation unit 11 acquires the name information MN output by the trained model LM.

Figure 8 is a diagram showing table Tb stored in memory unit 20. As shown in Figure 8, in table Tb, a heating time Ht and a heating intensity Hs are associated with name information MN. The heating control unit 12 determines the heating time Ht and heating intensity Hs for the heated object according to the estimation result of the estimation unit 11, i.e., the name information MN acquired by the estimation unit 11. For example, since the name information MN of the heated object shown in the cooking image HG shown in Figure 7 is estimated to be "pizza", the heating control unit 12 determines the heating time Ht to be "25 minutes" and the heating intensity Hs to be "250°C".

9 is a diagram showing an example of the operation of the heating unit 30. As shown in FIG. 9, the measurement result of the measuring unit 13 when the estimation unit 11 acquires the name information MN, that is, the heating time Ct of the heated object is 4 minutes. In other words, when 4 minutes have passed since the heating unit 30 started heating the heated object, the name information MN is acquired based on the food image HG, which is the imaging result. Then, the heating control unit 12 determines the heating time Ht to be "25 minutes" and the heating intensity Hs to be "250°C". Therefore, the heating control unit 12 determines the remaining heating time Rt for the heated object based on the measurement result (heating time Ct) of the measuring unit 13 and the determined heating time Ht. Then, the heating control unit 12 controls the heating unit 30 so that the heating of the heated object is terminated in response to the passage of the remaining heating time Rt after the estimation unit 11 acquires the name information MN.

Next, the process executed by the cooking device 1 will be described with reference to FIG. 10. FIG. 10 is a flowchart showing the process executed by the cooking device 1 according to the first embodiment. The process executed by the cooking device 1 includes steps S1 to S10.

As shown in FIG. 10, in step S1, the operation unit 50 receives an instruction to start heating by the heating unit 30. Then, the heating control unit 12 controls the heating unit 30 to heat the object to be heated. As a result, the heating unit 30 heats the object to be heated. In addition, as the heating unit 30 starts heating the object to be heated, the measurement unit 13 starts measuring the time for which the object to be heated is heated.

In step S2, the control unit 10 controls the imaging unit 40 to capture an image of the object to be heated. As a result, the imaging unit 40 captures an image of the object to be heated.

In step S3, the estimation unit 11 inputs the captured food image HG into the learned model LM.

In step S4, the estimation unit 11 determines whether or not the trained model LM has output name information MN based on the cooking image HG input in step S3. If a negative determination (No) is made in step S4, i.e., if the trained model LM has not output name information MN, or if the trained model LM has output "unknown", processing proceeds to step S2. On the other hand, if a positive determination (Yes) is made in step S4, i.e., if the trained model LM has output name information MN, processing proceeds to step S5. Cases where the trained model LM has not output name information MN and cases where the trained model LM has output "unknown" will be described later.

In step S5, the estimation unit 11 acquires the name information MN output by the trained model LM. Then, the estimation unit 11 inputs the acquired state information SB to the heating control unit 12.

In step S6, the heating control unit 12 determines the heating time Ht and heating intensity Hs for the heated object based on the estimation result of the estimation unit 11.

In step S7, the heating control unit 12 determines the remaining heating time Rt for the heated object based on the measurement result of the measurement unit 13 and the heating time Ht determined in step S6.

In step S8, the heating control unit 12 adjusts the heating intensity of the heating unit 30 based on the heating intensity Hs determined in step S6.

In step S9, the heating control unit 12 determines whether the remaining heating time Rt determined in step S7 has elapsed since the remaining heating time Rt was determined in step S7. If a positive determination (Yes) is made in step S9, the process proceeds to step S10. On the other hand, if a negative determination (No) is made in step S9, the process returns to step S9.

In step S10, the heating control unit 12 controls the heating unit 30 to stop heating the object to be heated. As a result, the heating unit 30 stops heating.

(Embodiment 2)
Next, a cooking device 1a according to a second embodiment will be described with reference to Fig. 11. Fig. 11 is a block diagram showing a cooking device 1a according to a second embodiment. In the second embodiment, the cooking device 1a is provided with a turntable 60, and the estimation unit 11a estimates the name of the object to be heated by pattern matching. The following mainly describes the differences between the second embodiment and the first embodiment.

The cooking device 1a includes a control unit 10a, a memory unit 20a, a heating unit 30, an imaging unit 40, an operation unit 50, and a turntable 60.

The object to be heated is placed on the turntable 60. In the second embodiment, while the heating unit 30 heats the object to be heated, the turntable 60 rotates, for example, at a substantially constant angular velocity. That is, the turntable 60 rotates with the object to be heated placed on it.

The control unit 10a includes a processor such as a CPU and an MPU. The processor of the control unit 10 executes a computer program stored in the storage device of the storage unit 20a to control the storage unit 20a, the heating unit 30, the imaging unit 40, the operation unit 50, and a motor (not shown) for rotating the turntable 60.

The processor of the control unit 10 executes a computer program stored in the storage device of the memory unit 20a, and functions as an estimation unit 11a, a heating control unit 12, a measurement unit 13, and a correction unit 14. In other words, the control unit 10 includes an estimation unit 11a, a heating control unit 12, a measurement unit 13, and a correction unit 14. The estimation unit 11a and the correction unit 14 will be described later.

The memory unit 20a stores pattern matching data PD and a table Tb. The pattern matching data PD includes a plurality of food images MG and name information MN.

Next, the pattern matching data PD stored in the storage unit 20a will be described with reference to Figures 12A to 12F. Figure 12A shows an example of the pattern matching data PD. Each of Figures 12B to 12F shows another example of the pattern matching data PD. Hereinafter, the pattern matching data PD shown in Figure 12A may be described as pattern matching data PD1, the pattern matching data PD shown in Figure 12B may be described as pattern matching data PD2, the pattern matching data PD shown in Figure 12C may be described as pattern matching data PD3, the pattern matching data PD shown in Figure 12D may be described as pattern matching data PD4, the pattern matching data PD shown in Figure 12E may be described as pattern matching data PD5, and the pattern matching data PD shown in Figure 12F may be described as pattern matching data PD6.

As shown in Figures 12A to 12F, the food images MG of the multiple pattern matching data PD each correspond to multiple dishes (e.g., pizza, toast, etc.). Furthermore, each of the multiple food images MG shows a single dish at a different time during the heating process.

For example, each of the dish images MG of the pattern matching data PD1 to PD3 shown in Figures 12A to 12C shows a pizza at a different time during the heating process.

As shown in FIG. 12A, the pizza shown in the dish image MG of the pattern matching data PD1 represents a pizza at an earlier stage than the pizza shown in the dish image MG of the pattern matching data PD2 and the pizza shown in the dish image MG of the pattern matching data PD3. In embodiment 2, the dish image MG of the pattern matching data PD1 is labeled with name information MN1 indicating "pizza."

Also, as shown in FIG. 12B, the pizza shown in the dish image MG of the pattern matching data PD2 represents a pizza at a stage between the pizza shown in the dish image MG of the pattern matching data PD1 and the pizza shown in the dish image MG of the pattern matching data PD3. In embodiment 2, the dish image MG of the pattern matching data PD2 is labeled with name information MN1 indicating "pizza."

Furthermore, as shown in FIG. 12C, the pizza shown in the dish image MG in the pattern matching data PD3 represents a later stage of pizza than the pizza shown in the dish image MG in the pattern matching data PD1 and the pizza shown in the dish image MG in the pattern matching data PD2. In embodiment 2, the dish image MG in the pattern matching data PD3 is labeled with name information MN1 indicating "pizza."

For example, each of the pattern matching data PD4 to PD6 shown in Figures 12D to 12F represents toast at a different time during the heating process.

As shown in FIG. 12D, the toast shown in the dish image MG of the pattern matching data PD4 shows an earlier stage of toast than the toast shown in the dish image MG of the pattern matching data PD5 and the toast shown in the dish image MG of the pattern matching data PD6. In embodiment 2, the dish image MG4 is labeled with name information MN2 indicating "toast."

Also, as shown in FIG. 12E, the toast shown in the dish image MG of the pattern matching data PD5 shows a stage of toast between the toast shown in the dish image MG of the pattern matching data PD5 and the toast shown in the dish image MG of the pattern matching data PD6. In embodiment 2, the dish image MG of the pattern matching data PD5 is labeled with name information MN2 indicating "toast."

Furthermore, as shown in FIG. 12F, the toast shown in the dish image MG of the pattern matching data PD6 shows a later stage of toast than the toast shown in the dish image MG4 of the pattern matching data PD4 and the toast shown in the dish image MG5 of the pattern matching data PD5. In embodiment 2, the dish image MG6 of the pattern matching data PD6 is labeled with name information MN2 indicating "toast."

Next, correction of the orientation of the imaging result will be described with reference to FIG. 13. FIG. 13 is a diagram showing a food image HG, which is the imaging result of the imaging unit 40, and a food image HGA, which is the imaging result after correction. As shown in FIG. 13, a case where the heated object is pizza Pz will be described as an example. That is, in the example shown in FIG. 13, pizza Pz is placed on the turntable 60.

The correction unit 14 corrects the orientation of the food image HG based on the angular velocity ω when the turntable 60 rotates. The food image HGA is the food image HG whose orientation has been corrected by the correction unit 14 based on the angular velocity ω and the measurement result of the measurement unit 13 when the heated object is heated. The measurement result of the measurement unit 13 is the time t when the heated object is heated by the heating unit 30. Specifically, the angle θ for correcting the orientation of the food image HG is calculated by θ = ω × t. The correction unit 14 corrects the food image HG by rotating the food image HG by the angle θ in the direction opposite to the direction in which the turntable 60 rotates.

The estimation unit 11a compares the corrected food image HGA with the food image MG stored in the memory unit 20a, and estimates the name of the heated object based on the comparison result. Therefore, the name of the heated object can be estimated more accurately than when the name of the heated object is estimated without correcting the orientation of the food image HG.

In this embodiment, the estimation unit 11a determines whether the matching rate indicated by the comparison result is equal to or greater than a threshold value. The matching rate indicates the matching ratio between the heated object indicated by the food image HGA and the food image MG stored in the memory unit 20a. The threshold value is, for example, 90%. In other words, if the matching rate between the food image HGA and the food image MG is 90% or greater, the estimation unit 11a estimates that the name of the heated object is the name information MN associated with the food image MG. The heating control unit 12 then controls the heating unit 30 according to the estimation result of the estimation unit 11a.

Next, the process executed by the cooking device 1a of the second embodiment will be described with reference to FIG. 14. FIG. 14 is a flowchart showing the process executed by the cooking device 1a.

As shown in FIG. 14, in step S21, the operation unit 50 receives an instruction to start heating by the heating unit 30. Then, the heating control unit 12 controls the heating unit 30 to heat the object to be heated. As a result, the heating unit 30 heats the object to be heated. In addition, as the heating unit 30 starts heating the object to be heated, the measurement unit 13 starts measuring the time for which the object to be heated is heated.

In step S22, the control unit 10 controls the imaging unit 40 to capture an image of the object to be heated. As a result, the imaging unit 40 captures an image of the object to be heated.

In step S23, the correction unit 14 determines whether or not the turntable 60 is rotating. If a positive determination (Yes) is made in step S23, that is, if the correction unit 14 determines that the turntable 60 is rotating, the process proceeds to step S24. On the other hand, if a negative determination (No) is made in step S23, that is, if the correction unit 14 determines that the turntable 60 is not rotating, the process proceeds to step S25. Here, the correction unit 14 may determine whether or not the turntable 60 is rotating by determining whether or not the motor that rotates the turntable 60 is driven.

In step S24, the correction unit 14 corrects the food image HG, which is the image capture result of the imaging unit 40 in step S22, based on the angular velocity ω when the turntable 60 rotates and the measurement result of the measurement unit 13.

In step S25, the estimation unit 11 compares the food image HG, which is the imaging result, with the food image MG. Note that if the food image HG, which is the imaging result, is corrected by the correction unit 14 in step S24, the estimation unit 11 compares the food image HGA, which is the imaging result after correction, with the food image MG.

In step S26, the estimation unit 11 determines whether the matching rate indicated by the comparison result in step S25 is equal to or greater than a threshold value. If a negative determination (No) is made in step S26, i.e., if the matching rate between the imaging result and the food image MG is less than the threshold value, processing proceeds to step S22. On the other hand, if a positive determination (Yes) is made in step S26, i.e., if the matching rate between the imaging result and the food image MG is equal to or greater than the threshold value, processing proceeds to step S27.

In step S27, the estimation unit 11 estimates the name of the object to be heated. Specifically, it obtains the name information MN associated with the food image MG whose matching rate is equal to or greater than a threshold value.

In step S28, the heating control unit 12 determines the heating time Ht and heating intensity Hs for the heated object based on the estimation result of the estimation unit 11.

In step S29, the heating control unit 12 determines the remaining heating time Rt for the heated object based on the measurement result of the measurement unit 13 and the heating time Ht determined in step S6.

In step S30, the heating control unit 12 adjusts the heating intensity of the heating unit 30 based on the heating intensity Hs determined in step S29.

In step S31, the heating control unit 12 determines whether or not the heating time Rt has elapsed since the remaining heating time Rt was determined in step S29. If a positive determination (Yes) is made in step S31, the process proceeds to step S32. On the other hand, if a negative determination (No) is made in step S31, the process returns to step S31.

In step S32, the heating control unit 12 controls the heating unit 30 to stop heating the object to be heated. As a result, the heating unit 30 stops heating.

(Modification)
Next, a cooking device 1 according to a modified example of the first embodiment will be described with reference to Fig. 15. Fig. 15 is a diagram showing learning data Tda obtained by machine learning of a trained model LM of the cooking device 1 according to a modified example of the first embodiment. In the modified example, the main difference between the modified example and the first embodiment is that one learning data Tda includes a plurality of learning target dish images LD. Below, the differences between the modified example and the first embodiment will be mainly described.

As shown in FIG. 15, one learning data set Tda includes multiple learning target dish images LD and name information MN. Specifically, the learning data Tda includes multiple learning target dish images LD1 to LDNX. To make it easier to understand, the browning of the dish is indicated by hatching. Specifically, the wider the spacing between the hatching, the lighter the browning, and the narrower the spacing between the hatching, the darker the browning.

The multiple learning target dish images LD each show a dish at a different time during the process of heating the dish. Specifically, the multiple learning target dish images LD each show a dish captured at a predetermined time interval during the process of heating the dish. In the modified example, the predetermined time is, for example, two seconds. In other words, the learning target dish images LD that the trained model LM in the modified example learns from are cooking videos.

The imaging unit 40 in the modified example captures an image of the heated object at predetermined time intervals, for example. Then, the estimation unit 11 inputs cooking videos, which are multiple imaging results captured at predetermined time intervals, to the trained model LM. That is, in the modified example, the input information of the trained model LM is the video captured by the imaging unit 40. The trained model LM outputs name information MN of the heated object indicated by the cooking video based on the input cooking video.

The above describes the embodiments and modifications of the present invention with reference to the drawings. However, the present invention is not limited to the above embodiments, and can be implemented in various forms without departing from the gist of the present invention. In addition, various inventions can be formed by appropriately combining multiple components disclosed in the above embodiments. For example, some components may be deleted from all components shown in the embodiments. The drawings are mainly schematic drawings of each component to make them easier to understand, and the number of each component shown may differ from the actual number due to the convenience of drawing. In addition, each component shown in the above embodiments is an example, and is not particularly limited, and various modifications are possible within a range that does not substantially deviate from the effects of the present invention.

(1) As described with reference to FIG. 7, the learned model LM outputs name information MN indicating the name of the heated object shown in the cooking image HG based on the cooking image HG input by the estimation unit 11. However, if the learned model LM cannot estimate the name of the heated object based on the cooking image HG input by the estimation unit 11, the learned model LM may not output name information MN, or may output name information MN indicating "unknown". If the learned model LM does not output name information MN or if the learned model LM outputs name information MN indicating "unknown", the estimation unit 11 controls the imaging unit 40 so that the imaging unit 40 images the heated object again. Specifically, the estimation unit 11 controls the imaging unit 40 so that the imaging unit 40 images the heated object a predetermined time after the imaging unit 40 images the heated object last time. That is, the imaging unit 40 images the heated object at predetermined time intervals until the estimation unit 11 estimates the name of the heated object. This prevents the object from being heated with a heating intensity and heating time that is inappropriate for the object.

(2) As described with reference to FIG. 9, the heating control unit 12 determined the remaining heating time Rt for the heated object based on the measurement result (heating time Ct) of the measurement unit 13 and the determined heating time Ht. Then, the heating control unit 12 controlled the heating unit 30 so that heating of the heated object would end in response to the remaining heating time Rt having elapsed after the estimation unit 11 acquired the name information MN. However, the heating control unit 12 may also control the heating unit 30 so that heating of the heated object would end after a certain amount of time has elapsed since the remaining heating time Rt had elapsed.

(3) The order of steps S1 to S10 in FIG. 10 may be changed as appropriate.

(4) The order of steps S21 in FIG. 14 to S32 in FIG. 15 may be changed as appropriate.

(5) As described with reference to Figures 4A to 4C, three learning target dish images LG for one dish were input to the learned model LM. However, as long as the learned model LM can estimate the name of the heated object, it may learn one or two learning target dish images LG for one dish, or it may learn four or more learning target dish images LG.

(6) As described with reference to Figs. 12 and 13, the cooking device 1a according to the second embodiment includes the turntable 60. However, as long as the cooking device 1a heats an object to be heated, the cooking device 1a does not need to include the turntable 60. In this case, the correction unit 14 does not correct the imaging result.

(7) As described with reference to Figures 12 and 13, the estimation unit 11a of the cooking device 1a according to embodiment 2 compares the food image HG, which is the imaging result, with the food image MG stored in the memory unit 20a, and estimates the name of the heated object based on the comparison result. However, as long as the estimation unit 11a can estimate the name of the heated object, the estimation unit 11a may compare the difference between the food images HG, which are multiple imaging results, with the difference between the multiple food images MG stored in the memory unit 20a, and estimate the name of the heated object based on the comparison result. In this case, the difference between the multiple food images MG stored in the memory unit 20a refers to the difference between the multiple food images MG showing a single dish at different times during the heating process.

The present invention can be used in the fields of cooking appliances and trained model generation methods.

Reference Signs List 1: Cooking device 11: Estimation unit 12: Heating control unit 13: Measurement unit 14: Correction unit 20: Memory unit 30: Heating unit 40: Imaging unit 50: Operation unit 60: Turntable HG: Food image Ht: Heating time Hs: Heating intensity LM: Trained model LG: Training target food image MG: Finished image ST: Status information TD: Training data

Claims (5)

A heating unit that heats the object to be heated;
An imaging unit that images the object to be heated while the object to be heated is being heated;
An estimation unit that estimates a name of the object to be heated based on an imaging result of the imaging unit while the object to be heated is being heated;
A memory unit that stores a trained model constructed by machine learning of training data;
a heating control unit that controls the heating unit in response to an estimation result of the estimation unit;
The estimation unit inputs the imaging result into the trained model,
The trained model outputs name information indicating a name of the heated object based on the input imaging result,
The learning data includes a plurality of imaging results of one object to be heated at different times during a process in which the object to be heated is heated,
The storage unit stores a table in which a name of a dish is associated with a heating time and a heating intensity corresponding to the dish,
The heating control unit is
When receiving an instruction to start heating, the heating unit starts heating the object to be heated at a predetermined heating intensity regardless of the object to be heated;
After starting heating of the object to be heated, a heating time and a heating intensity associated with the name of the dish in the table are determined based on the name of the dish corresponding to the name of the object to be heated estimated by the estimation unit;
Based on the determined heating time and the elapsed time from the start of heating of the object to the time when the name of the object is estimated, adjust the remaining heating time for the object from the time when the name of the object is estimated to the time when heating of the object is terminated;
adjusting the heating intensity of the heating unit based on the determined heating intensity;
Heating cooker. The learning data includes a learning target dish image showing a dish and name information showing a name of the dish,
The estimation unit inputs a food image that is the imaging result into the trained model,
The trained model outputs name information indicating the name of the heated object based on the input cooking image,
The cooking device according to claim 1 , wherein the estimation unit acquires the name information output by the trained model. The heating cooker according to claim 2, wherein each of the learning target dish images of the multiple learning data shows one of the dishes at different times during the process of heating the dish. A storage unit is further provided for storing a plurality of dish images and names of the dishes for each dish,
each of the plurality of food images shows the food at a different time during the process of the food being heated;
The cooking device according to claim 1 , wherein the estimation unit compares the imaging result with the food image and estimates the name of the object to be heated based on a comparison result. A turntable on which the object to be heated is placed and rotated;
A measuring unit that measures the time during which the object to be heated is heated;
a correction unit that corrects a direction of the imaging result based on an angular velocity when the turntable rotates and a measurement result of the measurement unit,
The cooking device according to claim 4 , wherein the estimation unit estimates a name of the object to be heated based on the corrected imaging result.

Images