DE102019120008B4 - Method for operating a cooking appliance and cooking appliance - Google Patents

Abstract

Method for operating a cooking appliance (1) with a cooking area (11) for preparing food by means of a treatment device (2). The food is monitored in the cooking area (11) during the cooking process. For this purpose, images of the surface of the food to be cooked are recorded over time by means of a camera device (3). The images each consist of a large number of picture elements and are evaluated by means of at least one processing device (4). At least one measure for shorter-term fluctuations (5) of an image parameter around a longer-term trend (6) caused by increasing cooking of the food is determined. An internal cooking state of the food to be cooked is derived as a function of the measure for the fluctuations (5).

Description

The present invention relates to a method for operating a cooking appliance with at least one cooking area and with at least one treatment device for preparing food to be cooked in the cooking area. To monitor the food to be cooked in the cooking area during the cooking process, images of at least part of the surface of the food to be cooked are recorded over time by means of at least one camera device.

In order to achieve the best possible cooking result, it is usually crucial to take certain properties of the food into account. Such information about the food to be cooked is particularly important for the reliable running of automatic programs. For example, for the preparation using an automatic function, it should be taken into account in which cooking or baking state the food is. It is particularly convenient if certain properties of the food can be recorded and taken into account by the cooking device during the cooking process and also independently.

Cooking devices have therefore become known which monitor the food to be cooked with a camera. However, the extraction of information from the recorded images is often unsatisfactory. For example, the recordings are subject to a large number of disruptive effects under cooking chamber conditions, which cannot be adequately calibrated or delimited in the case of a cooking appliance. In addition, the images usually cannot be used to infer the internal cooking state with sufficient reliability.

From the DE 102014114901 A1 a cooking device and a method for detecting a cooking process that takes place in a cooking device are known. Variable parameters of the food to be cooked are recorded by means of a sensor, in particular by means of a camera, and their changes over time are evaluated. The changes are in particular changes in the color contrast.

The DE 102016215550 A1 describes a method for determining the degree of browning of the food to be cooked, in which a reference image is first recorded of the food to be cooked, then two further measurement images at different brightnesses in the cooking space. A difference image is generated from the measurement images and compared with the reference image.

In the DE 102014113664 A1 a cooking device is disclosed in which the food is cooked by means of a radiation source, a sensor detecting parameters of the food and a controller controls the power of the radiation source as a function of the parameters detected so that the food is in a desired state. A method for cooking food is also disclosed, in which the sensor detects parameters in a local area of the food, and in which a control determines a state of the food on the other side from the recorded parameters.

In a method for operating a cooking appliance according to FIG DE 102017206056 A1 a cooking chamber is irradiated with light of different wavelengths, light reflected in the cooking chamber is measured, the measurement results are evaluated spectroscopically and the operation of the cooking appliance is changed as a function of this. The reflected light is recorded like a pixel by means of a camera.

It is therefore the object of the present invention to enable an improved monitoring and characterization of a product to be cooked in a cooking chamber during a cooking process with a camera device. In particular, the monitoring should take place particularly reliably and preferably also in a cost-optimized manner.

This object is achieved by a method with the features of claim 1 and by a cooking appliance with the features of claim 15. Preferred features are the subject matter of the subclaims. Further advantages and features emerge from the general description of the invention and the description of the exemplary embodiments.

The method according to the invention is used to operate a cooking appliance with at least one cooking area for the preparation of food by means of at least one treatment device. The food is monitored in the cooking area during the cooking process. For this purpose, images of at least part of the surface of the food to be cooked are recorded over time by means of at least one camera device. In particular, the images each consist of a multiplicity of image elements. The images are evaluated by means of at least one processing device. In this case, at least one measure is determined for shorter-term fluctuations of at least one image parameter by at least one longer-term trend caused by increasing cooking of the item to be cooked. At least one internal cooking state of the food to be cooked is derived as a function of the measure for the fluctuations. It is possible and preferred that at least one other cooking product parameter is derived as a function of the measure for the fluctuations.

The method according to the invention offers many advantages. The consideration of the short-term fluctuations and the determination of the internal cooking state from the measure for the fluctuations offers a considerable advantage. A particularly reliable and reproducible statement about the condition inside the food can be made from images of the surface. Since the observation of the surface is sufficient to make statements about the interior of the item to be cooked, complex and cost-intensive camera technology or measuring technology can be dispensed with in the invention. In addition, the method according to the invention is considerably more reliable and less prone to failure than, for example, methods which determine the cooking state from absolute measurement data and z. B. Derive color changes or use threshold value concepts. As a result, the invention can also be used advantageously with a particularly large number of types of food and preparation types.

The fluctuations are caused in particular by movements inside and / or on the surface of the item to be cooked. The fluctuations relate in particular to movements of the item to be cooked or of its components. For example, the fluctuations relate to dough movements and in particular a dough rise or crust formation, e.g. B. cracking of the crust, or the like. The fluctuations preferably relate to movements which occur as a result of increasing browning and / or crust formation. The movements can also be caused by the boiling of liquids in the interior of the item to be cooked and / or by their exit from the interior of the item to be cooked. Such movements are particularly characteristic of the internal cooking state.

The measure for the fluctuations preferably describes a temporal occurrence of extreme values in a temporal course of the image parameter. In particular, the measure for the fluctuations is determined from the occurrence of such extreme values over time. Such a configuration makes the method according to the invention particularly insensitive to interference. The trend preferably describes, in particular, a statistically smoothed temporal curve of the image parameter or is determined from such a curve. The measure for the fluctuations describes in particular fluctuations around a time average value over the cooking time. The fluctuations can be described by at least one statistical measure and, for example, a standard deviation from the time course of the trend. It is possible for the fluctuations to be calculated with the aid of fuzzy logic and, for example, the fuzzy index and / or the Fuji index and / or a model for machine learning (artificial intelligence) and / or a comparable algorithm.

In particular, the fluctuations have a number of sign changes that are many times greater than the trend. In particular, the fluctuations have random changes in sign. This enables a particularly reliable and inexpensive identification of the fluctuations from the image data. In particular, the fluctuations are random. The fluctuations are in particular fluctuations with different signs around the trend. The fluctuations show fluctuations in the direction of the trend as well as against the trend. For example, in the case of a trend towards increasing browning, fluctuations towards both lighter and darker color values are considered.

Such changes are particularly characteristic of internal changes in state and occur, for example, as a result of surface movements and the resulting changes in the angle of reflection.

In particular, the measure for the fluctuations is determined for each individual picture element of the image or is calculated from each individual picture element of the image. As a result, the fluctuations can be seen particularly clearly and do not get lost in the longer-term trend. It is also possible that the measure for the fluctuations is determined for at least one group of image elements, for example lines of an image sensor, or is calculated therefrom. In particular, the measure for the fluctuations is determined for a smaller number of picture elements than the trend. In particular, the trend is determined for the entire image or at least one area of the image assigned to the item to be cooked.

To determine the measure for the fluctuations, changes in at least one image parameter between at least two time-shifted images are preferably evaluated. In particular, the images have a time offset of not more than 5 minutes and preferably of less than 1 minute and particularly preferably of less than 30 seconds. As a result, the random fluctuations or fluctuations can be recorded particularly reliably. For the determination of the trend or for a conventional evaluation of images to detect changes in the cooking state, there are generally considerably fewer images sufficient as these are long-term changes. It is also possible that the images have a time offset of less than 10 seconds or less than 5 seconds. A time offset of a maximum of 1 second or less is also possible. In particular, images that follow one another in time are evaluated.

In particular, in order to determine the degree for the fluctuations, the image elements of the same location in the image of at least two images recorded with a time offset are compared with one another. In particular, the images are compared pixel by pixel. In particular, the image elements are at the same location, which were captured by the same position of an image sensor of the camera device and, for example, by the same sensor pixel. A comparison of a plurality and, for example, one line of image elements at the same location from at least two time-shifted images is also possible.

The trend is preferably a time curve of a mean value of the at least one image parameter or can be mapped by this. The trend here is in particular a time profile over the time of the cooking process. In particular, a time course of a moving average is provided. In order to describe the trend, in particular, a mean value is formed over a time interval. It is also possible that the trend is a time course of another suitable statistical value. It is possible and preferred that the trend is determined by at least one other mathematical method for calculating the trend. For example, the trend is calculated by fitting at least one function, in particular linear and / or polynomial. The trend calculation can also include the determination of a regression in the course of the at least one image parameter over time.

In an advantageous embodiment, the trend is a change in color value over time, which is caused by increasing cooking progress and in particular by an increasing browning process of the food to be cooked. The fluctuations here preferably correspond to color value changes and in particular to random color value changes around this trend. Alternatively or additionally, the trend can also be a change over time in a brightness value and / or contrast value and / or intensity value or another value of the image processing. The trend can also be a change in another image parameter over time, which is due to the increasing cooking of the item to be cooked. The fluctuations are then in particular random changes in value around such a trend.

In a likewise advantageous embodiment, the trend is a change in the distance value over time and / or a change in temperature value, which is caused by the increasing cooking progress of the item to be cooked. The fluctuations correspond in particular to changes in distance values and / or changes in temperature values around this trend. The changes in distance values and / or changes in temperature values can be random. In order to record the changes in distance values, the camera is designed in particular as a camera for recording three-dimensional object information, for example as a 3D camera or stereo camera. In order to detect changes in temperature values, the camera device is designed in particular as a thermal imaging camera or infrared camera and / or as a thermal radiation sensor or the like or comprises at least one such. Such temperature changes and changes in distance describe the movements of the food to be cooked in a particularly characteristic manner and can therefore be used particularly reliably to detect fluctuations.

In all of the refinements, it is particularly preferred that at least one course of the image parameter over time is registered. The fluctuations in the course of the image parameter over time are preferably corrected computationally from the trend in the course of the image parameter over time and, for example, subtracted. In this way, the fluctuations can be analyzed even more specifically isolated from the trend.

It is preferred that an internal cooking state with an average through-cooking and / or a ready time is assumed when a time course of the measure for the fluctuations reaches a minimum again after a maximum. Such a determination of the internal cooking state is carried out in particular when pasta or baked goods are used as the food to be cooked. However, it can also be used in the preparation of meat, fish, vegetables and / or fruit or other types of food. In order to identify cooking states before or after the mean thorough cooking, in particular temporal derivatives of the temporal course of the measure for the fluctuations are used. For example, the extreme values and / or zero crossings of such derivations are considered in time before or after a mean thorough cooking in order to identify cooking states. The internal cooking state can relate, for example, to the core state or a degree of cooking of the core and can also be referred to as the crumb state or degree of cooking of the crumb. The internal cooking state can relate to a mean internal thorough cooking and / or at least one other parameter for the internal cooking state.

It is possible that at least one period of time is stored or can be determined in which the food to be cooked is further cooked after reaching the medium through-cooking. For example, such a period is stored in an automatic program. It is possible that an assignment of the period to a type of food is stored. For example, a roast can be cooked for a few minutes after reaching medium through-cooking in order to achieve optimal preparation. In particular, by analyzing the fluctuations, the movements of the dough are observed and their maximum observed and then it is waited until the movement is again almost at zero. After the fluctuations have come to a standstill due to the movement of the dough, the cooking process can be ended or extended by a defined period of time.

In an advantageous further development, the treatment device is activated as a function of the determined internal cooking state. It is also possible that at least one adaptation of an automatic program is carried out as a function of the determined internal cooking state. For example, a cooking time stored in the automatic program can be shortened and / or lengthened or heating sources can be switched on or off. It is possible for rapid cooling and / or keeping warm by means of the treatment device when the finishing time is reached.

The image parameter is preferably taken from the group of the following image parameters or describes at least one such: color value, gray value, brightness value, intensity value, contrast value, temperature value of a thermal imaging camera, distance value of a 3D camera or a camera for capturing three-dimensional object information. For example, the image parameter can be a red value and / or green value and / or blue value or describes at least one such. It is also possible and preferred that the image parameter is or describes at least one other parameter of the image processing. The image parameter can also be derived arithmetically from such a parameter.

In particular, the image elements provide spatially resolved image information from the cooking area. The camera device comprises, in particular, a multiplicity of sensor segments. In this case, at least one image element from the cooking area can be detected in a spatially resolved manner with at least one sensor segment in each case. An image element is in particular assigned to at least one sensor segment of an image sensor of the camera device or can be mapped in at least one sensor segment. In the context of the present invention, picture element and pixel can be used synonymously. It is possible for at least one image parameter to be evaluated for each of the image elements.

The cooking appliance according to the invention is suitable and designed to be operated according to the method according to the invention. In particular, the cooking appliance comprises at least one lockable and preferably also heatable cooking space. In particular, the cooking area is provided by the cooking space. In particular, the cooking appliance comprises the components described above for carrying out the method according to the invention. The method according to the invention is preferably designed in such a way that the cooking device according to the invention can be operated with it. Such a cooking device offers many advantages and enables a particularly convenient and at the same time particularly reliable automated preparation of a wide variety of foods.

In the context of the present invention, a long-term trend is understood to mean, in particular, a trend over the entire cooking time. Short-term fluctuations are understood to mean, in particular, those fluctuations which occur during a fraction and, for example, a tenth or a hundredth or a thousandth of the cooking time. In particular, the fluctuations have a frequency or rate of change that is many times greater than the trend.

Further advantages and features of the present invention emerge from the exemplary embodiments, which are explained below with reference to the accompanying figures.

• 1 eine rein schematische Darstellung eines erfindungsgemäßen Gargerätes in einer Vorderansicht;
• 2 eine rein schematische Auftragung eines zeitlichen Verlaufs eines Farbwerts;
• 3 eine rein schematische Darstellung eines zeitlichen Verlaufs eines Trends;
• 4 eine rein schematische Darstellung eines zeitlichen Verlaufs von Fluktuationen; und
• 5 eine Auswertung des zeitlichen Verlaufs der Fluktuationen aus der 4.

In the figures show:

• 1 a purely schematic representation of a cooking device according to the invention in a front view;
• 2 a purely schematic plot of a time course of a color value;
• 3 a purely schematic representation of a time course of a trend;
• 4th a purely schematic representation of a temporal course of fluctuations; and
• 5 an evaluation of the fluctuations over time from the 4th .

The 1 shows one as an oven 100 or combination device designed according to the invention cooking device 1 . As a cooking area 11 here is a heatable and through a cooking chamber door 31 lockable cooking space 21 intended. The cooking appliance 1 is intended here as a built-in device. It can also be designed as a stand-alone device. The cooking appliance 1 is operated according to the method according to the invention.

A treatment device is used to prepare food 2 provided, which in the view shown here is not visible in the cooking space 21 or behind the door 31 is arranged. The treatment facility 2 includes e.g. B. a thermal heating device with several heat sources for heating the cooking space 21 . For example, an upper heat, a lower heat, a hot air heating source and / or a grill heating source can be provided as heating sources. The cooking appliance 1 can be equipped with a steam function. The treatment facility 2 can use a high-frequency generator to emit high-frequency radiation into the cooking chamber 21 for the preparation of food.

The cooking appliance 1 here includes a control device 12th for controlling or regulating device functions and operating states. Via the control device 12th preselectable operating settings and preferably also various automatic programs or program operating modes and other automatic functions can be executed. The control device 12th controls the treatment facility for this purpose 2 depending on a preselected automatic program.

For operating the cooking appliance 1 is an operating device 101 intended. For example, an operating mode or an automatic program or a program operating mode or other automatic functions can be selected and set via this. Via the operating device 101 Further user entries can also be made and, for example, menu control can be carried out. The operating device 101 also includes a display device 102 , via the user instructions and z. B. Prompts can be displayed. The operating device 101 can control elements and / or a touch-sensitive display device 102 or include a touchscreen.

The cooking appliance 1 is with one in the oven 21 arranged camera device 3 with a digital image sensor for capturing image information in the cooking space 21 located food. The image sensor has z. B. a resolution of 10 or 20 or 50 or more megapixels. One pixel corresponds to one picture element. The camera setup 3 is not visible here on the top of the cooking space 21 arranged. The one from the camera setup 3 recorded cooking area 11 but can also be on a hob or on a work area or on a worktop. Then is the camera setup 3 for example arranged above the hob or the worktop, for example on a wall cabinet or an extractor hood.

The camera setup 3 is used to place the food in the oven 21 monitor during the cooking process. For this purpose, images of the surface of the food are recorded over the time of the cooking process. The images are processed by means of a processing device 4th evaluated to determine the internal doneness of the food. From the inner boiling state, z. B. can be derived when the food is cooked as desired by the user or as provided in the automatic program. When the desired degree of cooking has been reached, the cooking process is ended and rapid cooling and / or keeping warm can take place.

To determine the internal cooking state, a measure for shorter-term fluctuations of an image parameter and, for example, a color value around a longer-term trend is determined. The longer-term trend is, for example, a time course of a moving average of the image parameter. The internal cooking state is then determined from the measure of the fluctuations. Thus, the invention offers a core state detection based on images of the surface and without an absolute measurement or calibration being necessary, because z. B. Time-determined extreme values from a curve discussion.

With reference to the 2 to 5 an exemplary determination of the internal cooking state according to the method according to the invention will now be described. This is in the 2 an image parameter, here a color value as an example 200 , over time 201 applied. The thinner curve here corresponds to the raw data of the measured color values 200 . The color value 200 was taken from the images of the food surface recorded over time as part of an image analysis.

A longer-term trend and, for example, a time course of a moving average of the color values are then derived from the raw data 200 calculated, which is shown here in bold. The color value 200 fluctuates around the longer-term trend. Due to the high frequency of the image acquisition, there are short-term fluctuations in the color value 200 particularly easy to see.

In the 3 is a time plot of the trend adjusted for fluctuations 6th shown. The trend of the color values 200 provides helpful information about the browning of the surface of the food, for example, so that the browning can also be determined in addition to the internal cooking state.

The 4th shows an isolated representation of the fluctuations 5 over time 201 . For this purpose, the temporal fluctuation of the measurement data is separated from the trend values (browning, distance, radiation temperature or others) from the raw data using suitable mathematical operations. For example, this is done by subtracting the trend 6th from the raw data.

In the 5 is a measure of the fluctuations 5 over time 201 shown. For this purpose, using suitable mathematical methods, for example, the strength and / or speed of the oscillations for the fluctuations is calculated for a time interval sliding over the time axis. For example, the standard deviation of the fluctuation is used for this purpose 5 determined in this time interval.

In addition, the point in time at which the degree of fluctuation is marked is marked here 5 reached a maximum (here denoted 100%). This point in time corresponds, for example, to a maximum movement in the food and, for example, in the dough. At the beginning of the cooking process, fluctuation is low because the dough movement has not yet started. After the maximum, the fluctuation then falls below a fraction of the maximum fluctuation (here denoted by x%, for example 10%). Then the food is baked through in volume or in the core. It is possible that a table which has been experimentally determined for different foodstuffs and degrees of cooking is stored so that the threshold for x can be set optimally depending on the food to be cooked and the degree of cooking desired. A suitable mathematical method can also be used to determine when the fluctuation is again close to zero after reaching its maximum. The volume of the food is then medium through-baked there.

The camera device records in a purely exemplary method sequence described below 3 here at each measurement time t _j a camera image with its mxn pixel-by-pixel raw data for the color values in m rows and n columns. As an alternative to the color value, other measurement data from a 2D, 3D or IR camera can also be used. It can be z. B. also be a distance value (3D) or radiation temperature (IR). Color value (2D) usually consists of several pieces of information. B. the brightness or intensity. In the following, only one or more of the color value information is used. $$\begin{array}{llllll}\hfill & {\text{S.}}_{11}\left({\text{t}}_{\text{j}}\right)\hfill & {\text{S.}}_{\mathrm{12th}}\left({\text{t}}_{\text{j}}\right)\hfill & {\text{S.}}_{\mathrm{13th}}\left({\text{t}}_{\text{j}}\right)\hfill & \mathrm{...}\hfill & {\text{S.}}_{1\text{n}}\left({\text{t}}_{\text{j}}\right)\hfill \\ \left({\text{S.}}_{\text{mn}}\left({\text{t}}_{\text{j}}\right)\right)\text{=}\hfill & {\text{S.}}_{21}\left({\text{t}}_{\text{j}}\right)\hfill & {\text{S.}}_{\mathrm{22nd}}\left({\text{t}}_{\text{j}}\right)\hfill & {\text{S.}}_{23}\left({\text{t}}_{\text{j}}\right)\hfill & \mathrm{...}\hfill & {\text{S.}}_{2\text{n}}\left({\text{t}}_{\text{j}}\right)\hfill \\ \mathrm{...}\hfill & \hfill & \hfill & \hfill & \hfill & \hfill \\ \hfill & {\text{S.}}_{\text{m}1}\left({\text{t}}_j\right)\hfill & {\text{S.}}_{\text{m}2}\left({\text{t}}_j\right)\hfill & {\text{S.}}_{\text{m}3}\left({\text{t}}_j\right)\hfill & \mathrm{...}\hfill & {\text{S.}}_{\text{mn}}\left({\text{t}}_{\text{j}}\right)\hfill \end{array}$$

The indices m and n describe the positions of the color values on the image sensor or on the imaged object. t _j stands for the different measurement times within a measurement sequence from the start to the end of the sequence. During the cooking process, measurement sequences with the internal time scale from t ₁ to t _N take place at different times. There is also the special case N = 1. Then the sequence consists of only one measurement. These are then individual measurements at times τ during the cooking process. At each point in time t _{j of} a sequence, m × n color values are therefore measured. The measured values of the image matrix are now processed accordingly.

At each point in time t _j , moving average values for each pixel color value S _mn (t _j ) for a fixed (but variable) location mxn on the object surface are calculated over a time interval [t _j − Δt, t _j]. Δt is typically in the range of seconds or a few minutes. The mean values are thus temporal mean values over the time interval of length Δt lying before the measurement time t _j , individually for each pixel color value at one of the m × n spatial positions. The mean values averaged over time over Δt are given by M _mn (t _j ) | _Δt denotes.

The time course of a mean value for a pixel describes the trend of the color value in this pixel. The differences S _mn (t _j ) - M _mn (t _j ) | _Δt describe the scattering of the color values in the pixel mn around their mean value over time over the interval [t _j - Δt, t _j ]. If the scatter of the non-averaged color values S _mn (t _j ) around their time average is very small, the differences are very small, and if there is a strong scatter, they are correspondingly large. A different preceding time interval of length Δt and a different associated mean value M _mn (t _j ) | belong to each measurement time t _{j Δ} . The time course of the mean value can be called trend in its generalized form will. In its general form, the trend arises not only through averaging over a time interval. Other mathematical methods for calculating trends are also possible (e.g. fit of a mathematical function linear, polynomial, etc.).

With fluctuation, a possible change in the variables in both directions over time is in all cases. B. from light to dark and from dark to light, from small to large distance and back again, from high surface temperature to lower and vice versa, everything in constant and early change, everything superimposed by a general trend towards brown (darker, red) , higher (rise baked goods) or flatter (shrinkage meat) or hotter (surface temperature).

The difference Smn-Mmn is a measure of the temporal fluctuation of the color values Smn. The fluctuation is based on a real physical effect. In the case of baked goods made from dough, this is the rise and the resulting rolling, swelling and cracking movement in the dough and its surface. By changing the local geometry on the surface of the dough, the angle of incidence and reflection of the light from the oven lighting change. Light and shadow areas are shifted by the movement of the dough. Even without the browning of the surface of the food to be cooked having to increase, the brightness and color value in the pixels fluctuate from light to dark and vice versa with the extent of the movement of the dough surface.

The movement of the dough surface, on the other hand, is linked to the cooking of the dough in volume. At the beginning of the baking, the dough is still cold and there is very little movement on the surface of the dough. It then runs to a maximum and comes to a standstill again at the end of the baking in volume. The degree of doneness is then just medium baked. It is therefore advantageous to observe the movement of the dough, wait for its maximum and recognize when the movement is again almost at zero.

After the fluctuating movement of the dough has come to a standstill, a slight shrinkage in volume is also observed. This means that the rather statistical swelling of the dough surface has then changed into a shrinkage of a solid surface structure. The shrinkage movement of a solid crust can be differentiated from a statistical movement of the dough. One of the standard mathematical methods can be used to describe the fluctuating dough movement. A first step is to separate the change in color value due to the continuous browning process (trend line) and the fluctuation of the color values in the pixels of the food due to movement of the dough and change in the surface geometry (e.g. as described below).

For example, when baking, the intensity of the movement in the dough volume and on its surface is initially low. As long as the dough has not yet baked through, the surface of the dough will simmer slowly. This leads to local fluctuations in the pixel-by-pixel color values (2D), distances (3D) and radiation temperatures (IR). Brightness or color value fluctuations (in the pixels of the camera image) due to the changing reflection of the cooking space light due to changing shadows, due to a changing geometry of the dough surface during dough baking At the start and end of baking, there is very little movement in the dough and on its surface.

A separation of trend and fluctuation can be carried out as follows: $$\left(\text{Smn}\right)\left(\text{t}\right)-\left(\text{Trend mn}\right)\left(\text{t}\right)$$

Trend mn (t) is the trend of the color value Smn (t) for the point mn of the image matrix, extended to the measurement time t, determined from the time course of the color value Smn (t) in the past, using a known, customary method, t ist the current measurement time. Smn (t) is the color value in the image matrix at row m (between 1 and m) and column n (between 1 and n) at the current measurement time. Trendmn (t) is the trend value associated with position mn, calculated from the measured values of Smn (t) over a previous time interval and usually up to t.

The trend can be calculated as trend mn (t) for each pixel of the image matrix. $$\left(\text{Smn}\right)\left(\text{t}\right)-\left(\text{Trend mn}\right)\left(\text{t}\right)$$

However, a larger number of pixels can also be used at the same point in time t, e.g. B. one line of the image matrix in order to calculate the trend from it. Then the trend matrix shrinks to a column with m rows and fluctuates less. For this, the correction is not optimally matched to pixel mn. $$\left(\text{Smn}\right)\left(\text{t}\right)-\left(\text{Trend mn}\right)\left(\text{t}\right)$$

Alternatively, the larger number of pixels for the trend calculation can also be generated by combining a column of the image matrix. Then the trend matrix shrinks to a row with n columns. $$\left(\text{Smn}\right)\left(\text{t}\right)-\left(\text{Trend mn}\right)\left(\text{t}\right)$$

It is even more extreme, e.g. B. to use the entire image matrix at one point in time for trend calculation. The trend matrix then only consists of one value that is to be applied to the color values of all pixels. $$\left(\text{Smn}\right)\left(\text{t}\right)-\left(\text{Trend mn}\right)\left(\text{t}\right)$$

Trend mn (t) is the trend of the color value Smn (t) for the point mn of the image matrix, extended to the measurement time t, determined from the time course of the color value Smn (t) in the past.

t ist der aktuelle Messzeitpunkt. Smn (t) ist der Farbwert in der Bildmatrix an der Stelle Zeile m (zwischen 1 und m) und Spalte n (zwischen 1 und n) zur aktuellen Messzeit. Der Mittelwert ist der zu Position mn zugehörige Mittelwert über das vom Messzeitpunkt um Δt zurückliegende und an t angrenzende Zeitintervall.For a pixel-by-pixel separation of trend and fluctuation, the mean value of the pixel value at the point mn is determined over a previous time interval Δt passed through in k steps. $$\text{Mmn}\left(\text{t}\right):=\text{Mean values}\left(\text{Smn}\left(\text{t}-\Delta \text{t}\right)\right),\text{Smn}\left(\text{t}-\Delta \text{t}+\Delta \text{t / k}\right)\mathrm{,\; ...,}\text{Smn}\left(\text{t}-\Delta \text{t}+\text{k}\Delta \text{t / k}\right)$$

t is the current measurement time. Smn (t) is the color value in the image matrix at row m (between 1 and m) and column n (between 1 and n) at the current measurement time. The mean value is the mean value associated with position mn over the time interval that was Δt back from the measurement time and is adjacent to t.

Trend-adjusted color value matrix: $$\left(\text{Smn}\right)\left(\text{t}\right)-\left(\text{Mmn}\right)\left(\text{t}\right)$$

A line-by-line or column-wise separation of trend and fluctuation is also possible.

In particular, the rows or columns of the picture elements of the image or of the image sensor are considered. A global mean value over the color values of all pixels of the image matrix at time t can also be determined as a trend at time t. A global clearing of the trend then takes place again at each point in time t.

To capture the raw data (e.g. how with reference to the 2 described) color values are measured at each point in time t mn. It is measured at time intervals Δτ (rough time scale). The mn (simultaneously determined) color values are distributed for evaluation on the time axis at a distance Δτ / mn between t and t + Δτ. This is followed by the next mn color values measured at t + Δτ in the same way. According to a known method, the trend (t) is then isolated from the raw data and the measure for the fluctuations is determined (as with reference to the 3 to 5 described).

Trend and fluctuation can be separated by filtering (e.g. high, low or band pass). The trend changes only slowly compared to the fluctuation. Disturbances that are very fast compared to the dough fluctuation can also be filtered out (switching light on / off, shadows cast by places in front of the oven door, etc.). A similar technique is digital filtering, e.g. B. by FFT (Fourier) of the color values (t) in the frequency. E.g. range, clipping to the permissible band (for the fluctuations) and FFT ^-1 .

At each reference point in time t ₀ , mn color values can be measured, which form the reference color value matrix. It is measured anew at (larger) time intervals of length Δτ. At the same time, with each reference measurement, a sequence of images is created in a preceding time interval of length Δt recorded. If the sequence contains N pictures, the time interval between the pictures in the sequence is equal to Δt / N.

The images of the sequence are compared, each individually, with the reference. Autocorrelation, covariance and variances determine the similarity of the two color value matrices. From the temporal course of the similarity, when t moves away from the reference time t ₀ , an assessment is made of how much movement the images of each sequence contain. The score of a sequence is the result of the autocorrelation, a number from (0.1) for each of the N images of the sequence. As a function of time, it is a measure of how much the image becomes dissimilar due to the movement in the dough surface within the duration of the sequence. The falling course can be described with a time constant.

At each current time t ₀ , the measured values from the current image S _mn (t ₀ ) can be multiplied pixel-by-pixel with the measured values of an image S _mn (t) at the point in time t. This is a pairwise comparison of two images in which the similarity of the two images is measured by the mathematical operation. For discrete measurement times, t is an element from (t ₁ , t ₂ , ..., t _N ). The t are measurement times within a sequence of the total duration Δt. In the example shown, the sequence has N measurement times in Δt. t ₁ is adjacent to t _0. The higher the index j of t _j , the further the image is for reference at t ₀ .

$${\text{t}}_1={\text{t}}_0-1\Delta \text{t/N}$$

$$\begin{array}{l}{\text{t}}_2={\text{t}}_0-2\Delta \text{t/N}\\ \\ \text{, }\mathrm{...,}\end{array}$$

$${\text{t}}_{\text{N}}={\text{t}}_0-\text{N}\Delta {\text{t/N=t}}_0-\Delta \text{t}$$

During a sequence, it is determined how quickly an image itself becomes more dissimilar as the distance increases, a measure of the movement and change in the image over the duration of the sequence. A new sequence of food images is measured and evaluated over and over again during a cooking process in order to observe how the speed of change in the dough surface varies over the cooking time (low -> high -> low = medium baked through). $${\text{t}}_0={\text{t}}_0-0\Delta \text{t}/\text{N}={\text{t}}_0$$

$${\text{<figure-callout id="1" label="t" filenames="DE102019120008B4\_0014.png" state="{{state}}">t</figure-callout>}}_1={\text{t}}_0-1\Delta \text{t / N}$$

$$\begin{array}{l}{\text{<figure-callout id="2" label="t" filenames="DE102019120008B4\_0014.png" state="{{state}}">t</figure-callout>}}_2={\text{t}}_0-2\Delta \text{t / N}\\ \\ \text{,}\mathrm{...,}\end{array}$$

$${\text{t}}_{\text{N}}={\text{t}}_0-\text{N}\Delta {\text{t / N = t}}_0-\Delta \text{t}$$

All pixel measured values from the current image S _mn (t ₀ ) at the current reference time t ₀ are multiplied by the corresponding pixel measured values of an image S _{mn (t) located earlier in} time at time t. All products thus formed are added up.

The sum is normalized to the square root of the product of the variance of the measured values at time t and the variance of the measured values at time t _0.

The underlying mathematical operation is called the autocorrelation Cor [(S _mn (t ₀ )), (S _mn (t))].

It is the covariance Cov [(S _mn (t ₀ )), (S _mn (t))] of the image matrices (S _mn (t ₀ )) and (S _mn (t)), based on the product of the standard deviations over all elements of (S _mn (t ₀ )) and separately over all elements of (S _mn (t)). $$\text{Cor}\left[\left({\text{S.}}_{\text{mn}}\left({\text{t}}_0\right)\right),\left({\text{S.}}_{\text{mn}}\left(\text{t}\right)\right)\right]=\text{Cov}\left[\left({\text{S.}}_{\text{mn}}\left({\text{t}}_0\right)\right),\left({\text{S.}}_{\text{mn}}\left(\text{t}\right)\right)\right]/\left[\sigma \left(\left({\text{S.}}_{\text{mn}}\left({\text{t}}_0\right)\right)\right)*\sigma \left(\left({\text{S.}}_{\text{mn}}\left(\text{t}\right)\right)\right)\right],$$

The autocorrelation is often referred to in the literature as g ₂ (t). t is the time on a scale within the sequence and describes the distance from the reference time.

It describes how quickly the measured values of the images change when t moves away _{from the reference time t 0.} The values of g2 (t) lie in the interval [0.1] and g ₂ = 1 if the images are identical, g ₂ = 0 if the images are not similar.

The observation is that the surface of the food to be cooked changes only slightly immediately after the start of cooking, because the food has to be heated up first. The evaluation of the autocorrelation g2 (t) at this early point in time τ in the baking process leads to a large time constant, ie changes take place on a large time scale. g ₂ can also be written as g ₂ (x, t), where τ describes the reference time on an “outer” time scale and t the inner time within the sequence at this reference time. The values of g2 (t) lie between [0.1] and start at 1. Depending on how much the starting value falls in the observed sequence time interval in the direction of 0, the greater the change. The change is small here.

As soon as the cooking process affects the food more, there is more movement in the surface of the food. In equally long sequence time intervals, the change in the surface of the food to be cooked increases from picture to picture, up to a maximum. The time constant determined from the autocorrelation decreases, i. H. Change in less time. The autocorrelation drops to a lower value in the observed interval, i. In other words, the change is also greater in the observed interval. After that, the change in the surface slows down again and comes to a standstill in the medium-baked volume near the cooking state. The time constant from the autocorrelation increases again. The current state of volume baking can be determined by discussion of curves, e.g. B. the temporal course of the time constants can be recognized from the autocorrelation.

At the beginning of baking, the images hardly change, in the middle of the baking time for volume baking, the images change strongly and quickly, when the cooking state is “medium-volume baked”, the change goes back to zero, except for the continuous browning process.

The calculation of the similarity of time-shifted data records of a sequence in comparison to the reference data record can be carried out with multivariate methods of pattern recognition such as e.g. B. Scatterplot or Scatterplot Matrix. Representation of the descriptor as a function of the time offset to the reference.

The fuzzy index can be determined to calculate movement in the camera image. For each pixel of the color value image at position mn with “Smn = color value at position mn”, the absolute amount of the color value difference at time tj and tj + 1 is calculated and normalized to their sum. This is done in pairs for all images in a sequence. Here, for the fuzzy index, τ is the evaluation point in time after recording the N + 1 images of the sequence. The τ time axis is larger than the t time axis. nm is the pixel position in the image matrix. t _{j + 1} - t _j is the time interval between the images in a sequence. The t-time axis describes small time intervals. The color value S can e.g. B. the brightness (L * in the L * a * b * system).

The numerator of the index describes how strongly the color value fluctuates from image to image within a sequence. F _mn (t) describes the fluctuation for pixel mn at time τ and supplies spatially resolved information. If you also add up over all pixels, the location information is lost and a value is obtained that describes the entire surface of the food being cooked.

The sum of the normalized amounts of the picture-to-picture changes over the N pictures of the sequence for a single pixel mn is added up over all pixels of the picture and is a measure of the fluctuation in the picture at time τ. If the color values fluctuate greatly from picture to picture, the index F (τ) is large. The index is a measure of the rate of fluctuation of the color values. The index F (τ) is observed over the time τ and has a clear and global maximum structure: at the beginning of baking, F (τ) is small, approximately in the middle of the baking time, and almost again when it is “medium cooked” in volume baking Zero. Other states of volume baking, before and after the middle baking, result from the mathematical analysis of the time derivatives of the index F (τ), e.g. B. by discussion of curves. Measuring the index provides information about the core condition during baking.

At each (reference) point in time t0 there is a sequence with N temporally adjacent camera images that were recorded at points in time t1, t2, ..., tN. The time increases from tN through t1 to t0. tN is furthest in the past, t0 is the current point in time. For the information sought at time t0, the camera image at time t0 is compared with the previous images from t1 to tN, with an increasing time interval. A numerical value is mathematically determined for the extent to which the structure of the images from tN through t1 to t0 is mainly caused by shrinkage when the structure is retained. That is, in the pixel-by-pixel comparison of the two camera images to be compared, a correlation remains.

The correlation increases due to a (global) change in the measured property that can be observed across all pixels (e.g. the average brightness (decreases during cooking), the average red-green value (increases), the height of the food (distance from the camera), the average surface temperature, etc. And it also decreases because the image of the surface of the food to be cooked shrinks during a cooking phase.

To correct for the first effect, the pixel content of a camera image is normalized to the mean value over all pixels. To correct for the second effect, the newer camera image is enlarged in each case until the correlation with the older camera image of the surface of the food to be cooked is at its maximum.

When baking baked goods, the correlation of the camera images is high at the start. The food has not changed yet because the dough is still too cold. Then the correlation of the current camera image to its N temporal ancestors decreases more and more. The more the volume cooking is completed, the less the camera image sequences change due to fluctuations in the pixel values. The correlation increases again. It is (again) maximally at the time of completion. If the baking is done beyond the time of completion, the correlation remains high, but the camera images of the surface shrink. With a suitable enlargement factor, the earlier image can be created.

As long as the baked good is cooking in volume, the surface of the baked good is in an (expanding) movement. When baking in the dough volume is complete, this movement comes to a standstill. Movements can be observed over the surface of the dough, which occur locally at different speeds, in different directions and at different times. When the volume is completely baked, a (slight) movement of the surface can also be observed. However, this is much less pronounced and the surface moves as a whole / unit. In the case of gratinating, the surface structure (apart from that, its appearance can no longer be changed over time) slowly shrinks, while it expands during volume cooking until the time of completion (and changes locally in different ways).

After starting the baking process, the dough appears z. B. for a 2D digital camera often first to become lighter, because the dough surface is caused by melting fat and a. pulls smooth and looks more like a mirror. Then the browning process begins and the brightness of the dough surface steadily decreases with a characteristic changing speed. It's a slow process. What is more important here, however, is the temporal course of the fluctuation in brightness (in each pixel), which takes place on a shorter time scale than browning and alternates in both directions at one location (pixel) (light-> dark and dark-> light).

This second, faster process causes the brightness of the dough to increase and decrease over time from the perspective of a pixel. The brightness recorded with the camera results from the reflection of the light from the cooking chamber on the milli- and microstructure of the dough surface. This movement and deformation leads to the fact that other places on the object surface are continuously pushed into the viewing angle of a pixel and that the geometry and the reflection properties of these object locations also change, which leads to changes in the scattering angle and the scattered intensity of the cooking space lighting in the fixed pixel . A similar movement is known from baking a casserole or pizza and boiling water.

In each pixel of the image sensor, the brightness fluctuates due to these effects on a short time scale, compared to the browning, which only leads to a slow darkening. Right at the start of baking, the fluctuation you are looking for is still very small because the dough is cold. As soon as the dough is hot and starts to bake, the fluctuation increases, is at its maximum around the middle of the baking time and comes to a standstill when the crumb is completely baked, i.e. H. when the dough is baked in volume.

The time of completion for baking in volume is given by the cessation of the movements on the surface. Both effects “decrease in brightness due to browning” and “characterization of fluctuation” are separated from each other in order to be able to differentiate between each of the browning and volume baking effects. The fluctuation is thus well characterized in order to identify the finished state of the volume baking. The browning is removed from it by measurement or mathematics. In the case of a brightness measurement, on the other hand, the fluctuation is a disturbance that makes it difficult to recognize the exact brightness value, which one would therefore like to get rid of. So the situation is exactly the opposite.

This relationship applies accordingly to distances (3D camera). As long as the surface of the dough, similar to a boiling water surface, wobbles, moves up and down and back and forth, each pixel measures different distances over time. This fluctuation effect can also be clearly seen with an IR camera: gas emerges irregularly from the surface of the food and leads to temperature changes on the surface of the food.

List of reference symbols

1

Cooking appliance

2

Treatment facility

3rd

Camera setup

4th

Processing facility

5

Fluctuations

6th

trend

11

Cooking area

12th

Control device

21

Cooking space

31

door

100

oven

101

Control device

102

Display device

200

Color value

201

time

Claims (15)

Method for operating a cooking appliance (1) with at least one cooking area (11) for the preparation of food by means of at least one treatment device (2), whereby the food in the cooking area (11) is monitored during the cooking process and, for this purpose, by means of at least one camera device (3) Over time, images of at least a part of the surface of the food to be cooked are recorded and the images each consist of a large number of image elements and are evaluated by means of at least one processing device (4), characterized in that at least one measure for shorter-term fluctuations (5) of at least one image parameter at least one longer-term trend (6) caused by increasing cooking of the food is determined and that at least one internal cooking state of the food is derived as a function of the measure for the fluctuations (5). Method according to the preceding claim, characterized in that the fluctuations (5) are caused by movements in the interior and / or on the surface of the food to be cooked. Method according to one of the preceding claims, characterized in that the measure for the fluctuations (5) describes a temporal occurrence of extreme values in the temporal course of the image parameter or is determined from such and preferably that the trend describes a statistically smoothed temporal course of the image parameter or is determined from such. Method according to one of the preceding claims, characterized in that the fluctuations (5) have a number of sign changes that are many times greater than the trend and in particular that the fluctuations (5) have random sign changes. Method according to one of the preceding claims, characterized in that the measure for the fluctuations (5) is determined for each individual picture element of the picture and / or for each at least one group of picture elements, for example lines. Method according to one of the preceding claims, characterized in that to determine the measure for the fluctuations (5) changes in the image parameter between at least two time-shifted images are evaluated and that the images have a time shift of no more than five minutes and preferably less than one minute and particularly preferably have less than 30 seconds. Method according to one of the preceding claims, characterized in that, in order to determine the measure for the fluctuations (5), the image elements of the same location in the image of at least two time-shifted images are compared with one another. Method according to one of the preceding claims, characterized in that the trend (6) is a time course of a mean value of the at least one image parameter. Method according to one of the preceding claims, characterized in that the trend (6) is a change in color value over time due to the increasing cooking progress, in particular the browning process, of the food and that the fluctuations (5) correspond to random changes in color value around this trend (6). Method according to one of the preceding claims, characterized in that the trend (6) is a temporal change in distance value and / or temperature value change due to the increasing cooking progress of the food and that the fluctuations (5) correspond to random distance value changes and / or temperature value changes around this trend (6) . Method according to one of the preceding claims, characterized in that a temporal course of the image parameter is registered and that the fluctuations (5) in the temporal course of the image parameter are arithmetically corrected from the trend (6) in the temporal course of the image parameter. Method according to one of the preceding claims, characterized in that an internal cooking state with average through-cooking and / or a ready time is assumed when a time course of the measure for the fluctuations (5) reaches a minimum again after a maximum. Method according to the preceding claim, characterized in that the treatment device (2) is controlled depending on the determined internal cooking state and / or that at least one adaptation of an automatic program is carried out depending on the determined internal cooking state. Method according to one of the preceding claims, characterized in that the image parameter is taken from the group of the following image parameters: color value, gray value, brightness value, intensity value, contrast value, temperature value of a thermal imaging camera, distance value of a 3D camera. Cooking appliance (1), suitable and designed to be operated according to the method according to one of the preceding claims.

Images