WO2022045049A1

WO2022045049A1 - Information processing device, information processing method, and program

Info

Publication number: WO2022045049A1
Application number: PCT/JP2021/030747
Authority: WO
Inventors: 和伸太田; 正晃上坂; 成治服部; 祐太坪井; 翔太佐々木
Original assignee: Ａｒｉｔｈｍｅｒ株式会社
Priority date: 2020-08-24
Filing date: 2021-08-23
Publication date: 2022-03-03
Also published as: JP6931511B1; JP2022036866A

Abstract

[Problem] To provide an information processing device, etc. that make it possible to suitably support sanitary management. [Solution] A program of an information processing device according to the present invention acquires a video capturing, from above, a table on which a food product is placed, detects a first target object and a second target object from a table area in the video, the table area corresponding to the table, generates, on the basis of the detection result of the first target object, a heat map in which values are set for individual areas in the table area, and, when the second target object has been detected, causes a computer to execute processing for changing the value of the heat map for an area in which the second target object has been detected.

Description

Information processing equipment, information processing methods and programs

The present invention relates to an information processing device, an information processing method and a program.

From the viewpoint of prevention of food poisoning, a method of supporting hygiene management at a restaurant that provides food has been proposed (for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2010-287166

By the way, due to the epidemic of viral infectious diseases these days, hygiene management is becoming more and more important in restaurants and the like. For example, in a restaurant, it is required to keep food and other supplies, tableware, and in-store equipment clean, but it may be difficult to determine whether these are kept clean.

The program according to one aspect acquires a moving image of a table on which food is placed from above, and detects a first object and a second object from the table area corresponding to the table in the moving image. , A heat map in which the values of each region in the table area are set is generated based on the detection result of the first object, and when the second object is detected, the region where the second object is detected is detected. The computer is made to execute the process of changing the value of the heat map.

On one side, hygiene management can be adequately supported.

It is explanatory drawing which shows the configuration example of the hygiene management system. It is a block diagram which shows the configuration example of a server. It is explanatory drawing which shows the outline of Embodiment 1. FIG. It is explanatory drawing which shows the outline of Embodiment 1. FIG. It is explanatory drawing which shows the display screen example of a heat map. It is a flowchart which shows an example of the processing procedure which a server executes. It is explanatory drawing which shows the outline of Embodiment 2. It is a flowchart which shows an example of the processing procedure executed by the server which concerns on Embodiment 2. FIG. It is explanatory drawing which shows the outline of Embodiment 3. FIG. It is a flowchart which shows an example of the processing procedure executed by the server which concerns on Embodiment 3. FIG. It is explanatory drawing which shows the outline of Embodiment 4. It is a flowchart which shows the procedure of the calculation process of a degree of cleanliness. It is a schematic diagram which shows the structure of the control part 11 which concerns on Embodiment 5. It is a flowchart which shows an example of the processing procedure executed by the server which concerns on Embodiment 5.

Hereinafter, the present invention will be described in detail with reference to the drawings showing the embodiments thereof.
(Embodiment 1)
FIG. 1 is an explanatory diagram showing a configuration example of a hygiene management system. In the present embodiment, a hygiene management system that generates a heat map showing the hygiene state of the table from a moving image of a table on which food (object) is placed will be described. The hygiene management system includes an information processing device 1, a terminal 2, and a camera 3. Each device is communicated and connected via the network N. The heat map referred to here is represented by continuously changing numerical data in individual regions in an image, and is represented by, for example, continuously changing colors and their shades.

The information processing device 1 is an information processing device capable of transmitting and receiving various types of information processing and information, and is, for example, a server computer, a personal computer, or the like. In the present embodiment, it is assumed that the information processing apparatus 1 is a server computer, and in the following, it will be read as server 1 for the sake of brevity. As will be described later, the server 1 generates a heat map showing the hygiene state of the table from a moving image obtained by capturing the table on which the food is placed from above. The table is a table used when providing food in the form of a buffet, a buffet, or the like at a restaurant or the like, and is a table used by an unspecified number of customers (persons). The server 1 detects the customer's hand, finger, or equipment such as tongs used when taking food from the moving image of the table as the first object, and heat map showing the cleanliness of each area on the table. (See FIG. 3A). Then, the server 1 detects an object related to the cleaning work of the table as a second object, such as a cleaning tool (duster or the like) used by the employee who cleans the table, or a glove worn by the employee, and the second object. The value of the heat map in the area where an object is detected is changed (reset) (see FIG. 3B).

The server 1 may be a cloud server communication-connected to the terminal 2 or the like via the Internet or the like, or may be a local server communication-connected to the terminal 2 or the like via a LAN (Local Area Network) or the like.

The terminal 2 is an information processing terminal used by an employee, for example, a tablet terminal, a personal computer, a smartphone, or the like. The server 1 outputs the generated heat map to the terminal 2, displays it, and presents it to the employee.

The camera 3 is an image pickup device that captures an image of a table on which food is placed. For example, the camera 3 is installed directly above the table as shown in FIG. 1, and continuously images the table from directly above. The installation position of the camera 3 is not limited to directly above the table, and may be installed diagonally above the table, for example, and may be installed at least at a position higher than the mounting surface of the table. The server 1 acquires a moving image from the camera 3 and generates a heat map.

FIG. 2 is a block diagram showing a configuration example of the server 1. The server 1 includes a control unit 11, a main storage unit 12, a communication unit 13, and an auxiliary storage unit 14.
The control unit 11 has an arithmetic processing unit such as one or a plurality of CPUs (Central Processing Units), MPUs (Micro-Processing Units), GPUs (Graphics Processing Units), and stores the program P stored in the auxiliary storage unit 14. By reading and executing, various information processing, control processing, etc. are performed. The main storage unit 12 is a temporary storage area for SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), flash memory, etc., and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. Remember. The communication unit 13 is a communication module for performing processing related to communication, and transmits / receives information to / from the outside. The auxiliary storage unit 14 is a non-volatile storage area such as a large-capacity memory or a hard disk, and stores a program P and other data necessary for the control unit 11 to execute processing.

The server 1 may be a multi-computer composed of a plurality of computers, or may be a virtual machine virtually constructed by software. The server 1 may be virtualized by the container technology.

Further, in the present embodiment, the server 1 is not limited to the above configuration, and may include, for example, an input unit that accepts operation input, a display unit that displays an image, and the like. Further, the server 1 includes a reading unit for reading a portable storage medium 1a such as a CD (CompactDisk) -ROM, a DVD (DigitalVersatileDisc) -ROM, and reads and executes a program P from the portable storage medium 1a. You can do it. Alternatively, the server 1 may read the program P from the semiconductor memory 1b.

FIG. 3 is an explanatory diagram showing an outline of the first embodiment. FIGS. 3A and 3B conceptually show tables when viewed from the viewpoint (directly above) of the camera 3, respectively. An outline of the present embodiment will be described with reference to FIG.

As described above, the server 1 acquires a moving image of the table on which the object is placed from above from the camera 3. The table is, for example, a table installed in a restaurant (store), and is a table on which food is placed when food is provided in the form of a buffet, a buffet, or the like. The server 1 acquires a moving image from a camera 3 installed on the ceiling of a store, for example, and generates a heat map.

In this embodiment, a table used by an unspecified number of customers such as a buffet and a buffet will be described as an example, but individual tables used by customers for meals may be targeted. Further, the target table is not limited to the table used by the customer, and may be a table used by employees, for example, a workbench used in a kitchen.

Further, in the present embodiment, the explanation will be given using a restaurant as an example, but for example, a table (furniture, etc.) on which products are displayed at a retail store, a general store, or the like may be targeted. That is, the object placed on the table is not limited to food, and may be other objects.

The server 1 identifies the table area corresponding to the table from the moving image obtained by capturing the table. Then, the server 1 divides the table area into a plurality of grids in a grid pattern. The division unit (shape) of the table area is not limited to the grid-like grid, and may be any division unit suitable for generating a heat map.

Server 1 detects a specific object from the table area. The object includes a first object and a second object. The first object is, for example, a customer's hand, finger, or an instrument such as tongs used when picking up food, and is an object other than the second object, that is, an object that comes into contact with a table for a purpose other than cleaning work. .. The second object is, for example, a cleaning tool (duster or the like) used by the employee during cleaning, or a glove worn by the employee during cleaning, and is an object related to the cleaning work of the table.

FIG. 3A illustrates how a heat map is generated based on the detection result of the first object. In FIG. 3A, for convenience of illustration, the portion displayed in color on the heat map is shown by hatching. Further, in FIGS. 3A and 3B, the objects (food) on the table are not shown. When the first object is detected from the table area, the server 1 generates a heat map in which the values of each area in the table area are set based on the detection result of the first object.

Specifically, the server 1 generates a heat map in which the value of the area where the first object is detected is set according to the time when the first object is detected. The "region in which the first object is detected" refers to, for example, a grid on which the first objects overlap and a grid around the grid, but is a circle within a certain distance from the center of gravity of the detected first object. You may target the area. The same applies to the second object described later.

The server 1 sets the value of the heat map of the area (grid) where the first object is detected larger as the time when the first object is detected is longer. For example, the server 1 increases the increase width of the value of the grid on which the first object overlaps so that the value of the heat map becomes larger as it gets closer to the position where the first object is detected, and the grid around the grid. Set a small increase in the value of. The server 1 sets the increase width of each grid according to the detection time, and generates a heat map.

In the above, the value of the heat map is set according to the time when the first object is detected, but the present embodiment is not limited to this. For example, the server 1 may set the value of the heat map according to the number of times that the first object is detected in addition to the detection time of the first object or instead of the detection time. Further, the time and the number of times that the first object is detected is an example of the setting standard, and other setting criteria (for example, the type of the first object, or the presence or absence of contact as in the second embodiment described later). You may set the value of the heat map according to. In this way, the server 1 may be able to generate a heat map based on the detection result of the first object.

Further, in the present embodiment, the heat map is generated only from the moving image, but the present embodiment is not limited to this. For example, the server 1 may generate a heat map based on environmental values such as temperature and humidity around the table. For example, sensors such as a temperature sensor and a humidity sensor are installed near the table, and the server 1 acquires environmental values such as temperature and humidity together with a moving image. Then, the server 1 weights the value of the heat map based on the acquired environment value. For example, when the temperature, humidity, or the like is equal to or higher than a certain value, the server 1 sets a large increase range of the heat map value due to the detection of the first object. For example, the server 1 may uniformly increase the values of all the grids in the table area regardless of whether or not the first object is detected. In this way, the heat map may be generated in consideration of the environment in which the food is placed.

FIG. 3B illustrates how the value of the heat map is changed when the second object is detected. When the second object is detected, the server 1 changes the value of the heat map in the area where the second object is detected. Specifically, the server 1 resets the value of the heat map in the region. Note that "resetting the heat map value" means setting the initial value (for example, "0") before the first object is detected. The server 1 resets the value of the heat map of the area where the second object is detected, that is, the area where the second object has passed by the wiping operation or the like, and erases the color display of the area.

Note that, for example, the server 1 may recognize the operation of the second object from the moving image and reset the value of the heat map according to the recognized operation. Specifically, the server 1 recognizes an operation of moving the second object at a constant speed or higher, an operation of repeatedly moving the second object, and the like, and when the operation is recognized, the second object passes through. Reset the heatmap value for the area you have created. As a result, for example, the case where a cleaning tool (duster or the like) is only placed can be excluded, and the value of the heat map can be appropriately reset.

Further, for example, when the value of the heat map is reset, the server 1 may notify the terminal 2 that there is an uncleaned area when the area where the value is not reset remains in the table area. .. As a result, it is possible to prevent unwiped residue and the like, and to more preferably support the cleaning work.

Further, in the present embodiment, the value of the heat map is reset when the second object is detected, but the value is not completely reset to the initial value before the detection of the first object, but is a constant value. You may try to subtract only the value. That is, when the server 1 detects the second object, the server 1 may change the heat map value so as to be the same as or closer to the state (value) before the first object detection, and reset the heat map value. The configuration to do is not mandatory.

As described above, the server 1 generates a heat map according to the detection result of the first object, and when the second object is detected, the value of the heat map of the region where the second object is detected is used. Reset. As a result, it is possible to visualize the area requiring cleaning and appropriately support hygiene management.

FIG. 4 is an explanatory diagram showing an example of a heat map display screen. FIG. 4 illustrates an example of a display screen when the terminal 2 displays the above heat map. The terminal 2 receives the output of the heat map from the server 1 and displays the screen. The screen includes a table selection field 41 and a heat map 42. The terminal 2 displays the heat map 42 corresponding to the table selected in the table selection field 41. Specifically, the terminal 2 displays an image in which the heat map 42 of the selected table is superimposed on the moving image captured by the camera 3. The server 1 sequentially generates a heat map from the moving image, superimposes it on the moving image being imaged, and displays it on the terminal 2.

Further, the terminal 2 displays the table information field 43 on the screen. The table information column 43 is a list showing information of each object (food) placed on the table. In the table information column 43, the terminal 2 displays the name, position, and the like of each food placed on the table, as well as the placing time of each food and the necessity of replacement.

In the present embodiment, the server 1 detects the first object and the second object from the table area, and also detects (recognizes) the food placed on the table. Then, the server 1 counts the placing time of each food and determines whether or not the food should be replaced. For example, the server 1 compares the placement time with a predetermined threshold value, and if it is equal to or more than the threshold value, determines that the server 1 should be replaced. For example, the server 1 may change the threshold value according to the type of detected food (whether or not it is fresh food, etc.).

In the above, food is mentioned as an object for determining the necessity of replacement, but the present embodiment is not limited to this. For example, the server 1 may target tableware (dish) prepared on the table, tableware such as tongs used when taking food.

Further, in the above, the necessity of replacement is determined only by the mounting time, but the present embodiment is not limited to this, and the necessity of replacement may be determined in consideration of the value of the heat map. .. For example, the server 1 sets a lower threshold value as the value of the area where the food is placed is larger, and sets a higher threshold value as the value of the area where the food is placed is smaller. Compare. In this way, by referring not only to the mounting time but also to the value of the heat map, it is possible to more preferably determine the necessity of replacement.

The terminal 2 displays the food placement time counted by the server 1 and the necessity of replacement in the table information column 43. For example, the terminal 2 displays a list of the placement times of each food and highlights the foods determined to be replaced. In FIG. 4, for convenience, the highlighted state is shown in bold. As a result, it is possible to support not only the hygiene management of the table but also the replacement of food.

FIG. 5 is a flowchart showing an example of a processing procedure executed by the server 1. The processing contents executed by the server 1 will be described with reference to FIG.
The control unit 11 of the server 1 acquires a moving image of the table on which the object is placed from above from the camera 3 (step S11). The control unit 11 identifies a table area corresponding to the table from the moving image, and detects a specific object and an object placed on the table from the table area (step S12). As described above, the object includes the first object and the second object. The first object is, for example, a customer's hand, a finger, or an instrument such as tongs used when taking food, and is an object other than the second object, that is, an object that comes into contact with a table for a purpose other than cleaning. The second object is, for example, a cleaning tool, gloves, or the like used by an employee during cleaning, and is an object related to table cleaning work. The object placed on the table is, for example, food, which needs to be replaced regularly.

The control unit 11 determines whether or not the first object is detected from the moving image (step S13). When it is determined that the first object has been detected (S13: YES), the control unit 11 generates a heat map showing the hygiene state of the table based on the detection result of the first object (step S14). Specifically, the control unit 11 generates a heat map in which the value of the region where the first object is detected is set according to the time, the number of times, and the like when the first object is detected. The control unit 11 may generate a heat map by referring to the environmental values (temperature, humidity, etc.) around the table in addition to the detection result of the first object.

After executing the process of step S14, or when NO in step S13, the control unit 11 determines whether or not the second object is detected from the moving image (step S15). When it is determined that the second object has been detected (S15: YES), the control unit 11 resets the heat map value in the region where the second object is detected (step S16).

After executing the process of step S16, or if NO in step S15, the control unit 11 determines whether or not the predetermined time has elapsed (step S17). When it is determined that the predetermined time has not elapsed (S17: NO), the control unit 11 repeats the processes of steps S11 to S16 until the predetermined time elapses. When it is determined that the predetermined time has elapsed (S17: YES), the control unit 11 counts the placement time of the object (food) detected from the moving image in step S12 (step S18). The control unit 11 determines whether or not the object placed on the table should be replaced based on the counted placement time (step S19). In response to a request from the terminal 2, the control unit 11 outputs a heat map of the table, a placement time of each object placed on the table, a determination result relating to the necessity of replacement, and the like (step S20). The process returns to step S11.

In the flowchart of FIG. 5, after the processing of steps S11 to S16 (generation of heat map) is executed, the processing of steps S18 to S20 (counting of the mounting time) is continuously executed, but steps S11 to S11 to The process of S16 and the process of steps S18 to S20 may be executed independently.

From the above, according to the first embodiment, by generating and presenting a heat map showing the hygiene state of the table, it is possible to suitably perform hygiene management of the table on which the object (food) is placed.

Further, according to the first embodiment, it is possible to more appropriately support the employees of the store by controlling the placement time at the same time as generating the heat map and determining the necessity of exchanging the object (food). can.

Further, according to the first embodiment, by generating a heat map based on the environmental value around the table, it is possible to present a heat map considering the environment in which the object (food) is placed.

(Embodiment 2)
In this embodiment, a mode for determining whether or not the first object and the second object have come into contact with the table by estimating the depths of the first object and the second object will be described. In the following description, the same reference numerals will be given to the contents overlapping with the first embodiment, and the description thereof will be omitted.

FIG. 6 is an explanatory diagram showing an outline of the second embodiment. FIG. 6 illustrates a case where the first object (customer's hand, tongs, etc.) is approaching the table when viewed from the side of the table. An outline of the present embodiment will be described with reference to FIG.

The server 1 detects the first object (and the second object) from the table area and generates a heat map, as in the first embodiment. In this case, the server 1 estimates the depth (distance) from the camera 3 to the first object (and the table), and determines whether or not the first object has touched the table based on the estimated depth.

For example, in the present embodiment, the camera 3 is configured as a depth camera equipped with a depth sensor. The server 1 acquires a distance image (moving image) in which a depth is added to each pixel in the image from the camera 3 and refers to the depth of the pixel corresponding to the position where the first object is detected.

The depth camera is an example of a depth estimation means, and the present embodiment is not limited to this. For example, a plurality of cameras 3 (stereo cameras) may be installed, and the depth of the first object may be estimated from the parallax of each camera 3. Alternatively, the server 1 may prepare a trained machine learning model (for example, a neural network) so as to estimate the depth of the first object when a moving image is input, and estimate the depth using the model. good. As described above, the depth estimation means is not limited to the depth camera.

For example, the server 1 estimates the depth from the camera 3 (the imaging point of the moving image) to the first object and the table, respectively. The depth of the table is not estimated from the image, but may be a fixed value. The server 1 compares the depth of the first object with the depth of the table and determines whether or not the first object has touched the table. The term "contact with the table" includes not only the case of contact with the table body but also the case of contact with an object (food) placed on the table. Not only when the first object is actually in contact with the table, but also when the first object is sufficiently close to the table (when the difference between the depth of the first object and the depth of the table is a certain value or less). Can also be included.

Server 1 generates a heat map according to the above determination result. That is, when the server 1 determines that the first object has contacted, the server 1 increases the value of the region in which the first object has contacted according to the detection time (contact time), the number of times, and the like, and generates a heat map.

Even when the second object is detected from the moving image, the server 1 performs the same processing. That is, the server 1 estimates the depth from the camera 3 to the second object, and determines whether or not the second object is in contact with the table. If it is determined that the table is in contact, the server 1 resets the value of the heat map of the area in which the second object is in contact.

FIG. 7 is a flowchart showing an example of a processing procedure executed by the server 1 according to the second embodiment.
When it is determined that the first object has been detected (S13: YES), the control unit 11 of the server 1 estimates the depth of the first object, and the first object comes into contact with the table based on the estimated depth. Whether or not it is determined (step S201). When it is determined that the first object is in contact (S201: YES), the control unit 11 generates a heat map in which the value of the region in which the first object is in contact is increased (step S202). After executing the process of step S202, or if NO in steps S13 and S201, the control unit 11 shifts the process to step S15.

When it is determined that the second object has been detected (S15: YES), the control unit 11 estimates the depth of the second object, and whether or not the second object has touched the table based on the estimated depth. Is determined (step S203). When it is determined that the second object is in contact (S203: YES), the control unit 11 resets the heat map value of the region in which the second object is in contact (step S204). After executing the process of step S204, or if NO in steps S15 and S203, the control unit 11 shifts the process to step S17.

From the above, according to the second embodiment, it is possible to more preferably generate a heat map by determining whether or not the first object and / or the second object is in contact with the table.

(Embodiment 3)
In this embodiment, a mode for generating a heat map based on the information of a person existing around the table will be described.

FIG. 8 is an explanatory diagram showing an outline of the third embodiment. FIG. 8 illustrates a case where a customer (person) exists around the table as viewed from the side of the table. An outline of the present embodiment will be described with reference to FIG.

In this embodiment, not only the camera 3 but also the thermography sensor 4 is installed above the table. The thermography sensor 4 is a sensor that measures a thermal image showing a temperature distribution around a table. The server 1 acquires a moving image from the camera 3 and also acquires a thermal image from the thermography sensor 4.

When generating a heat map, the server 1 identifies peripheral person information about a person existing around the table and refers to it when generating the heat map. Peripheral person information includes, for example, the density and behavior of people around the table, as well as the body temperature of the surrounding people. The server 1 recognizes the people existing around the table from the moving image, and specifies the density (number of people) of the surrounding people. Further, the server 1 identifies the behavior of a peripheral person from the moving image. The behavior to be identified is, for example, conversation, coughing, sneezing, etc. Further, the server 1 identifies the body temperature of the person recognized from the moving image from the thermal image.

Server 1 generates a heat map based on the peripheral person information specified above. For example, when the density of people existing around the table is a certain value or more, the server 1 weights the value of the heat map and sets a large increase width of the area where the first object is detected. Alternatively, when the server 1 identifies an action such as a conversation of a surrounding person, the server 1 sets a large value of the heat map of the table area within a certain distance from the person. Alternatively, when the body temperature of a surrounding person is equal to or higher than a certain value, the server 1 sets a large value of the heat map of the table area within a certain distance from the person. The above process is an example, and it suffices if a heat map can be generated based on peripheral person information.

FIG. 9 is a flowchart showing an example of a processing procedure executed by the server 1 according to the third embodiment. After executing the process of step S12, the server 1 executes the following process.
The control unit 11 of the server 1 acquires a thermal image showing the temperature distribution around the table (step S301). The control unit 11 identifies peripheral person information regarding a person (customer) existing around the table based on the moving image acquired in step S11 and / or the thermal image acquired in step S301 (step S302). Specifically, as described above, the control unit 11 specifies the body temperature of the surrounding person from the thermal image in addition to the density of the person around the table, the behavior (presence or absence of conversation), and the like. The control unit 11 shifts the process to step S13.

When it is determined that the first object has been detected from the moving image (S13: YES), the control unit 11 generates a heat map based on the detection result of the first object and the peripheral person information (step S303). .. Specifically, the control unit 11 determines the heat map of each region in the table region according to the density, behavior, body temperature, etc. of the person around the table, in addition to the time, the number of times, etc., when the first object is detected. Set the value. The control unit 11 shifts the process to step S15.

Based on the above, according to the third embodiment, the sanitary state of the table is determined by recognizing the person around the table from the moving image, specifying the information on the surrounding person related to the person, and referring to the heat map when generating the heat map. It can be managed more preferably.

Further, according to the third embodiment, by installing the thermography sensor 4 and specifying the body temperature of the surrounding person, the person who may be suffering from a disease or the like is specified, and the heat map is more preferably generated. be able to.

(Embodiment 4)
In this embodiment, a form of calculating the cleanliness of a store from the heat map of each table in the store and outputting it to the outside such as a website for posting a review of the store (hereinafter referred to as "review site") will be described. do.

FIG. 10 is an explanatory diagram showing an outline of the fourth embodiment. FIG. 10 illustrates how the cleanliness of the store is posted on the review site. An outline of the present embodiment will be described with reference to FIG.

As described in the first embodiment, the server 1 generates a heat map showing the hygiene state of the table in the store where the object (food) is placed. In the present embodiment, it is assumed that a plurality of

cameras

3, 3, 3, ... Are installed corresponding to each of the plurality of tables installed in the store, and a heat map of each table is generated. The server 1 calculates the cleanliness of the entire store based on the heat map data generated for each table.

For example, the server 1 calculates the cleanliness of the store based on the update history of the heat map of each table. The specific method for calculating the cleanliness is not particularly limited, but for example, the server 1 calculates the cleanliness according to the area (the number of grids) of the region where the value of the heat map is equal to or larger than the threshold value. Specifically, the server 1 calculates the total area of the area where the heat map value is equal to or greater than the threshold value in units of the past week, one month, etc., and divides the calculated total area by the area of the entire table. , Calculate the ratio (area ratio) of the table area above the threshold value for each table. Then, the server 1 calculates the cleanliness by taking the average value, the median value, etc. of the ratios of all the tables. The cleanliness may be expressed numerically, or may be expressed by a plurality of stages of evaluation such as "A rank", "B rank", "C rank", and so on.

Note that the above calculation method is an example, and the present embodiment is not limited to this. For example, the server 1 may calculate the cleanliness based on the number of times the heat map is reset by detecting the second object. Specifically, the server 1 counts the number of resets for each table in units of the past week, one month, etc., and calculates the cleanliness by taking the average value, the median value, etc. of the reset times of all the tables. As described above, the method of calculating the cleanliness is not particularly limited.

The server 1 outputs the calculated cleanliness of the store to a Web server (not shown) that manages the review site, and displays it in the review site. For example, as shown in FIG. 10, on the review site, the cleanliness of each store using this system is posted as "A rank", "B rank" and the like. As a result, it is possible to present to the users of the review site data that objectively evaluates the hygiene management efforts of each store.

FIG. 11 is a flowchart showing the procedure of the cleanliness calculation process. Based on FIG. 11, the processing contents executed by the server 1 in the present embodiment will be described.
The control unit 11 of the server 1 acquires heat map data (update history) of each of a plurality of tables installed in the store (step S401). The control unit 11 calculates the cleanliness of the store based on the acquired heat map data (step S402). For example, the control unit 11 calculates an index value for each table according to the area of the table area where the heat map value is equal to or larger than the threshold value, the number of times the heat map value is reset, and the like, and the average value, median value, etc. of each table. The cleanliness of the entire store is calculated by taking. The control unit 11 outputs the calculated cleanliness to an external Web server (step S403), and ends a series of processes.

From the above, according to the fourth embodiment, the cleanliness of the store can be objectively evaluated based on the heat map of each table.

(Embodiment 5)
In the present embodiment, a mode of recognizing the object as a specific object based on the movement of the object reflected in the moving image for a certain period of time will be described.

FIG. 12 is a schematic diagram showing the configuration of the control unit 11 according to the fifth embodiment. The control unit 11 has the same function as that of the first embodiment. Here, the control unit 11 functions as an acquisition unit 11a, a detection unit 11b, a generation unit 11c, and a change unit 11d by reading and executing the program P stored in the auxiliary storage unit 14.

The acquisition unit 11a acquires a moving image of the table on which the food is placed from above from the camera 3.

The detection unit 11b detects the first object and the second object from the table area corresponding to the table in the moving image. Specifically, the detection unit 11b detects a second object related to the cleaning work of the table and a first object other than the second object from the table area. Here, the detection unit 11b may detect the first object and / or the second object based on the movement of the object reflected in the moving image for a certain period of time.

The generation unit 11c generates a heat map in which the values of each area in the table area are set based on the detection result of the first object. Specifically, the generation unit 11c generates a heat map in which the value of the region where the first object is detected is set according to the time or the number of times when the first object is detected.

When the second object is detected, the change unit 11d changes the value of the heat map in the area where the second object is detected. Specifically, when the second object is detected, the changing unit 11d resets or subtracts the value in the region where the second object is detected.

The control unit 11 may further function as the depth estimation unit 11e. The depth estimation unit 11e estimates the depth from the imaging point where the moving image is captured to the first object or the second object. Further, the depth estimation unit 11e determines whether or not the first object or the second object has come into contact with the table based on the estimated depth. When the control unit 11 functions as the depth estimation unit 11e, the generation unit 11c generates or changes the value of the heat map according to the determination result.

Further, the control unit 11 may further function as the food determination unit 11f. The food determination unit 11f detects the food placed on the table from the moving image, and determines whether or not the food should be replaced based on the food placement time.

Further, the control unit 11 may further function as the recognition unit 11g. The recognition unit 11g recognizes a person existing around the table from the moving image. In addition, the recognition unit 11g identifies peripheral person information regarding the recognized person. When the control unit 11 functions as the recognition unit 11g, the generation unit 11c generates a heat map based on the peripheral person information and the detection result of the first object. The recognition unit 11g may acquire a thermal image showing the temperature distribution around the table and specify the body temperature of the person based on the thermal image. In this case, the control unit 11 generates a heat map according to the body temperature of the person.

Further, the control unit 11 may further function as a cleanliness calculation unit 11h. The cleanliness calculation unit 11e calculates the cleanliness of the store based on the heat maps of each of the plurality of tables as described in the fourth embodiment.

Note that the control unit 11 may acquire the environmental values around the table. In this case, the generation unit 11c generates a heat map based on the environmental value and the detection result of the first object.

FIG. 13 is a flowchart showing an example of a processing procedure executed by the server 1.
The control unit 11 of the server 1 detects an object reflected in the moving image in step S12. Here, when the control unit 11 detects the first object from the moving image, it does not determine that the first object has been detected immediately, but determines that the object that is a candidate for the first object has been detected. (S501). Then, the control unit 11 stores in the main storage unit 12 the time-series data of the first target operation indicating the operation of the object that is a candidate for the first object for a certain period of time. Next, the control unit 11 analyzes the time-series data of the first target operation by comparing with a predetermined operation set in advance (S502). When the control unit 11 determines that the time-series data of the first object operation is unique to the first object, the control unit 11 considers the candidate of the first object to be the first object, and determines that the candidate of the first object is the first object. Is detected (S503). When the control unit 11 determines that the first object has been detected (S503: YES), the control unit 11 generates a heat map showing the sanitary state of the table (S504). Here, the control unit 11 generates a heat map in which the value of the region where the first object is detected is set according to the time, the number of times, etc. from the time when the candidate of the first object is detected.

After executing the process of step S504, or when NO in step S503, the control unit 11 determines whether or not the second object is detected from the moving image. Here, when the control unit 11 detects the second object from the moving image, it does not determine that the second object has been detected immediately, but determines that the object that is a candidate for the second object has been detected. (S505). Then, the control unit 11 stores in the main storage unit 12 the time-series data of the second target operation indicating the operation of the object that is a candidate for the second object for a certain period of time. Next, the control unit 11 analyzes the time-series data of the second target operation by comparing with a predetermined operation set in advance (S506). When the control unit 11 determines that the time-series data of the second target operation is unique to the second target, the control unit 11 considers the candidate for the second target to be the second target, and determines that the candidate for the second target is the second target. Is detected (S507). When the control unit 11 determines that the second object has been detected, the control unit 11 changes the value of the heat map of each area in the table area according to the time, the number of times, etc. from the time when the candidate for the second object is detected. (Reset, etc.) (S508).

As described above, the server 1 according to the fifth embodiment detects the first object and / or the second object based on the movement of the object reflected in the moving image for a certain period of time. The detection accuracy can be improved. Supplementally, in the method of specifying the object based on the features obtained from the shape and color without using the time series data, there is a possibility that the detection omission of the object whose shape changes dynamically may occur. In the server 1 according to the fifth embodiment, even if the object has a dynamically changing shape such as a "hand", the object is specified by using the time-series data, so that the omission of detection of the object can be avoided. .. As a result, the detection accuracy of the object can be improved.

In addition, for example, in the method of specifying an object based on the feature amount obtained from the shape and color, when trying to detect the object such as "white duster", an object of a color similar to the object (white paper napkin, etc.) May be falsely detected. Since the server 1 according to the fifth embodiment specifies the operation of wiping the table with the duster based on the shape of the hand, the color of the cloth under the hand, the time-series data of the position of the hand to be wiped, and the like, the time-series data. It is possible to avoid false detection of an object as compared with a method that does not use. As a result, the detection accuracy of the object can be improved.

In the fifth embodiment, even if the first object and the second object are composed of the same object, they can be distinguished from each other. Supplementally, in the method of specifying the object based on the feature amount obtained from the shape and color, the first object that is the hand that touches the table and the second object that is the hand that wipes the table are confused and detected. Can happen. This is because since the first object and the second object are composed of "hands" which are the same object, there may be no difference in the feature amounts of the two obtained from the shape and color. On the other hand, in the server 1 according to the fifth embodiment, since the object is identified by using the time series data, even if it is the first object and the second object composed of the same object, those objects are identified. It is possible to distinguish and detect the operation.

The embodiments disclosed this time should be considered to be exemplary in all respects and not restrictive. The scope of the present invention is indicated by the scope of claims, not the above-mentioned meaning, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims. Further, in the present disclosure, various disclosures can be formed by an appropriate combination of the plurality of components disclosed in each of the above embodiments. For example, some components may be removed from all the components shown in the embodiments. Further, the components may be appropriately combined in different embodiments.

1 Server (information processing device)
11 Control unit 12 Main storage unit 13 Communication unit 14 Auxiliary storage unit P Program 2 Terminal 3 Camera

Claims

An acquisition unit that acquires a moving image of a table on which food is placed from above,
A detection unit that detects a first object and a second object from a table area corresponding to the table in the moving image.
Based on the detection result of the first object, a generation unit that generates a heat map in which the values of each region in the table region are set, and a generation unit.
When the second object is detected, the change part that changes the value of the heat map in the region where the second object is detected, and
An information processing device equipped with.
The information processing according to claim 1, wherein the generation unit generates the heat map in which the value of the region where the first object is detected is set according to the time or the number of times when the first object is detected. Device.
The detection unit detects the second object related to the cleaning work of the table and the first object other than the second object from the table area.
The information processing apparatus according to claim 1 or 2, wherein the changing unit resets or subtracts a value in a region where the second object is detected when the second object is detected.
The depth from the imaging point where the moving image was imaged to the first object or the second object is estimated, and based on the estimated depth, whether or not the first object or the second object comes into contact with the table. Further equipped with a depth estimation unit to determine whether
The information processing apparatus according to any one of claims 1 to 3, wherein the generation unit generates or changes the heat map according to a determination result.
Further provided is a food determination unit that detects the food placed on the table from the moving image and determines whether or not the food should be replaced based on the placing time of the food.
The information processing apparatus according to any one of claims 1 to 4.
A recognition unit that recognizes a person existing around the table from the moving image and identifies peripheral person information about the recognized person is further provided.
The information processing device according to any one of claims 1 to 5, wherein the generation unit generates the heat map based on the peripheral person information and the detection result of the first object.
The recognition unit acquires a thermal image showing the temperature distribution around the table, and identifies the body temperature of the person based on the thermal image.
The generation unit generates the heat map according to the body temperature of the person.
The information processing apparatus according to claim 6.
The generation unit generates the heat map for each of the plurality of tables installed in the store.
A cleanliness calculation unit that calculates the cleanliness of the store based on the heat map of each of the plurality of tables and outputs the cleanliness of the store to the outside is further provided.
The information processing apparatus according to any one of claims 1 to 7.
The acquisition unit acquires the environmental values around the table and obtains them.
The information processing apparatus according to any one of claims 1 to 8, wherein the generation unit generates the heat map based on the environment value and the detection result of the first object.
The detector is
The first object and / or the second object is detected based on the movement of the object reflected in the moving image for a certain period of time.
The information processing apparatus according to any one of claims 1 to 9.
Acquire a moving image of the table on which food is placed from above,
The first object and the second object are detected from the table area corresponding to the table in the moving image, and the first object and the second object are detected.
Based on the detection result of the first object, a heat map in which the values of each area in the table area are set is generated.
An information processing method in which a computer executes a process of changing the value of the heat map in a region where the second object is detected when the second object is detected.
A program for causing a computer to execute the information processing method according to claim 11.