KR102708485B1 - Apparatus and method for monitoring eating - Google Patents

Abstract

The present invention relates to a meal monitoring device and method. A meal monitoring device according to one embodiment of the present invention comprises: a storage unit; a communication unit for receiving an image including a dining space where at least one user eats and dishes within the dining space; And a control unit which obtains meal location information of the user within the dining space based on the user terminal, recognizes food contained in the dish within the image, generates meal information based on the food, and then controls to link and store the image and the meal information, and controls to store the meal location information and acquisition time information of the meal location information within meta information within the image; wherein the meal location information is obtained by using at least one of a first method for obtaining the user terminal by recognizing at least one recognition means within the dining space, a second method for obtaining the information based on image analysis, a third method for obtaining the information based on GPS location information of the user terminal, and a fourth method for obtaining the information based on the communication strength of the user terminal for the currently connected wireless Internet network, and the meal location information may include at least one of information on an area where the user is located within the dining space, information on a table at which the user is seated among at least one table arranged within the dining space, and information on a seat at which the user is seated at a table arranged within the dining space.

Description

{APPARATUS AND METHOD FOR MONITORING EATING}

The present invention relates to a meal monitoring device and method, and more particularly, to a meal monitoring device and method that can provide useful information to users and store managers through the user's intake information.

The amount of food waste discarded each year is increasing worldwide, and the disposal of food waste is emerging as an important social issue. In order to reduce the amount of food waste generated in stores such as restaurants that sell food, various attempts are being made to protect the environment and achieve positive effects on store management.

Meanwhile, as the amount of food waste increases, the inconsistent quality of food sold in stores is becoming a problem.

Therefore, there is a need to develop a technology that can be used not only in stores such as restaurants but also at home, and that can provide health information to users through their meal information, and information that can be used for food waste management and promotion.

Korean Patent Publication No. 10-2020-0113564 (Published on October 7, 2020)

The present invention has been proposed to solve the above-mentioned problems, and provides a meal monitoring device and method that generates and provides useful meal information through a video or image of a user's meal, thereby enabling the user to utilize the information for managing his or her diet and health.

In addition, the present invention provides a meal monitoring device and method that enables a store manager to manage the uniform quality of food sold in the store and manage the cooking amount by generating and providing store information including the amount of leftover food or food preferences based on various information about users of the store.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

According to one embodiment of the present invention for solving the above-described problem, a meal monitoring device comprises: a storage unit; a communication unit for receiving an image including a dining space where at least one user eats and dishes within the dining space; And a control unit for obtaining meal location information of the user within the dining space based on the user terminal, recognizing food contained in the dish within the image to generate meal information based on the food, and then controlling the image and the meal information to be linked and stored, and controlling the meal location information and acquisition time information of the meal location information to be stored within meta information within the image; wherein the meal location information is obtained by using at least one of a first method for obtaining the user terminal by recognizing at least one recognition means within the dining space, a second method for obtaining the information based on image analysis, a third method for obtaining the information based on GPS location information of the user terminal, and a fourth method for obtaining the information based on the communication strength of the user terminal for the currently connected wireless Internet network, and the meal location information may include at least one of information on an area where the user is located within the dining space, information on a table at which the user is seated among at least one table arranged within the dining space, and information on a seat at which the user is seated at a table arranged within the dining space.

Meanwhile, a meal monitoring method according to one embodiment of the present invention comprises the steps of: obtaining an image including a dining space where at least one user is eating and tableware within the dining space; obtaining meal location information of the corresponding user within the dining space based on a user terminal; recognizing food contained in the tableware within the image and generating meal information based on the food; And a step of controlling to store the image and the meal information in a linked manner, and including the meal location information and the acquisition time information of the meal location information in the meta information of the image and storing them; wherein the meal location information is obtained by using at least one of a first method in which the user terminal recognizes at least one recognition means in the dining space, a second method in which the user terminal is obtained based on the image analysis, a third method in which the user terminal is obtained based on the GPS location information of the user terminal, and a fourth method in which the user terminal is obtained based on the communication strength of the user terminal for the currently connected wireless Internet network, and the meal location information may include at least one of information on an area in which the user is located in the dining space, information on a table at which the user is seated among at least one table arranged in the dining space, and information on a seat at which the user is seated at a table arranged in the dining space.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention, useful meal information is generated and provided through a video or image of a user's meal, thereby enabling the user to utilize it for managing his or her diet and health.

In addition, according to the present invention, store information including leftovers and food preferences, etc. are generated and provided based on various information about users of the store, thereby enabling the store manager to manage the uniform quality of food sold in the store and manage the cooking amount at the same time.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

Figure 1 is a floor plan of a store using a meal monitoring device according to an embodiment of the present invention.
Figure 2 is a block diagram showing the configuration of a meal monitoring device according to an embodiment of the present invention.
FIG. 3 is a drawing showing an example of a table in a store using a meal monitoring device according to an embodiment of the present invention.
FIG. 4 is a drawing showing an example of a camera that can be applied to a store using a meal monitoring device according to an embodiment of the present invention.
FIG. 5 is a drawing showing another example of a camera that can be applied to a store using a meal monitoring device according to an embodiment of the present invention.
Figure 6 is a flowchart showing a meal monitoring method according to one embodiment of the present invention.
FIG. 7 is a diagram showing an example of meal information transmitted from a meal monitoring device according to one embodiment of the present invention being output through a user interface of a user terminal.

The advantages and features of the present invention, and the methods for achieving them, will become clear with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and these embodiments are provided only to make the disclosure of the present invention complete and to fully inform a person skilled in the art of the scope of the present invention, and the present invention is defined only by the scope of the claims.

The terminology used herein is for the purpose of describing embodiments only and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. The terms "comprises" and/or "comprising" as used in the specification do not exclude the presence or addition of one or more other components in addition to the mentioned components. Like reference numerals refer to like components throughout the specification, and "and/or" includes each and every combination of one or more of the mentioned components. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it should be understood that a first component mentioned below may also be a second component within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with the meaning commonly understood by those skilled in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries shall not be ideally or excessively interpreted unless explicitly specifically defined.

The spatially relative terms "below," "beneath," "lower," "above," "upper," and the like can be used to easily describe the relationship between one component and other components as depicted in the drawings. The spatially relative terms should be understood to include different orientations of the components when used or operated in addition to the orientations depicted in the drawings. For example, if a component depicted in the drawings is flipped over, a component described as "below" or "beneath" another component may be placed "above" the other component. Thus, the exemplary term "below" can include both the above and below orientations. The components may also be oriented in other directions, and the spatially relative terms may be interpreted accordingly.

The term "part" or "module" as used in this specification means a software or hardware component such as an FPGA or ASIC, and the "part" or "module" performs certain functions. However, the "part" or "module" is not limited to software or hardware. The "part" or "module" may be configured to reside on an addressable storage medium and may be configured to execute one or more processors. Thus, by way of example, the "part" or "module" includes components such as software components, object-oriented software components, class components, and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functionality provided in the components and "parts" or "modules" may be combined into a smaller number of components and "parts" or "modules" or further separated into additional components and "parts" or "modules."

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with the meaning commonly understood by those skilled in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries shall not be ideally or excessively interpreted unless explicitly specifically defined.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

Figure 1 is a floor plan of a store using a meal monitoring device according to an embodiment of the present invention.

A store (1) may be a restaurant or cafe that sells food to customers, and may be a place that provides food to users (customers), such as a general restaurant, cafeteria, in-house restaurant, meal service, buffet, take-out specialty store, or delivery specialty store, and there are no restrictions on its type or form. However, this is only one example, and may be a general home rather than a store (1), and the number of tables and their arrangement structure may be changed.

Referring to Fig. 1, a store (1) can be divided into a hall (2) where customers can eat food and a kitchen (3) where food is cooked. At least one table (10) where customers can eat food can be placed in the hall (2). The location information of the user can be obtained based on one of the recognition means (11), such as a table number, tag, QR code, barcode, NFC, and RFID, equipped (attached) to the table (10) through the user terminal (200), which is the user's terminal. At this time, the location information can further include store information, user information, whether the user is seated at a table, information on the seated table, information on the seat of the user at the table, and time information.

Specifically, each table (10) may be provided with an identification symbol as a recognition means (11), for example, an identification symbol such as a table number or a QR code may be displayed on each table (10). Meanwhile, a tag for short-range communication in which an identification symbol is stored as a recognition means (11) may be attached to each table (10), and the communication technology used in the tag for short-range communication may be NFC (Near Field Communication) or RFID. The types of the recognition means and the communication technology applied are not limited to those described above.

For example, when a user tags a user terminal (200) in a store (1) with a tag for short-range communication as a recognition means (11) attached to a table (10), user information including not only table information but also user identification information can be collected. For example, when a user simply places his/her user terminal (200) on a position of a table 101) where a tag for short-range communication is attached, the table information and information about the user can be provided to the meal monitoring device (100) from the user terminal (200) due to tagging. Therefore, according to the present embodiment, the meal monitoring device (100) can collect user information without causing inconvenience to the user.

However, this is only one embodiment, and the user's location information may be obtained by using at least one of a method based on the GPS location information of the user's terminal, a method of analyzing an image captured by the user's terminal within the store or an image captured by a camera installed in the hall (2), and a method based on the communication strength of the user's terminal with respect to a wireless Internet network such as Wi-Fi/beacon used in the store, and the method for obtaining the location information is not limited. That is, the dining location information may be obtained by using at least one of a first method of obtaining it by recognizing at least one recognition means within the dining space, a second method of obtaining it based on analysis of the captured image, a third method of obtaining it based on the GPS location information of the user's terminal, and a fourth method of obtaining it based on the communication strength of the user's terminal with respect to the currently connected wireless Internet network.

In addition, each method can be utilized stepwise to obtain the user's location information more accurately. For example, the approximate location of the user's dining area is identified based on the GPS location information of the user's terminal, and the location of the meal within the dining area is identified based on the communication strength of the user's terminal with respect to the wireless Internet network to which the user's terminal is connected.

By utilizing each method in stages like this, the user's dining location can be accurately identified and linked, thereby allowing more comprehensive management of the user's visit information. Specifically, by analyzing the video captured by the camera, it is possible to identify not only the table at which the user sat within the dining location, but also the menus (food) ordered at that table. In other words, it is possible to confirm which restaurant the user visited, which table, which seat, and which menu he or she consumed.

Meanwhile, an auxiliary device (not shown) such as a kiosk may be placed in the store (1), and the user may order food through the auxiliary device. In addition, the user may order food through a store employee, such as an employee of the store (1), or through a user terminal (200) such as a smartphone. When the user orders food through an auxiliary device or a user terminal (200), the order for food may be made through voice recognition using a microphone equipped in the auxiliary device or a microphone (not shown) equipped in the user terminal (200).

Here, when a user places an order using a user terminal (200) at a table (10), an identification symbol such as a table number or QR code may be used. In some embodiments, an order may be placed by accessing the menu using GPS information using the user terminal (200).

In some embodiments, a user may order food only through a store employee, such as an employee of the store (1), and in such a case, a customer who wants healthcare information may present an identification symbol, such as a QR code of the user terminal (200), to the store employee, and after the store employee scans or verifies the identification symbol, the user may enter table information, so that food information for the table information may be recorded for the user by matching it with the table information.

Furthermore, the auxiliary device or user terminal (200) can collect user information through the user's payment process at the store (1), and the user information collected through the payment process together with the table information can be used to provide a service including healthcare information to the user based on the collected and analyzed food information. Since the information collected from the auxiliary device or user terminal (200) and the information analyzed by the meal monitoring device (100) can be stored for the customer based on the collected user information, a service can be provided for the user based on the collected and analyzed food information.

Meanwhile, although not shown in Fig. 1, at least one camera may be placed at multiple locations within the store to obtain image data about the user and/or the store, or to recognize the user. At this time, at least one camera may be placed by mixing various types of cameras. However, since each camera is for photographing the user or dishes, etc., they may be placed at a height where they can photograph the user's face, table, floor, countertop, etc.

Meanwhile, at least one camera may be provided in at least one of the following forms: a first form in which the camera is provided as a portable type that can be moved and placed on each table so that it can be used; a second form in which the camera is installed on at least one of the ceiling and the wall; a third form in which the camera is provided in a kiosk or a point of sales (POS); and a fourth form in which the camera is provided in a terminal used by a store (hall) manager or employee. In other words, the type, shape, and form of the camera are not limited.

For example, when a user enters a store (1), the user can be recognized by obtaining an image (video) containing the user's face through at least one camera, the user's movement path can be confirmed to obtain meal location information, and a meal video (video taken while eating) that serves as the basis for confirming the user's meal information can be obtained.

Accordingly, various food information can be collected within the store (1). After the user finishes eating, the food remaining on the dishes can be photographed to collect post-meal image information about the user, and the post-meal image information can be used to generate leftover information about the user. For example, the user can photograph the food remaining on the dishes using at least one of the cameras, auxiliary devices, and user terminals (200) installed in the store, but the store staff can also photograph the food remaining on the dishes before clearing the table (10).

In addition, as food information, food in dishes may be photographed before a user eats, and pre-meal image information for the user may be collected. In addition, while the user is eating, the process of consuming food in dishes may be photographed, and image information during the meal may be collected for the user. In addition, the process of cooking food in the kitchen (3) may be photographed, and cooking image information may be collected.

According to an embodiment of the present invention, the store (1) can analyze the user's leftover food information to provide additional services, and can also use the analysis results of food information including the leftover food information for the operation of the store (1).

Figure 2 is a block diagram showing the configuration of a meal monitoring device according to an embodiment of the present invention.

Referring to FIG. 2, a meal monitoring device (100) according to an embodiment of the present invention may include a communication unit (110), a storage unit (120), and a control unit (130).

The communication unit (110) is for performing communication with the user's terminal and transmits and receives wireless signals in a communication network according to wireless Internet technologies.

Wireless Internet technologies include, for example, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), etc., and the meal monitoring device (100) transmits and receives data according to at least one wireless Internet technology, including Internet technologies not listed above.

As for short range communication, short range communication can be supported using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies. Such short range wireless communication networks (Wireless Area Networks) can support wireless communication between a meal monitoring device (100) and a user terminal (200). At this time, the short range wireless communication network can be a short range wireless personal area network (Wireless Personal Area Networks).

The storage unit (120) stores data supporting various functions of the meal monitoring device (100). It can store a plurality of application programs (or applications) driven by the meal monitoring device (100), data for the operation of the meal monitoring device (100), and commands. At least some of these application programs can be downloaded from an external server via wireless communication. Meanwhile, the application programs can be stored in the storage unit (120), installed on the meal monitoring device (100), and driven to perform operations (or functions) by the control unit (130).

In addition to the operations related to the application program, the control unit (130) typically controls the overall operation of the meal monitoring device (100). The control unit (130) can provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the components discussed above or by operating an application program stored in the storage unit (120).

Specifically, the control unit (130) receives an image including a dining space where at least one user is eating and tableware in the dining space, analyzes the image to generate and store meal information of the user. At this time, the control unit (130) stores the meal information in association with the image, and includes previously acquired meal location information and acquisition time information in the meta information of the image. Here, the meal location information may include at least one of information on the area where the user is located in the dining space, information on the table at which the user is seated among at least one table arranged in the dining space, and information on the seat at which the user is seated among the tables arranged in the dining space. In addition, the meal information may include at least one of the type of food (food), the user's preference for the food, the amount consumed, the calories consumed, and leftover information. Here, the leftover information may be used by the meal monitoring device (100) to manage leftovers for the corresponding store in the future, and the preference may be used by the meal monitoring device (100) to manage the user's diet as well as statistics on menus sold in the corresponding store. Through this, management can be carried out, such as adjusting the amount of food for each menu to minimize leftovers, or organizing the menu or recipe by considering food preferences.

Meanwhile, the image including the dining space where at least one user is eating and the tableware within the dining space may be an image captured by a camera installed within the dining space or an image captured via a user terminal (200). Here, there may be at least one camera installed within the dining space.

Meanwhile, the control unit (130) must confirm (identify) meal location information to identify the user before analyzing the video. To this end, the meal location information can be obtained based on information transmitted from the user terminal (200) before shooting the video, or the meal location information can be confirmed first before generating the meal information based on the shot video.

For example, if the image includes multiple tables in a dining space, the image is divided into multiple areas based on each table, the area in which the user is included is recognized among the divided areas, and the relative position of the recognized area in the image is determined to obtain the dining location information of the user based on this. In addition, if at least one of the codes and tags attached to the table is recognized by the user terminal (200) or the camera provided in the dining space, the recognized identification information can be received through the communication unit (110), and the control unit (130) checks the seat information corresponding to the identification information among the pre-stored seat information. The confirmed seat information can be obtained as the dining location information of the user.

Meanwhile, when a request to write/register a review for a store corresponding to a dining space is received from a user terminal (200), the control unit (130) determines whether or not to allow the writing/registration of a review by using pre-stored information to determine whether the user has actually visited the store.

Specifically, the control unit (130) determines whether the user has visited the store before by checking the visit date, visit time, order details, etc. based on the video and meta information pre-stored in the storage unit (120), and decides whether to write/register a review based on the determination result. Meanwhile, when a review video including a shooting location and shooting time in the meta information is received as a request for writing/registering a review, the control unit (130) determines whether a video related to the shooting location and shooting time included in the meta information of the review video among the videos pre-stored in the storage unit (120) exists, and decides whether to write/register a review based on the determination result. If it is confirmed that there is a visit history, the writing or registration of the review is permitted, and if it is confirmed that there is no visit history or that the visit history does not match, the writing or registration of the review is not permitted.

A meal monitoring device (100) according to one embodiment of the present invention may be a service server for providing a meal monitoring service. Meanwhile, a user terminal (200) or a store terminal (not shown) for using the meal monitoring service may be a computer, an Ultra Mobile PC (UMPC), a workstation, a netbook, a PDA (Personal Digital Assistants), a portable computer, a web tablet, a wireless phone, a mobile phone, a smart phone, a pad, a smart watch, a wearable terminal, an e-book, a portable multimedia player (PMP), a portable game console, a navigation device, a black box, a digital camera, or other mobile communication terminals, on which a user can install and execute a plurality of application programs (i.e., applications) desired by the user. Accordingly, the user terminal (200) may need to install a separate program or application in order to receive the meal monitoring service. However, this is only one example, and you may also receive meal monitoring services by accessing the webpage.

FIG. 3 is a drawing showing an example of a table in a store using a meal monitoring device according to an embodiment of the present invention, illustrating a case where four users (20-1 to 20-4) sit around one table (10) and eat.

Referring to Fig. 3, one table (10) is formed with four seats for sitting and eating, and each seat may be equipped (attached) with an identification means (11-1 to 11-4) including identification information for each seat.

Although the user terminal of each user is not shown in Fig. 3, each user can recognize the recognition means equipped on the seat where he or she is seated through his or her terminal, thereby allowing the meal monitoring device (100) to confirm which seat at which table he or she is sitting.

Meanwhile, assuming that a video of a meal is captured through a camera installed on the upper ceiling of the table, the video captured by the camera is transmitted to the meal monitoring device (100), and the meal monitoring device (100) can analyze the video to generate meal information for each of the first user (20-1) to the fourth user (20-4).

Meanwhile, when four users (20-1 to 20-4) order food, the order information may be transmitted together to the meal monitoring device (100). At this time, the order may be made using the terminal of one of the four users (20-1 to 20-4), or may be made using a device provided at each table for ordering, a kiosk installed in the store, or may be made by the store (hall) manager or employee entering the order information into his/her terminal or POS. In this way, the meal monitoring device (100) may be enabled to generate meal information by referring to the order information during image analysis. For example, the meal monitoring device (100) may check the type of food consumed by each user, or calculate the amount consumed, calories consumed, etc. through the order information. To this end, the meal monitoring device (100) may store a list of food items sold by the store (hall) manager, the quantity of each food item, calorie information, etc. as pre-stored data.

As illustrated in FIG. 3, when the first user (20-1) to the fourth user (20-4) order the first food (30-1) to the fourth food (30-4) as main foods, respectively, and the fifth food (30-5) is provided as shared food, the meal monitoring device (100) analyzes the acquired images to confirm the food types of the main food and shared food of each user, and generates meal information for each user. Here, for the main food, the volume and weight before the meal can be compared with the volume and weight after the meal to derive relevant data, and for the shared food, data such as the intake amount of each user can be derived individually through image analysis.

FIG. 4 is a drawing showing an example of a camera that can be applied (installed) in a dining space using a meal monitoring device according to an embodiment of the present invention, and FIG. 5 is a drawing showing another example of a camera that can be applied to a store using a meal monitoring device according to an embodiment of the present invention.

Specifically, FIG. 4 illustrates a case where the camera (300) is installed on the ceiling in a dining space, and FIG. 5 illustrates a case where the camera (300) is configured as a portable type that can be moved. In the latter case, it can be used on a table placed in a dining space, or it can be used when the store manager moves it as needed. As illustrated in FIG. 5, even when configured as a portable type, it can be formed in various forms such as (a) and (b).

Meanwhile, as described above, at least one camera may be installed in various types and forms in a store using the meal monitoring device (100) according to the embodiment of the present invention. For example, at least one camera may include at least one of a 2D camera, a 3D camera, a ToF (Time of Flight) camera, a light field camera, a stereo camera, an event camera, an infrared camera, a lidar sensor, and an array camera, and there is no limitation on its configuration as long as it can measure image information and depth information. In addition, a plurality of cameras may be arranged in a stereo structure to acquire left and right images for implementing a stereoscopic image.

Figure 6 is a flowchart showing a meal monitoring method according to one embodiment of the present invention.

First, when the user's seating in the dining space is detected through a verification (analysis) operation such as recognizing a recognition means through a user terminal (200), a camera installed in the dining space, or an administrator terminal, or tracking the user (S201), then, based on the recognized information or verification result, the seat information at which the user is seated at a table arranged in the dining space is confirmed and stored as dining location information for the user (S203). At this time, the dining location information and the acquisition time information may be stored together.

Thereafter, the image captured while the user is eating is received through the user terminal (200), a camera installed in the dining space, or an administrator terminal (S205), and the image, the meta information within the image, the previously acquired dining location information, and the acquisition time information are matched and stored (S207).

The meal monitoring device (100) generates and stores meal information for the user based on the information stored in step S207 (S209) and transmits the meal information to the user or a preset terminal (S211).

Although not shown in Fig. 6, the meal monitoring device (100) may separate and analyze information about each store based on the information stored in step S207 and transmit the amount of leftover food, sales volume, preferences, etc. as store information to the manager terminal of the store. As a result, the manager of the store can utilize the store information for store operation.

Meanwhile, after step S211, if the meal monitoring device (100) receives a request for writing/registering a review for a specific menu or store from a user terminal (200) (S213), it verifies whether the user has actually consumed the menu or actually visited the store based on the information stored in step S207 (S215). Depending on the result of the verification, the meal monitoring device (100) allows or rejects the writing/registration of a review for the user (S217). Specifically, if the user has actually consumed the menu or actually visited the store, the writing/registration of a review by the user terminal (200) is permitted, and if the user has no history of actually consuming the menu or actually visiting the store, the writing/registration of a review by the user terminal (200) is rejected.

This ensures that only reviews from actual visitors are registered, thereby providing more useful review information to other consumers.

FIG. 7 is a diagram showing an example of meal information transmitted from a meal monitoring device according to one embodiment of the present invention being output through a user interface of a user terminal.

Referring to FIG. 7, the user terminal (200) may be equipped with a camera unit (201), an audio output unit (203), and a display unit (205).

The camera unit (201) can be used to capture images while eating, or to capture images before and after eating and transmit them to the meal monitoring device (100).

At this time, if the camera unit (201) includes a depth camera such as a 3D camera or a ToF (Time of Flight) camera capable of obtaining depth information, more accurate meal information can be generated from the meal monitoring device (100).

Meanwhile, meal information transmitted by the meal monitoring device (203) can be output as sound through the sound output unit (203), and the meal information can be displayed on the user interface so that the user can visually confirm the meal information through the display unit (205).

The steps of a method or algorithm described in connection with the embodiments of the present invention may be implemented directly in hardware, implemented in a software module executed by hardware, or implemented by a combination of these. The software module may reside in a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, a CD-ROM, or any other form of computer-readable recording medium well known in the art to which the present invention pertains.

Above, while the embodiments of the present invention have been described with reference to the attached drawings, it will be understood by those skilled in the art that the present invention may be implemented in other specific forms without changing the technical idea or essential features thereof. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

1: Store 2: Hall
3: Kitchen 10: Table
11: Recognition means 100: Meal monitoring device
110: Communications Department 120: Storage Department
130: Control Unit 20-1: User 1
20-2: 2nd user 20-3: 3rd user
20-4: 4th user 30-1: 1st food
30-2: Second food 30-3: Third food
30-4: 4th food 30-5: Shared food
200: User terminal 201: Camera section
203: Sound output section 205: Display section
300: Camera

Claims (10)

In the meal monitoring device,
storage;
A communication unit that receives an image including a dining space where at least one user is eating and tableware within the dining space; and
A control unit is provided for obtaining meal location information of the user based on the user's terminal (hereinafter, "user terminal"), recognizing food contained in the dish in the image, generating meal information based on the recognized food, and then controlling the image and the meal information to be linked and stored, while controlling the storage of the meal location information and acquisition time information of the meal location information within the meta information in the image.
The above meal location information is,
It is obtained by using at least one of the following: a first method in which the user terminal is obtained by recognizing at least one recognition means within the dining space, a second method in which the user terminal is obtained based on analysis of the image, a third method in which the user terminal is obtained based on GPS location information, and a fourth method in which the user terminal is obtained based on the communication strength of the user terminal for the currently connected wireless Internet network.
The above dining location information is characterized in that it includes at least one of information on the area where the user is located within the dining space, information on the table at which the user is seated among at least one table arranged within the dining space, and information on the seat at which the user is seated among the tables arranged within the dining space.
The above control unit,
When a request to write a review for the first menu at the first store corresponding to the dining space is received from the user terminal above,
Based on the video and meta information previously stored in the storage unit, whether the user actually visited the first store, the time of the visit to the first store, and the order details at the first store are confirmed.
Based on the above verification results, whether or not the request for writing a review is accepted is determined based on whether the user actually visited the first store and actually consumed the first menu at the first store.
In the case where there is food shared by multiple users (hereinafter referred to as “shared food”), the video is analyzed to confirm the main food ordered by each user and the type of food of the shared food.
A meal monitoring device characterized in that it calculates the intake amount of the shared food of each user through the above image analysis and generates meal information related to the shared food.
In the first paragraph,
The above video,
A meal monitoring device, characterized in that the video is captured by the user terminal or captured by a camera installed in the dining space.
In the first paragraph,
The cameras installed in the above dining space are:
A meal monitoring device characterized in that it is provided in at least one of the following forms: a first form that can be provided as a portable type that can be moved and placed so that it can be used on each table, a second form that is installed on at least one of a ceiling and a wall, a third form that is provided in a kiosk or a point of sales (POS), and a fourth form that is provided in a terminal used by a manager or employee of the dining space.
In the first paragraph,
At least one of the above recognition means,
Each of which is provided on at least one table placed within the dining space, and which can be attached to each seat formed on the table, and which includes at least one of a code and a tag including identification information for the seat,
The above control unit,
A meal monitoring device characterized in that, when receiving identification information recognized by the user terminal or a camera installed in the dining space, the device checks the seat information corresponding to the identification information among the pre-stored seat information, and acquires the checked seat information as the dining location information of the corresponding user.
In the first paragraph,
The above video,
The photograph was taken including at least one table placed within the above dining space,
The above control unit,
The above at least one table divides the image into multiple regions, based on each of the tables,
Recognize the area in which the user is included among the above-mentioned separated areas,
Identify the relative position of the recognized area within the above image,
A meal monitoring device characterized by obtaining the meal location information based on the above relative location.
In the first paragraph,
The above control unit,
Based on the GPS location information of the third method, the location of the dining area is identified,
A meal monitoring device characterized in that it identifies the meal location within the dining space based on the communication strength of the user terminal of the fourth method and obtains final meal location information.
delete delete In a method of meal monitoring performed by a device,
A step of acquiring an image including a dining space where at least one user is eating and tableware within the dining space;
A step of obtaining meal location information of a user based on the user's terminal (hereinafter, 'user terminal');
A step of recognizing food contained in the dish in the above image and generating meal information based on the food; and
A step of controlling the above image and the above meal information to be stored in a linked manner, and including the meal location information and the acquisition time information of the meal location information in the meta information within the image;
The above meal location information is,
It is obtained by using at least one of the following: a first method in which the user terminal is obtained by recognizing at least one recognition means within the dining space, a second method in which the user terminal is obtained based on analysis of the image, a third method in which the user terminal is obtained based on GPS location information, and a fourth method in which the user terminal is obtained based on the communication strength of the user terminal for the currently connected wireless Internet network.
The above dining location information includes at least one of information on the area where the user is located within the dining space, information on the table at which the user is seated among at least one table arranged within the dining space, and information on the seat at which the user is seated among the tables arranged within the dining space.
The above device,
When a request to write a review for the first menu at the first store corresponding to the dining space is received from the user terminal above,
Based on the previously stored video and meta information, check whether the user actually visited the first store, the time of the visit to the first store, and the order details at the first store.
Based on the above verification results, whether or not the request for writing a review is accepted is determined based on whether the user actually visited the first store and actually consumed the first menu at the first store.
In the case where there is food shared by multiple users (hereinafter referred to as “shared food”), the video is analyzed to confirm the main food ordered by each user and the type of food of the shared food.
A meal monitoring method characterized by generating meal information related to the shared food by calculating the intake amount of each user for the shared food through the above image analysis.
A computer program stored in a computer-readable recording medium for executing the meal monitoring method of Article 9, coupled with a computer.

Images