CN109615358B

CN109615358B - Deep learning image recognition-based restaurant automatic settlement method and system

Info

Publication number: CN109615358B
Application number: CN201811295435.8A
Authority: CN
Inventors: 方炼
Original assignee: Beijing Vizum Intelligent Technology Co ltd
Current assignee: Beijing Vizum Intelligent Technology Co ltd
Priority date: 2018-11-01
Filing date: 2018-11-01
Publication date: 2020-11-03
Anticipated expiration: 2038-11-01
Also published as: CN109615358A

Abstract

A restaurant automatic settlement method and system based on deep learning image recognition are provided, and the system comprises a monocular camera, an image processing module, a pricing program module and a payment system. The method comprises the steps that a monocular camera is used for collecting complete images of a dinner plate identification area, the complete images are transmitted to an image processing module and then are calculated through an ENet Object Detection algorithm model to obtain positioning frame and category label data of a dinner plate, the data are output to a pricing program module, a pricing program in the pricing program module calculates and displays a total price according to the category and the number of dishes, and a customer can pay conveniently through a payment system. The method adopts the image recognition technology based on deep learning to automatically settle accounts for the restaurants, has the advantages of low settlement cost, flexible deployment, strong robustness and the like of the common visual technology, overcomes the problem of insufficient accuracy of settlement recognition of the common visual technology under the influence of adverse factors such as angles, light rays, shielding and the like, and has more recognizable contents and stronger practicability.

Description

Deep learning image recognition-based restaurant automatic settlement method and system

Technical Field

The invention relates to the technical field of catering settlement, in particular to a restaurant automatic settlement method and system based on deep learning image recognition.

Background

The catering is an essential link in the life of people, and along with the continuous acceleration of the rhythm of urban life, more and more people solve the diet problem through restaurants. The mainstream practice of the restaurant is to autonomously select and then uniformly settle accounts for charging. Generally, the current restaurant settlement methods mainly include:

(1) the traditional manual settlement method comprises the following steps:

in the traditional manual settlement, the category and the cost of dishes are manually distinguished and summed, so that more people have to eat at a dining peak, the distinguishing and calculating efficiency of manual settlement lines is low, and errors are easily caused.

(2) The settlement method based on bar code identification comprises the following steps:

the bar code label is pasted on the dinner plate and is associated with the corresponding dish price, the corresponding price is obtained by reading the dish price by an infrared or laser bar code scanning gun at the charging terminal, and then the total amount is calculated manually or automatically to carry out settlement. Compared with the traditional manual settlement, the result is more accurate and reliable, but the settlement efficiency is improved to a limited extent due to the fact that dinner plates need to be scanned one by one, and the barcode labels are easy to damage, so that the effect is not ideal in practical use.

(3) The settlement method based on RFID identification comprises the following steps:

embedding an RFID chip in a dinner plate in advance, associating the dish price with a corresponding chip, and reading the content of the chip at a charging terminal to obtain the corresponding dish price. The method has the disadvantages that special tableware is required to be used, the settlement system is high in cost and complex in deployment, and the embedded chip is easy to damage.

(4) The settlement method based on image recognition comprises the following steps:

in the existing settlement method based on image recognition, the pixel characteristics of dishes or the color, shape and pattern characteristics of a dinner plate are associated with the price of the dishes, and the corresponding price of the dishes is obtained at a charging terminal by the image recognition method. Wherein: the dish identification model is complex, the identification rate is low, errors are easy to occur, and the complex model needs to be updated again when a new dish is pushed out; the settlement system based on the dinner plate color and shape recognition has single recognition characteristic, is easily influenced by light, angles and dinner plate shielding factors, and has poor robustness in the face of poor scene recognition environment; for the method for identifying and settling accounts by printing special patterns on the dinner plate, special tableware is required to be used, and new tableware added in the later period is not universal.

Disclosure of Invention

Aiming at the defects that the existing settlement method based on image recognition is complex in model, low in recognition rate, poor in robustness in certain use scenes and high in actual use cost, the invention provides an automatic restaurant settlement method and system based on deep learning image recognition.

In order to achieve the purpose, the technical scheme adopted by the invention is as follows:

a restaurant automatic settlement method based on deep learning image recognition is characterized by comprising the following steps:

(1) a monocular camera is used for collecting a complete image of the dinner plate identification area;

(2) inputting the acquired complete image into an ENet Object Detection model system, firstly extracting basic features through a backbone network ENet, filtering through an FFWA to screen out high-quality features convenient for positioning and identification, and then outputting a confidence level (conf) and a preliminary positioning frame and a category label;

(3) carrying out post-processing by a self-adaptive NMS algorithm, and screening out a positioning frame with lower credibility (conf);

(4) outputting the final positioning frame and the category label data;

(5) transmitting the final positioning frame and the category label data to a pricing program;

(6) the pricing program converts the final positioning frame and the category label data into the type and the number of the dishes, and the total price of the dishes is calculated and displayed according to the dish price which is input in advance;

(7) and (4) the client completes payment through the payment system and returns to the step (1).

The generation method and the function of the positioning frame and the category label are respectively as follows: after the dinner plate identification area image is input, the algorithm model identifies the dinner plate from the whole image and marks the dinner plate by using the block diagram, so that the quantity of the dinner plate is conveniently recorded, and a 'positioning box' is generated; at the same time, the algorithm identifies different types of plates and labels the plates to facilitate distinguishing between the different types, which generates a "class label".

The positioning frame and the category label data belong to characteristic data, the characteristic data correspond to specific dish types and quantities, and when the specific positioning frame and the category label data are generated, the corresponding dish types and quantity data are generated.

A system for implementing the method, comprising:

(1) the monocular camera is used for collecting a complete image of the dinner plate identification area;

(2) the image processing module is used for processing the complete image acquired by the monocular camera and outputting a positioning frame and category label data;

(3) the pricing program module is used for converting the final positioning frame and the category label data into the category and the number of the dishes, and calculating and displaying the total price of the dishes according to the dish price input in advance;

(4) and the payment system is used for completing payment by the client according to the total price of the dishes provided by the pricing program module.

The image processing module is an ENet Object Detection model.

The ENet Object Detection model comprises:

(a) a backbone network ENet module for extracting the basic features of the complete image;

(b) the FFWA module is used for filtering and screening out high-quality features convenient to locate and identify;

the complete image is processed by an ENet module and an FFWA module to output credibility (conf) and preliminary positioning frames and category labels;

(c) and the self-adaptive NMS algorithm module is used for carrying out post-processing on the data output by the ENet module and the FFWA module, screening out the positioning frame with lower reliability (conf), and outputting the final positioning frame and the category label data.

The image processing module, the pricing program module and the payment system are installed in a computer system to operate.

The invention has the beneficial effects that: by adopting the visual technology settlement system based on deep learning, the advantages of low cost, flexible deployment, strong robustness and the like of using the visual technology compared with other settlement modes are achieved, meanwhile, the characteristics of expressive force and generalization force of the dinner plate generation are better achieved by utilizing the specific deep learning technology, the robustness is better in the scene with poorer recognition environment, the dinner plate recognition accuracy is higher, and the recognizable dinner plate types are more. In addition, the recall rate of the detection of the shielding object is improved through a specific self-adaptive NMS algorithm, and the problem of shielding of the dinner plate can be effectively solved. Moreover, specific loss functions are designed aiming at the algorithm system of the invention, and the model can be trained to solve the problems of difficult cases, shielding among objects and the like.

Drawings

FIG. 1 is a flow chart showing the operation of the restaurant settlement system of the present invention

FIG. 2 is a flowchart illustrating the operation of the ENet Object Detection algorithm model in the restaurant settlement system of the present invention;

FIG. 3 is a schematic explanatory view of Double Shuffle (Double Channel Shuffle);

FIG. 4 is a schematic illustration of the Channel Attention mechanism;

FIG. 5 is a schematic illustration of FFWA (Fast FPN with attachment).

Detailed Description

Referring to fig. 1 to 5, the invention relates to a restaurant automatic settlement method based on deep learning image recognition, and the basic idea is as follows: a monocular camera is adopted to obtain a dinner plate image, the dinner plate image is input into an ENet Object Detection algorithm model (system), after the ENet Object Detection model is operated after the trained (the system is used for carrying out repeated training for identifying the dinner plate and is a general training process for deep learning), data of a 'positioning frame' and a 'category label' of the dinner plate are output, a pricing program processes the data to obtain the quantity and the category of the dinner plate, the total price of dishes is calculated and displayed according to the dish price input in advance, and the payment of a customer is facilitated. The positioning frame and the category label are respectively as follows: after the dinner plate identification area image is input, the algorithm model can identify dinner plates from the whole image and mark the dinner plates by using a block diagram, so that the quantity of the dinner plates is conveniently recorded, and a 'positioning box' is generated; at the same time, the algorithm identifies different types of plates and labels the plates to facilitate distinguishing between the different types, which generates a "class label".

The specific method of the invention comprises the following steps:

(1) a monocular camera is used for collecting a complete image of the dinner plate identification area; the dinner plate identification area refers to an area which can be shot by the monocular camera, and a complete dinner plate image can be shot in the area.

(2) Inputting the acquired complete image into an ENet Object Detection model system, firstly extracting basic features through a backbone network ENet, then filtering through FFWA to screen out high-quality features convenient for positioning and identification, and then outputting credibility (conf) and preliminary positioning frames and category labels.

(3) And (4) carrying out post-processing through a self-adaptive NMS algorithm, and screening out the positioning frames with lower credibility (conf). The specific screening method is to run the self-adaptive NMS algorithm, and the self-adaptive NMS algorithm selected by the invention is the soft-NMS algorithm, which is an algorithm specially used for post-processing.

(4) And outputting the final positioning frame and the category label data.

(5) And transmitting the final positioning frame and the category label data to a pricing program.

(6) And the pricing program converts the final positioning frame and the category label data into the type and the number of the dishes, and calculates and displays the total price of the dishes according to the dish price recorded in advance.

The database of the system stores the data of the positioning frame and the category label data corresponding to the specific dish type and quantity, and when the system generates the specific positioning frame and the category label data, the corresponding dish type and quantity data can be generated.

The independently designed EfficientNet backbone network (namely the ENet backbone network in the figure) algorithm model reduces the parameter quantity and improves the operation speed through the effective combination of the DepthWise Conv and the Group Conv; in order to prevent the expression force from weakening, the probability of generating high-quality characteristics is improved through twice Shuffle, and then the advantage characteristics are enhanced and the disadvantage characteristics are weakened through a Channel Attention mechanism.

Two Shuffle (Double Channel Shuffle):

as shown in fig. 3, it is assumed that 8 convolution channels are provided, and then divided into 4 groups (two adjacent columns are one group), each group is respectively convolved, as shown in the second row, so that the memory access rate can be reduced, the operation is improved, but the possibility of generating high-quality features is reduced, therefore, before convolution, the arrangement order of the 8 channels is disturbed, so that more combinations are generated, and the possibility of generating high-quality features is also improved; meanwhile, after convolution is finished, the sequence of 8 channels is disturbed, and then the 8 channels are reweighed by combining Channel Attention to serve as final convolution characteristic output.

Channel Attention mechanism:

referring to fig. 4, a group of channels are evaluated by a Sigmoid function, and the obtained value can be regarded as the weight of each channel, and then multiplied by each channel by using the weight, which is equivalent to weighted average, thereby increasing the weight of the channels with good quality.

The two-way Attention mechanism is an operation of the Channel Attention mechanism in two directions, as shown in fig. 2 and 5, and the following two paragraphs of text also have specific explanations for the two-way Attention mechanism.

Referring to fig. 5, a Feature Pyramid Network (FPN) can improve the performance of small object detection, and we design an FFWA (Fast FPN with attachment) on this basis, i.e., using semantic information of Up Layer to supervise the screening of appearance features of Down Layer, and using Channel attachment to screen each Layer of features, thereby extracting high-quality features; however, the FPN increases the model operation amount, and the parameter amount is reduced and the speed advantage is ensured by utilizing the EfficientNet backbone network. Each block in the figure refers to a feature of an intermediate process.

FFWA(Fast FPN with Attention)：

The upward operation process, namely normal convolution operation; and in the downward operation process, the convolution operation is firstly carried out on the layer of convolution to generate a channel with the same size as the lower layer, then the Sigmoid is used for evaluating the channel to be used as the weight of the lower layer channel, and finally the lower layer convolution is multiplied by the weight to obtain the output of the layer.

The lower layer convolution has more bottom layer characteristics, the upper layer convolution has more semantic characteristics, so that the detection of the significant object can be effectively improved, and the characteristics of the bottom layer are also favorable for detecting more small objects, thereby improving the recall rate in the whole detection process and avoiding missing detection.

Because the food in the dinner plate is changed all the time and great difficulty is added to the model identification dinner plate, the self-adaptive Normalization is designed to solve the difficulty brought to the dinner plate identification by the appearance change, the Batchnormalization and the Instance Normalization are mainly combined and then multiplied by the weight respectively, and then the two results are added.

Adaptive Normalization:

BN (batch normalization): the content is invariable, which is beneficial to the judgment of the dinner plate type; IN (example normalization): the appearance is invariable, and the appearance migration under the condition of changeable appearance characteristics such as color, shape, size and the like is facilitated. Aiming at the characteristics that food is variable and the dinner plate is not changed in the dinner plate identification process, the combination of the two technologies becomes feasible. The combination method comprises the following steps: out is alpha BN + beta IN. The two parameters of alpha and beta are automatically adjusted in size according to the data per se to obtain a final normalization result, so that the advantages of the two parameters are effectively combined.

Because certain shielding exists among dinner plates, the use of a common NMS algorithm can cause the deletion of a subsequent dinner plate preliminary positioning frame, so that the use of a self-adaptive NMS algorithm for post-processing, namely soft-NMS, adopts linear weighting on the basis of the original NMS algorithm function:

wherein M is the positioning frame with the largest current score, Nt is the inhibition threshold, and Si is more than.

Thanks to soft-NMS, candidate frames which do not meet the conditions are reserved and given smaller scores, so that recall rate of detecting the shielding objects is improved, and the problem of dinner plate shielding can be effectively solved.

In the process of model training, the effect of identifying partial dinner plates is poor, a loss function is improved, and the model can better learn the characteristics of the difficult cases:

Y＝-(1-p)*Log(X)

wherein: x is the model output, p is the probability of dinner plate correspondence, and Y is the loss function

The classical loss function does not have (1-p) this factor, and the factor we add can increase the loss of difficult cases, thus giving the model more attention to these difficult cases.

In the automatic settlement method, after the monocular camera is used for collecting the complete image of the dinner plate required in the step (1), simple preprocessing (filtering, noise reduction, white balance, distortion processing, radiation change and the like) can be carried out on the image so as to improve the image identification accuracy.

The invention can acquire the dinner plate image by the monocular camera, simultaneously acquire the face image of the user by other cameras, and compare the acquired face image with the pre-stored user image to confirm the account information of the current user.

Claims

1. A restaurant automatic settlement method based on deep learning image recognition is characterized by comprising the following steps:

(2) inputting the acquired complete image into an ENet Object Detection model system, firstly extracting basic features through a backbone network ENet, filtering and screening high-quality features convenient for positioning and identification through an FFWA (fringe field effect transistor), and then outputting credibility and a preliminary positioning frame and category label, wherein the FFWA is a fast Object Detection algorithm based on a feature pyramid network and a bidirectional attention mechanism, the FFWA monitors the screening of the appearance features of a lower layer by utilizing semantic information of an upper layer, and screens the features of each layer by utilizing the bidirectional attention mechanism to extract the high-quality features;

(3) carrying out post-processing through a self-adaptive NMS algorithm, and screening out a positioning frame with lower reliability;

(4) outputting the final positioning frame and the category label data;

2. The method for automatically settling a restaurant based on deep learning image recognition as claimed in claim 1, wherein the generating method and the role of the "positioning frame" and the "category label" are respectively: after the dinner plate identification area image is input, the ENETObject Detection model system identifies the dinner plate from the whole image and marks the dinner plate by using a block diagram, so that the number of the dinner plates is conveniently recorded, and a 'positioning frame' is generated; at the same time, the ENet Object Detection model system identifies different types of dinner plates and applies labels to the dinner plates to facilitate distinguishing between different types, which generates a "class label".

3. The method as claimed in claim 1, wherein the positioning frame and category label data belong to a kind of feature data, the feature data are associated with specific dish type and quantity, and when the specific positioning frame and category label data are generated, the corresponding dish type and quantity data are generated.

4. An automatic restaurant settlement system based on deep learning image recognition, comprising:

5. The deep learning image recognition-based restaurant automatic settlement system according to claim 4, wherein the image processing module is an ENet Object Detection model.

6. The deep learning image recognition-based restaurant automatic settlement system of claim 5, wherein the ENet Object Detection model comprises:

the complete image is processed by an ENet module and an FFWA module and then a credibility, a primary positioning frame and a primary category label are output;

(c) and the self-adaptive NMS algorithm module is used for carrying out post-processing on the data output by the ENet module and the FFWA module, screening out the positioning frame with lower reliability and outputting the final positioning frame and the category label data.

7. The deep learning image recognition-based restaurant automatic settlement system of claim 4, wherein the image processing module, the pricing program module and the payment system are installed in a computer system for operation.