CN109508974B - A shopping checkout system and method based on feature fusion - Google Patents

Abstract

The invention discloses a shopping checkout system based on feature fusion, which comprises: the commodity identification checkout terminal comprises a processor, a commodity camera, a binocular camera, a pressure sensor array module, a storage table, a communication module, a loudspeaker, a sound acquisition module and a touch display screen, and the cloud server comprises a commodity database, a transaction record management module, a checkout module, a commodity identification module, a face recognition module and a voiceprint identification module; the surface of the object placing table is stuck with a pressure sensor array module. The customer does not need to additionally take the bulk commodity to a weighing position for weighing, but directly takes the bulk commodity to a commodity identification and checkout system for checkout, so that the settlement mode of the bulk packaged commodity is unified. And this scheme can discern all commodity simultaneously, need not to place commodity in proper order, and the customer only need put all commodity on the mesa simultaneously, and no matter this commodity is packing commodity or bulk commodity, homoenergetic unification discernment.

Description

A shopping checkout system and method based on feature fusion

technical field

The invention relates to the technical field of shopping checkout, in particular to a shopping checkout system and method based on feature fusion.

Background technique

The current mainstream commodity settlement method is to identify commodities by scanning barcodes one by one. Scanning one by one takes a lot of time, and manual scanning requires a lot of manpower expenditure. In addition, the self-service QR code scanning does not support the settlement of bulk commodities, resulting in the settlement of bulk commodities relying on the weigher, and the checkout of commodities depends on the cashier. This has the disadvantages of separate settlement of bulk packaged goods, large expenditure of manpower and material resources in supermarkets, low economic benefits of supermarkets, and excessive consumption of customers' time. However, the recently emerging self-service cash registers and self-service weighing platforms only relieve the pressure on cashiers and weighers when there is a large flow of people, and do not significantly improve the shopping experience of customers. Therefore, there is an urgent need in the industry for a system or method that simultaneously identifies all commodities, uniformly settles bulk and packaged commodities, and solves the problem of separation of settlement methods for bulk commodities and packaged commodities.

Contents of the invention

The object of the present invention is to provide a shopping checkout system based on feature fusion in order to overcome the shortcomings of the above prior art.

Another object of the present invention is to provide a shopping checkout method based on feature fusion in order to overcome the shortcomings of the above prior art.

The purpose of the present invention is achieved through the following technical solutions:

A shopping checkout system based on feature fusion, including: a product recognition checkout terminal and a cloud server, the product recognition checkout terminal includes a processor, a product camera, a binocular camera, a pressure sensor array module, a storage table, a communication module, a speaker, Sound collection module and touch display screen, described cloud server includes commodity database, transaction record management module, checkout module, commodity recognition module, face recognition module and voiceprint recognition module; Described commodity camera, binocular camera, pressure sensor array Module, loudspeaker, sound acquisition module, touch display screen and one end of the processor are connected, and the other end of the processor is connected through the communication module and commodity database, transaction record management module, checkout module, commodity recognition module, face recognition module, sound The fingerprint recognition modules are all connected; the surface of the storage table is pasted with a pressure sensor array module.

Preferably, the left side of the storage table is a box body, a processor and a communication module are arranged inside the box body, a commodity camera is arranged on the lower surface of the corner protrusion of the box body, and a touch display is arranged on the inclined surface of the front side of the box body A small round hole is provided under the touch display screen and on the inclined surface, a speaker and a sound collection module are arranged inside the small round hole, and a binocular camera is arranged above the touch display screen and on the inclined surface.

Preferably, the pressure sensor array module includes: a pressure sensor array unit and an analog-to-digital conversion circuit; the pressure sensor array unit is arranged on the surface of the storage table, and one end of the pressure sensor array unit is connected to the analog-to-digital conversion circuit, so The other end of the analog-to-digital conversion circuit is connected to the processor.

Preferably, the communication module is any one of 4G IoT communication module, WIFI communication module and Ethernet communication module, and the processor is any one of microcomputer, workstation or embedded control board.

Preferably, the checkout module includes a two-dimensional code payment unit and a biological payment unit, the face recognition module includes a face identity identification unit and a blink detection unit, and the product identification module includes a product positioning unit and a product recognition unit, so The voiceprint recognition module includes a voice preprocessing unit, a voice feature extraction unit and a voice classification unit connected in sequence.

Another object of the present invention is achieved through the following technical solutions:

A shopping checkout method based on feature fusion, comprising:

S1, the binocular camera captures the face image of the customer, and the sound collection module collects the payment keywords entered by the customer; the touch screen receives the electronic payment account information entered by the customer, and completes the binding of the customer and the electronic payment account.

S2, the customer places the product on the shelf, and the cloud server identifies the product, and the product includes bulk products;

S3, when the customer chooses biometric payment, the customer points his face at the binocular camera, blinks to confirm the living body, says "confirm payment" to confirm the transaction, and the bound electronic payment account or virtual payment account is automatically deducted;

S4, the customer takes away all the commodities on the shelf.

Preferably, step S2 includes:

The pressure sensor array module obtains the weight distribution information of the goods on the storage table, and constructs a weight distribution map;

The product camera takes photos of all the products on the shelf to obtain product pictures;

The processor uploads the product picture and weight distribution map to the cloud server;

The product recognition module of the cloud server locates the position of the product on the product picture, identifies the type, and obtains the weight w_k of the product on the weight distribution map according to the weight distribution map;

After obtaining the unit price of commodity a_k from the commodity database, combine the weight w_k to calculate the price of the commodity a_k;

Use k to traverse all the values from 1 to n to identify all the commodities on the shelf;

Preferably, the product identification module of the cloud server obtains the weight w_k of the product on the weight distribution map according to the position of the product on the product picture, including: the product identification module of the cloud server locates the position of the product on the product picture Finally, obtain the sub-area r_k corresponding to the commodity a_k in the gravity distribution map; affine transform the sub-region r_k on the gravity distribution map to the sub-region s_k of the product image, and obtain the corresponding position s_k in the product image; the cloud server according to the weight distribution map , Integrate the gravity in the sub-area r_k to obtain the weight w_k of the commodity a_k; the cloud server integrates the gravity in the sub-area r_k according to the weight distribution map to obtain the weight w_k of the commodity a_k, including: performing the pressure sensor array unit in advance Pressure calibration to obtain a calibration curve; for the value of any point p_j in the sub-region r_k in the gravity distribution diagram, use the calibration curve to calibrate the value to obtain the real pressure value rw_j; where p_j is any point in the region r_k, j=1...m; for all j from 1 to m, sum rw_j to obtain the weight w_k on the sub-region r_k; the product identification module of the cloud server locates the position of the product on the product picture, including: cloud server The product recognition module of the product preprocesses the weight distribution map to obtain the position of the product on the product picture; performs post-processing on the product position obtained after the preprocessing to obtain the weight distribution map sub-area and the picture sub-area; the product on the cloud server The identification module identifies the type of commodity on the commodity image, including: inputting the subregion of the weight distribution diagram into the convolution layer, extracting the feature vector of the subregion of the weight distribution diagram; inputting the subregion of the image into the feature extraction layer, extracting the subregion of the image The feature vector; wherein said convolutional layer and feature extraction layer are deep convolutional neural network structures.

Preferably, the preprocessing steps are: binarize the weight distribution map with the background weight as the threshold to obtain a binarized map; perform template matching on the binarized map to determine the rough position of the commodity; Expand the range of the location area to obtain the location of the product, so that the product image completely falls within the positioning range;

The post-processing steps are: cutting the weight distribution map according to the position of the commodity to obtain the area after the clipping of the weight distribution map; according to the affine transformation relationship between the weight distribution map and the picture, performing affine transformation on the position of the commodity to obtain the weight distribution map The corresponding position of the product position in the product picture; crop the product picture according to the corresponding position in the product picture to obtain the cropped area of the picture; perform median filtering on the cropped area of the picture; After the image is cropped, the area is stretched and transformed to the input size matched by the convolutional neural network, and the weight distribution map sub-area and the image sub-area are obtained.

Preferably, the face recognition module includes a face identification unit and a blink detection unit; the face identification unit is used to combine traditional visible light three-channel images with near-infrared single-channel images to form four-channel images, The four-channel image is used as the input of the convolutional neural network CNN, and is sent to the convolutional neural network CNN for face recognition and classification to obtain the identity information of the customer's face; the blink detection unit is used to extract the feature points of the eyes , sending the feature vector describing the feature point into the machine learning classifier for classification training to obtain the recognition model of blink detection; the voiceprint recognition module is used to pre-emphasize, frame and window the collected voice signal ; Carry out endpoint detection to identify the start moment, transition stage, noise segment and end moment of the speech signal, wherein the endpoint detection algorithm is to use the double threshold endpoint detection method based on short-term energy and short-term zero-crossing rate to calculate each frame of speech The Mel cepstral coefficients and Gammatone frequency cepstral coefficients of the signal are combined to form speech fusion features.

Compared with the prior art, the present invention has the following advantages:

In this solution, customers do not need to take the bulk goods to the weighing place for weighing, but directly take them to the product identification checkout system for checkout, which unifies the settlement method of bulk packaged goods. And this solution can identify all commodities at the same time, without placing the commodities sequentially, customers only need to put all the commodities on the table at the same time, and regardless of whether the commodities are packaged commodities or bulk commodities, they can be uniformly identified. Using the feature fusion method combining weight distribution map and pictures, it can better locate the position of the product and extract more complete product features, so it has high product recognition accuracy. Replacing the traditional barcode recognition with visual recognition: the process of repeatedly looking for the barcode of the product is eliminated, and the recognition process is faster. Customer self-checkout for goods: eliminating the need to hire cashiers, saving supermarket operating costs.

Description of drawings

Fig. 1 is a structural block diagram of the shopping checkout system based on feature fusion of the present invention.

Fig. 2 is a structural diagram of the commodity identification checkout terminal of the present invention.

Fig. 3 is a flow chart of the shopping checkout method based on feature fusion of the present invention.

Fig. 4 is a structure diagram of the commodity identification module of the present invention.

Fig. 5 is a structure diagram of the face recognition module of the present invention.

Fig. 6 is a structure diagram of the voiceprint recognition module of the present invention.

Fig. 7 is a residual module diagram of the present invention.

Among them, 1: storage table; 2: pressure sensor array unit; 3: cabinet; 4: commodity camera; 5: small round hole; 6: touch display screen; 7: binocular camera.

Detailed ways

The present invention will be further described below in conjunction with drawings and embodiments.

Referring to Figures 1-2, a shopping checkout system based on feature fusion includes: a product recognition checkout terminal and a cloud server, and the product recognition checkout terminal includes a processor, a product camera 4, a binocular camera 7, a pressure sensor array module, Shelter 1, communication module, loudspeaker, sound collection module and touch display screen, said cloud server includes commodity database, transaction record management module, checkout module, commodity identification module, face recognition module and voiceprint recognition module; Camera 4, binocular camera 7, pressure sensor array module, loudspeaker, sound collection module, touch display and one end of the processor are connected, and the other end of the processor is connected through the communication module and commodity database, transaction record management module, checkout module , product recognition module, face recognition module, and voiceprint recognition module are all connected; the surface of the storage table 1 is pasted with a pressure sensor array module.

In this embodiment, the commodity camera 4 is used to collect image information of commodities. The binocular camera 7 is used to collect customer face images and payment two-dimensional code images. The binocular camera 7 is a binocular camera 7 that passes through different light wave frequency bands. The loudspeaker is used to play voice prompts of commodity identification information and payment information. The sound collection module includes a microphone and a corresponding driving circuit for collecting voiceprint information. The touch display screen is a color display screen with capacitive or resistive touch, which is used to display a list of identified commodities, including commodity name, weight, quantity, unit price and total price information, and display commodity payment information, including total price, payment QR codes, payment options to enable human-computer interaction with customers. The commodity database stores the name, weight, unit price and price information of each commodity. The transaction record management module is used for recording, checking and managing commodity transaction records. The voiceprint recognition module uses the voiceprint information collected by the voice collection module to realize speaker identification through the voiceprint recognition algorithm. Commodity identification module uses the weight distribution information obtained by the pressure sensor array module, combined with the product image obtained by the product camera 4, to identify the product through product positioning and product recognition algorithms.

More specifically, the range of said processors covers microcomputers, workstations or embedded control motherboards. The processor is used to coordinate and process the work and operation of other sub-modules in the commodity identification checkout system, and is used for preliminary processing of collected data. Commodity camera 4: the commodity camera 4 is a color high-definition camera. The commodity camera 4 is used to collect the image information of the commodity, and then the processor uploads it to the cloud server through the communication module for identification. Binocular camera 7: The binocular camera 7 is a binocular camera 7 with different light wave frequency bands, one of which is a visible light camera and the other is a near-infrared light camera. The binocular camera 7 is used to collect the customer's face image information, which is uploaded to the cloud server by the processor through the communication module for identification, and is also used to collect the two-dimensional code image for payment. Pressure sensor array module: the pressure sensor array module is composed of a pressure sensor array unit and an analog-to-digital conversion circuit. The pressure sensor array unit is used to collect the weight distribution information of the product. The weight distribution information is processed by the processor to obtain the weight distribution map. At the same time, the processor uploads the obtained weight distribution map to the cloud server through the communication module for auxiliary product identification. Storage table 1: The storage table 1 is a solid-color flat plate with a pressure sensor array unit attached to the surface. The storage stand 1 is used to place the identified commodities and provide a uniform background with uniform color for photographing commodity images. Communication module: the communication module includes a 4G Internet of Things communication module, a WIFI communication module and an Ethernet communication module. The communication module is used to transmit images and other data information to realize communication with the cloud server. Speaker: The speaker is used to play voice prompts for product identification information and payment information. Sound collection module: the sound collection module includes a microphone and a corresponding driving circuit. The sound collection module is used to collect voiceprint information. Touch display screen: the touch display screen is a color display screen with capacitive or resistive touch. The touch display screen is used to display the list of identified commodities, including commodity name, weight, quantity, unit price and total price information, and is used to display commodity payment information, including total price, payment QR code, payment options, to realize personal communication with customers machine interaction. Commodity database: the commodity database includes the name, weight, unit price and price information of each commodity. The commodity database is used to store the name, weight, unit price and price of the commodity. Transaction record management module: The transaction record management module is used to record, view and manage commodity transaction records. Checkout module: The checkout module includes two-dimensional code payment, biological payment and other mainstream payment methods. The checkout module is used to combine the product recognition results of the product recognition module, the customer identity information identified by the face recognition module and the voiceprint recognition module, or the account information of the payment QR code, to call the corresponding payment interface, from the customer's electronic payment account or Automatic debit checkout in the virtual payment account. Product identification module: the basic process of the product identification module is product positioning and product identification. The product recognition module is used to locate the product position and identify the product type from the product image uploaded to the cloud server. The face recognition module is composed of a face identification unit and a blink detection unit. Wherein, the face identification module is used for identifying the face image uploaded to the cloud server, and the blink detection unit is used for identifying the living body of the face image uploaded to the cloud server. Voiceprint recognition unit: the voiceprint recognition module includes a voice preprocessing unit, a voice feature extraction unit and a voice classification unit connected in sequence. The voice preprocessing unit is used to preprocess the collected voice signal, the voice feature extraction unit is used to extract the preprocessed voice features, and the voice classification unit is used to classify the extracted voice features.

In this embodiment, referring to Fig. 2, the left side of the storage table 1 is a box body 3, a processor and a communication module are arranged inside the box body 3, and a commodity camera 4 is arranged on the lower surface of the sharp corner protrusion of the box body 3 , the inclined surface of the front of the box body 3 is provided with a touch display 6, the bottom of the touch display 6 and on the inclined surface is provided with a small round hole, the inside of the small round hole is provided with a speaker and a sound collection module, and the top of the touch display 6 And the binocular camera 7 is arranged on the inclined surface.

In this embodiment, the pressure sensor array module includes: a pressure sensor array unit 2 and an analog-to-digital conversion circuit; One end of the circuit is connected, and the other end of the analog-to-digital conversion circuit is connected to the processor. The pressure sensor array module is used for collecting weight distribution information of commodities. The storage table 1 is a solid-color plane plate with a pressure sensor array unit 2 attached on its surface.

In this embodiment, the communication module is any one of a 4G Internet of Things communication module, a WIFI communication module and an Ethernet communication module, and the processor is any one of a microcomputer, a workstation or an embedded control board.

In this embodiment, the checkout module includes a two-dimensional code payment unit and a biological payment unit, the face recognition module includes a face identity identification unit and a blink detection unit, and the commodity identification module includes a commodity positioning unit and a commodity recognition unit , the voiceprint recognition module includes a voice preprocessing unit, a voice feature extraction unit and a voice classification unit connected in sequence.

Referring to Fig. 3, the feature fusion-based shopping checkout method applicable to the above-mentioned feature fusion-based shopping checkout system includes:

S1, the binocular camera 7 captures the face image of the customer, and the sound collection module collects the payment keywords input by the customer; the touch screen 6 receives the electronic payment account information input by the customer, and completes the binding of the customer and the electronic payment account.

S2, the customer places the product on the storage table 1, and the cloud server identifies the product, and the product includes bulk products;

S3, when the customer chooses biometric payment, the customer points his or her face at the binocular camera 7, blinks to confirm the living body, says "confirm payment" to confirm the transaction, and the bound electronic payment account or virtual payment account is automatically debited;

S4, the customer takes away all the commodities on the shelf 1 .

From the customer's point of view, the operation of step S1 is as follows: the customer enters the customer registration interface by touching the touch screen 6, and operates according to the prompts on the screen. During face entry, the customer stands in front of the binocular camera 7 with his or her face facing the camera, and completes the entry of the face image by turning the head or the like. When the voiceprint is collected, the customer follows the screen prompts and repeats the payment keyword (such as the payment keyword "confirm payment") several times to complete the entry of the voiceprint information. The product recognition checkout terminal uploads the face image and voice information to the cloud server, calls the face recognition module and the voiceprint recognition module to complete the training of the face recognition module and the voiceprint recognition module. The customer completes the binding of the face and the electronic payment account by inputting personal electronic payment account information on the touch screen 6; or uses the existing electronic payment account (such as Alipay, WeChat, bank card, etc.) to recharge the virtual payment account . From the customer's point of view, the operation of step S2 is: the customer brings one or more commodities to the commodity identification checkout system. The commodities include bulk commodities and packaged commodities. The customer places all the products on the storage table 1 at the same time, and of course they can also place them on the storage table 1 in sequence. Note that the products cannot be stacked. The processor obtains the identified product information from the cloud server through the communication module, and displays the product information on the touch screen 6 . From the customer's point of view, the operation of step S3 is: when confirming the settlement, the customer selects biometric payment or two-dimensional code payment from the touch screen 6 . Biometric payment requires the customer's personal face and voiceprint information registered in advance to be used. When choosing biometric payment, the customer points his or her face at the binocular camera 7, blinks to confirm the living body, says "confirm payment" to confirm the transaction, and the bound electronic payment account or virtual payment account is automatically deducted. In this process, the commodity recognition checkout system will upload face video images and voice information to the cloud server in real time, call the face recognition module for face recognition and blink detection, and call the voiceprint recognition module for speaker recognition. And combined with the customer's bound electronic payment account or virtual payment account for automatic deduction. When choosing two-dimensional code payment, the customer aligns the payment two-dimensional code (such as Alipay, WeChat and other payment two-dimensional codes) with the binocular camera 7, and the electronic payment account will automatically debit the money. In this process, the commodity identification checkout system analyzes and recognizes the captured QR code pictures, and calls the official payment interface to automatically debit the electronic payment account. After the transaction deduction is successful, the transaction record management module of the cloud server will record the transaction information of the commodity.

In this embodiment, step S2 includes: the pressure sensor array module obtains the weight distribution information of the commodities on the storage table 1, and constructs a weight distribution map; the commodity camera 4 takes pictures of all the commodities on the storage platform 1 to obtain commodity pictures; Upload the product picture and weight distribution map to the cloud server; the product identification module of the cloud server locates the position of the product on the product picture, identifies the type, and obtains the weight w_k of the product on the weight distribution map according to the weight distribution map; from the product database After obtaining the unit price of the product a_k, combine the weight w_k to calculate the price of the product a_k; traverse all the values of k from 1 to n, and identify all the products on the shelf 1.

Wherein, referring to FIG. 4 , the product identification module of the cloud server obtains the weight w_k of the product on the weight distribution map according to the position of the product on the product picture by the product identification module of the cloud server. After positioning the position, obtain the sub-area r_k corresponding to the product a_k in the gravity distribution map; when there are multiple products on the shelf 1, record the number of products as n, and record the products as a_i (i=1...n) in order to distribute the gravity The sub-area r_k on the figure is affinely transformed into the sub-area s_k of the product picture, and the corresponding position s_k in the product picture is obtained; the cloud server integrates the gravity in the sub-area r_k according to the weight distribution map, and obtains the weight w_k of the product a_k; if it is For packaged goods, obtain the price of the product directly from the product database; if it is a bulk product, obtain the unit price of the product from the product database and combine the weight w_k to calculate the price of the product a_k. By traversing k through all the values from 1 to n, all the commodities on shelf 1 can be identified.

According to the weight distribution diagram, the cloud server integrates the gravity in the sub-region r_k to obtain the weight w_k of the commodity a_k including: performing pressure calibration on the pressure sensor array unit in advance to obtain a calibration curve; The value of any point p_j, use the calibration curve to calibrate the value to obtain the real pressure value rw_j; where p_j is any point in the area r_k, j=1...m; for all j from 1 to m, sum rw_j, get the weight w_k on the sub-region r_k;

The commodity recognition module of the cloud server locates the position of the commodity on the commodity picture, including: the commodity recognition module of the cloud server preprocesses the weight distribution map to obtain the position of the commodity on the commodity picture; The position is post-processed to obtain the sub-region of the weight distribution map and the sub-region of the picture; the steps of the preprocessing are: binarize the weight distribution map with the background weight as a threshold to obtain a binarized map; Template matching, determining the rough position of the product; expanding the range of the rough position area of the product to obtain the position of the product, so that the product image completely falls within the positioning range; the post-processing step is: according to the position of the product, the weight distribution map is Cutting, to obtain the cropped area of the weight distribution map; according to the affine transformation relationship between the weight distribution map and the picture, perform affine transformation on the product position, and obtain the corresponding position of the product position in the weight distribution map in the product picture; according to the affine transformation relationship in the product picture Crop the product image at the corresponding position to obtain the cropped area of the image; perform median filtering on the cropped area of the image; perform scaling transformation on the cropped area of the weight distribution map and the cropped area of the product image, and transform it to the one matched by the convolutional neural network Enter the size to get the weight distribution map sub-area and picture sub-area.

The product recognition module of the cloud server identifies the type of the product on the product picture, including: inputting the sub-region of the weight distribution graph into the convolution layer, extracting the feature vector of the sub-region of the weight distribution graph; inputting the sub-region of the image into the feature extraction layer , extracting feature vectors of image sub-regions; wherein the convolutional layer and feature extraction layer are deep convolutional neural network structures. The convolutional layer and feature extraction layer often use variants of existing classification deep convolutional neural networks. Common existing classification deep convolutional neural networks include VGG, ResNet, Inception, DenseNet, ZFNet, and AlexNet.

This convolutional layer is a variant of the VGG16-based classification deep convolutional neural network. Denote n convolution kernels of a*a size as conv_a_n, such as 64 convolution kernels of 3*3 size as conv_3_64. Copy the single-channel weight distribution map sub-region twice to form a three-channel image, and input it into the subsequent convolutional layer. The convolutional layer input is a 224*224 three-channel color image. With the help of the symbolic expression of the convolution kernel, the neural network structure of this VGG16 variant can be described as: conv_3_64, conv_3_64, maxpool, conv_3_128, conv_3_128, maxpool, conv_3_256, conv_3_256, conv_3_256, maxpool, conv_3_512, conv_3_512, conv_3_ 512, maxpool, conv_3_512, conv_3_512, conv_3_512, maxpool, FC_4096, FC_4096, FC_1000. Among them, maxpool is a pooling layer with a size of 2*2 and a step size of 2, FC_4096 represents a fully connected layer with 4096 neurons, and FC_1000 is a fully connected layer with 1000 neurons. The feature extraction layer of the last fully connected layer FC_1000 is a variant of the classification deep convolutional neural network based on ResNet50. We remember the convolutional structure shown in Figure 7 as a residual module. The residual module is sequentially composed of a 1*1 convolution kernel, b 3*3 convolution kernel and c 1*1 convolution kernel. In the structure shown in Figure 6, the selection of a, b and c The values are 64, 64, and 256, in that order. The residual module directly bridges the input before the three layers to the output terminal and adds them, and then activates the relu activation function as the output of the residual module. Each convolution uses a relu activation function. We abbreviate the residual module with parameters a, b and c as the symbol bottleneck_a_b_c, for example, the residual module shown in Figure 7 is recorded as bottleneck_64_64_256. Then one embodiment structure of the feature extraction layer is: conv_7_64, maxpool, 3 bottleneck_64_64_256, 4 bottleneck_128_128_512, 6 bottleneck_256_256_1024, 3 bottleneck_512_512_2048, avgpool, FC_1000. Among them, conv_7_64 represents 64 convolution kernels of 7*7, maxpool represents the maximum pooling layer with a size of 3*3 and a step size of 2, avgpool represents the average pooling layer, and FC_1000 is a fully connected layer with 1000 neurons. Finally, the output of the fully connected layer FC_1000 is used as the output of the feature extraction layer, and the output is a 1000-dimensional feature vector.

Combining the weight feature and image feature to obtain a synthetic feature, the dimension of the synthetic feature is 2000.

The synthetic features are input into the classification deep neural network, which consists of a fully connected layer and a softmax classification layer, and the product with the highest probability of the softmax output layer is used as the product recognition result.

The classification deep neural network is: the structure of the neural network is FC_2000 and softmax in sequence. Among them, FC_2000 represents the fully connected layer with 2000 neurons, and softmax is the classification output layer with the number of output nodes equal to the number of commodity types. The activation function uses the relu activation function.

In the present embodiment, referring to Fig. 5, the face recognition module includes a face identification unit and a blink detection unit; the face identification unit adopts a convolutional neural network (CNN) for combining traditional visible light three-channel images with near-infrared The single-channel images are combined to form a four-channel image, and the four-channel image is used as the input of the convolutional neural network CNN, and then sent to the convolutional neural network CNN for face identification and classification to obtain the identity information of the customer's face; The blink detection unit is used to extract the feature points of the eyes, and the feature vector describing the feature points is sent to the machine learning classifier for classification training to obtain the recognition model of blink detection; the purpose of the blink detection unit is to fight against non-living objects, only blinking It is possible for the measured object to be identified as a living object. The blink detection algorithm can be selected as a real-time blink detection algorithm using eye feature extraction technology. The blink detection algorithm has certain robustness against external attacks. The blink detection algorithm can obtain the recognition model of blink detection by extracting the feature points of the eyes, and then sending the feature vectors describing the feature points into a machine learning classifier (such as a support vector machine (SVM)) for classification training.

Referring to Fig. 6, the voiceprint recognition module is used to pre-emphasize, frame and window the collected voice signal; perform endpoint detection to identify the start moment, transition phase, noise segment and end moment of the voice signal, The endpoint detection algorithm adopts the double-threshold endpoint detection method based on short-term energy and short-term zero-crossing rate to calculate the Mel cepstral coefficient and Gammatone frequency cepstral coefficient of each frame of speech signal, and combine them to form speech fusion features.

The features are trained using a deep neural network whose output layer is a softmax classification layer. The one with the highest output probability of the softmax layer is taken as the recognition result.

Compared with the prior art, this solution has the following advantages:

(1) Putting down unified identification: Compared with the current method of scanning products one by one, with this system, customers only need to put all the products on the table at the same time. The system is able to identify all products at the same time without placing them sequentially, regardless of whether the product is packaged or loose.

(2) Unify the settlement method of bulk packaged goods: With this system, customers do not need to take bulk goods to the weighing place for weighing, but directly take them to the commodity identification checkout system for checkout.

(3) Replacing the traditional barcode recognition with visual recognition: the process of repeatedly looking for the barcode of the product is eliminated, and the recognition process is faster.

(4) Customer self-service checkout: no need to hire cashiers, saving supermarket operating costs.

(5) Higher product recognition accuracy: Compared with the traditional machine learning algorithm, the feature fusion method combining weight distribution map and picture can better locate the product position and extract more complete product features, so it has a higher High product recognition accuracy.

(6) Higher face recognition accuracy: Compared with traditional machine learning algorithms, the use of feature fusion and convolutional neural network can describe face features more completely, so it has higher face recognition accuracy .

(7) The access method of the cloud server reduces the cost of image recognition: the access method of the cloud server makes it unnecessary to write the visual recognition model inside the checkout system, which facilitates the modification of the visual recognition model and reduces the hardware overhead of the product recognition checkout system .

(8) The wired network deployment cost of the device is eliminated: 4G Internet of Things access makes the device no need for network connection deployment. But this system also supports WIFI and Ethernet communication interface.

(9) Reduce the possibility of customers checking out by mistake: the triple verification method of face recognition, blink detection and voiceprint recognition greatly reduces the possibility of customers making wrong checkouts due to misoperation. But compared with today's checkout method, it is more convenient and easy to operate.

(10) The cost of hiring manpower in the supermarket is saved: the supermarket does not need to hire weighers and cashiers, which saves labor expenses. If the supermarket originally employs 7 cashiers, 3 staff with titles, and the monthly income of the employees is 5,000, the introduction of this system is equivalent to bringing an annual profit of 600,000 yuan to the supermarket, and the economic benefits are considerable.

The specific implementation described above is a preferred embodiment of the present invention, and does not limit the present invention. Any other changes or other equivalent replacement methods that do not deviate from the technical solution of the present invention are included in the scope of protection of the present invention. within.

Claims (7)

1. A shopping checkout system based on feature fusion, characterized in that it includes: a commodity recognition checkout terminal and a cloud server, and the commodity recognition checkout terminal includes a processor, a commodity camera, a binocular camera, a pressure sensor array module, and a storage table , a communication module, a loudspeaker, a sound collection module and a touch display screen, wherein the cloud server includes a commodity database, a transaction record management module, a checkout module, a commodity recognition module, a face recognition module and a voiceprint recognition module; One end of the commodity camera, binocular camera, pressure sensor array module, loudspeaker, sound collection module, touch display and processor is connected, and the other end of the processor is connected through the communication module and commodity database, transaction record management module, checkout module, product recognition module, face recognition module, and voiceprint recognition module are all connected; the surface of the storage table is pasted with a pressure sensor array module; The processor is used for the preliminary processing of the collected data; Commodity cameras are used to collect image information of commodities; The binocular camera is a binocular camera with different light wave frequency bands, one is a visible light camera, and the other is a near-infrared light camera, which is used to collect customer face images and payment QR code images; The pressure sensor array module includes a pressure sensor array unit and an analog-to-digital conversion circuit, the pressure sensor array unit is arranged on the surface of the storage table, one end of the pressure sensor array unit is connected to the analog-to-digital conversion circuit, and the other end of the analog-to-digital conversion circuit is connected to the processor , used to collect the weight distribution information of the product and build a weight distribution map; The storage table is a solid-color flat board, which is used to place the identified products and provide a uniform background with uniform color for product image shooting; The communication module is used to transmit images and other data information to realize communication with the cloud server; the speaker is used to play voice prompts for commodity identification information and payment information; The sound collection module includes a microphone and a corresponding drive circuit for collecting voiceprint information; The touch display screen is a color display screen with capacitive or resistive touch, which is used to display the list of recognized goods and the payment information of goods; The commodity database is used to store the name, weight, unit price and price information of each commodity; The transaction record management module is used to record, view and manage commodity transaction records; The checkout module includes a two-dimensional code payment unit and a biological payment unit. The checkout module is used to combine the product identification result of the product identification module, the customer identity information identified by the face recognition module and the voiceprint recognition module or the account information for paying the two-dimensional code. Call the corresponding payment interface to automatically debit the payment from the customer's electronic payment account or virtual payment account; The product identification module includes a product positioning unit and a product identification unit, and the product identification module is used to locate the product position and identify the product type from the product image uploaded to the cloud server; The face recognition module includes a face identity identification unit and a blink detection unit. The face identity identification module is used to identify the face image uploaded to the cloud server, and the blink detection unit is used to identify the face image uploaded to the cloud server. identification of living organisms; The voiceprint recognition module includes a voice preprocessing unit, a voice feature extraction unit and a voice classification unit connected in sequence, the voice preprocessing unit is used to preprocess the collected voice signal, and the voice feature extraction unit is used to extract the preprocessed voice Features, the speech classification unit is used to classify the extracted speech features. 2. The shopping checkout system based on feature fusion according to claim 1, characterized in that, the left side of the storage table is a box body, and a processor and a communication module are arranged inside the box body, and the corner protrusion of the box body The lower surface is provided with a commodity camera, the inclined surface of the front of the box is provided with a touch display screen, and a small round hole is provided under the touch display screen and on the inclined surface, and a speaker and a sound collection module are arranged inside the small round hole, and the touch display A binocular camera is set above the screen and on an inclined surface. 3. The shopping checkout system based on feature fusion according to claim 1, wherein the communication module is any one of a 4G Internet of Things communication module, a WIFI communication module and an Ethernet communication module, and the processor It is any one of microcomputer, workstation or embedded control board. 4. A shopping checkout method based on feature fusion, characterized in that, comprising: S1, the binocular camera captures the face image of the customer, and the sound collection module collects the payment keywords entered by the customer; the touch screen receives the electronic payment account information entered by the customer, and completes the binding of the customer and the electronic payment account; S2, the customer places all the bulk commodities and packaged commodities on the storage table simultaneously or sequentially, and the cloud server identifies the commodities, including: The pressure sensor array module obtains the weight distribution information of the goods on the storage table, and constructs a weight distribution map; The product camera takes photos of all the products on the shelf to obtain product pictures; The processor uploads the product picture and weight distribution map to the cloud server; The product recognition module of the cloud server locates the position of the product on the product picture, identifies the type, and obtains the weight w_k of the product on the weight distribution map according to the weight distribution map; After obtaining the unit price of commodity a_k from the commodity database, combine the weight w_k to calculate the price of the commodity a_k; Use k to traverse all the values from 1 to n to identify all the commodities on the shelf; S3, when the customer chooses biometric payment, the customer points his face at the binocular camera, blinks to confirm the living body, says "confirm payment" to confirm the transaction, and the bound electronic payment account or virtual payment account is automatically deducted; During the bio-payment process, the product recognition checkout terminal uploads face video images and voice information to the cloud server, calls the face recognition module for face recognition and blink detection, and calls the voiceprint recognition module for speaker identification; S4, the customer takes away all the commodities on the shelf. 5. The shopping checkout method based on feature fusion according to claim 4, characterized in that, the commodity recognition module of the cloud server obtains the weight w_k of the commodity on the weight distribution diagram according to the weight distribution diagram to the position of the commodity on the commodity picture include: After the product identification module of the cloud server locates the position of the product on the product picture, it obtains the sub-region r_k corresponding to the product a_k in the gravity distribution map; Affine transform the sub-area r_k on the gravity distribution map to the sub-area s_k of the commodity image to obtain the corresponding position s_k in the commodity image; According to the weight distribution diagram, the cloud server integrates the gravity in the sub-area r_k to obtain the weight w_k of the product a_k; According to the weight distribution diagram, the cloud server integrates the gravity in the sub-region r_k, and obtains the weight w_k of the commodity a_k including: Perform pressure calibration on the pressure sensor array unit in advance to obtain the calibration curve; For the value of any point p_j in the sub-region r_k in the gravity distribution diagram, use the calibration curve to calibrate the value to obtain the real pressure value rw_j; where p_j is any point in the region r_k, j=1...m; For all j from 1 to m, sum rw_j to get the weight w_k on the sub-region r_k; The product recognition module of the cloud server locates the position of the product on the product picture, including: The product recognition module of the cloud server preprocesses the weight distribution map to obtain the position of the product on the product picture; Perform post-processing on the product position obtained after pre-processing to obtain the sub-area of the weight distribution map and the sub-area of the image; The product recognition module of the cloud server identifies the type of the product on the product picture, including: Input the sub-region of the weight distribution map into the convolution layer, and extract the feature vector of the sub-region of the weight distribution map; The picture sub-region is input into the feature extraction layer, and the feature vector of the picture sub-region is extracted; wherein the convolution layer and the feature extraction layer are deep convolutional neural network structures. 6. The shopping checkout method based on feature fusion according to claim 5, characterized in that, the preprocessing step is: Binarize the weight distribution map with the background weight as the threshold to obtain a binarized map; Perform template matching on the binary image to determine the rough position of the product; Expand the range of the rough location area of the product to obtain the product position, so that the product image completely falls within the positioning range; The steps of the post-processing are: Cut the weight distribution map according to the position of the product to obtain the cut area of the weight distribution map; According to the affine transformation relationship between the weight distribution diagram and the picture, affine transformation is performed on the commodity position, and the corresponding position of the commodity position in the weight distribution diagram in the commodity picture is obtained; Crop the product picture according to the corresponding position in the product picture to obtain the cropped area of the picture; Perform median filtering on the cropped area of the image; Scale and transform the cropped area of the weight distribution map and the cropped area of the product image to the input size matched by the convolutional neural network to obtain the sub-area of the weight distribution map and the sub-area of the picture. 7. the shopping settlement method based on feature fusion according to claim 4, is characterized in that, face recognition module comprises face identification unit and blink detection unit; The face identity identification unit is used to combine the traditional visible light three-channel image with the near-infrared single-channel image to form a four-channel image, and the four-channel image is used as the input of the convolutional neural network CNN and sent to the convolutional neural network. The network CNN performs face recognition and classification, and obtains the identity information of the customer's face; The blink detection unit is used to extract the feature points of the eyes, send the feature vector describing the feature points into the machine learning classifier for classification training, and obtain the recognition model of blink detection; The voiceprint recognition module is used to pre-emphasize, frame and window the collected voice signal; perform endpoint detection to identify the start moment, transition stage, noise segment and end moment of the voice signal, wherein the endpoint detection algorithm In order to adopt the dual-threshold endpoint detection method based on short-term energy and short-term zero-crossing rate, the Mel cepstral coefficient and Gammatone frequency cepstral coefficient of each frame of speech signal are calculated and combined to form speech fusion features.