JP2025082224A - Commodity purchase settlement system and commodity purchase settlement method - Google Patents

Abstract

[Problem] To provide a product purchase settlement system and a product purchase settlement method that store large and large volumes of data and manage information more securely, with the aim of automating, reducing manpower, and preventing crime in stores such as convenience stores, supermarkets, and drug stores. [Solution] The product purchase settlement system 0 uses cameras attached to the shopping basket or the cargo carrying the shopping basket, and cameras that photograph the product shelves, to determine whether a customer has purchased a product, and at the same time, uses cameras installed at the gates at the entrance and exit of the store to link the customer and the shopping basket and make the settlement, while at the same time monitoring whether the product the customer picked up from the product shelf was properly placed in the shopping basket. [Selected Figure] Figure 2

Description

The present invention relates to payment methods and crime prevention for purchasing goods in stores such as supermarkets, convenience stores, and drug stores, and relates to a system that uses image processing, measurement, and information processing technologies to realize unmanned stores and labor-saving stores.

In recent years, the decline in the productive age population has led to a labor shortage problem. To address this labor shortage, stores are rapidly moving toward automation or labor-saving. Examples of such systems are proposed in, for example, Patent Document 1 and Non-Patent Document 1.

For example, Patent Document 1 discloses that a customer registers a dedicated app on their mobile phone in advance, and when the customer enters a store, the store's entrance is identified by a reading sensor or the like, and the store uses a number of video cameras placed around the store to determine the customer's movements and behavior as well as whether or not products have been purchased through image processing. When the customer has finished purchasing products and is about to leave the store, they pass through a gate, which indicates that the product purchase is complete, the total amount of the purchased products is calculated, and payment for the product purchase is made using a credit card or prepaid card registered in the app.

Non-Patent Document 1 also discloses that, like Patent Document 1, multiple cameras are placed in the store to analyze customer behavior while determining whether a product has been purchased based on its correspondence with weight sensors attached to product shelves.

On the other hand, in stores that use cameras like those mentioned above to process images of customers entering the store and determine whether they have made a purchase, the image processing burden is heavy. This requires placing many cameras in the store and simultaneously using many computers to process the images, which requires a large initial investment. Also, as the number of customers in a store increases, it becomes difficult to track them with the cameras, so it becomes necessary to limit the number of customers who can enter the store at the same time, which results in customers having to wait to enter the store.

JP 2023-014207 A JP 2020-135210 A

"Touch To Go" unmanned AI payment store at Takanawa Gateway Station is operating steadily for one month since its opening!!" [Retrieved October 24, 2023], Internet <https://prtimes.jp/main/html/rd/p/000000001.000057612.html>

As an alternative to these methods, Patent Literature 2 discloses a system that uses a shopping basket. However, Patent Literature 2 requires that a camera attached to the shopping basket and a customer terminal on which an app has been downloaded must be manually linked before shopping, which makes shopping difficult.

In view of the above problems, the present invention aims to provide a new technology that allows customers to enter and exit a store and purchase products without feeling uncomfortable.

In order to solve the above-mentioned problems, the present invention provides a product purchase settlement system that allows customers to make payments by themselves in a store, the purchase settlement system comprising a customer photographing camera installed in the store, a purchased product bin in which a purchased product photographing camera that photographs products to be placed in the purchased product bin is installed, a list registration unit, a list update unit, a settlement unit, a product identification unit, and a database, the database stores the customer's payment information used for product payment for each customer, and if the purchased product bin is not linked to a customer identified based on an image acquired via the customer photographing camera, the list registration unit registers the purchased product bin identified based on an image acquired via the customer photographing camera and held by the identified customer, and A customer purchase list having the combination of the identified customers is registered, and/or, if the purchased item container is linked to a customer identified based on an image acquired via the customer photographing camera, a customer purchase list having the purchased item container linked to the identified customer and the combination of the identified customer is registered, the product identification unit identifies product information of the product placed in the purchased item container based on the image acquired via the purchased item photographing camera, the list update unit updates the purchase list of the customer purchase list linked to the purchased item container in which the purchased item photographing camera that photographed the product is installed based on the product information, and the payment unit makes payment for the product based on the payment information and the customer purchase list.

With this configuration, customers can shop simply by picking up the shopping container to put the items they want to buy, allowing them to purchase products smoothly and without any awkwardness.

In a preferred embodiment of the invention,
The settlement unit settles for merchandise based on the payment information linked to a customer identified based on an image acquired by a customer photography camera installed at an exit of the store, and based on the purchase list.

With this configuration, customers can pay for products based on their own registered payment information. This allows customers to pay for products without having to take the time to select the payment method they want to use when making a payment.

In a preferred embodiment of the present invention, the purchase container has an electronic circuit that provides power to the purchase item photographing camera, and a switch that turns the power OFF when the purchase containers are placed on top of each other and turns the power ON when a customer removes the purchase container.

This configuration allows the products to be placed in the purchased items bin to be identified only when the customer is shopping.

In a preferred embodiment of the present invention, the purchased goods container is provided with a plurality of purchased goods photographing cameras, and at least a first purchased goods photographing camera is provided with a second purchased goods photographing camera in a position opposite the first purchased goods photographing camera, and the viewing angles of the first purchased goods photographing camera and the second purchased goods photographing camera overlap with the viewing angle of the purchased goods photographing camera adjacent to the first purchased goods photographing camera or the second purchased goods photographing camera.

This configuration makes it possible to more reliably capture the characteristic surfaces of products placed in the purchased product container, improving the accuracy of product identification. This makes it possible to increase the accuracy of product identification without requiring a high-precision camera, and reduces the cost of installing the cameras.

In a preferred embodiment of the present invention, when a customer has multiple purchase containers in his/her hands, the list registration unit registers the customer purchase list having a combination of the multiple purchase containers and the identified customer, identified based on images of the multiple purchase containers captured via the customer photography camera, and the list update unit updates the purchase list of the customer purchase list linked to each purchase container based on product information of the products identified by the product identification unit and placed in each purchase container.

With this configuration, if a customer intends to buy many items, they will enter the store with multiple purchase containers. Even in such a case, it is possible to identify which customer is using each of the multiple purchase containers, and appropriate product payment can be made for each customer.

In a preferred embodiment of the present invention, the list registration unit identifies the customer holding the purchase item container as the payer based on an image captured by the customer photography camera, and the payment unit makes payment for the items based on the customer purchase list and the payment information of the customer identified as the payer.

With this type of configuration, customers who come to shop may enter the store in groups. In such cases, if payment for products is made for all customers in the group, calculations such as how to distribute the payment become complicated. On the other hand, by identifying the customer who holds the purchase item container as the payment maker and making payment for products based on that customer's payment information, it is possible to reduce calculation processing such as payment distribution and make payment for products smoothly.

In a preferred embodiment of the present invention, the product purchase and payment system further includes a gate control unit, and when a combination of an image of the customer and a purchased product container picked up by the customer is acquired, the gate control unit controls the entrance gate of the store.

This configuration allows the customer to enter the store when the appropriate payment for the product is possible. This avoids situations where the product in the shopping cart is not linked to the customer and payment cannot be made, ensuring that payment for the product can be made reliably.

In a preferred embodiment of the present invention, when a combination of a customer and the purchase container is registered in advance by the customer purchasing or renting the purchase container, the list registration unit registers the customer purchase list including the customer who enters the specified store based on the image acquired via the customer photography camera and the combination of the purchase container associated with the customer.

With this configuration, the customer can be identified and the purchased item container can be identified, so the customer purchase list can be registered more reliably and missed item payments can be prevented.

In a preferred embodiment of the present invention, the list registration unit registers the customer purchase list for each customer, and if the first customer belongs to a group, the list registration unit registers the customer purchase list of another second customer in the group by linking it to the first customer, and when the payment unit receives input related to payment from the first customer to which the customer purchase list of the other second customer is linked, the payment unit notifies the first customer as to whether or not to make payment for the product based on the customer purchase list of the other second customer.

This configuration allows customers who visit the store in groups to pay for products appropriately. In other words, it is possible to accommodate various types of payments, such as when a group wants to make a collective payment or when customers want to make payments for each shopping basket.

In a preferred embodiment of the present invention, the product purchase settlement system further includes a customer identification unit that identifies a customer based on an image acquired via the customer photography camera, determines whether the customer has registered customer-related information in advance, and, if the customer has not registered customer-related information in advance, further acquires an image of the customer via the customer photography camera and registers the image as information about the customer.

This configuration makes it possible to reliably capture images of customers who have not registered their customer information in advance. This helps prevent missed product payments.

In a preferred embodiment of the present invention, the product purchase and payment system further includes a customer position measuring camera that measures the position of the customer within the store, a display product photography camera that photographs the displayed products, and a customer identification unit, the product identification unit identifies the product that has been removed from the product shelf based on the image acquired via the display product photography camera, and the customer identification unit identifies customers who are near the product shelf from which the product has been removed and whose purchase list has not been updated based on the product information of the product.

This configuration makes it possible to identify customers who attempt to commit crimes such as shoplifting by removing products from shelves and not placing them in a purchased item container.

In a preferred embodiment of the present invention, the product purchase and payment system further includes a notification unit, and when a customer is identified who is near a shelf from which a product has been removed and whose purchase list has not been updated based on the product information of the product, the notification unit notifies a manager, employee, or security guard of the store that owns the purchased product container associated with the customer.

With this configuration, if a customer attempts to shoplift, a notification can be sent to the store staff, preventing the crime from occurring.

In a preferred embodiment of the present invention, the product purchase and payment system further includes a display product photography camera that photographs the products on display, and the product identification unit executes a process of identifying the product removed from the purchased product container by the customer based on the image acquired via the display product photography camera, and a process of identifying that the product is placed on the product shelf based on the image acquired via the display product photography camera within a predetermined time after the product is removed, whereby the list update unit updates the purchase list of the customer purchase list linked to the purchased product container based on the product information of the product.

With this configuration, it is possible to prevent a customer's purchase list from being updated unless an item that has been placed in the shopping basket is returned to the shelf. This means that even if a customer attempts to shoplift an item that has been removed from the shopping basket by hiding it in another location, the purchase list is not updated and the item is paid for, preventing shoplifting.

The present invention has the effect of providing a new technology that allows customers to enter and exit a store and purchase products without feeling uncomfortable.

FIG. 1 is an example of a perspective view showing an internal configuration of a supermarket in one embodiment of the present invention. 1 is a system configuration diagram according to an embodiment of the present invention; FIG. 2 is a hardware configuration diagram of a system according to an embodiment of the present invention. 13 is an example of a process flow when registering payment information in advance in one embodiment of the present invention. 4 is an example of a process flow when passing through an entrance gate in one embodiment of the present invention. FIG. 2 is a diagram illustrating a customer passing through an entrance gate in one embodiment of the present invention. 1 is an example of a perspective view showing a shopping basket in one embodiment of the present invention. 1 is an example of a perspective view showing a shopping basket in which cameras for photographing purchased products are set at the four corners of the upper part of the shopping basket in one embodiment of the present invention; 1 is an example of an elevational view showing a shopping basket in one embodiment of the present invention. FIG. 2 is a hardware configuration diagram of a purchased product photographing camera according to an embodiment of the present invention. 1 is an example of a process flow when a customer makes a purchase according to an embodiment of the present invention. 4 is an example of a series of images captured by a purchase item photographing camera attached to a shopping basket as a customer places items in the shopping basket in one embodiment of the present invention. 1 is an example of a series of images captured by a purchase item photographing camera attached to a shopping basket as a customer removes items from the shopping basket in one embodiment of the present invention. 11 is an example of an image showing whether a product captured by a camera photographing a purchased product is being put into or removed from a shopping cart in one embodiment of the present invention. 4 is an example of a series of images captured by a purchase item photographing camera attached to a shopping basket as a customer places items in the shopping basket in one embodiment of the present invention. 13 is an example of a screen of a customer terminal that prompts a customer to take a picture with a camera in one embodiment of the present invention. 13 is an example of a process flow at an exit gate in one embodiment of the present invention. FIG. 4 is an example of a perspective view showing details of an exit gate in one embodiment of the present invention. FIG. 11 is an example of a perspective view showing an internal configuration of a supermarket according to another embodiment of the present invention. FIG. 11 is an example of a perspective view showing an internal configuration of a supermarket according to another embodiment of the present invention. FIG. 11 is a system configuration diagram according to another embodiment of the present invention. 11A and 11B are perspective views showing an example of a product shelf before and after a product purchase in another embodiment of the present invention. FIG. 13 is an example of a perspective view showing a product shelf according to another embodiment of the present invention. 11 is an example of an image taken from a camera attached to a product shelf in another embodiment of the present invention. FIG. 2 is an example of a perspective view showing a cart in one embodiment of the present invention. 1 is an example of a shopping basket according to an embodiment of the present invention.

(Embodiment 1)
The present invention will now be described more fully with reference to the accompanying drawings, in which there is shown a preferred embodiment, but which may be embodied in many different forms and is not limited to the embodiments set forth herein.

For example, in this embodiment, the configuration, operation, etc. of a product purchase payment system will be described, but a similarly configured method, computer program, etc. can also achieve the same effects. The program may also be stored on a recording medium. By using this recording medium, for example, the program can be installed on a computer, thereby configuring a product purchase payment system. Here, the recording medium storing the program may be a non-transitory recording medium such as a CD-ROM.

In this embodiment, the term "part" may include, for example, a combination of hardware resources implemented by a circuit in the broad sense and software information processing that can be specifically realized by these hardware resources. In this embodiment, "information" is represented, for example, by the physical value of a signal value representing a voltage or current, the high or low of a signal value as a binary bit set consisting of 0 or 1, or a quantum superposition (so-called quantum bit), and communication and calculation can be performed on a circuit in the broad sense. A circuit in the broad sense is a circuit realized by appropriately combining a circuit, circuitry, processor, memory, etc. That is, it includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), etc.

The present invention relates to a system that allows customers to make payments themselves in a store. FIG. 1 is a perspective view showing the internal configuration of a supermarket in this embodiment. The basic procedure for a customer to purchase an item and make a payment at a store is as follows: first, the customer enters the store, picks up a purchase item container designated by the store at the entrance gate, and enters. Next, the customer places the items in a purchase item container BA inside the store, and finally makes the payment before passing through the exit gate OUT, returns the purchase item container BA, and leaves the store. Note that in this embodiment, the purchase item container BA is a shopping basket or cart, but the following describes a shopping basket.

The present invention focuses on the relationship between the shopping basket provided to the customer and the customer. In this embodiment, a customer ID is assigned to a customer who enters the store through the entrance gate, and the customer's behavior is tracked while in the store. In addition, an ID is assigned in advance to each purchased item container in the shopping basket, and a camera placed near the entrance gate of the store associates the customer ID with the shopping basket held by the customer, and stores the association in a server (hereinafter referred to as the store server).

Even if a customer does not have a shopping basket, their face is recognized and stored in the store server in the same way as a customer with a shopping basket. This is done to track customers and understand their behavior, since customers may pick up items in the store even if they do not have a shopping basket.

In the present invention, the shopping basket is equipped with an electronic circuit having a communication function, and is configured to be able to communicate with the store server. In the present invention, a camera attached to the shopping basket is used to take a picture of a customer who enters the store putting products into the shopping basket, and the image is sent to the store server. The store server stores in advance information about the products, such as the price, product specifications, and in the case of food, the expiration date, as well as images of various postures of each product sold in the store, and the store server uses existing image processing technology (for example, pattern matching, template matching, machine learning methods that extract contours, calculate features, and perform discriminant analysis from the information, supervised machine learning such as convolutional neural networks (CNN), and unsupervised machine learning methods such as clustering) to identify the products put into the shopping basket, and update and register the customer's payment information. Since a customer may cancel the purchase of a product, when a product is removed from the shopping basket associated with the customer, the store server recognizes that the product once put into the shopping basket is not to be purchased, deletes the product from the customer's purchase list, and updates the purchase list.

Finally, when a customer has finished shopping, they pass through the store's exit gate OUT to leave the store. In the present invention, when paying for products, if the customer is a registered member and has a registered credit card, they check the details on their mobile device or other device in front of the exit gate OUT and press an "approval" button to complete the payment. On the other hand, if the customer is not a registered member, the payment is completed using a self-checkout register installed in front of the exit gate OUT, and after completing the payment, the customer passes through the exit gate OUT to leave the store, for example by issuing a card with a passcode printed on it and presenting the card to a store clerk or having the card read mechanically.

1. System configuration of the first embodiment
2 is a block diagram showing the configuration of the system of this embodiment. As shown in FIG. 2, the product purchase settlement system 0 includes a store server 1, a customer terminal device 2, a purchase product photographing camera 3 for photographing products to be put into a shopping basket, an entering customer photographing camera 4 (customer photographing camera) installed at an entrance gate IN for photographing a customer entering the store and the shopping basket picked up by the customer, an exiting customer photographing camera 5 (customer photographing camera) installed at an exit gate OUT for photographing a customer about to leave the store, a self-checkout 6, a display product photographing camera 7 (not shown) for photographing the display state of products, a customer position measuring device 8 (not shown) for photographing the inside of the store to detect the position of the customer, a store management terminal device 9 (not shown) carried by the store manager, employee, and security guard, a card settlement intermediate company server (in the illustrated example, the intermediate company server), and a card company server, which are configured to be able to communicate with each other via a communication network NW. In this embodiment, the entering customer photographing camera 4 is installed at the entrance gate IN, but it may be installed separately from the entrance gate IN. In the present embodiment, the customer position measuring device 8 is a customer position measuring camera, but it may be a transmitter installed in a shopping basket.

In this embodiment, the store server 1 can be a general-purpose server computer, a personal computer, a smartphone, a tablet terminal, a wearable device, etc. The store server 1 is a server provided for each store chain, and stores product information for products displayed at each store chain, as well as customer registration information for customers who are members of the store chain. Alternatively, there may be a single server that stores product information and customer registration information for all stores.

The intermediary company server registers and manages customer card information, and the card company server is a company server that approves various card payments, and accepts requests for card payment processing from customer terminals via the store server.

In this embodiment, the communication network NW is an IP (Internet Protocol) network, but there are no restrictions on the type of communication protocol, and there are also no restrictions on the type or scale of the network.

<1.1. Hardware configuration>
2 is a diagram showing a hardware configuration of the store server 1. As shown in FIG. 3A, the store server 1 includes a processing unit 101, a storage unit 102, and a communication unit 103.

The processing unit 101 has a processor such as a CPU capable of executing an instruction set, and controls the overall operation and processing of the store server 1 by executing the purchase product settlement program according to the present invention, the OS, and other applications.
The storage unit 102 includes a volatile memory such as a RAM capable of storing an instruction set, and a non-volatile recording medium such as an HDD or SSD capable of recording an OS and a purchased product settlement program, etc.
The communication unit 103 has a communication interface device for connecting to a network, and controls communication with the communication network NW to input and output information.

As shown in FIG. 3(b), the terminal device 90 (customer terminal device 2, store management terminal device 9) has a processing unit 901, a memory unit 902, a communication unit 903, an input unit 904, and an output unit 905.

The processing unit 901 has a processor such as a CPU capable of executing an instruction set, and controls the overall operation and processing of the terminal device 90 by executing an OS and other applications such as a shopping application for utilizing the services provided by this system.
The storage unit 902 includes a volatile memory such as a RAM capable of storing an instruction set, and a non-volatile recording medium such as an HDD or SSD capable of recording an OS and the like.
The communication unit 903 has a communication interface device for connecting to a network, and controls communication with the communication network NW to input and output information.
The input unit 904 has an input device capable of input processing, such as a keyboard or a touch panel.
The output unit 905 has a display device capable of display processing, such as a display.

<1.2. Functional configuration of store server 1>
2, the store server 1 includes a list registration unit 11, a product identification unit 12, a basket identification unit 13, a customer identification unit 14, a list update unit 15, a settlement unit 16, a display processing unit 17, and a database 18. This is a concrete implementation of information processing by software (stored in a storage unit 102) using hardware (the processing unit 101).

The customer terminal device 2 may include some or all of the functional components (units) included in the store server 1. For example, the customer terminal device 2 may include a list registration unit 11, a product identification unit 12, a basket identification unit 13, a customer identification unit 14, a list update unit 15, a payment unit 16, and a display processing unit 17, and the store server 1 may be a cloud storage that records and manages various information related to travel, etc.

<1.3. Database 18>
The database 18 stores product information, image information, and customer registration information.

<1.3.1. Product information＞
Product information is master information about products displayed in a store. The product information includes product features including color information, in addition to a product ID that uniquely identifies the product, product name, product type, price, etc. In addition to or instead of the product features, information for calculating the product features may include image information of multiple products photographed from various angles and distances. In addition, when the product information is stored in one store server, the product information may further include a store ID.

<1.3.2. Imaging information>
The imaging information is image information captured by a camera. The imaging information includes multiple images of the subject captured at various angles and distances, as well as the date and time of capture, and is stored in association with the cameras that captured the images (the camera 3 capturing purchased products, the camera 4 capturing customers entering the store, and the camera 5 capturing customers leaving the store).

<1.3.3. Customer Registration Information>
The customer registration information is information about a customer who uses a shopping application, and is information registered on the application. The customer registration information includes a customer ID that uniquely identifies the customer, the customer's name, gender, address, contact information (phone number, email address, etc.), payment information indicating the type of payment (credit card, prepaid card, cash, etc.), financial information such as credit card, as well as features such as body shape and face. In this embodiment, the body shape and facial features are information acquired before entering a store using a machine learning model or the like launched by the shopping application or other applications.

<1.3.4. Store Customer Information>
Entering customer information is information about customers who enter a store. Entering customer information includes an entering customer ID that uniquely identifies the entering customer, the date and time of entry, a pre-registration flag, and customer features. Here, the pre-registration flag is data indicating whether or not the customer has registered customer registration information in advance, and stores "1" if customer registration information has been registered, and stores "0" if not. In this embodiment, entering customer IDs are assigned to all customers photographed by a customer photography camera in order to track the customer's movements within the store.

<1.3.4. Customer Purchase List>
The customer purchase list is information registered for a customer who enters a store, and is information to which the purchase list of products to be paid for is added and deleted. The customer purchase list includes a customer payment ID that uniquely identifies the customer's payment, a store-entered customer ID, a purchase list, and the store entry date and time, and is stored in association with the shopping cart. Here, the purchase list includes the product ID, product name, product quantity, and product price. The purchase list may also include the total amount to be paid by the customer.

<1.3.5. Shopping Basket Information>
The shopping basket information is information related to a shopping basket. The shopping basket information includes a basket ID that uniquely identifies the shopping basket and a camera ID that photographs the purchased products. Preferably, when a customer purchases or rents a shopping basket, customer registration information is linked to the shopping basket information and registered. Also preferably, the shopping basket may be linked to each store.

<1.4. List registration unit 11>
The list registration unit 11 registers a customer purchase list for each customer. Based on a customer identified based on an image captured by the store-entering customer photography camera 4 and a shopping basket held by the customer, the list registration unit 11 registers a customer purchase list having a combination of the customer and the shopping basket. In addition, the list registration unit 11 links and registers the customer purchase list and customer registration information of a customer identified by the customer identification unit 14 as a customer for which customer registration information is to be registered.

In addition, in this embodiment, a customer may be linked to a shopping basket by purchasing or renting the shopping basket. In this case, the list registration unit 11 registers a customer purchase list including a combination of a customer and a shopping basket identified based on an image acquired via the customer photographing camera 4 entering the store and/or the shopping basket held by the customer.

<1.5. Product identification section 12>
The product identification unit 12 identifies product information based on product images acquired from the purchased product photographing cameras 3 and the displayed product photographing cameras 7. The product identification unit 12 identifies the product type and product price by analyzing the image information of the product acquired from the purchased product photographing cameras 3 and the displayed product photographing cameras 7 stored in the database 18 using an image processing model. In this embodiment, multiple purchased product photographing cameras 3 are installed in one shopping basket, and when a product is put into the shopping basket, the product identification unit 12 identifies the product based on the product image and the photographing date and time of the image information acquired by the purchased product photographing cameras 3 linked to the shopping basket.

In addition, when the product identification unit 12 acquires image information of a product (such as a customer's personal belongings) whose features cannot be acquired, or when the product identification unit 12 cannot identify the product from the image information, it sends an instruction to the display processing unit 17 to display an error message on the customer terminal device 2.

<1.6. Cage Identification Unit 13>
The basket identification unit 13 identifies a shopping basket from the database 18 based on an image of the shopping basket held by a customer acquired via the store-entering customer photography camera 4. The basket identification unit 13 acquires an image of the shopping basket and identifies a marker (character string, code, tag, etc.) attached to the shopping basket in the image, thereby identifying the photographed shopping basket.

<1.7. Customer identification department 14>
The customer identification unit 14 identifies a customer based on images captured by the entering customer photography camera 4 and the leaving customer photography camera 5. The customer identification unit 14 assigns a entering customer ID to the customer identified based on images captured by the entering customer photography camera 4, and registers the entering customer information.

The customer identification unit 14 also identifies whether or not the customer captured via the customer-entering camera 4 is a customer who is to register customer registration information. The customer identification unit 14 acquires the features of the customer by analyzing the image of the customer captured via the customer-entering camera 4 stored in the database 18 using existing image processing technology. The customer identification unit 14 then compares the features of the body type, face, etc. in the customer registration information with the features acquired from the image, and registers a pre-registration flag for the customer-entering information depending on whether the customer has the corresponding features or whether the difference between the features exceeds a predetermined threshold. When the customer identification unit 14 registers "0" as the pre-registration flag, the customer image (image information) captured via the customer-entering camera 4 is linked to the customer ID and stored in the database 18.

<1.8. List update unit 15>
The list update unit 15 updates the purchase list based on the product information of the product placed in the shopping basket. The list update unit 15 updates the purchase list of the customer purchase list linked to the shopping basket in which the purchased product photography camera 3 that photographed the product is installed based on the product information of the product identified by the product identification unit 12.

In a more preferred embodiment, the product identification unit 12 executes a process of identifying the product removed from the shopping basket by the customer based on an image acquired via the purchased product photography camera 3, and a process of identifying that the product is placed on a product shelf based on an image acquired via the display product photography camera 7 within a predetermined time after the product is removed. The list update unit 15 then updates the customer purchase list linked to the shopping basket in which the purchased product photography camera 3 that photographed the product is installed based on the product information of the product.

<1.6. Settlement unit 16>
The settlement unit 16 makes payment for products based on the customer purchase list. The settlement unit 16 accepts input related to payment for products, and when it identifies the customer who made the input related to payment, it makes payment for products based on the payment information in the customer registration information of the customer and the purchase list in the customer purchase list of the customer. In this embodiment, when the customer makes input related to payment on the shopping application, the settlement unit 16 makes payment based on the payment information linked to the customer ID of the shopping application and the purchase list.

In addition, in this embodiment, when the payment unit 16 receives information identifying a customer identified by the customer identification unit 14 based on an image captured by the exit customer camera 5 at the exit gate OUT as payment-related input, the payment unit 16 makes payment for the product based on the customer's payment information and purchase list. Note that the payment unit 16 may not make payment for the product, but may notify the identified customer that payment will be made.

When the input related to the payment is received, if the customer's payment information has not been registered or if the payment information is cash, the payment unit 16 notifies the customer that payment will be made at the self-register 6. In this case, a customer photography camera (not shown) is installed in the self-register 6, and information related to the payment is displayed on the screen of the self-register 6 based on a customer purchase list of the customer identified by the customer identification unit 14 based on an image acquired from the customer photography camera. When the input related to the payment is received, if the customer's payment information has not been registered or if the payment information is cash, a two-dimensional code or the like for making the payment is displayed on the customer terminal device 2, and the payment may be made by having the self-register 6 read the code.

<1.9. Display Processing Unit 17>
The display processing unit 17 processes display of various information related to payment on the customer terminal device 2 on which the shopping application is installed. Note that in this embodiment, the display processing refers to a process in which the display processing unit 17 executes a process of generating necessary information, transmits the generated information to the output unit 905, and the output unit 905 displays the generated information. On the other hand, the display processing may also be a process in which the display processing unit 17 transmits a processing command to the terminal device 90 to generate information necessary for display, and the terminal device 90 generates information necessary for display and displays the generated information.

<3. Flowchart for registering customer information>
The registration process of customer registration information in this embodiment will be described below with reference to Fig. 4. Fig. 4 shows a flow when a customer of a store registers a credit card, a prepaid card, etc. in advance through the customer terminal device 2.

First, in step S101 (hereinafter, "step SXXX" will be simply referred to as "SXXX"), the customer first accesses the website of the store they wish to use and downloads a dedicated shopping application. Next, in S102, the customer enters personal information such as name, address, contact details, and gender from the customer registration information, as well as payment information. Then, in S103, the customer enters financial information such as the card number, expiration date, and security code of their credit card or prepaid card.

In S104, the information entered in S102 and S103 is sent to the store server, and in S105, the store waits for review of the relevant credit card, etc. Next, in S106, the store uses the sent information to inquire at a credit information agency, etc., to confirm the validity of the card. If the validity of the card can be confirmed (YES in S106), in S107, the store server 1 registers the customer as a member and issues a customer ID. The store may also issue a dedicated card to the customer registered as a member and send it by mail. On the other hand, if the validity of the card cannot be confirmed (NO in S106), in S108, the store informs the customer that membership registration is not possible.

It is also possible to complete this series of membership registrations by filling out application forms available at the store. In this case, store staff will send the customer's information to the store server via the store's terminal.

<4. Entry flow>
Next, with reference to FIG. 5, a process when a customer enters a store in this embodiment will be described.

First, when a customer enters a store (S201), in S202 the customer is notified that he or she will be picking up a shopping basket. In this embodiment, the customer is notified that he or she will be picking up a shopping basket by the customer-photographing camera 4 installed at the entrance gate IN photographing the customer, but the customer may also be notified via audio or text that is periodically broadcast. Next, in S203, the customer-photographing camera 4 photographs an overall image of the customer about to pass through the entrance gate IN and transmits the image to the store server 1. Then, in S204, the customer identification unit 14 analyzes the features of the image acquired by the customer-photographing camera 4, such as clothing, body shape (height, etc.), and facial features, to identify the customer with the shopping basket, assign a customer-photographing ID to the customer, and register the customer's entry date and time and customer features as part of the customer-photographing information.

In addition, in S205, the basket identification unit 13 performs image processing on the image acquired through the customer photographing camera 4 to determine whether or not the customer has a shopping basket. If the customer has a shopping basket, the basket identification unit 13 identifies the identification information attached to the shopping basket and identifies the basket. Note that in this embodiment, the processing of the basket identification unit 13 and the processing of the customer identification unit 14 are performed based on images acquired through the same customer photographing camera 4, but may be performed based on images acquired through different customer photographing cameras 4.

Then, in S206, the list registration unit 11 registers a customer purchase list having the combination of the customer identified in S204 and the shopping basket identified in S205.

Next, in S207, the customer identification unit 14 identifies whether the identified customer is a customer who will register customer registration information. Then, in S208, if the identified customer is a customer who will register customer registration information (YES in S207), the pre-registration flag in the store customer information is set to "1" and the customer is allowed to enter as a registered customer. Then, in S209, the list registration unit 11 links the customer registration information and the customer registration product list and registers them.

If the customer is not a customer who registers customer registration information (NO in S207), the customer identification unit 14 determines whether the customer registration information has been accepted via a customer card that records information about the customer (S210). In this embodiment, a customer card that records information about the customer is given to the customer. The customer identification unit 14 then identifies the customer registration information of the customer by having the customer hold the customer card or shopping application over a card reader installed at the entrance gate IN when entering the store (YES in S210). Then, proceed to S208. On the other hand, if the customer does not have a customer card (NO in S210), the customer is allowed to enter the store as a guest customer (S211), and the customer characteristics and customer purchase list of the customer are linked and registered (S212).

The customer entry camera 4 includes a camera used to identify the customer and a camera used to register information about the customer. If the first customer is not a customer who registers customer registration information, the customer identification unit 14 may notify the first customer that he or she will be moving to a location different from other customers, obtain an image of the first customer via a camera used to register information about customers installed at that location, and register the image as information about the first customer. In addition, in S207, the customer identification unit 14 may proceed to S208 if there is no customer feature in the customer registration information that matches the customer feature analyzed from the customer image obtained via the customer entry camera 4, or if the difference between the customer features does not exceed a predetermined threshold. Here, the notification is made using a screen of a terminal installed at the entrance gate that notifies the app, a voice announcement, etc.

Specific examples of this embodiment include a case where a single customer enters a store (FIG. 6(a)) and a case where a group of multiple customers enters the store (FIG. 6(b)). In either case, the customer identification unit 14 analyzes the characteristics of all customers, such as clothing, body shape (e.g., height), and facial features, to identify the customer and assign a customer ID to the customer who entered the store. On the other hand, when multiple people enter the store as in FIG. 6(b), there is a problem of who should be the person who makes the payment. In this embodiment, the list registration unit 11 identifies a customer with a shopping basket as the person who makes the payment based on an image acquired by the customer image capturing camera 4, and registers the customer registration information and customer purchase list of the customer identified as the person who makes the payment in association with each other. The payment unit 16 also makes payment for the product based on the payment information in the customer registration information and the customer purchase list.

In addition, one customer or one group may have multiple shopping baskets. In such a case, the list registration unit 11 registers a customer purchase list having a combination of multiple shopping baskets and identified customers identified based on images of multiple shopping baskets acquired via the store-entering customer photography camera 4. The list update unit 15 then updates the purchase list of the customer purchase list linked to the shopping basket based on the product information of the products identified by the product identification unit 12 and placed in each shopping basket. The payment unit 16 then makes payment for the products based on the payment information of the customers linked to the customer purchase list.

In addition, when multiple customers belonging to one group each have a shopping basket in their hands, the list registration unit 11 registers a customer purchase list for each customer, which includes a combination of the customer identified based on the image acquired via the customer-entering camera 4 and the shopping baskets held by the customer. In addition, the list registration unit 11 registers a customer purchase list of another second customer in the group in association with a first customer belonging to the one group. The settlement unit 16 accepts input related to the settlement of goods, and when the customer purchase list of another second customer is linked to the first customer making the settlement, it notifies the first customer asking whether there is another shopping basket (customer purchase list). Then, when the settlement unit 16 accepts input from the customer that there is another shopping basket, it settles for goods based on the customer purchase list of the first customer and the customer purchase list of the other second customer linked to the first customer. The settlement unit 16 may display identification information of a shopping basket linked to the first customer so that it can be selected and input, and may make payment for products based on the customer purchase list linked to the selected shopping basket. Also, when input is received from the first customer that there are no other shopping baskets, the settlement unit 16 makes payment for products based on the customer purchase list of the first customer. Here, the notification may be displayed on the customer terminal device 2 of the first customer making the payment, or may be displayed on the screen of the self-checkout if the customer makes the payment at a self-checkout, or may be displayed on the screen of a dedicated store terminal for making notifications.

In this embodiment, a customer purchase list is linked and registered for each shopping basket, but one customer purchase list may be registered for multiple shopping baskets. In this case, when the list registration unit 11 links and registers the customer purchase list of another second customer in a group to a first customer belonging to the group, the list registration unit 11 notifies the first customer asking whether the first customer has any other shopping baskets, and when it receives input from the first customer that there are other shopping baskets, it registers one customer purchase list that brings together the customer purchase lists linked to the first customer.

As described above, the shopping basket and customer are identified when entering the store, and the shopping basket and customer are linked, allowing the customer to enter the store. In a more preferred embodiment, only customers holding a shopping basket designated for each store may be allowed to enter the store. In this case, the shopping basket and the entering customer photographing camera 4 are linked to each store, and the basket identification unit 13 determines whether the store of the photographing information acquired via the entering customer photographing camera 4 is the same as the store linked to the shopping basket identified based on the image information of the photographing information, and if they are not the same, notifies the customer instructing them to pick up a shopping basket designated by the store. This makes it possible to deny entry to the store if a customer enters the store with a shopping basket that does not have a purchased item photographing camera 3 installed.

In a more preferred embodiment, the customer purchases or rents a shopping basket, and the customer and the shopping basket are linked and managed. If the shopping basket is not linked to the customer identified in S204, the list registration unit 11 registers a customer purchase list that includes the combination of the identified customer and the shopping basket identified in S205. If the shopping basket is linked to the customer identified in S204, the list registration unit 11 registers a customer purchase list that includes the shopping basket linked to the identified customer and the combination of the identified customer.

In addition, when a customer purchases or rents a shopping basket and the customer and the shopping basket are linked and managed, the shopping basket information is linked to the customer registration information, and the list registration unit 11 may acquire the customer registration information registered on a card issued for each store via a card reader installed at the entrance gate IN, without identifying the customer and the shopping basket based on an image acquired via the store-entering customer photography camera 4, and may register a customer purchase list containing combinations of customers and shopping baskets.

＜5. Product Purchase＞
Then, a customer who has passed through the entrance gate IN and entered the store can shop within the store. In the store, the customer places the purchased items in the shopping basket that was picked up when the customer passed through the entrance gate IN, and then does the shopping.

5.1. Shopping basket configuration
As shown in Fig. 7, a purchased product photographing camera 3 for photographing the purchasing status of products is attached to the top of the shopping basket, and information for identifying the shopping basket is attached. The purchased product photographing camera 3 is composed of a single focus lens. This is because the range photographed by this camera is the inside of the shopping basket, and the purpose is to photograph the state of products entering and leaving the shopping basket. In this embodiment, it is preferable to ignore the movements of other customers in the vicinity of the target shopping basket outside the shopping basket, and it is preferable that the purchased product photographing camera 3 has a distance measuring function that can capture only the products inside the shopping basket.

Also, FIG. 8 shows a structure in which four camera modules are attached to the four corners of the upper part of a shopping basket, and each of the purchased product photographing cameras 3 is installed facing each other. Even if it is difficult to obtain an image of the characteristic side of the product, the price, or an image of the product with one purchased product photographing camera 3, it is possible to photograph the characteristic image and price display of the product using two or more purchased product photographing cameras 3 arranged facing each other. Specifically, in the shopping basket, at least the second purchased product photographing camera 3 is installed at a position facing the first purchased product photographing camera 3 with respect to the first purchased product photographing camera 3, and the first purchased product photographing camera 3 and the second purchased product photographing camera 3 are arranged so that the viewing angle of the first purchased product photographing camera 3 and the second purchased product photographing camera 3 overlap with the viewing angle of the purchased product photographing camera 3 adjacent to the first purchased product photographing camera 3 or the second purchased product photographing camera 3. As a result, it is possible to reliably obtain the photographing information necessary to identify the product, and the accuracy of identifying the product is increased. When multiple cameras are used in this way to take pictures, the images taken by each purchased product photography camera 3 are sent to the store server 1 as independent images, and the product identification unit 12 identifies the product and reads the price based on the product image and the photography date and time in the imaging information.

As shown in FIG. 9, the shopping basket is provided with a contact-type limit switch, and electricity is supplied from the built-in battery to the electronic circuit and the camera through this limit switch. In this embodiment, the shopping basket has an electronic circuit that supplies power to the purchased product photography camera 3, and a switch that turns off the power when the purchased product containers are stacked and turns on when the customer removes the purchased product container. Therefore, when the shopping basket is not held by the customer, but is stacked or placed on a floor or desk with a protrusion as shown in FIG. 9, the electronic circuit and the purchased product photography camera 3 built into the shopping basket are not energized. When the customer removes the shopping basket, that is, when the customer holds the shopping basket in their hand or places the shopping basket on a flat floor, the power is turned on, and the image captured by the camera module is sent to the electronic circuit built into the shopping basket. By configuring the shopping basket in this way, the product can be identified even if the customer places the product in the basket while it is on the floor.

Figure 10 shows the hardware configuration of the purchased product photographing camera 3. As shown in Figure 10, the purchased product photographing camera 3 has a processing unit 301, an output unit 302, an input unit 303, a photographing unit 304, a memory unit 305, and a communication unit 306.

The processing unit 301 has a processor such as a CPU capable of executing an instruction set, and controls the overall operation and processing of the purchased product photographing camera 3 by executing an OS and other applications.
The output unit 302 has a display device capable of display processing, such as a display.
The input unit 303 has an input device capable of input processing, such as a touch panel.
The photographing unit 304 photographs an image of an item to be placed in the shopping cart.
The storage unit 305 includes a volatile memory such as a RAM capable of storing an instruction set, and a non-volatile recording medium such as an HDD or SSD capable of recording an OS and the like.
The communication unit 306 has a communication interface device for connecting to a network, executes communication control with the communication network NW, and outputs the image captured by the photographing unit 304 and the photographing date and time received via the processing unit 301.

In addition to or instead of the camera for photographing purchased items, a terminal device with a camera, such as a smartphone, may be installed in the shopping basket. The terminal device has a display function for displaying various information, and displays advice to the customer when the customer is unable to identify an item to be placed in the shopping basket. The terminal device can also display details of the purchase list and the total price of the purchased items when the customer is in the store to purchase items or when the customer has completed purchasing the items. In addition, in the case of a customer who does not have a terminal with a shopping application installed, a notification regarding payment may be sent via the terminal device.

<5.2. Product purchase flow>
Next, a flow when a customer purchases a product in this embodiment will be described with reference to FIG.

First, when a customer picks up a product in S301, the store server 1 acquires the customer's position in the store via the customer position measuring camera 8 in S302. Then, if the customer purchases a product (YES in S303) and the product picked up by the customer is to be placed in the shopping cart (YES in S304), the customer places the product in the shopping cart (S305).

Next, in S306, the purchased product photographing camera 3 photographs images of the products at a predetermined interval (t0 [seconds]). Then, in S306, the store server 1 acquires the images photographed by the purchased product photographing camera 3 and stores them in the database 18.

Here, the products that consumers purchase include, for example, products whose prices can be identified by their appearance, such as paper-packaged beverages and plastic bottles. Figures 12(a) to 12(e) show a series of images captured by the purchased product photography camera 3 when a consumer selects a paper-packaged beverage as an example of such a product and places it in a shopping cart.

Each image in FIG. 12 is taken at a fixed time interval. If FIG. 12(a) is the image at the time when the product is first seen, the four images from FIG. 12(b) onwards show images taken by the purchased product photography camera 3 at fixed time intervals. When a consumer purchases a product, the consumer first picks up the product from the shelf. When the consumer places the product in the shopping basket, the image taken by the purchased product photography camera 3 attached to the top of the shopping basket shows part of the product, and in some cases part of the consumer's hand. In FIG. 12(a), the bottom of the purchased beverage pack is seen first, and FIG. 12(b) is an image taken a fixed time after FIG. 12(a), and the consumer's hand about to purchase the product becomes visible in the image shown in FIG. 12(a). In addition, in the images shown in FIG. 12(c) to FIG. 12(e), the product and hand are visible in the same way as in FIG. 12(b).

In addition, there are cases where a consumer returns an item that they have purchased by returning it to the shelf. In such a case, the images taken by the purchased item photographing camera 3 will be in the opposite order to the images shown in Figures 12(a) to 12(e) because the item will have left the shopping basket. Figures 13(a) to 13(e) show images of this situation. As with the images shown in Figures 12(a) to 12(e) described above, these images are shown lined up at regular intervals. In this case, the same algorithm as described above can be used to identify items that are returned to the shelf without being purchased.

As shown in Fig. 14, whether a product is being put into or taken out of a shopping cart can be determined by finding a characteristic shape, such as a pattern on the product, and capturing its movement over time. In the example shown in Fig. 14, whether a product is being put into or taken out of a shopping cart is understood from the movement over time of an illustrated image of an orange on the surface of an orange juice carton. In Fig. 14(a), a product is being put into the shopping cart, and in Fig. 14(b), a product is being taken out of the shopping cart.
In this manner, in an embodiment of the present invention, images are taken at regular time intervals, and by calculating the difference over time in the position of the product's characteristic image, the direction in which the product is moving can be detected and it can be determined whether the customer is planning to buy the product or return it to the shelf.

In addition, products do not necessarily have the same price depending on the product's appearance, like paper drink cartons. For example, vegetables and fish have identifiable shapes, but the prices of individual products vary. In such cases, it is necessary to identify the product and at the same time read the price written on the price tag. Existing image processing techniques such as deep learning methods can also be used to recognize printed characters such as prices, or handwritten characters.

Figures 15(a) to 15(e) show images acquired by the purchased product photography camera 3 when purchasing a pumpkin, taking the example of a pumpkin. When purchasing a pumpkin, the bottom part of the pumpkin is first visible as shown in Figure 15(a), and as time progresses from Figure 15(b) to Figure 15(e), the pumpkin is placed in the shopping cart. From the movement of the pumpkin, it is possible to determine whether the pumpkin is being purchased or returned, and it is possible to read the letters showing the price from these images.

Next, in S308, the store server 1 determines whether the customer is in a sales area for single-price items based on the customer's location acquired in S302. If the customer is in a sales area for single-price items (YES in S308), the product identification unit 12 analyzes the product features based on the product images acquired from the purchased product photography camera 3 (S309) and compares the features with the features of the product information stored in the database 18 (S310).

Then, in S311, the product identification unit 12 determines whether there is a product for which the degree of match between the feature acquired in S309 and the feature of the product information exceeds a predetermined threshold (S311). If there is a product with matching feature (YES in S311), the product identification unit 12 identifies the product name and price of the product (S312) and acquires the product name and price (S313). Then, in S314, the list update unit 15 updates the purchase list of the customer purchase list linked to the shopping basket in which the purchased product photography camera 3 that photographed the product in S306 is installed, based on the product name and price acquired in S313.

Also, if the customer is not located in a sales area for single-price items in S308 (NO in S308), the product identification unit 12 identifies the price display of the product captured via the purchased product photography camera 3 (S315), identifies the product name and price of the product (S316), and proceeds to S314.

In addition, if the product that the customer is trying to purchase is a product that cannot be placed in the shopping cart in the first place (NO in S304), the product identification unit 12 acquires a product image via the customer terminal device 2 or a terminal device installed in the shopping cart, and identifies the product, thereby determining the product name and price of the product (S317).

In addition, when a customer puts an item in or takes out of the shopping basket, if the price, product information, or the entire product cannot be photographed because the customer's hand is covering it, the product identification unit 12 will prompt the customer to use the camera of the customer's customer terminal device 2 or a terminal device attached to the shopping basket to photograph the price, product information, and the entire product without covering it with their hand. The product identification unit 12 will then obtain product images via the customer terminal device 2 or a terminal device installed in the shopping basket, and identify the product, thereby determining the product name and price of the product (S317).

Figure 16 shows an example of a screen that prompts the user to take a picture with the camera of the customer terminal device 2. In the example shown in Figure 16, if the user is trying to purchase a pumpkin and is able to identify the type of product but is unable to read the price or product information, an image of a pumpkin will be displayed and the user will be prompted to take a picture of the price and product information including the price.

As a result, customers can shop without feeling uncomfortable, simply by placing items in the shopping basket. In this embodiment, multiple cameras are attached to the shopping basket and positioned so that each camera can see the entire product in its field of view, allowing the entire product to be captured, and since one of the cameras photographs the product, the price of the product can be identified. In addition, by identifying the position of each customer in the store and taking into account the distance between the position where the product was purchased and the position of the relevant customer, and the difference between the time when the product was picked up by the customer from its display state and the time when it was placed in the shopping basket, the information to be identified is limited, and products can be identified with high accuracy.

<6. Exit Flow>
Next, the flow when a customer leaves the store will be described with reference to FIG. 17. First, in S401, the exiting customer photography camera 5 takes an image of a customer leaving the store at the exit gate OUT (FIG. 18). In S402, the customer identification unit 14 determines whether the customer is a customer who has registered customer registration information. In this embodiment, the customer identification unit 14 analyzes customer features based on the customer image acquired via the exiting customer photography camera 5, and determines whether there is a customer whose customer features and the features of the customer registration information match to a predetermined threshold value. If there is a customer whose customer features exceed the predetermined threshold value, the customer identification unit 14 identifies the customer as a customer whose customer registration information has been registered (YES in S402).

When it is determined that the customer has registered customer registration information, in S403 the store server 1 notifies the customer asking whether or not to make payment using the payment method registered in the payment information, and waits for input from the customer as to whether or not to make payment using the registered payment method. Then, when the customer inputs that they will make payment using the registered payment information (YES in S403), the payment unit 16 makes payment for the product based on the purchase list in the customer purchase list linked to the customer and the payment information (S404).

If the customer inputs that they will not make the payment using the registered payment information (NO in S403), the payment unit 16 notifies the customer that they will be guided to the self-register 6. The payment unit 16 then identifies the customer based on a customer image acquired by a payment customer photography camera installed in the self-register 6, and displays information related to the payment of the products (purchase list, total amount, payment method, etc.) based on the customer purchase list linked to the customer (S407). The payment unit 16 then makes the payment for the products using the payment method desired by the customer (cash, credit card, prepaid card, etc.). The payment unit 16 may notify the customer that they will be guided to the self-register 6 and may also display a two-dimensional code for the customer to make the payment.

If the customer is not identified as a customer whose customer registration information has been registered (NO in S402), the customer identification unit 14 determines whether there is a customer whose customer characteristics exceed a predetermined threshold value and the customer characteristics of the store-entered customer information match. If there is a customer whose customer characteristics exceed the predetermined threshold value, the process proceeds to S407. On the other hand, if there is no customer whose customer characteristics exceed the predetermined threshold value, the customer identification unit 14 notifies a manager, employee, or security guard to request assistance for the customer.

As a result, the customer can make the payment without feeling uncomfortable and leave the store. In this embodiment, the customer identification unit 14 identifies the customer by acquiring an image of the customer via the exiting customer photography camera 5, and the payment unit 16 executes the processing related to the payment, but the customer identification unit 14 may also determine whether the customer has completed the payment by acquiring an image of the customer via the exiting customer photography camera 5. In this case, the payment unit 16 notifies the customer who is identified as not having completed the payment that they will make the payment. The processing related to the payment is the same as S403, S404, and S407.

In addition, in this embodiment, to allow customers to enter the store without feeling uncomfortable, the entrance gate IN is opened while linking the customer to the shopping basket, but the store may also have a gate control unit that controls the gate so that it does not open until the combination of the customer and shopping basket is acquired.

When paying for products purchased at the exit gate OUT, the customer may find that the purchase price is too high compared to the customer's estimate, or that the customer reviews the purchased products and decides that they are unnecessary, and cancels the purchase. In such cases, the customer removes the products that were once placed in the shopping basket from the shopping basket and returns them to the product shelf, but it would be convenient for the customer if there was a space such as a return shelf to return the purchased products at the exit gate OUT. In this case, if it can be determined that the product that the customer removed from the shopping basket is on the return shelf, it can be confirmed that it has been returned without fail, even if the customer is outside the store. In this embodiment, when the product identification unit 12 identifies a product based on a product image of the product acquired via the purchased product photography camera 3, and also identifies the product based on a product image of the product acquired via the display product photography camera 7 within a predetermined time after identifying the product, the list update unit 15 updates the customer's purchase list linked to the purchased product photography camera 3, assuming that the product has been returned to the product shelf.

In addition, payment priorities may be registered as payment information linked to a customer. Specifically, payment priorities for family members may be registered as payment information, and it may be possible to register which payment information is used as the basis for priority payment when making a payment. The payment unit 16 then makes payment for products for the customer based on the payment priorities. In this way, even if a customer who comes to shop does not have the ability to pay, payment for products can be made based on the payment priorities linked to the customer.

(Embodiment 2)
Next, in the second embodiment, the present invention will be described for use in crime prevention such as preventing shoplifting. When a certain product is not in the shopping basket of any customer in the store even though the number of products has decreased from the state where it was displayed, it is often the case that the product is not in the shopping basket but in a pocket or the like. For example, if a certain time, for example, 5 minutes, has passed since the number of products changed, and the product is not in the shopping basket of any customer, it is possible that the product has been shoplifted. In such a case, it is highly likely that the customer who was in a place close to the display shelf of the product at the time when the change in the number of products occurred picked up the product. Therefore, it is preferable to prevent shoplifting by having a store employee or security guard call out to customers who were in a close distance to the sales floor of the product at the time when the change in the number of products occurred and urge them to put the product they have picked up in their shopping basket. Also, more preferably, it is possible to limit the placement of the camera that photographs the shelves displaying products to shelves displaying products that are easy to shoplift. It is known that products that are shoplifted tend to be alcohol, cosmetics, sweets, sashimi, meat, rice balls, plastic bottle drinks, side dishes, diapers, supplements, etc. By focusing on the shelves displaying such products with a camera, changes in the number of products can be captured by observing the difference in images taken over time.

In the first embodiment, the customer's shopping list was updated by identifying the products that the customer put in the shopping cart, and payment was made based on the shopping list. In contrast, in this embodiment, a customer who places products in a location where the shopping list is not updated and makes payment without updating the shopping list is identified. Note that in the following explanation, the system according to this embodiment operates each functional configuration, and parts common to the first embodiment will be omitted.

Figure 19 is a perspective view showing the internal configuration of a supermarket in this embodiment. The difference between Figure 1 and Figure 19 is that many customer position measuring cameras 8 are installed in the store. The purpose of installing the customer position measuring cameras 8 is to understand the display status of products and the movement of customers, to ensure payment for purchased products, and to prevent crimes such as shoplifting. The customer position measuring cameras 8 capture the status of the store while capturing customer movements. The images captured by each camera are stored in the database 18 of the store server for a certain period of time. In this embodiment, images of products on the product shelves are acquired via the display product photographing cameras 7, but they may also be acquired via the customer position measuring cameras 8.

For example, even if a mother and child enter the store as shown in FIG. 6(b) above, the characteristics of the mother with a shopping basket and the child without a shopping basket can be recognized. Therefore, the movement of each customer in the store can be understood by analyzing the image of the customer position measuring camera 8 shown in FIG. 19. In the perspective view shown in FIG. 19, the mother and child are very close to each other, but in the perspective view shown in FIG. 20, the child is in a different location from the mother. In this case, the customer position measuring camera 8 is placed as shown in FIG. 19, and the child's movement can also be understood by analyzing the image.

7. Functional configuration of the second embodiment
As shown in FIG. 21 , the product purchase payment system 0 in this embodiment includes a store server 1, a customer terminal device 2, a camera 3 for photographing purchased products, a camera 4 for photographing customers entering the store, a camera 5 for photographing customers leaving the store, a self-checkout register 6, a camera 7 for photographing displayed products, a customer position measuring camera 8 (not shown), a store management terminal device 9 (not shown), a card payment intermediary company server, and a card company server, which are configured to be able to communicate with each other via a communications network NW.

<7.1. Functional configuration of store server 1>
The store server 1 in the second embodiment further includes a notification unit 19 in addition to a list registration unit 11, a product identification unit 12, a basket identification unit 13, a customer identification unit 14, a list update unit 15, a payment unit 16, a display processing unit 17, and a database 18.

7.2. Product Identification Unit 12 in the Second Embodiment
The product identification unit 12 identifies the product taken out from the product shelf based on the image acquired via the display product photography camera 7.

<7.3. Customer identification unit 14 in embodiment 2>
The customer identification unit 14 identifies a customer based on a customer image acquired via the customer position measuring camera 8. In this embodiment, when the product identification unit 12 identifies a product that has been taken out from a product shelf, the customer identification unit 14 identifies, among the customers, customers who are near the product shelf from which the product has been taken out and whose purchase list has not been updated based on the product information of the product.

<7.4. Notification section 19>
The notification unit 19 notifies a manager, employee, or security guard of the store. When a customer who is near the shelf from which the product has been removed and whose purchase list has not been updated based on the product information of the product is identified, the notification unit 19 notifies a manager, employee, or security guard of the store who owns the purchased product container associated with the customer.

8. Examples of this embodiment
An example of this embodiment will be described below with reference to Figures 22 to 24. When a customer purchases a product and places it in a shopping cart, the corresponding product will disappear from the product shelf on which it is displayed.

Figure 22(a) is an oblique view showing the state of the product shelves before a purchase, and a similar image is captured by the product display camera 7 installed in the store. On the other hand, when a product is placed in a shopping cart after a specified time, the product shelves will look like those shown in Figure 22(b). Taking the difference between Figure 22(b) and Figure 22(a) results in the image shown in Figure 22(c). In this embodiment, the product identification unit 12 can determine the product that a customer has taken from the display shelves by taking the difference between the images such as Figure 22(b) and Figure 22(a) captured via the customer position measurement camera 8.

In addition, depending on the product shelf, when a product such as a small-capacity beverage is located at the back of the product shelf, the customer position measurement camera 8 may be unable to see whether the product has been picked up or not. For example, in the case of a showcase displaying beverages as shown in the second product shelf from the bottom in FIG. 23, in addition to the customer position measurement camera 8, a display product photography camera 7 that photographs the state of the product shelf is attached to the upper left end of the second product shelf from the bottom. FIG. 24 shows the state of photography from the display product photography camera 7, where FIG. 24(a) shows the state of beverages displayed before the customer takes out the product, photographed by the display product photography camera 7, and FIG. 24(b) shows the state of beverages displayed after the customer takes out the beverage at the center of the back, as viewed from the display product photography camera 7. The product identification unit 12 in this embodiment can identify the beverage taken by the customer by calculating the difference between the images in FIG. 24(b) and FIG. 24(a).

In this embodiment, the product removed from the display shelf is identified based on the difference between images of the display shelf at two points in time acquired via the customer position measuring camera 8 and/or the displayed product photographing camera 7. On the other hand, the product identification unit 12 may identify the product removed from the display shelf by acquiring product images of the product removed from the display shelf via the customer position measuring camera 8 and/or the displayed product photographing camera 7.

If an item removed from a shelf is not placed in the shopping basket of a customer near the shelf within a predetermined time after the item has been identified, the item may have been placed in the customer's pocket, etc. In this embodiment, if the purchase list of a customer near the shelf of the identified item is not updated within a predetermined time after the item identification unit 12 identifies the item removed from the shelf based on an image acquired via the display item photography camera 7, the customer identification unit 14 identifies the customer. When a customer whose purchase list has not been updated is identified, the notification unit 19 notifies the store management terminal device 9 of the store manager, employee, or security guard who owns the shopping basket linked to the customer to request assistance with the customer.

In this embodiment, if a predetermined time passes without the purchase list being updated and the customer is identified, the store manager, employee, or security guard is notified. On the other hand, if a customer makes payment for purchased items and passes through the exit gate OUT without updating the purchase list, or if the customer passes through the exit gate OUT as a customer who has registered with the store, the manager, employee, or security guard may be notified.

As described above, the customer position measuring camera 8 and/or the display product photographing camera 7 are used to identify the display shelf or the product to be removed, and the customer position measuring camera 8 is used to identify the customer near the display shelf, making it possible to monitor whether the product removed from the display shelf has been placed in a shopping basket, thereby preventing shoplifting.

In a more preferred embodiment, the product purchase and payment system may include a reset unit that resets the image information. For example, if an earthquake or a collision with a customer's cart causes a displayed product to fall or the display to become disarrayed, even if the store clerk returns the product to its original state, it may not be possible to completely return it to its original state. In such a case, the image of the display shelf may change even if the product has not been removed by the customer, and the customer may be erroneously identified. In order to solve this problem, the reset unit receives an instruction for resetting from a store clerk or the like, and sets the image information before receiving the instruction from the store clerk or the like as image information that is not subject to the image information for taking the image difference. In other words, the image information acquired through the display product image camera 7 at the time the instruction is received from the store clerk or the like is stored as image information in the initial state. The product identification unit 12 then calculates the image difference based on the image information acquired after receiving the instruction from the store clerk or the like.

Note that the purchased product photographing camera 3 may be attached not to the shopping basket, but to a cart for carrying the shopping basket. Consumers who purchase large quantities of products, or consumers who have difficulty walking around the store holding products due to old age, often use a manual transport device called a cart with rollers attached to the bottom. Figure 25 is a perspective view of such a cart. In the example of the cart shown in Figure 25, the shopping baskets are placed in two levels. In this case, products placed in the shopping basket placed in the lower level are photographed by the purchased product photographing camera 3 attached to the cart, facing downward. Products placed in the shopping basket placed in the upper level are photographed by the purchased product photographing camera 3 mounted on the cart, with a field of view similar to that described above.

In addition, after shopping, it is troublesome to transfer purchased items from the shopping basket into a shopping bag or plastic bag that customers have brought with them. To address such needs, it is possible to sell store-designated shopping baskets or to rent them out on a monthly basis. Figure 26 shows examples. Figure 26(a) shows a shopping basket used in stores that has a cover attached so that the contents of the basket cannot be seen. Figure 26(b) shows a shopping basket that can be carried in a carrier with wheels for elderly people, etc.

Here, in this embodiment, the customer identification unit 14, based on the image acquired via the customer position measurement camera 8, identifies customers who are near the product shelf from which the product was taken out and whose purchase list has not been updated based on the product information of the product, and identifies the location of the customer.

Specifically, for example, the customer is identified based on the face image or shopping basket of the customer that appears in the image acquired through the customer position measurement camera 8. The customer identification unit 14 then inputs the acquired image using a position estimation model that has been trained using the aberration in the acquired image and the distance to the subject (customer or shopping basket) as input, and calculates the distance from the customer position measurement camera 8 to the customer. The customer's position in the store is then identified using the position (coordinates) in the store of the customer position measurement camera 8 and the calculated distance. As another process for identifying the customer's position, for example, the customer's position may be identified based on multiple captured images and the position in the store of the customer position measurement camera 8 that captured the images. For example, the customer's position may be identified based on the position (coordinates) in the store of the product shelf that appears in the captured image, the distance between the product shelf and the customer, and the position in the store of the customer position measurement camera 8 that captured the image. Note that the information regarding the position may be global coordinates.

On the other hand, the customer identification unit 14 may acquire the identification information of the shopping basket via a transmitter installed in the shopping basket, identify a customer whose purchase list has not been updated based on the product information of the product, and identify the customer's location based on the radio waves emitted from the transmitter.

As a specific process for identifying a customer's location, multiple receivers that receive radio waves from a transmitter via wireless communication (e.g., Bluetooth (registered trademark) or Wi-Fi, etc.) are installed in the store, and the customer identification unit 14 calculates customer location information based on the strength of each of the radio waves received by the multiple receivers. Note that a transmitter that transmits radio waves may be installed in the store, and a receiver installed in the customer terminal device 2 or shopping basket may receive the radio waves, and the customer identification unit 14 may identify the customer's location based on the strength of each of the received radio waves. The customer identification unit 4 may also identify the customer's location by directly acquiring GPS (Global Positioning System) information from the customer terminal device 2. Note that there are no restrictions on the wireless communication standard.

0: Product purchase settlement system 1: Store server 2: Customer terminal device 3: Camera for photographing purchased products 4: Camera for photographing entering customers 5: Camera for photographing leaving customers 6: Self-checkout register 7: Camera for photographing displayed products 8: Customer position measurement camera 9: Store management terminal device 10: Server 101: Processing unit 102: Memory unit 103: Communication unit 301: Processing unit 302: Output unit 303: Input unit 304: Photographing unit 305: Memory unit 306: Communication unit 90: Terminal device 901: Processing unit 902: Memory unit 903: Communication unit 904: Input unit 905: Output unit NW: Communication network 11: List registration unit 12: Product identification unit 13: Basket identification unit 14: Customer identification unit 15: List update unit 16: Settlement unit 17: Display processing unit 18: Database 19: Notification unit IN ：Entrance gate OUT ：Exit gate BA ：Place for purchased items (shopping basket)

Claims (16)

A product purchase payment system that allows customers to make payments by themselves in a store,
The purchase settlement system includes a customer photographing camera installed in a store, a purchase item container in which a purchase item photographing camera that photographs items to be placed in the purchase item container is installed, a list registration unit, a list update unit, a settlement unit, a product identification unit, and a database,
The database stores customer payment information used for payment of products for each customer,
The list registration unit includes:
If the purchase container is not associated with the customer identified based on the image captured by the customer camera, a customer purchase list is registered that includes a combination of the purchase container identified based on the image captured by the customer camera and held by the identified customer, and the identified customer; and/or
When the purchased item container is associated with a customer identified based on an image acquired via the customer photographing camera,
registering a customer purchase list having a combination of the purchase item container associated with the identified customer and the identified customer;
The product identification unit identifies product information of the product placed in the purchased product container based on the image acquired via the purchased product photography camera,
The list update unit updates the customer purchase list associated with the purchased item container in which the purchased item photography camera that photographed the item is installed based on the item information,
The payment unit performs payment for the product based on the payment information and the customer purchase list. The product purchase payment system according to claim 1, wherein the payment unit pays for products based on the payment information linked to a customer identified based on an image captured by a customer-photographing camera installed at an exit of a store, and the purchase list. The product purchase settlement system according to claim 1, wherein the purchase container has an electronic circuit that supplies power to the purchase product photography camera, and a switch that turns the power OFF when the purchase containers are placed on top of each other and turns the power ON when the customer removes the purchase container. The product purchase settlement system according to claim 1 or claim 2, wherein a plurality of the purchased product photographing cameras are installed in the purchased product container, and at least a second purchased product photographing camera is installed in a position opposite the first purchased product photographing camera, and the viewing angles of the first purchased product photographing camera and the second purchased product photographing camera overlap with the viewing angle of the purchased product photographing camera adjacent to the first purchased product photographing camera or the second purchased product photographing camera. If the customer has multiple purchase containers,
the list registration unit registers the customer purchase list including a combination of the plurality of purchase containers identified based on the images of the plurality of purchase containers acquired via the customer photography camera and the identified customer;
The product purchase payment system of claim 1 , wherein the list update unit updates the purchase list of the customer purchase list linked to each purchased product container based on product information of the products identified by the product identification unit and placed in each purchased product container. The list registration unit identifies the customer who has the purchase item container in hand as a payment person based on the image captured by the customer photography camera,
The product purchase settlement system according to claim 1 , wherein the settlement unit settles the product based on the customer purchase list and the settlement information of the customer specified as the settler. The product purchase settlement system further includes a gate control unit,
2. The product purchase settlement system according to claim 1, wherein the gate control unit controls an entrance gate of the store when a combination of an image of the customer and a purchased product container picked up by the customer is acquired. When the customer purchases or rents the purchase container, the combination of the customer and the purchase container is registered in advance.
The product purchase payment system of claim 1, wherein the list registration unit registers the customer purchase list including a combination of customers entering a store identified based on images captured by the customer photography camera and the purchased product containers associated with the customers. the list registration unit registers the customer purchase list for each customer,
If the first customer belongs to a group, registering the customer purchase list of another second customer in the group in association with the first customer;
The product purchase payment system of claim 1, wherein when the payment unit receives payment input from the first customer to which the customer purchase list of another second customer is linked, the payment unit notifies the first customer as to whether or not to make payment for the product based on the customer purchase list of the other second customer. The product purchase settlement system further includes a customer identification unit,
The customer photographing camera includes a camera used to identify a customer and a camera used to obtain information about the customer,
the customer identification unit identifies a customer based on an image acquired via a camera used for identifying the customer, and identifies whether or not the customer has registered customer-related information in advance;
2. The product purchase payment system of claim 1, further comprising: if the customer has not registered customer information in advance, an image of the customer is acquired via a camera used to acquire information about the customer, and the image is registered as information about the customer. The product purchase settlement system further includes a customer position measuring device that measures a position of a customer in a store, a display product photographing camera that photographs displayed products, and a customer identification unit;
The product identification unit identifies a product taken out from the product shelf based on an image acquired via the display product photography camera,
2. The product purchase settlement system according to claim 1, wherein the customer identification unit identifies, among the customers, a customer who is near the product shelf from which the product has been taken out and whose purchase list has not been updated based on product information of the product. The customer location measuring device is a customer location measuring camera,
The product purchase settlement system according to claim 11 , wherein the customer identification unit identifies a customer whose purchase list has not been updated and a location of the customer based on an image acquired via the customer position measuring camera. The customer location measuring device is a transmitter installed in the purchase container,
The product purchase payment system according to claim 11, wherein the customer identification unit acquires identification information of the purchase container via the transmitter to identify a customer whose purchase list has not been updated, and identifies the location of the customer based on radio waves transmitted from the transmitter. The product purchase settlement system further includes a notification unit,
When a customer who is near the shelf from which the product is taken out and whose purchase list has not been updated based on the product information of the product is identified,
The product purchase settlement system according to claim 9 , wherein the notification unit notifies a manager, an employee, or a security guard of a store that owns the purchased product container associated with the customer. The product purchase and settlement system further includes a display product photography camera for photographing the displayed products,
The product identification unit,
A process of identifying an item removed from the purchase container by a customer based on an image captured by the purchase item photographing camera;
and executing a process of identifying that the product is placed on the product shelf based on an image acquired via the display product photography camera within a predetermined time period after the product is removed,
The product purchase settlement system according to claim 1 , wherein the list update unit updates the customer purchase list linked to the purchased product container based on product information of the product. A payment method for purchasing goods that allows customers to make payments by themselves in a store,
The computer
Stores customer payment information used for payment of products for each customer,
If a purchase container is not associated with a customer identified based on an image captured by a customer-photographing camera installed in a store, a customer purchase list is registered that includes a combination of the purchase container identified based on an image captured by the customer-photographing camera and held by the identified customer, and the identified customer; and/or
When the purchased item container is associated with a customer identified based on an image acquired via the customer photographing camera,
registering a customer purchase list having a combination of the purchase item container associated with the identified customer and the identified customer;
Identifying product information of the product placed in the purchase container based on an image acquired via a purchase product photography camera that photographs the product to be placed in the purchase container;
Based on the product information, a purchase list of the customer purchase list linked to the purchased product container in which the purchased product photography camera that photographed the product is installed is updated;
A product purchase settlement method, comprising: making payment for the product based on the settlement information and the purchase list.