JP7741025B2 - Information processing device and program - Google Patents

Description

Embodiments of the present invention relate to an information processing device and a program.

Conventionally, payment systems have been known in supermarkets and other stores that allow customers to register the products they purchase using mobile devices such as smartphones and tablet computers (see, for example, Patent Document 1).

In this payment system, customers pick up the product they wish to purchase at the sales floor and scan the barcode attached to the product with a scanner. The mobile device registers the product based on the product code read by the scanner.

Once a customer has completed product registration for all items they wish to purchase, they proceed to the checkout area where a checkout device is installed. Based on the customer's operation, the mobile terminal sends checkout information, including the registered product information, to the checkout device. Based on the received checkout information, the checkout device carries out checkout processing, including payment of the customer's purchase price. In the conventional checkout system described above, product registration is not performed at the checkout device where payment is made, so processing at the checkout device can be completed in a short time, thereby reducing the wait time at the register.

However, with the above-mentioned conventional technology, customers who are unfamiliar with registering products register products while moving around the sales floor, which means there is a risk of registering products unintentionally. For example, when a customer is moving around the sales floor in an area where scanning is possible, they may accidentally scan the barcodes attached to products displayed on the shelves.

The problem that this invention aims to solve is to provide an information processing device and program that can prevent products from being registered unintentionally by the customer.

The information processing device of this embodiment is operated by a customer moving around a store, and includes a first acquisition unit that acquires product identification information that identifies a product, a second acquisition unit that acquires a customer image that is an image of the customer, a registration unit that registers product information of a product identified by the product identification information acquired by the first acquisition unit as a transaction target, and a determination unit that determines whether to permit registration by the registration unit based on the customer image acquired by the second acquisition unit.

FIG. 1 is a diagram showing an overview of a transaction system including a mobile terminal according to a first embodiment. FIG. 2 is a block diagram showing the main hardware configuration of the mobile terminal of the first embodiment. FIG. 3 is a diagram showing the data structure of the product information section stored in the mobile terminal of the first embodiment. FIG. 4 is a diagram illustrating an example of a display on the display unit of the mobile terminal according to the first embodiment. FIG. 5 is a block diagram showing the main functional configuration of the control unit of the mobile terminal of the first embodiment. FIG. 6 is a flowchart showing the flow of the registration process by the control unit of the mobile terminal according to the first embodiment. FIG. 7 is a diagram showing the data structure of the product information section stored in the mobile terminal of the second embodiment. FIG. 8 is a flowchart showing the flow of registration processing by the control unit of the mobile terminal according to the second embodiment.

The following describes an information processing device and a program according to an embodiment, with reference to the drawings. Note that the present invention is not limited to the embodiments described below. For example, in the embodiments described below, an example is described in which the information processing device is a mobile terminal such as a smartphone owned by a customer, but the information processing device may also be a mobile terminal such as a tablet terminal provided by the store.

Furthermore, in each of the embodiments described below, an example will be described in which the mobile terminal is equipped with a first imaging unit that captures product images for recognizing product codes and a second imaging unit that captures customer images, which are images of the customer capturing the product images, but this is not limited to this. The mobile terminal may not be equipped with the first imaging unit and the second imaging unit, and may instead acquire the product images and customer images from an external device, such as a camera installed in a shopping cart.

(First embodiment)
The following describes an information processing device and program according to the first embodiment. Fig. 1 is a diagram showing an overview of a checkout system 100. The checkout system 100 of this embodiment is applied to a store that sells products, such as a supermarket.

The accounting system 100 includes a mobile terminal 1, an accounting device 2, and a server device 3. The accounting device 2 and the server device 3 are connected to each other via a network N such as a LAN (Local Area Network) so that they can communicate with each other. The mobile terminal 1 is connected to the accounting device 2 and the server device 3 via a wireless LAN or the like so that they can communicate with each other.

Mobile terminal 1 is configured, for example, as a smartphone owned by customer C. A shopping application program (hereinafter also referred to as a "shopping app") is installed on mobile terminal 1. By running the shopping app inside a store, mobile terminal 1 becomes able to connect to the store's wireless LAN and perform processes for shopping at the store. Mobile terminal 1 is an example of an information processing device.

For example, the mobile terminal 1 captures an image of a code symbol such as a barcode (hereinafter also referred to as a "product barcode") attached to a product by customer C's operation, and reads the product code indicated by the product barcode. The product code is information that identifies the product and is an example of product identification information. The image of the product barcode is also an example of a product image for recognizing product identification information. The mobile terminal 1 may also read the product code using general object recognition (object recognition), in which case an image of the product's exterior becomes the product image for recognizing product identification information.

When the mobile terminal 1 determines that the read product code is a transaction target, it registers the product based on that product code. Here, product registration means acquiring product information (product name, price, etc.) based on the product code determined to be a transaction target and registering (storing) it in the memory unit. The mobile terminal 1 functions as a product registration device.

The mobile terminal 1 captures an image of customer C capturing a product barcode (hereinafter also referred to as the "customer image") and determines, based on the customer image, whether the read product code is a transaction target. In other words, when the mobile terminal 1 reads a product code, it determines, based on the customer image, whether to allow product registration of the product identified by the product code.

For example, if customer C, who is taking an image of a product barcode, is looking at the screen displaying the product barcode being captured, it is determined that customer C is intentionally reading the product code, and product registration based on that product code is permitted. Conversely, if customer C, who is taking an image of a product barcode, is not looking at the screen displaying the product barcode being captured, it is determined that customer C is unintentionally reading the product code, and product registration based on that product code is not permitted.

This prevents the mobile terminal 1, which moves around the sales floor with customer C, from inadvertently registering products that customer C does not intend to purchase. For example, it can prevent the product barcode of a product displayed on a display shelf from being read and resulting in unintended product registration. Whether customer C, who is capturing an image of the product barcode, is visually viewing the screen displaying the product barcode being captured is determined by analyzing the customer image using known image processing technology.

Multiple accounting devices 2 are installed in the accounting area within the store. These accounting devices 2 are so-called self-service accounting devices where, for example, customer C performs the operations related to paying for purchases themselves. The accounting device 2 performs accounting processing related to payment by cash payment or cashless payment such as credit card payment based on the accounting information (including product registration information) received from the mobile terminal 1. Accounting processing is the process performed by the accounting device 2 to pay for the products purchased by customer C. The accounting device 2 may also be a POS (Point of Sales) terminal operated by a store clerk. Furthermore, the mobile terminal 1 may also be equipped with accounting processing functions such as credit card payment, in which case the accounting device 2 may not be installed.

The server device 3 functions as a store server that manages store information. The server device 3 receives payment information, such as product information for products paid for in the checkout process and payment methods, from each checkout device 2, and manages store sales information. The server device 3 also stores a product master that associates product codes with product information. The server device 3 transmits the product master to the mobile terminal 1 as appropriate.

Next, the configuration and functions of the mobile terminal 1 will be described in detail. Figure 2 is a block diagram showing the main hardware configuration of the mobile terminal 1. The mobile terminal 1 comprises a control unit 10, a display unit 11, an operation unit 12, an outer camera 13, an inner camera 14, a communication unit 15, and a memory unit 16. The control unit 10, display unit 11, operation unit 12, outer camera 13, inner camera 14, communication unit 15, and memory unit 16 are connected to each other via a bus 17, etc.

The control unit 10 is configured as a computer equipped with a CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, and RAM (Random Access Memory) 103. The CPU 101, ROM 102, and RAM 103 are connected to each other via a bus 17.

CPU 101 controls the overall operation of mobile terminal 1. ROM 102 stores various programs, such as programs used to drive CPU 101, and various data. RAM 103 is used as a work area for CPU 101, and loads various programs and data stored in ROM 102 and memory unit 16. Control unit 10 executes various control processes for mobile terminal 1 by having CPU 101 operate in accordance with control programs stored in ROM 102 and memory unit 16 and loaded into RAM 103.

The RAM 103 also includes a product information section 1031. The product information section 1031 stores product information and the like relating to product codes that are determined to be the subject of a transaction among the product codes read by the outer camera 13. Product registration is performed by storing product information for the product to be traded in the product information section 1031. Figure 3 is a diagram showing the data configuration of the product information section 1031. The product information section 1031 stores product information in association with the product code that identifies the product. The product information includes the product name, price, and quantity. Note that the product information stored in the product information section 1031 is not limited to the example shown in the figure, and may include, for example, an illustration showing the appearance of the product.

The display unit 11 is composed of, for example, a liquid crystal panel and displays various information. For example, the display unit 11 displays product information for registered products. The display unit 11 also displays a registration screen for registering products that do not have product barcodes, as well as controls for entering various information.

Furthermore, the display unit 11 displays the product image captured by the outer camera 13. Specifically, the display unit 11 displays the captured image including the product barcode captured by the outer camera 13 in order to read the product code. Figure 4 shows an example of a display screen that displays the captured image captured by the outer camera 13.

The display unit 11 displays an image captured by the outer camera 13, which is provided facing the back side of the display unit 11. In other words, the display unit 11 displays an image of the back side of the display unit 11 captured by the outer camera 13. At this time, a message 111, such as "Please align the barcode within the frame," and a frame 112 are displayed on the display unit 11.

When the product barcode is positioned within frame 112, outer camera 13 is able to recognize the product barcode and read the product code. While visually checking display 11, customer C moves mobile terminal 1 so that the product barcode of the product they wish to purchase is aligned within frame 112. When the product barcode is positioned within frame 112 and outer camera 13 reads the product code, display 11 displays a screen showing product information and various controls related to the read product code.

The operation unit 12 is used to input various information into the control unit 10 and is composed of a touch panel mounted on the surface of the display unit 11. When customer C operates an operator displayed on the display unit 11, the operation unit 12 inputs information corresponding to the operator into the control unit 10. For example, the operation unit 12 inputs the product code of a product that does not have a product barcode into the control unit 10. The operation unit 12 also inputs a transaction information transmission instruction to the control unit 10 to instruct the control unit 10 to send transaction information including registered product information to the accounting device 2.

The outer camera 13 is provided facing the rear side of the display unit 11. The outer camera 13 is equipped with an optical imaging element such as a CCD (Charge Coupled Device) and is an example of a first imaging unit that captures product images. The outer camera 13 functions as a scanner that reads product codes from product barcodes. Specifically, the outer camera 13 captures and recognizes the product barcode of the product purchased by customer C, and decodes the product barcode to read the product code. Note that the recognition of the product barcode based on the product image captured by the outer camera 13 and the decoding of the product barcode may be performed by the control unit 10. The outer camera 13 may also capture an image of the exterior of the product, in which case the product code may be read (recognized) using general object recognition (object recognition).

The in-camera 14 is provided facing the front surface of the display unit 11. The in-camera 14 is equipped with an optical imaging element such as a CCD (Charge Coupled Device), and is an example of a second imaging unit that captures customer images. Specifically, the in-camera 14 captures an image of the face of customer C who is visually viewing the display unit 11 and capturing an image of the product barcode.

The communication unit 15 is an interface for communicating with external devices such as the accounting device 2 and the server device 3. By connecting to external devices via the communication unit 15, the control unit 10 can send and receive information (data) with the external devices.

The memory unit 16 is composed of storage media such as a HDD or flash memory, and retains its contents even when the power is cut off. The memory unit 16 stores a control program 161 and a product master 162.

The control program 161 is a shopping app or the like that causes the mobile terminal 1 to function as a product registration device. The product master 162 is a master file that stores information about products sold by the store. As described above, the product master 162 stores product codes in association with product information including product names and prices. As the products handled by the store change daily, the product master 162 is updated as appropriate by the server device 3.

The storage unit 16 may be configured not to store the product master data 162. In this case, the mobile terminal 1 can handle the product master data of the server device 3 in the same way as in a configuration in which the product master data 162 is stored, by appropriately communicating with the server device 3 and referencing the product master data stored in the server device 3.

Next, the main functional configuration of the control unit 10 of the mobile terminal 1 will be described. Figure 5 is a block diagram showing the functional configuration of the control unit 10 of the mobile terminal 1. The control unit 10 functions as a reception unit 1001, an acquisition unit 1002, a determination unit 1003, a registration unit 1004, a transmission/reception unit 1005, and a display control unit 1006, as the CPU 101 operates in accordance with a control program stored in the ROM 102 or the storage unit 16. Note that each of these functions may be configured using hardware such as a dedicated circuit.

The reception unit 1001 receives various information from the operation unit 12. For example, the reception unit 1001 receives instructions to launch a shopping app, instructions to start scanning, instructions to send transaction information, and so on from the operation unit 12. The shopping app launch instruction is information that instructs the device to launch the shopping app. The scan start instruction is information that instructs the outer camera 13 to start scanning product barcodes, i.e., to start reading product codes. The transaction information send instruction is information that instructs the device to send transaction information, including information registered in the product information unit 1031, to the transaction device 2.

The acquisition unit 1002 acquires product identification information that identifies a product. Specifically, the acquisition unit 1002 acquires the product code read by the outer camera 13 from the outer camera 13. The acquisition unit 1002 functions as a first acquisition unit. Note that if the recognition of the product barcode based on the product image captured by the outer camera 13 and the decoding of the product barcode are performed by a decoding processing unit (not shown) of the control unit 10, the acquisition unit 1002 can acquire the product code from the decoding processing unit.

The acquisition unit 1002 also acquires a customer image, which is an image of customer C capturing a product image for recognizing product identification information. Specifically, the acquisition unit 1002 acquires a customer image, such as customer C's face, captured by the in-camera 14 from the in-camera 14. The customer image is an image of the product barcode captured by customer C using the out-camera 13. The acquisition unit 1002 also functions as a second acquisition unit.

The determination unit 1003 determines whether or not to permit registration by the registration unit 1004 based on the customer image acquired by the second acquisition unit (acquisition unit 1002). Specifically, the determination unit 1003 performs image processing on the customer image acquired by the acquisition unit 1002 from the in-camera 14, and determines whether customer C, who is the subject of the customer image, consciously read the product code. The determination unit 1003 then determines that the product code that is determined to have been consciously read by customer C is the subject of a transaction.

For example, if it is determined that customer C is looking at the display unit 11 in the customer image when the acquisition unit 1002 acquires the product code, i.e., the customer image when customer C reads the product code, the determination unit 1003 allows the product to be registered by the registration unit 1004. In other words, in this case, the determination unit 1003 determines that the product code acquired by the acquisition unit 1002 was intentionally read by customer C and is the product code of the product that customer C intends to purchase, and allows the product to be registered.

On the other hand, if it is determined that customer C is not visually viewing the display unit 11 in the customer image when the acquisition unit 1002 acquires the product code, the determination unit 1003 does not permit product registration by the registration unit 1004. In other words, in this case, the determination unit 1003 determines that the product code acquired by the acquisition unit 1002 was read unintentionally by customer C, and does not permit product registration.

Here, the determination of whether customer C is looking at the display unit 11 is made using known image processing technology. For example, the determination unit 1003 extracts a facial area from the customer image, and if the eye area can be recognized, identifies the viewing direction from the eye area or the position of the pupils. If the eye area cannot be recognized, it identifies the viewing direction from the position of the nose or mouth. The determination unit 1003 then determines whether customer C is looking at the display unit 11 based on the identified viewing direction. Furthermore, if the determination unit 1003 cannot extract a facial area from the customer image, it determines that customer C is not looking at the display unit 11.

The determination unit 1003 may use another method to determine whether the product code acquired by the acquisition unit 1002 was intentionally read by customer C. For example, if the determination unit 1003 recognizes that the image captured by the outer camera 13 is an image of a person's hand holding a product, it may determine that the product barcode was intentionally read by customer C. This is because when customer C captures an image of the product barcode of a product they wish to purchase, they are likely to pick up the product and then capture the image of the product barcode.

The determination of whether customer C intentionally scanned the product barcode may also be performed using machine learning. Machine learning is a well-known technique that uses large amounts of training data to learn patterns and then uses the training results to infer unknown data. For example, the determination unit 1003 may use a learning model that infers whether the product with the scanned product code was purchased based on a customer image.

The registration unit 1004 registers the product information of the product identified by the product identification information acquired by the first acquisition unit as a transaction target. Specifically, the registration unit 1004 executes product registration for products for which the determination unit 1003 has permitted the product information to be registered, i.e., products determined by the determination unit 1003 to be a transaction target. More specifically, the registration unit 1004 reads product information corresponding to a product code that is determined to have been intentionally read by customer C from the product master 162, and registers it in the product information unit 1031. When the registration of one product is complete, the registration unit 1004 outputs an alert sound from a speaker unit (not shown) to notify customer C that the product has been registered.

Note that the registration unit 1004 may also store product information for products determined not to be eligible for trading in the product information unit 1031. In this case, the registration unit 1004 stores product information for products determined to be eligible for trading separately from product information for products determined not to be eligible for trading. In this specification, storing product information for products determined not to be eligible for trading in the product information unit 1031 is not included in the concept of product registration.

Furthermore, if the product master 162 is not stored in the memory unit 16, the registration unit 1004 can also communicate with a server device 3 that stores the product master 162 and obtain product information from the server device 3.

The transmission/reception unit 1005 transmits and receives information to and from external devices. For example, the transmission/reception unit 1005 transmits transaction information, including product information registered in the product information unit 1031, to the checkout device 2. The transmission/reception unit 1005 also receives a payment completion notification from the checkout device 2 indicating that payment has been completed in the transaction process. The transmission/reception unit 1005 also receives a product master from the server device 3.

The display control unit 1006 displays various information on the display unit 11. For example, the display control unit 1006 displays product information registered by the registration unit 1004 on the display unit 11. The display control unit 1006 also displays product information for products for which registration by the registration unit 1004 has been denied by the determination unit 1003, along with information indicating that the product will not be registered, on the display unit 11. The display control unit 1006 also displays product images captured by the outer camera 13 on the display unit 11. The display control unit 1006 also displays other information, such as various controls for inputting information such as instructions to launch an app or to send transaction information, on the display unit 11.

Next, we will explain the process by which the mobile terminal 1 configured as described above registers products. Figure 6 is a flowchart showing the flow of the registration process by the control unit 10 of the mobile terminal 1. Here, we will assume that the shopping app is running on the mobile terminal 1.

The control unit 10 determines whether the reception unit 1001 has received a scan start instruction from the operation unit 12 (step S1), and if the scan start instruction has not been received (N in step S1), the control unit 10 returns to the processing of step S1 and waits.

When the reception unit 1001 receives a scan start instruction (Y in step S1), the control unit 10 determines whether the acquisition unit 1002 has acquired a product code from the outer camera 13 (step S2). If a product code has not been acquired (N in step S2), the control unit 10 returns to the processing of step S2 and waits. Note that the processing of step S1 may be omitted; in this case, the control unit 10 may execute the processing from step S2 onwards when the shopping app is launched.

Once the acquisition unit 1002 acquires the product code (Y in step S2), it acquires a customer image from the in-camera 14 (step S3). The acquisition unit 1002 acquires the image of customer C capturing the product barcode using the out-camera 13 as the customer image.

Next, the determination unit 1003 determines whether the customer image acquired by the acquisition unit 1002 is an image of customer C looking at the display unit 11 (step S4). If the image is not of customer C looking at the display unit 11 (N in step S4), the display control unit 1006 causes the display unit 11 to display an unregistered product display (step S5). The unregistered product display is a display indicating that product registration has not been performed based on the product code acquired by the acquisition unit 1002, i.e., the product code read by the outer camera 13. For example, the unregistered product display displays a message indicating that product registration has not been performed based on the product code read by the outer camera 13, along with product information for the product associated with that product code.

After displaying the unregistered product in step S5, the control unit 10 returns to the processing of step S1. Displaying the unregistered product prevents customer C from mistakenly recognizing an unregistered product as a registered product. For example, even if customer C reads the product code of a product he or she wishes to purchase without visually checking the display unit 11, displaying the unregistered product allows customer C to recognize that the product is not registered. Therefore, customer C can re-read the product barcode of the product displayed as an unregistered product.

In this embodiment, when a product is registered, an audible alert is emitted to notify customer C that the product has been registered. As a result, if customer C reads the product code without visually checking the display unit 11, customer C can recognize that the product has not been registered because no audible alert is emitted. However, the audible alert may not be heard in a noisy sales floor. Alternatively, if the mobile terminal 1 is a smartphone and is in silent mode, the audible alert is not emitted. Therefore, when registering a product using customer C's smartphone in a sales floor, it may be difficult for customer C to recognize that the product has not been registered based on the audible alert. In such an environment, it is particularly effective to notify customer C that the product has not been registered by displaying an "unregistered" message.

In addition, in this embodiment, customer C issues a scan start command each time a product code is read. As a result, customer C visually checks the display unit 11 each time a product code is read. This prevents customer C from overlooking the unregistered product display, and allows customer C to more reliably recognize that the product is not registered.

If, in step S4, the customer image acquired by the acquisition unit 1002 is an image of customer C looking at the display unit 11 (Y in step S4), the registration unit 1004 registers the product (step S6). Next, the display control unit 1006 causes the display unit 11 to display product information for the registered product (step S7). At this time, the display control unit 1006 also causes the display unit 11 to display an operator for entering a command to start scanning and an operator for issuing a command to send transaction information.

The control unit 10 then determines whether the reception unit 1001 has received an instruction to send accounting information from the operation unit 12 (step S8), and if an instruction to send accounting information has not been received (N in step S8), the control unit 10 returns to the processing in step S1.

When the reception unit 1001 receives the instruction to send transaction information (Y in step S8), the transmission/reception unit 1005 reads the information stored in the product information unit 1031 and sends transaction information including that information to the transaction device 2 (step S9). The transaction information may be sent to the transaction device 2 via network N, or may be sent to the transaction device 2 using short-range wireless communication such as NFC (Near Field Communication). The control unit 10 then ends the registration process.

Through the above registration process, the mobile terminal 1 registers only products with product codes that are determined to have been read intentionally by customer C. This prevents product registrations that customer C did not intend, reducing the chance of mismatches between the products that customer C places in the shopping cart as purchases and the products that are actually registered. This makes it possible to prevent products purchased by customer C from being purchased at an incorrect transaction price.

Second Embodiment
Next, the second embodiment will be described. The second embodiment differs from the first embodiment in that product information for products not included in the transaction despite product codes acquired by the acquisition unit 1002 is displayed on the display unit 11 after an instruction to send transaction information is given, allowing customer C to confirm it. Below, descriptions of configurations and functions that are similar to those of the first embodiment may be omitted. This is to avoid unnecessary redundancy in the following description and to make it easier for those skilled in the art to understand.

In this embodiment, the product information unit 1031 stores product information for all products associated with product codes read by the outer camera 13. In particular, the product information unit 1031 distinguishes between products that are subject to trading and products that are not subject to trading, and stores product information for these products. Figure 7 is a diagram showing the data structure of the product information unit 1031. The product information unit 1031 stores product information and a trading target flag in association with the product code that identifies the product.

Product information includes the product name and price. The trading target flag is information indicating whether the product has been determined by the determination unit 1003 to be a trading target. The trading target flag is registered as "1" for products determined to be trading targets, and "0" for products determined not to be trading targets.

Next, the registration process in this embodiment will be described. Figure 8 is a flowchart showing the flow of the registration process performed by the control unit 10 of the mobile device 1. Processes that are the same as those in the first embodiment are assigned the same reference numerals. Also, as in the first embodiment, it is assumed that the shopping app is running on the mobile device 1.

The processing of steps S1 to S4 is the same as in the first embodiment. In the processing of step S4, if the image does not show customer C visually looking at the display unit 11 (N in step S4), the registration unit 1004 performs transaction non-eligibility registration (step S11). Transaction non-eligibility registration is a process of storing the product information of the product associated with the product code acquired by the acquisition unit 1002 in association with information indicating that the product is not a transaction target.

Specifically, the registration unit 1004 reads product information corresponding to the product code acquired by the acquisition unit 1002 from the product master 162 and registers it in the product information unit 1031. At this time, the registration unit 1004 registers "0" in the transaction target flag associated with the product information, indicating that the product is not a transaction target. After registering the product as not a transaction target in step S11, the control unit 10 returns to the processing of step S1.

If, in the processing of step S4, the image shows customer C looking at the display unit 11 (Y in step S4), the control unit 10 executes the processing of steps S6 to S8 in the same manner as in the first embodiment.

When an instruction to send transaction information is issued in step S8 (Y in step S8), the display control unit 1006 displays product information for non-transaction eligible items on the display unit 11 (step S12). Displaying product information for non-transaction eligible items displays product information for items stored in the product information unit 1031 whose transaction eligible flag is registered as "0." At this time, the display control unit 1006 may display a message in addition to the product information. For example, the display control unit 1006 may display a message encouraging product registration for unregistered items, such as "If you wish to purchase the displayed item, please scan the barcode again."

The control unit 10 then determines whether the reception unit 1001 has received confirmation input from the operation unit 12 (step S13). The confirmation input is input by customer C confirming the display of a product that is not eligible for a transaction and inputting information indicating that they will not purchase the product related to the displayed product information. In other words, the confirmation input is inputting information declaring that customer C will not purchase a product that is registered in the product information unit 1031 as not eligible for a transaction. The confirmation input may be input by any method, but may be input by, for example, operating a confirmation button (operator) that is displayed together with the product information in the display of a product that is not eligible for a transaction.

When the reception unit 1001 receives confirmation input from the operation unit 12 (Y in step S13), the transmission/reception unit 1005 reads out the information stored in the product information unit 1031 for which the transaction target flag is registered as "1" and sends the transaction information including that information to the checkout device 2 (step S9). In other words, the transmission/reception unit 1005 sends information about the products registered in the product information unit 1031 as transaction targets to the checkout device 2. The control unit 10 then ends the registration process.

If the reception unit 1001 does not receive confirmation input during the processing of step S13 (N in step S13), the control unit 10 determines whether the reception unit 1001 has received correction input (step S14). Correction input is input when customer C checks a product display that is not subject to a transaction and purchases the product related to the displayed product information, and involves inputting information that identifies the product to be purchased. In other words, correction input is input of information that identifies the product that customer C will purchase from among the products registered in the product information unit 1031 as not subject to a transaction. The method for inputting correction is not critical, but it may be performed, for example, by operating a purchase button (operator) that is displayed in association with the product information in the product display that is not subject to a transaction.

If no corrections are made (N in step S14), the control unit 10 returns to the processing of step S13. If corrections are made (Y in step S14), the registration unit 1004 performs additional product registration for the product that has been corrected as a product to be purchased (step S15). Specifically, the registration unit 1004 rewrites the transaction target flag in the product information unit 1031 for the product designated as a product to be purchased from "0" to "1." Products whose transaction target flags have been rewritten to "1" are registered as products to be traded. In other words, the products are registered. Once the additional product registration has been performed, the control unit 10 returns to the processing of step S12.

As with the first embodiment, the above registration process makes it possible to prevent transactions at an incorrect transaction price for products purchased by customer C. Furthermore, before sending transaction information to the checkout device 2, the mobile terminal 1 displays product information for products associated with product codes acquired by the acquisition unit 1002 that are determined not to be eligible for the transaction. The mobile terminal 1 then accepts product registration for the products for which product information is displayed. This allows customer C to confirm that the product they are attempting to purchase is not registered and to register the product before paying for it. Therefore, this also makes it possible to prevent transactions at an incorrect transaction price.

As described above, the mobile terminal 1 in each embodiment is an information processing device operated by a customer C moving around a store, and includes a first acquisition unit (acquisition unit 1002) that acquires product identification information (product code) that identifies a product, a second acquisition unit (acquisition unit 1002) that acquires a customer image, which is an image of customer C taking a product image for recognizing the product identification information, a registration unit 1004 that registers product information of a product identified by the product identification information acquired by the first acquisition unit as a transaction subject, and a determination unit 1003 that determines whether to permit registration by the registration unit 1004 based on the customer image acquired by the second acquisition unit.

This allows the mobile terminal 1 to prevent customer C from registering products that he or she does not intend, and prevents transactions from being made at an incorrect transaction amount.

In addition, the mobile terminal 1 in each embodiment further includes a display unit 11 that displays product information for a product that the determination unit 1003 has denied from being registered by the registration unit 1004.

As a result, if the judgment unit 1003 determines that the product that customer C is attempting to purchase is not eligible for the transaction, customer C will be able to recognize this. This allows customer C to re-register the displayed product, further preventing transactions at an incorrect transaction price.

Furthermore, the mobile terminal 1 in each embodiment further includes a first imaging unit (outer camera 13) that captures product images and a second imaging unit (inner camera 14) that captures customer images, and the display unit 11 displays the product images captured by the first imaging unit.

This allows the mobile terminal 1 to provide good operability when customer C reads product codes.

In addition, in each embodiment, the determination unit 1003 of the mobile terminal 1 determines to permit registration by the registration unit 1004 if the customer image acquired by the second acquisition unit (acquisition unit 1002) is an image of customer C looking at the display unit 11.

This allows the mobile terminal 1 to easily determine whether or not the product associated with the product code acquired by the acquisition unit 1002 is a product to be traded. Specifically, by noting that when customer C reads the product code, customer C visually checks the display unit 11 displaying the product image, the determination unit 1003 can easily determine whether or not the product is a product to be traded. More specifically, the determination unit 1003 can determine whether or not to permit registration by the registration unit 1004 by utilizing a configuration that improves operability when customer C reads the product code, namely the display unit 11 displaying the product image.

In addition, the determination unit 1003 of the mobile terminal 1 in each embodiment recognizes the orientation of customer C's face in the customer image.

This allows the mobile terminal 1 to use known image processing technology to determine whether the product associated with the product code acquired by the acquisition unit 1002 is a product eligible for transaction, making this determination easier.

In the above embodiment, the control program executed by the mobile terminal 1 may be configured to be recorded on a computer-readable recording medium such as a CD-ROM and provided. The control program executed by the mobile terminal 1 in the above embodiment may also be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network, or may even be configured to be provided via a network such as the Internet.

The above describes an embodiment of the present invention, but this embodiment is presented as an example and is not intended to limit the scope of the invention. This embodiment can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention.

1. Mobile terminal (information processing device)
11 Display unit 13 Outer camera (first imaging unit)
14 In-camera (second imaging unit)
1002 Acquisition unit (first acquisition unit, second acquisition unit)
1003 Determination unit 1004 Registration unit

Japanese Patent Application Laid-Open No. 2021-124894

Claims (6)

An information processing device operated by a customer moving around a store,
a first acquisition unit that acquires product identification information that identifies a product;
a second acquisition unit that acquires a customer image that is an image of a customer;
a registration unit that registers product information of a product identified by the product identification information acquired by the first acquisition unit as a trading object;
a determination unit that determines whether or not to permit registration by the registration unit based on the customer image acquired by the second acquisition unit;
An information processing device comprising: a display unit that displays product information of a product that has been denied registration by the registration unit by the determination unit,
The information processing device according to claim 1 . a first imaging unit that captures a commodity image for recognizing the commodity identification information;
a second imaging unit that captures the customer image,
The display unit displays the product image captured by the first imaging unit.
The information processing device according to claim 2 . the determination unit determines to permit registration by the registration unit when the customer image acquired by the second acquisition unit is an image of the customer looking at the display unit.
The information processing device according to claim 3 . the determination unit recognizes the orientation of the customer's face in the customer image;
The information processing device according to claim 4 . A program for controlling, by a computer, an information processing device operated by a customer moving around a store,
The computer
a first acquisition unit that acquires product identification information that identifies a product;
a second acquisition unit that acquires a customer image that is an image of a customer;
a registration unit that registers product information of a product identified by the product identification information acquired by the first acquisition unit as a trading object;
a determination unit that determines whether or not to permit registration by the registration unit based on the customer image acquired by the second acquisition unit;
A program that makes it work.