JP6458239B1 - Image recognition system - Google Patents

Abstract

【Task】
An object of the present invention is to provide an image recognition system for specifying a product displayed in a store or the like.
[Solution]
A product tag specifying processing unit for specifying a product tag area in image information showing a display shelf for displaying a product, and a product for specifying information described in the product tag by performing OCR recognition processing in the specified product tag area And an in-tag information specifying processing unit, the product tag specifying processing unit specifies a product tag region using the brightness information of the image information, and the in-product tag information specifying processing unit specifies the specified product tag region. The box is specified by binarizing and labeling processing is performed, among the identified boxes, the boxes satisfying the predetermined condition are identified as blocks, the OCR recognition processing is executed on the identified blocks, and the OCR recognition is performed. This is an image recognition system that specifies the tax-excluded price and tax-included price by executing the logical consistency judgment process for the two price information specified by .
[Selection] Figure 1

Description

  The present invention relates to an image recognition system for specifying a product displayed in a store or the like.

  In the retail industry, it is known that the display status of products affects the sales of products. For this reason, as a product manufacturer and sales company, it is possible to connect to the development strategy and sales strategy of their own products by grasping what products of their own or other companies are displayed in the store.

  On the other hand, in order to achieve this, it is important to accurately identify the products displayed in stores. Therefore, it is conceivable that a store display shelf is photographed, and a product displayed on the basis of the image information is manually specified by a human. In this case, the product can be specified almost accurately. However, in order to keep track of the display status of products, it is necessary to monitor the display status at regular intervals. Humans specify products every time from image information of store shelves. Is burdensome and inefficient.

  Therefore, it is required to automatically identify the products displayed on the display information of the store shelves and grasp the display status of the products. For example, there is a method of using an image recognition technique for an image obtained by photographing a display shelf of a store based on a sample image for each product. As these conventional techniques, for example, there is a system that manages the display status of products using techniques as shown in Patent Documents 1 to 3 below.

JP-A-5-342230 JP-A-5-334409 International Publication 2012/029548

  The invention of Patent Document 1 is a system that supports a person who has no knowledge on which display shelf a product should be displayed. For this reason, it is impossible to grasp the products that are actually displayed. Patent Document 2 is a system that supports the input of product images in a shelf allocation support system that supports the display of products. However, the system of Patent Document 2 only supports the input of product images when using the shelf allocation support system, and even if this system is used, the products actually displayed cannot be grasped. .

  Patent Document 3 is an invention that, when there is a vacant space in a display shelf, specifies a product to be displayed in the vacant space, or notifies the user when a product to be displayed in the display shelf is misplaced. Even in this invention, the image matching process between the product and the product displayed on the display shelf is performed to identify the product displayed on the display shelf, but the reality is that the recognition accuracy is low.

  When using an image recognition technique based on a sample image of a product for an image obtained by photographing a display shelf, the accuracy and processing load of the image recognition become a problem. For example, in a store where display shelves are installed, the illumination state is not uniform, and if the difference from the illumination of the sample image is large, the accuracy of image recognition decreases. For this reason, it may not be easy to specify the products displayed on the display shelf from the captured image. In addition, since a large processing load is generated in image recognition, a great amount of resources are required. As a result, invested capital increases.

  The inventor pays attention to the product tag for displaying the price of the product as one method for identifying the product shown in the image of the display shelf, and accurately displays the displayed product. Invented an image recognition system that can be specified. In addition, an image recognition system that can accurately identify the displayed product by recognizing the product on the display shelf, recognizing the information on the product tag, and collating them. Invented.

The first invention is described in the product tag by performing a OCR recognition process in the product tag identification processing unit that identifies the product tag region in the image information in which the display shelf displaying the product is reflected, and the identified product tag region A product tag information specifying processing unit for specifying the information that has been processed, wherein the product tag specifying processing unit specifies a product tag region using lightness information of the image information, and the product tag information The identification processing unit binarizes the identified product tag area and identifies a box by performing a labeling process, identifies a box satisfying a predetermined condition among the identified boxes as a block, and identifies the identified block the running OCR recognition process, the two price information identified by the OCR recognition process by executing the logical consistency of the determination process with respect to, tax To determine the rating and the price including tax, is an image recognition system.

  By using the image recognition system of the present invention, the information written on the product tag can be recognized with high accuracy, so that the product displayed correspondingly can be specified with high accuracy.

According to a second aspect of the present invention, a product tag identification processing unit for identifying a product tag area in image information showing a display shelf for displaying a product, and an OCR recognition process on a part or all of the identified product tag area And an in-product tag information specifying processing unit for specifying information described in the product tag in the product tag specifying processing unit, the rise of a histogram obtained by integrating the brightness information of the image information in the horizontal direction, By specifying the falling edge, the upper side position and the lower side position of the product tag are specified, and the rising edge and falling edge of the histogram obtained by integrating the brightness information of the image information in the vertical direction are specified, and within a predetermined distance from the rising edge. This is an image recognition system that identifies the position of the left side and the right side of a product tag with a pair of falling edges .

As in the first aspect of the invention, by using the image recognition system of the present invention, the information written on the product tag can be recognized with high accuracy, so that the product displayed correspondingly can be specified with high accuracy. it can.
In order to accurately recognize the information written on the product tag, it is necessary to specify the product tag area as accurately as possible. Generally, since the color of the ground of the product tag is mostly white, when the display shelf is photographed, the product tag portion is bright. Focusing on this point, as in these inventions, the product tag region can be specified with high accuracy by integrating the brightness information of the image information in the vertical and horizontal directions.

According to a third aspect of the present invention, a product tag identification processing unit that identifies a product tag area in image information in which display shelves displaying products are shown, and an OCR recognition process is performed on a part or all of the identified product tag area. And an in-product tag information specifying processing unit for specifying information described in the product tag in the product tag, wherein the product tag specifying processing unit has a rising edge of a histogram obtained by integrating the brightness information of the image information in the vertical direction. Of the rises that could not form a pair, if there is no tag area within a predetermined distance on the right side of the rise, it is identified as a product tag area, and among the falling edges of the histogram obtained by integrating the brightness information of the image information in the vertical direction, identifying a product tag area if there is no tag area within distance predetermined to the left about the fall could not be formed a pair, image The recognition system.

As in the first aspect of the invention, by using the image recognition system of the present invention, the information written on the product tag can be recognized with high accuracy, so that the product displayed correspondingly can be specified with high accuracy. it can.
In some cases, the product tag is attached with a part of it overlapped. In this case, the histogram obtained by integrating the brightness information continuously rises or falls. Therefore, by executing the processing of the present invention, it is possible to specify the product tag region for the product tag attached in an overlapping manner.

According to a fourth aspect of the present invention, a product tag specifying processing unit for specifying a product tag area in image information showing a display shelf for displaying a product, and an OCR recognition process in the specified product tag area are described on the product tag. A product tag internal information specification processing unit for specifying the specified information, and the product tag internal information specification processing unit executes binarization processing in the specified product tag region, and performs the binarization By performing a labeling process on the processed image information, consecutive pixels are grouped to specify a box, and among the specified boxes, adjacent boxes having a predetermined similarity are merged and specified as a block. The image recognition system executes the OCR recognition process by estimating the block having the highest height among the identified blocks as a price region .

As with the first invention, by using the image recognition system of these inventions, the information written on the product tag can be recognized with high accuracy, so that the product displayed correspondingly can be specified with high accuracy. Can do.
In order to recognize the information written on the product tag, OCR recognition processing may be executed for all of the product tag areas. However, in that case, a lot of noise or the like is detected, resulting in deterioration of accuracy. Connected. Also, the processing time increases. Therefore, it is preferable to execute the OCR recognition process only on a part of the product tag area. The recognition accuracy can be improved by executing the OCR recognition process on the block for which the process of the present invention is specified. In addition, since the price is most important for the purchaser of the product, the block with the highest height is often the price area. Therefore, the price can be recognized with high accuracy by estimating the block as a price region and executing the OCR recognition process.

According to a fifth aspect of the present invention, a product tag specifying processing unit for specifying a product tag area in image information showing a display shelf for displaying a product and an OCR recognition process in the specified product tag area are described on the product tag. A product tag internal information specifying processing unit for specifying the information that has been processed, and the product tag internal information specifying processing unit, as the price information recognized as a tax-excluded price by the OCR recognition processing, This is an image recognition system for identifying the price information and the tax-inclusive price by executing the judgment process of the magnitude and / or tax rate consistency of the recognized price information .

As with the first invention, by using the image recognition system of these inventions, the information written on the product tag can be recognized with high accuracy, so that the product displayed correspondingly can be specified with high accuracy. Can do.
In general, the product tag includes a tax-excluded price and a tax-included price. Therefore, the accuracy can be further improved by determining whether the logical consistency of the prices recognized as the tax-excluded price and the tax-included price is established.

According to a sixth aspect of the present invention, a product tag specifying processing unit for specifying a product tag area in image information showing a display shelf for displaying a product, and performing OCR recognition processing on a part or all of the specified product tag area In-product tag information specifying processing unit for specifying information described in the product tag in the product tag, the in-product tag information specifying processing unit performs logical processing on the two price information specified by the OCR recognition processing . This is an image recognition system that identifies the tax-excluded price and the tax-included price by executing the process of determining the physical consistency.

As with the first invention, by using the image recognition system of these inventions, the information written on the product tag can be recognized with high accuracy, so that the product displayed correspondingly can be specified with high accuracy. Can do.

According to a ninth aspect of the present invention, there is provided a product tag recognition processing unit for specifying product identification information in a product tag area in image information in which display shelves displaying products are shown, and product identification information of products displayed in the image information. A recognition result for comparing the identified display product recognition processing unit, the product identification information in the specified product tag area, and the product identification information of the specified displayed product and checking whether the recognition results match. A recognition processing unit, and the recognition result verification processing unit takes a group of sections in which faces having high similarity are lined up by a recognition process in the display product recognition processing unit, and each group in each shelf An image recognition system for associating product tag recognition results by the product tag recognition processing unit with each other.

According to the present invention, the result of recognizing the product shown on the display shelf can be collated with the recognition result of the information written on the product tag. As a result, it is possible to accurately identify the displayed product.
Further, by using the present invention, it is possible to associate the recognition result of the displayed product with the recognition result of the product tag.

In the above-described invention, the image recognition system further includes a photographed image information alignment processing unit that performs a keystone correction process on the image information obtained by photographing the display shelf.置化processing unit includes a first image information, based on the correspondence between the second image information, the vertices used in the keystone correction process in the first image information, projected to the second image information In addition, it can be configured as an image recognition system that executes a trapezoidal correction process of the second image information based on the projected vertex.

Photographed image information, it is also often not be the image information taken from the front. When processing is performed from the captured image information itself, the accuracy is low. For this reason, it is preferable to perform trapezoidal correction processing to make it in-place. Therefore, the person in charge has to specify the vertex that is the standard for the trapezoidal correction process, which is a heavy burden. By using the present invention, it is possible to automatically specify a vertex as a reference for trapezoidal correction processing, and to reduce the work load on the person in charge.

In the above-described invention, the image recognition system further includes a position specifying processing unit that specifies a product tag placement region in the image information that has been subjected to the trapezoidal correction process in the captured image information normalization processing unit, and the specified It can be configured as an image recognition system having a product tag placement area alignment processing unit that performs trapezoidal correction processing in the product tag placement area.

  Whereas the display shelf surface is vertical, the product tag surface is often mounted upward (towards customers) than the vertical surface. This is to make it easier for the customer to visually recognize the product tag. Therefore, the product tag area can be specified with higher accuracy by performing the normalization processing on the product tag arrangement area. This is because, for example, the rise and fall of the histogram in the lightness information (information on the amount of light when the image information is decomposed into two-dimensional information of hue, lightness, and saturation) is steep. Further, according to the present invention, it is possible to improve the accuracy of the OCR recognition process in the product tag area.

  In the above-described invention, the product tag placement area alignment processing unit performs edge detection in the product tag placement area, detects a straight line group that satisfies a predetermined condition of a vertical angle, and detects the detected straight line group. Of these, at least two straight lines can be specified, and a trapezoidal correction process can be performed using the two specified straight lines.

  By executing the process of the present invention, the inclination angle for use in the trapezoid correction process can be specified, so that the trapezoid correction process for the product tag placement area can be executed.

The image recognition system of the first invention can be realized by reading the image recognition program of the present invention into a computer and executing it. That is, the product tag specifying processing unit for specifying the product tag area in the image information in which the display shelf displaying the product is displayed, and the OCR recognition process is performed on the specified product tag area, and the product tag is indicated. An image recognition program that functions as an in-product tag information specifying processing unit that specifies information, wherein the product tag specifying processing unit specifies a product tag area using lightness information of the image information, and includes in the product tag The information identification processing unit identifies the box by binarizing the identified product tag area and performing a labeling process, identifies the boxes satisfying a predetermined condition among the identified boxes as a block, and identifies the identified executing the OCR recognition process for a block, the two price information identified by the OCR recognition process, logical By executing the determining process of compatibility, specifying the tax price and price including tax, an image recognition program.

The image recognition system of the second invention can be realized by reading the image recognition program of the present invention into a computer and executing it. That is, the computer uses a product tag specifying processing unit for specifying a product tag area in image information showing a display shelf displaying products, and performs OCR recognition processing on a part or all of the specified product tag area. An image recognition program for functioning as an in-product tag information specifying processing unit for specifying information described in a product tag by performing integration of brightness information of the image information in a horizontal direction. By specifying the rising and falling edges of the histogram, the top and bottom positions of the product tag are specified, and the rising and falling edges of the histogram obtained by integrating the brightness information of the image information in the vertical direction are specified. left position of the product tag falling as a pair is within a predetermined distance from, it identifies the right position, an image recognition program That.
The image recognition system according to the third aspect of the invention can be realized by causing the computer to read and execute the image recognition program of the present invention. That is, the computer uses a product tag specifying processing unit for specifying a product tag area in image information showing a display shelf displaying products, and performs OCR recognition processing on a part or all of the specified product tag area. An image recognition program that functions as an in-product tag information identification processing unit that identifies information described in a product tag by performing integration of brightness information of the image information in a vertical direction. Among the rising edges of the histogram, the rising edge that could not form a pair is identified as a product tag area if there is no tag area within a predetermined distance on the right side of the rising edge, and the brightness information of the image information is integrated vertically. Among the falling edges, the falling edge that could not form a pair is within a predetermined distance on the left side. Identifying a product tag area Without area, an image recognition program.

The image recognition system according to the fourth aspect of the invention can be realized by causing the computer to read and execute the image recognition program of the present invention. That is, the product tag specifying processing unit for specifying the product tag region in the image information in which the display shelf displaying the product is shown, and the product tag by performing OCR recognition processing in the specified product tag region An image recognition program that functions as an in-product tag information specifying processing unit that specifies information described in the item, wherein the in-product tag information specifying processing unit executes binarization processing in the specified product tag area , By performing a labeling process on the binarized image information, a box is specified by grouping consecutive pixels, and adjacent boxes having a predetermined similarity are merged among the specified boxes. Block as a block, and among the identified blocks, the block with the highest height is estimated as the price range and OCR recognition is performed. Processing is executed, an image recognition program.
The image recognition system according to the fifth aspect of the invention can be realized by causing the computer to read and execute the image recognition program of the present invention. That is, the product tag specifying processing unit for specifying the product tag region in the image information in which the display shelf displaying the product is shown, and the product tag by performing OCR recognition processing in the specified product tag region An image recognition program that functions as an in-product tag information specifying processing unit that specifies information described in the item, wherein the in-product tag information specifying processing unit recognizes price information as a tax-excluded price by the OCR recognition processing. The image recognition program identifies the price and tax-excluded price information by executing the process of determining the size of the price and / or the consistency of the tax rate for the price information recognized as the tax-included price .

The image recognition system according to the sixth aspect of the present invention can be realized by causing the computer to read and execute the image recognition program of the present invention. That is, the computer uses a product tag specifying processing unit for specifying a product tag area in image information showing a display shelf displaying products, and performs OCR recognition processing on a part or all of the specified product tag area. An image recognition program that functions as an in-product tag information specifying processing unit that specifies information described in a product tag by performing the two operations specified by the OCR recognition processing . This is an image recognition program that identifies the price without tax and the price with tax by executing logical consistency judgment processing for price information.

The image recognition system according to the ninth aspect of the present invention can be realized by causing the computer to read and execute the image recognition program of the present invention. That is, the computer uses a product tag recognition processing unit for specifying product identification information in the product tag area in the image information in which the display shelf displaying the product is shown, and the product identification information of the product displayed in the image information A display product recognition processing unit for identifying the product, a product identification information in the region of the identified product tag and a product identification information of the identified product to be identified, and a recognition for verifying whether the recognition results match An image recognition program that functions as a result collation processing unit , wherein the recognition result collation processing unit is a group of sections in which faces having high similarity are arranged by a recognition process in the display merchandise recognition processing unit. An image recognition program for associating a product tag recognition result by the product tag recognition processing unit with each group in a stage .

  By using the image recognition system of the present invention, the products displayed on the display shelf can be accurately identified.

It is a block diagram which shows typically an example of the system configuration | structure of the image recognition system of this invention. It is a block diagram which shows typically an example of the hardware constitutions of the computer used with the image recognition system of this invention. It is a flowchart which shows an example of the whole processing process in the image recognition system of this invention. It is a flowchart which shows an example of the process of the regularization process of a goods tag arrangement | positioning area | region. It is a flowchart which shows an example of the process of the specific process of merchandise tag image information. It is a flowchart which shows an example of the process of the specific process of the information in goods tag. It is a figure which shows an example of the in-place image information which performed the in-place processing to the picked-up image information. It is a figure which shows another example of the in-place image information which performed the in-place processing to the picked-up image information. It is a figure which shows an example of the state which designated the goods tag arrangement | positioning area | region to the in-place image information with respect to the in-place image information of FIG. It is a figure which shows an example of the state which designated the product tag arrangement | positioning area | region to the regular image information with respect to the regular image information of FIG. It is a figure which shows an example of the image information which performed the normalization process with respect to the goods tag arrangement | positioning area | region in FIG. It is a figure which shows the state which detected the straight line of the perpendicular direction used as the reference | standard of trapezoid correction | amendment processing in the product tag arrangement | positioning area regularization process part. It is a figure which shows typically the process which specifies the upper side position and lower side position of a product tag area | region in a product tag specific process part. It is a figure which shows typically the process which specifies the position of the horizontal direction of the product tag area | region in a product tag specific process part. It is a figure which shows typically the state which pinpointed the product tag area | region from the image information of the product tag arrangement | positioning area | region which was placed in FIG. It is a figure which shows typically the image information which binarized the product tag area | region in the process in the information identification process part in a product tag. It is a figure which shows typically the state which produced | generated the box in the process in the information identification process part in goods tag. It is a figure which shows typically the state which produced | generated the block in the process in the information specific process part in a product tag. It is a figure which shows an example of the product name and edit distance with the shortest edit distance among the product names described in the product tag and the product name memorize | stored in the product dictionary of the result of OCR recognition. It is a figure which shows an example of the correspondence of the edit distance which may be decided for every number of characters of a final candidate. It is a figure which shows an example of picked-up image information typically. It is a figure which shows an example of picked-up image information typically. FIG. 10 is a block diagram schematically illustrating an example of a system configuration of an image recognition system according to a second embodiment. It is a block diagram which shows typically an example of a structure of a goods tag recognition process part. It is a block diagram which shows typically an example of a structure of a display goods recognition process part. 12 is a flowchart illustrating an example of an overall processing process in the image recognition system according to the second embodiment. It is a figure which shows typically the Nth process in a merchandise identification information specific process part. An example of the sample image information memorize | stored in a sample image information storage part is shown. It is a figure which shows the state by which the shelf level area | region and the goods tag arrangement | positioning area were specified. It is a figure which shows the state by which the shelf level area | region and the goods tag arrangement | positioning area were specified. An example of a state in which a feature amount collection area is set for N-1th captured image information is shown. It is a figure which shows an example of the state which set the feature-value collection area | region with respect to the captured image information of the Nth time. It is a figure which shows the relationship of the pair of the N-th feature-value collection area | region and the N-th feature-value collection area | region. It is a figure which shows the state which projects the shelf position C in the N-1th picked-up image information as the shelf position D in the N-th picked-up image information by the function F. It is a figure which shows an example of picked-up image information. It is a figure which shows the state which set the feature-value collection area | region to the N-1th picked-up image information. It is a figure which shows the state which set the feature-value collection area | region to the N-th picked-up image information. It is a figure which shows an example in the state which specified shelf position D1 thru | or D4 in the Nth picked-up image information.

  An example of the system configuration of the image recognition system 1 of the present invention is shown in FIG. The image recognition system 1 uses a management terminal 2 and a captured image information input terminal 4.

  The management terminal 2 is a computer used by an organization such as a company that operates the image recognition system 1. The photographed image information input terminal 4 is a terminal for inputting image information obtained by photographing a display shelf in a store.

  The management terminal 2 and the captured image information input terminal 4 in the image recognition system 1 are realized using a computer. FIG. 2 schematically shows an example of the hardware configuration of the computer. The computer is capable of inputting information, such as an arithmetic device 70 such as a CPU for executing arithmetic processing of a program, a storage device 71 such as a RAM or a hard disk for storing information, a display device 72 such as a display for displaying information, and the like. It has an input device 73 such as a keyboard and a mouse, and a communication device 74 that transmits and receives processing results of the arithmetic device 70 and information stored in the storage device 71 via a network such as the Internet or a LAN.

  When the computer includes a touch panel display, the display device 72 and the input device 73 may be configured integrally. The touch panel display is often used in, for example, a portable communication terminal such as a tablet computer or a smartphone, but is not limited thereto.

  The touch panel display is an apparatus in which the functions of the display device 72 and the input device 73 are integrated in that input can be performed directly on the display using a predetermined input device (such as a touch panel pen) or a finger.

  The photographed image information input terminal 4 may include a photographing device such as a camera in addition to the above devices. As the photographed image information input terminal 4, a portable communication terminal such as a mobile phone, a smartphone, or a tablet computer can be used.

  Each means in the present invention is only logically distinguished in function, and may be physically or virtually the same area. The processing order of each means of the present invention can be changed as appropriate. Further, a part of the processing may be omitted. For example, the in-place processing can be omitted. In this case, it is possible to execute processing for image information that has not been subjected to normalization processing.

  The management terminal 2 in the image recognition system 1 can transmit / receive information to / from the captured image information input terminal 4 via the network.

  The image recognition system 1 includes a photographed image information input reception processing unit 20, a photographed image information storage unit 21, a photographed image information regularization processing unit 22, a position specifying processing unit 23, and a product tag arrangement area extraction processing unit. 24, a product tag arrangement area regularization processing unit 25, a product tag identification processing unit 26, and an in-product tag information identification processing unit 27.

  The photographed image information input acceptance processing unit 20 accepts input of image information (photographed image information) of a store display shelf photographed by the photographed image information input terminal 4 and stores it in a photographed image information storage unit 21 described later. From the photographed image information input terminal 4, in addition to the photographed image information, it is preferable to accept input together with photographing date and time, store identification information such as a store name, image information identification information for identifying image information, and the like.

  The photographed image information storage unit 21 stores the photographed image information received by the photographed image information input acceptance processing unit 20, the photographing date and time, the store identification information, the image information identification information, and the like in association with each other. The captured image information only needs to be image information to be subjected to trapezoidal correction processing, and includes image information that is synthesized as a single image information when a single display shelf is captured in multiple sheets. . Also, the image information after the distortion correction processing is executed is included in the captured image information.

  The captured image information normalization processing unit 22 generates correct image information obtained by performing trapezoidal correction processing on the captured image information stored in the captured image information storage unit 21 so as to be aligned. The trapezoidal correction process is a correction process performed so that the shelf of the display shelf shown in the photographed image information is horizontal and the product tag for the product displayed there is vertical.

  The trapezoid correction process executed by the captured image information normalization processing unit 22 receives input of designation of four vertices in the captured image information, and executes the trapezoid correction process using each vertex. The four vertices that accept the designation may be the four vertices of the shelf of the display shelf or the four vertices of the shelf position of the display shelf. Further, it may be the four vertices of a group of two or three shelves. Any four points can be designated as the four vertices.

  FIG. 7 and FIG. 8 show an example of photographed image information (orientated image information) subjected to the in-place processing. FIG. 7 shows in-place image information in which beverage cans such as beer are displayed on the upper and lower shelves as display shelves. FIG. 8 is a hanging shelf that displays a product such as a toothbrush in a manner in which the product is suspended from the display shelf, and shows the in-place image information in which the product is displayed in two upper and lower stages.

  The position specifying processing unit 23 is an area (product tag arrangement) in which the product tag may be attached in the in-place image information obtained by performing the trapezoidal correction process on the captured image information in the captured image information alignment processing unit 22. Area). In other words, the display shelf is shown in the photographed image information and the in-place image information, but there is a possibility that the shelf area where the product is displayed and the product tag for the product displayed there are attached to the display shelf. There are some product tag placement areas. Therefore, the product tag placement area is specified from the in-place image information. For specifying the product tag placement area, the operator of the management terminal 2 may manually specify the product tag placement area, which may be received by the position specification processing unit 23, or the product tag that has been manually input for the first time. Based on the information of the arrangement area, the product tag arrangement area may be automatically specified after the second time.

  FIG. 9 schematically shows a state where an input for specifying a product tag arrangement area is received for the in-place image information of FIG. FIG. 10 schematically shows a state in which an input for specifying a product tag placement area is received for the in-place image information of FIG.

  The product tag arrangement area extraction processing unit 24 extracts the image information of the product tag arrangement area specified by the position specification processing unit 23 as the product tag arrangement area image information. The product tag arrangement area cutout processing unit 24 may actually cut out as image information, or may cut out virtually without cutting out as image information. Virtually cutting out image information means that processing is executed with a specified region, for example, a range of a product tag placement region as a processing target.

  The product tag placement area corrector processing unit 25 executes a trapezoidal correction process for normalizing the product tag placement region image information cut out by the product tag placement region cutout processor 24. While the display shelf surface is vertical, the product tag surface is often facing upwards from the vertical surface so that it can be easily seen by customers. Therefore, the recognition accuracy is improved by arranging the image information in the product tag placement area. FIG. 11 shows an example of the image information of the product tag arrangement area that has been placed in place with respect to the image information of the product tag arrangement area of FIG. FIG. 11A is the image information of the product tag arrangement area in which the image information of the product tag arrangement area of the upper shelf in FIG. 9 is arranged, and FIG. 11B is the image information of the lower shelf in FIG. It is the image information of the product tag arrangement area obtained by arranging the image information of the product tag arrangement area.

  The product tag placement area regularization processing unit 25 performs the following processing to regularize the image information of the product tag placement area. That is, edge detection is performed in the image information of the product tag placement region, and a vertical outline (for example, 70 degrees to 110 degrees, vertical (90 degrees), for example, 70 degrees to 110 degrees at a position near the left and right ends. ) To an outline within an angle within a predetermined range. In addition, although it is preferable to extract the contour line of the location near both the left and right ends, the present invention is not limited to this. In the case of the image information of the product tag arrangement area in FIG. 9, L1 to L4 are specified as shown in FIG. 12A shows the product tag placement area on the upper shelf in FIG. 9, and FIG. 12B shows the product tag placement area on the lower shelf in FIG. Note that the contour lines L1 to L4 to be specified are not actually drawn on the image information of the product tag arrangement area. Then, trapezoidal correction processing is performed on the image information of the product tag placement area so that L1 and L2 in FIG. 12A and L3 and L4 in FIG. By executing such processing, the image information of the product tag placement area is placed in the correct position, and the image information of the placed product tag placement area shown in FIG. 11 can be obtained. In addition, since the accuracy of the product tag identification processing unit 26 and the in-product tag information identification processing unit 27 can be improved by executing the processing of the product tag arrangement area alignment processing unit 25, the processing is executed. Although preferred, it can be omitted. In this case, the product tag identification processing unit 26 and the in-product tag information identification processing unit 27 are executed for the product tag arrangement area cut out by the product tag arrangement area extraction processing unit 24.

  The product tag specification processing unit 26 specifies the region (product tag region) of each product tag from the product tag placement region image information that has been placed in a normal position. Two types of methods can be mainly used for the product tag area specifying process. The first method is a method of specifying a product tag area based on the contour line, and the second method is to match the overall characteristics such as the overall light and dark distribution with the image information of the template of the product tag. In this way, the product tag area is specified. Methods other than the first method and the second method can also be used.

  The first method uses the fact that the background color of the product tag is much white (brighter than the background). That is, first, a histogram is generated by integrating the brightness information of the image information in the horizontal direction out of the image information of the product tag placement area that has been placed in a normal position. Then, the rising and falling positions of the histogram are specified, and the upper side position A and the lower side position B of the product tag are specified. This process is schematically shown in FIG. A rising edge is a point that increases steeply (over a predetermined ratio) from black to white in the histogram, and a falling edge is a point that sharply decreases from white to black in the histogram. .

  Then, between the image information of the placed product tag placement area, a portion between the upper side position A and the lower side position B is cut out, and a histogram in which the brightness information of the placed product tag placement area image information is vertically integrated is generated. To do. Then, the rising and falling positions of the histogram are specified, and the product tag area is specified with the rising and the falling in the predetermined distance range on the right as a pair, and the left side position and the right side position respectively. If the product tag region does not exist within a predetermined distance, the product tag region is specified on the right side of the rise when the pair could not be formed and on the left side of the fall where the pair could not be formed. FIG. 14 schematically shows this processing.

  FIG. 15 shows a state in which the product tag region is specified from the image information of the product tag placement region in FIG. A rectangular area constituted by the upper side position A, the lower side position B, the left side position (rising edge) U, and the right side position (falling edge) D is the specified product tag area.

  The second method is so-called template matching. That is, image information of a product tag is registered in advance as a template, and the product tag region is specified by matching the template with image information of the product tag placement region that has been placed in place.

  The product tag includes a tax-excluded price, a tax-included price, product identification information (product name, etc.), manufacturer name, rating, and the like. Therefore, if product identification information such as the product name, specific numbers and characters of the price are included in the image information of the product tag that is the template, that part is also included in the image matching process, and that part is mosaicked. It is preferable to neutralize or exclude from the determination target by deleting. Neutralization means not assigning a high point or a low point for any input, and exclusion means that the part is excluded from the matching target in the image matching process.

  The in-product tag information specification processing unit 27 executes processing for specifying information described in the product tag area specified by the product tag specification processing unit 26 by OCR recognition or the like. When performing OCR recognition, it can be performed for all or part of the area specified as the product tag area. Since it is highly possible that erroneous recognition will occur due to noise or the like if it is performed on all the areas specified as the product tag areas, it is preferable to limit the target areas for OCR recognition. In this case, processing corresponding to each of the first method and the second method used in the product tag identification processing unit 26 is executed.

  When the product tag identification processing unit 26 uses the first method, binarization processing is first performed in the identified product tag area. Then, a labeling process is executed on the binarized image information. The labeling process is a process of assigning the same number (identification information) to pixels in which white or black continues in binary image information, thereby forming consecutive islands into one island (group). is there. Then, a rectangular area (box) including islands detected by the labeling process is generated, and the height, width, and baseline of the box are obtained. When generating a box, it is preferable to generate a rectangle surrounded by vertical, horizontal line segments surrounding an area labeled with the same number, but this is not restrictive. When a box is generated, islands that do not satisfy the predetermined threshold height and width are not generated as noise but are removed from the processing target as they are. For example, islands that are too small can be horizontal ruled lines or dust on the image, while islands that are too wide can be logos, which are removed as noise.

  The characters used in product tags are generally bold, such as Gothic. Therefore, even if there is a slight blur in the image information, a character group forming one character string can be detected as an island having a baseline and a height.

  The in-product tag information identification processing unit 27 merges adjacent boxes having a predetermined similarity to form a block. That is, a block is formed by merging consecutive boxes whose base line and height are within a predetermined range and whose height and width are within a certain threshold range. At this time, small boxes and the like between the blocks to be merged are collectively configured as one block. As a result, a muddy point, a semi-turbid point, a hyphen, and the like are also included in one block. A block is an area to be subjected to OCR recognition. Then, the block having the highest height among the blocks is estimated as a price area (a price area excluding tax), and OCR recognition is performed. Similarly, OCR recognition is performed for other blocks. By executing the processing as described above, it is possible to execute the character recognition processing with higher accuracy than when performing OCR recognition corresponding to a document freely laid out over a plurality of lines in the product tag area. FIG. 16 to FIG. 18 schematically show this processing. FIG. 16 shows binarized image information, and FIG. 17 shows a state in which a box (area indicated by a broken line) is generated. FIG. 18 is a diagram showing a state in which blocks (areas indicated by broken lines) are generated.

  As described above, the in-product tag information identification processing unit 27 can perform character recognition on the information described in the product tag.

  When the second method is used in the product tag identification processing unit 26, product identification information such as a tax-excluded price, a tax-included price, a manufacturer name, a product name, and a rating are described in advance in the image information of the product tag as a template. The position and size (height and width) of each character frame are set. Therefore, the image information of the corresponding part is cut out from the product tag area specified by the product tag specification processing unit 26, and the OCR recognition process is executed. At this time, the product identification information such as price, manufacturer name, product name, etc., and the character type used by the rating (for example, numbers, Roman letters, symbols, character strings, etc.) are defined as constraints, so that the OCR recognition process can be performed. Accuracy can be improved.

  Further, the in-product tag information identification processing unit 27 executes processing for confirming the consistency of the read information. As the consistency check process, it is preferable to perform two types of consistency check process by dictionary collation and logical check.

  Consistency confirmation processing by dictionary collation is executed as follows, for example. The image recognition system 1 includes a product dictionary (not shown) that stores product identification information such as product names of products that may be displayed on display shelves and corresponding code information (for example, JAN code) in association with each other. )). Then, the edit distance (Levenshtein) between the character string other than the character string read from the area indicating the price recognized by the in-product tag information identification processing unit 27 and the product identification information such as all the product names registered in the product dictionary. Find the distance. If the minimum editing distance is one of the obtained editing distances, the product identification information such as the product name of the editing distance is set as the final candidate. An allowable edit distance is determined in advance for the length of the character string of the product identification information such as the product name that is the final candidate, and the product identification information such as the product name is identified if it is within the allowable edit distance. To do. If the allowable edit distance is exceeded, the read character string is unconfirmed. If there are multiple minimum edit distances, the read character string is unconfirmed.

  Edit distance is a type of distance that indicates how different two character strings are. Specifically, one character string is transformed into the other by inserting, deleting, or replacing one character. This is the minimum number of steps required to do this. FIG. 19 shows an example of the product name written on the product tag, the result of OCR recognition, and the product name with the shortest edit distance and the edit distance among the product names stored in the product dictionary. FIG. 20 shows a table of correspondence relationships of edit distances that can be determined for each number of characters of the final candidate. In this specification, the case where processing is performed using the edit distance will be described, but a distance function obtained by performing processing for shortening the replacement distance in the calculation of the edit distance may be used. This distance function is a function for calculating a distance indicating how different two character strings are, and includes the above-described editing distance.

  After obtaining the product identification information such as the product name of the minimum editing distance, a rating dictionary (not shown) is provided separately for the character string obtained by excluding the corresponding part of the product identification information such as the product name from the OCR recognition result. Edit distance to each rating stored in (1) and edit distance to each manufacturer name stored in a manufacturer name dictionary (not shown). For example, in the example of “350 ml” in FIG. 19, the “350 ml” portion matches “350 ml” in the rated dictionary with an edit distance of 0, and the character string of the rated portion is identified. Similarly, the edit distance is obtained for the manufacturer name, and the character string at the minimum edit distance is identified as the manufacturer name. Then, remove the character strings of the rated part and the manufacturer name part from the OCR recognition result, find the character string at the shortest edit distance in the product name dictionary for the removed character string, and confirm that the edit distance is acceptable. Determine whether or not. If it is within the allowable edit distance, the product identification information such as the product name, the manufacturer name, and the rating are determined. By performing such processing, it is possible to appropriately determine even when the product identification information such as the product name includes the rating and the manufacturer name. The rating dictionary is a dictionary that stores ratings (capacity, etc.) of products that may be displayed on the display shelf. The manufacturer name dictionary is a dictionary that stores manufacturer names of products that may be displayed on the display shelf.

  When displaying the recognition result to the user for selection, the confirmed character string and the unconfirmed character string may be displayed in a manner that allows each to be identified. For example, the colors of the confirmed character string and the unconfirmed character string are separated, and information indicating confirmation or unconfirmed is attached to the confirmed character string and / or the unconfirmed character string. With respect to an undetermined character string, if there are a plurality of candidates for product identification information such as a product name having the minimum edit distance, each candidate may be displayed.

  The logical check (determination process of the logical consistency of recognized information) in the product tag internal information identification processing unit 27 can be performed as follows. For example, when two prices are read as price areas, one is a tax-excluded price and one is a tax-included price. However, if the tax-excluded price is higher than the tax-included price, they are reversed and recognized. Let In addition, if the value obtained by multiplying the tax-excluded price by the consumption tax rate does not match the tax-included price, it is assumed that either or both are misrecognized. Further, when product identification information such as a product name is recognized, it is determined whether the product or product category falls within the normal price range. Further, it may be determined whether the correspondence relationship between the product identification information such as the product name, the manufacturer name, and the rating is the same.

  As described above, by confirming the consistency of information in the product tag, it is possible to determine information included in the product tag shown in the photographed image information. The information thus determined can be output in a table format, for example.

  Next, an example of the processing process of the image recognition system 1 of the present invention will be described with reference to the flowcharts of FIGS.

  The photographed image information obtained by photographing the display shelf of the store is input from the photographed image information input terminal 4, and the input is received by the photographed image information input reception processing unit 20 of the management terminal 2 (S100). 21 and 22 show an example of photographed image information. Also, input of image information identification information such as shooting date / time, store identification information, and shooting image information is accepted. Then, the photographed image information input acceptance processing unit 20 stores the photographed image information, the photographing date and time, the store identification information, and the image information identification information of the photographed image information, which are accepted, in the photographed image information storage unit 21 in association with each other.

  When the management terminal 2 receives a predetermined operation input, the in-place image information normalization processing unit extracts the photographic image information stored in the photographic image information storage unit 21, and is a shelf that is a vertex for performing trapezoid correction processing. The input of 4 points of the position (the position of the display shelf) is received, and the keystone correction process is executed (S110). An example of the captured image information (rectified image information) that has been subjected to the trapezoidal correction process in this way is shown in FIGS.

  And the position specific process part 23 specifies a goods tag arrangement | positioning area | region by accepting predetermined operation input in the management terminal 2 with respect to in-place image information (S120). That is, the input of the product tag arrangement area in the in-place image information is accepted. FIG. 9 and FIG. 10 are views showing a state in which the product tag placement area is specified. Then, the product tag placement area cutout processing unit 24 cuts out the image information of the product tag placement area specified in S120 (S130), and the product tag placement area corrective processing unit 25 executes the trapezoidal correction process, so that the product tag placement process is performed. Alignment processing is executed for the image information in the tag placement area (S140).

  As the normalization processing for the image information in the product tag placement area, first, edge detection is performed on the image information in the product tag placement area. Then, among the detected edges, a contour line that is not less than a certain length and is within a predetermined range of angle from the vertical is identified (S200). Then, the leftmost and rightmost contour lines are identified among the identified contour lines (S210). An example of the contour line thus identified is shown in FIG. Then, a trapezoid correction process is performed on the image information of the product tag placement area so that the identified contour lines (L1, L2 in FIG. 12A, L3, L4 in FIG. 12B) are vertical lines, respectively. (S220). By executing such processing, the image information of the product tag placement area is placed in the correct position, and the image information of the placed product tag placement area shown in FIG. 11 can be obtained.

  When the normalization processing of the image information in the product tag placement area is completed in S140, the product tag identification processing unit 26 uses the first method or the second method to determine the individual product tag from the image information in the product tag placement area. An area is specified (S150).

  In the case of the first method, a histogram in which the brightness information of the image information is integrated in the horizontal direction is generated from the image information in the product tag placement area that has been placed in the normal direction (S300), and the rising and falling positions of the histogram are determined. Identify. Then, the rising edge of the identified histogram is specified as the upper side position A of the product tag, and the falling position of the histogram is specified as the lower side position B of the product tag (S310).

  Next, a histogram obtained by cutting out between the upper side position A and the lower side position B from the image information of the product tag placement area that has been placed in a normal position and integrating the brightness information of the image information in the product tag placement area that has been placed in the vertical direction. Is generated (S320).

  The rising position U and the falling position D are specified in the generated histogram, the rising position U (left side position) and the falling position D (right side position) within a predetermined distance range to the right are specified as a pair, and the product tag The area is specified (S330).

  If there is no product tag area within a predetermined distance on the right side of the rising position U where the pair could not be formed and on the left side of the falling position D where the pair could not be formed, the product tag area is specified. (S340).

  FIG. 15 shows a state in which each product tag area is specified by the processing as described above.

  When the second method is used, the product tag identification processing unit 26 performs an image matching process between the image information of the product tag of the template registered in advance and the image information of the product tag placement area that has been placed in order. By executing, the product tag area is specified.

  When the product tag identification processing unit 26 identifies the product tag area as described above, the in-product tag information identification processing unit 27 identifies information in the product tag (S160).

  The first method in the in-product tag information specifying processing unit 27 binarizes the specified product tag area to obtain binarized image information (S400). Then, the box is specified by the labeling process in the binarized image information (S410). Note that islands that do not satisfy the predetermined threshold height and width do not generate a box as noise and are directly removed from the processing target.

  Then, the height, width, and baseline of the generated box are obtained, and by identifying adjacent boxes that match the baseline and height within a predetermined range, and the height and width are within a certain threshold range, The identified boxes are merged to form a block (S420). Of the blocks, the block with the highest height is estimated as a price area (a price area excluding tax), and OCR recognition is performed (S430). If price information cannot be obtained as a result of OCR recognition (if a numeric character string cannot be recognized) (S440), the block with the next highest height is estimated as the price area (tax-excluded price area), and similarly OCR Recognize.

  In the OCR recognition of S430, the accuracy of price reading can be improved by adding character recognition that constitutes the price display such as “number” and “comma” used for the price as a constraint.

  As described above, when price information is acquired from the block estimated as the price area (tax-excluded price area) by OCR recognition (S440), blocks other than the price information acquired by estimating the price area (tax-excluded price area) in S430 are obtained. Identification (S450), and OCR recognition is performed for each identified block (S460). For this OCR recognition, it is preferable to perform two types of OCR recognition processing. That is, a normal standard OCR recognition process and an OCR recognition process in which character recognition constituting price display is added as a constraint.

  The blocks specified in S450 include a price area block (including tax price area), a product identification information such as a manufacturer name and a product name, and an information area block such as a rating. Then, two types of OCR recognition processing are executed for each block. For the block of information areas such as product identification information such as manufacturer name, product name, and rating, the standard OCR recognition process recognizes character strings, and the OCR recognition process with constraints often includes characters that contain errors. It becomes a column. In this case, it is determined whether the recognition results of the two recognition processes are different from each other by a predetermined value or more. If they are different, the character string executed in the standard OCR recognition process is used as the recognition result, and the price area (price area including tax) It is determined that the block is other than this block. On the other hand, for the price area (tax-included price area) block, the character string of price information is recognized in the standard OCR recognition process, and the character string of price information is also recognized in the OCR recognition process with constraints. In this case, it is determined whether the recognition results of the two recognition processes are different from each other by a predetermined value or more. If they are not different, it is determined that the block is a price area (tax-included price area), and the restriction process is added. The character string is used as price information of the recognition result.

  As described above, the in-product tag information identification processing unit 27 can perform character recognition on the information described in the product tag.

  When the second method is used in the product tag identification processing unit 26, the product tag used as a template has product identification information such as a tax-excluded price, a tax-included price, a manufacturer name, a product name, and a rating in advance. The position and size of the character frame to be displayed are set. Therefore, it is only necessary to cut out the image information of the corresponding part from the product tag area specified by the product tag specification processing unit 26 and execute the OCR recognition process.

  Then, the in-product tag information specifying unit 27 executes a dictionary matching process with the specified product name or the like (S470). That is, the edit distance between the read character string and the product identification information such as each product name in the product dictionary is obtained, and the product identification information such as the product name of the minimum editing distance is specified, and it is within a predetermined distance. If so, it is identified as product identification information such as a product name. Then, the product name portion with the shortest edit distance is removed from the read character string, and the edit distance with each rating in the rating dictionary is obtained for the remaining partial character strings, and whether the minimum edit distance is within the predetermined distance. If it is within the predetermined distance, the portion is identified as a rated character string. Similarly, the product name part and the rated part with the minimum editing distance are removed from the read character string, and the editing distance with each manufacturer name in the manufacturer name dictionary is obtained for the remaining partial character string, and the minimum editing distance is determined as the predetermined distance. It is determined whether it is within the distance, and if it is within the predetermined distance, that portion is identified as a character string of the manufacturer name.

  Further, the in-product tag information identification processing unit 27 executes a logical check process for the character string that has been character-recognized (S480). That is, it is determined whether the character strings recognized as characters are logically consistent.

  If there is no contradiction as a result of the logical check, specify the product identification information, manufacturer, and rating such as tax-excluded price, tax-included price, product name, etc. for each recognized character string, and identify them with the shooting date and time, store identification The information is stored and output in a predetermined storage area in association with the image information identification information of the captured image information. For example, output in tabular form.

  In the first embodiment, the case where the information described in the product tag is read from the photographed image information has been described. However, it may be associated with the image recognition of the product displayed on the display shelf. That is, the product identification information such as the product name of the product to be displayed on the display shelf may be specified from the information described in the product tag and collated.

  In this case, the image recognition system 1 according to the first embodiment constitutes a part of the functions of the image recognition system 1 according to the second embodiment. The image recognition system 1 according to the second embodiment includes a captured image information input reception processing unit 20, a captured image information storage unit 21, a captured image information storage unit 21, a captured image information alignment processing unit 22, and a position specifying process. Unit 23, product tag recognition processing unit 28, display product recognition processing unit 29, sample image information storage unit 30, product identification information storage unit 31, consistency determination processing unit 32, and recognition result matching processing unit 33. And have. FIG. 23 shows an example of the system configuration of the image recognition system 1 in the second embodiment. FIG. 24 shows an example of the configuration of the product tag recognition processing unit 28, and FIG. 25 shows an example of the configuration of the display product recognition processing unit 29.

  The management terminal 2 in the image recognition system 1 can transmit / receive information to / from the captured image information input terminal 4 via the network.

  The captured image information input acceptance processing unit 20, the captured image information storage unit 21, and the captured image information normalization processing unit 22 are the same as those in the first embodiment.

  In addition to the function of the position specification processing unit 23 in the first embodiment, the position specification processing unit 23 specifies a shelf area (shelf stage area). That is, among the display shelves shown in the photographed image information and the in-place image information, there are a shelf area where products are displayed and a product tag placement area where product tags for the products displayed there are attached. . Therefore, the product tag placement area and the shelf area are specified from the in-place image information. For specifying the product tag placement area and the shelf area, the operator of the management terminal 2 may manually specify the product tag placement area and the shelf area, and the position identification processing unit 23 may accept them. The product tag placement area and the shelf area may be automatically specified after the second time based on the information of the product tag placement area and the shelf area that have been manually input.

  The product tag recognition processing unit 28 executes product tag recognition processing. That is, each process of the product tag arrangement area extraction processing unit 24, the product tag arrangement area regularization processing unit 25, the product tag identification processing unit 26, and the in-product tag information identification processing unit 27 in the image recognition system 1 of the first embodiment. Execute. The processing in each processing unit is the same as in the case of the image recognition system 1 of the first embodiment.

  The display merchandise recognition processing unit 29 executes a process of recognizing the merchandise displayed on the shelves in the display shelves shown in the photographed image information. The display product recognition processing unit 29 includes a shelf area extraction processing unit 291, a face identification processing unit 292, a product identification information identification processing unit 293, and a shelf image matching processing unit 294.

  The shelf area extraction processing unit 291 extracts the image information of the shelf area specified by the position specification processing unit 23 as the shelf area image information. The shelf area cutout processing unit 291 may actually cut out as image information, or may cut out virtually without cutting out as image information. When there are a plurality of shelves on the display shelf, each is cut out as shelf area image information.

  The face identification processing unit 292 identifies the face on which the product is placed (the region on which the product is placed) for each shelf in the shelf area in the in-place image information. The face identification processing unit 292 is divided into an initial face identification process and a second and subsequent face identification process.

  The first face identification processing in the face identification processing unit 292 is an area where products are placed within a range (preferably a rectangular area) composed of shelf coordinates identified by the position identification processing unit 23 (preferably a rectangular area). Face). Specifically, the narrow and narrow shadows that occur between products are specified, the image repeat pattern is specified, the top edge of the package is specified, the product width is the same, and so on. The face area is specified by specifying the position. As the face identification processing, any method can be adopted depending on the product category and the product form, and the method is not limited to the above. Moreover, you may receive the correction input by a person in charge with respect to the face identified automatically. Furthermore, an input of the face position from the person in charge may be accepted. The coordinates of the area constituting the identified face are the shooting area information, the store information, the image information identification information of the captured image information, the image identification information of the in-place image information, the face, and the coordinates of the face area in the in-place image information. The face identification information for identification is associated and managed. The coordinates indicating the face area are the coordinates of the vertices necessary for specifying the rectangular area. For example, the coordinates may be four points, or the coordinates of two points, upper right and lower left, upper left and lower right.

  In the face identification processing for the second time and thereafter in the face identification processing unit 292, the coordinates of the area of the face specified by the previous (N-1th) in-place image information for the same shelf of the same display shelf are the current ( The coordinates of the face area specified by the (Nth) position image information are used.

  The coordinates of the face area are relative coordinates with reference to a predetermined position in the image information (for example, the vertex C1 at the upper left of the display shelf) such as a display shelf in the in-place image information, similarly to the coordinates of the shelf position.

  The product identification information specifying processing unit 293 specifies the product identification information of the product displayed on the face for each shelf of the display shelf. The product identification information includes, in addition to the product name, a JAN code assigned to the product, but is not limited thereto. Any information can be used as long as it can identify the product.

  The product identification information specifying processing unit 293 executes the following processing. That is, for each face, the product identification information of the product displayed on the face is specified by matching the image information of the face with the sample image information of the product stored in the sample image information storage unit 30 described later. Specifically, first, the similarity between the image information of the area formed by the coordinates of the face to be processed and the sample image information stored in the sample image information storage unit 30 is determined, and the similarity is the highest. The product identification information corresponding to the sample image information is specified as the product identification information of the product displayed on the face constituted by the coordinates.

  Here, in order to determine the similarity between the face image information and the sample image information, the following processing is performed. First, in the process before the product identification information specifying process in the product identification information specifying processing unit 293, the image information of the face area in the shelf of the in-place image information and the sample image information have the same direction (the rollover or In addition, the size of each image information is approximately the same (if the size of the image information is different within a predetermined range, before each similarity determination, The image information is adjusted so that the size of the image information is within a predetermined range). The similarity between the face image information and the sample image information is determined by the image feature amount (for example, local feature amount) between the feature point based on the image feature amount (for example, local feature amount) of the face image information and the sample image information. ) Based on each feature point. Then, a pair having the highest similarity between the feature point of the face image information and the feature point of the sample image information is detected, and the difference between the coordinates of the corresponding points is obtained. Then, the average value of the differences is obtained. The average value of the difference indicates the total average movement amount between the face image information and the sample image information. Then, the coordinate differences of all feature point pairs are compared with the average coordinate difference, and pairs with a large degree of deviation are excluded. Then, the similarity is ranked by the number of remaining corresponding points.

  The similarity between the face image information and the sample image information can be calculated by the method as described above. Further, in order to improve the accuracy, an EMD (Earth Moves Distance) between color histograms may be obtained and used as a measure of similarity. As a result, it is possible to compare comparatively strong similarities to environmental changes such as brightness information of photographed image information, and to specify with high accuracy. Note that the similarity determination is not limited to the above. The identified product identification information is associated with the shooting date / time information, the store information, the image information identification information of the captured image information, the image identification information of the in-place image information, and the face identification information for identifying the face. 31.

  The product identification information specified as described above is associated with shooting date / time information, store information, image information identification information of captured image information, image identification information of in-place image information, and face identification information for identifying a face. Store in the product identification information storage unit 31.

  The shelf image matching processing unit 294 includes image information on the shelf area in the previous (N-1) th in-place image information, and image information on the shelf area in the current (Nth) in-place image information. If the similarity is high, the product identification information of each face on the shelf is determined to be the same. As described above, this similarity determination processing is performed in the image feature amount of the image information of the shelf area in the previous (N-1) th in-place image information and the current (Nth) in-place image information. Similarity determination based on image information of a shelf area may be used, or EMD between color histograms may be used. Moreover, it is not limited to them. Then, instead of the identification process for each face unit in the merchandise identification information identification processing unit 293, the merchandise identification information identification processing unit 293 stores the merchandise identification information of each face on the shelf in the N-th in-place image information as N- The product identification information storage unit 31 stores the same as the product identification information of each face in the same first shelf. As a result, it is possible to omit processing for a shelf that hardly changes, such as a shelf that does not move much or a shelf that is managed in a very short cycle. Note that the processing by the shelf image matching processing unit 294 may not be provided.

  The sample image information storage unit 30 stores sample image information for identifying which product is the product of each face in the shelf of the display shelf shown in the in-place image information. The sample image information is image information obtained by photographing a product that may be displayed on a display shelf from a plurality of angles such as up and down, left and right, and diagonally. FIG. 28 shows an example of sample image information stored in the sample image information storage unit 30. FIG. 28 shows the case where canned beer is photographed from various angles as sample image information, but is not limited to canned beer. The sample image information storage unit 30 stores sample image information and product identification information in association with each other.

  The specimen image information storage unit 30 stores information necessary for calculation of similarity extracted from the specimen image information together with the specimen image information or in place of the specimen image information, for example, image feature amounts and their positions. Pair information may be stored. The sample image information includes information necessary for calculating similarity. In this case, the product identification information specifying processing unit 293 does not have to calculate the image feature amount every time for the sample image information when performing the matching process between the image information of the face area and the sample image information. Time can be shortened.

  The product identification information storage unit 31 stores product identification information of products displayed on each face of the shelf of the display shelf. For example, in association with product identification information, product identification in association with shooting date / time information, store information, image information identification information of captured image information, image identification information of in-place image information, face identification information for identifying a face The information is stored in the information storage unit 31.

  The consistency determination processing unit 32 determines each shelf and each shelf about the recognition result of the information displayed on the product tag by the product tag recognition processing unit 28 and the recognition result of the product (product identification information) by the display product recognition processing unit 29. The consistency of whether or not the product is likely to be included in the column is determined. For example, in the recognition result of the information displayed on the product tag or the recognition result of the product, the product is recognized as a product with a rating of 350 ml, but a product of 500 ml is displayed on the display shelf or shelf of the product. If so, the rating of the same product name is changed to “500 ml”. The products placed on the display shelf or the shelf are preferably set in advance and are associated with the photographed image information stored in the photographed image information storage unit 21.

  The recognition result verification processing unit 33 matches the product identification information of the product for each face recognized by the display product recognition processing unit 29 with the product information (product identification information) recognized by the product tag recognition processing unit 28, and the recognition result. Checks if they match.

  Specifically, first, as a result of the recognition processing by the display merchandise recognition processing unit 29, the number of sections in which faces having high similarity are arranged as one group, and how many groups exist on one shelf is specified. Also, specify where the left and right positions of the shelves of each group are. And the information of the recognition result of the product tag by the product tag recognition process part 28 by which the left-right position corresponds with each group is matched with each group.

  When the group by face and the product tag are associated, the following processing is executed. First, as a result of product tag recognition by the product tag recognition processing unit 28, the read product name (product identification information) is converted into a candidate product list with likelihood. In this case, conversion is performed in correspondence with the probability such that the probability of the product A1 is p1, and the probability of the product A2 is p2. When the manufacturer name and rating information are obtained from the recognition result of the product tag, the likelihood of the product of the manufacturer and the product having the rating is set high. Further, when price information is obtained from the recognition result of the product tag, the likelihood of the product having the price range as the selling price is increased. In addition, if two prices, the tax-excluded price and the tax-included price, are read and the ratio between them is exactly the same as the presence or absence of the consumption tax, and the logical check matches, the likelihood is further increased. Set high. In addition, at this time, when the genres of the products displayed on the shelves of the display shelf are known, the likelihood of the products belonging to those genres is set to be high or low.

  As a result of the recognition by the display merchandise recognition processing unit 29, a candidate merchandise list with likelihood is similarly given to the information of the recognized merchandise based on the degree of image similarity. In this case, conversion is performed in correspondence with the probability such that the probability of the product B1 is Pb1, and the probability of the product B2 is Pb2.

  Then, the list of candidate products such as products A1 and A2 based on the recognition result of the product tag is compared with the list of candidate products such as products B1 and B2 based on the recognition result of the display product recognition processing unit 29. (Case 1), appearing only in group A (case 2), appearing only in group B (case 3), and for case 1, the likelihood of the products in each group A and group B Is a higher likelihood than any of the above. For cases 2 and 3, a composite function reflecting the accuracy of the product tag recognition result by the product tag recognition processing unit 28 and the accuracy of the product information recognition result by the display product recognition processing unit 29 is applied. By doing so, a final candidate commodity list with likelihood is generated.

  By generating a final candidate product list with likelihood as described above, the candidate products can be ranked and specified. For example, the top (first) candidate product is used as the product. You may confirm, the goods which become a candidate from the 1st place to a predetermined order may be displayed, and selection input of a visual judgment result may be received.

  Then, the likelihood of the read price is calculated again for the confirmed product. That is, the likelihood of the price is determined by determining the ratio between the price excluding tax and the price including tax, the range of the selling price of the product, and the coincidence or divergence with the frequent price of the product. If the likelihood is higher than a predetermined threshold value, the price may be automatically determined, and if the likelihood is lower, the fact may be reflected in the display and an input by selection may be accepted.

  Next, an example of the processing process of the image recognition system 1 in the second embodiment will be described with reference to the flowchart of FIG. Note that the description of the same processing as in the first embodiment is omitted.

  The captured image information (FIGS. 21 and 22) obtained by photographing the store shelves in the store is input from the captured image information input terminal 4 and received by the captured image information input reception processing unit 20 of the management terminal 2 (S100). . Also, input of image information identification information such as shooting date / time, store identification information, and shooting image information is accepted. Then, the photographed image information input acceptance processing unit 20 stores the photographed image information, the photographing date and time, the store identification information, and the image information identification information of the photographed image information, which are accepted, in the photographed image information storage unit 21 in association with each other.

  When the management terminal 2 accepts a predetermined operation input, the photographed image information normalization processing unit 22 extracts the photographed image information stored in the photographed image information storage unit 21 and is a shelf that is a vertex for performing trapezoid correction processing. The input of 4 points of the position (the position of the display shelf) is received, and the keystone correction process is executed (S110). An example of the captured image information (normal image information) on which the trapezoid correction processing is executed in this way is shown in FIGS.

  Then, by accepting a predetermined operation input in the management terminal 2 for the in-place image information, the position specification processing unit 23 specifies the shelf area and the product tag placement area (S120). That is, the input of the shelf area and the product tag arrangement area in the in-place image information is accepted. FIG. 29 and FIG. 30 are diagrams showing a state in which the shelf area and the product tag placement area are specified.

  As described above, when the shelf region and the product tag placement area are specified, the display product recognition processing unit 29 performs the display product recognition process in the shelf region, and the product tag recognition process in the product tag placement region. Each unit 28 executes. Note that the display product recognition processing in the display product recognition processing unit 29 and the product tag recognition processing by the product tag recognition processing unit 28 may be performed in parallel or at different timings.

  The product tag recognition process (S130 to S160) in the product tag recognition processing unit 28 is the same as that in the first embodiment. That is, the product tag placement area extraction processing unit 24 in the product tag recognition processing unit 28 cuts out image information of the product tag placement area specified in S120 (S130), and the product tag placement area alignment processing unit 25 performs the keystone correction process. Is executed to execute the normalization process (S140).

  When the normalization processing of the product tag placement area is completed in S140, the product tag identification processing section 26 in the product tag recognition processing section 28 identifies individual product tag areas from the image information of the placed product tag placement area. (S150).

  When the product tag specification processing unit 26 specifies the product tag area, the product tag internal information specification processing unit 27 specifies information in the product tag (S160). By this specification, it is possible to character-recognize information described in the product tag, such as information including tax-excluded price, tax-included price, product name (product identification information), and rating.

  Next, display product recognition processing by the display product recognition processing unit 29 will be described.

  The shelf area extraction processing unit 291 extracts the image information of the shelf area from the in-place image information based on the shelf area received in S120 (S170). Then, a process for specifying a face is executed for each shelf in the shelf area image information (S180). Specifically, for the shelves in the shelf area, identify the narrow and narrow shadow that occurs between the products within the rectangular area consisting of the coordinates of the four points. The face is specified by specifying a step on the upper side of the package, specifying a separation position based on constraints such as the same product width, and the like. Face identification information for identifying the face is attached to the specified face. The coordinates of each identified face are stored in association with the shooting date and time, the store identification information, the image information identification information of the photographed image information, the image information identification information of the in-place image information, and the face identification information for identifying the face. Let The coordinates of the face may be two points that can specify a rectangular area without storing four points.

  As described above, when each face of each shelf in the shelf position area image information of the in-place image information is specified, the product identification information specifying processing unit 293 stores the sample image stored in the sample image information storage unit 30 for each face. The information and the matching process are executed, and the product identification information of the product displayed on the face is specified (S190). That is, each image feature amount is determined from image information in a rectangular area of a face of a certain shelf (face identification information of the face of this area is X) and each sample image information stored in the sample image information storage unit 30. Is calculated, and similarity is determined by obtaining a pair of feature points. Then, the sample image information with the highest similarity is specified, and if the similarity at that time is equal to or greater than a predetermined threshold, the product identification information corresponding to the sample image information is based on the sample image information storage unit 30. Identify. Then, the specified product identification information is set as product identification information of the product displayed on the face of the face identification information X. Then, the product identification information identification processing unit 293 associates the identified product identification information with the shooting date and time, the store identification information, the image information identification information of the captured image information, the image information identification information of the in-place image information, and the face identification information. It memorize | stores in the merchandise identification information storage part 31 (S200).

  In addition, the product identification information of all faces cannot be specified. Therefore, for a face that has not been specified, input of product identification information is accepted, and the product identification information for which input has been accepted is taken as shooting date / time, store identification information, image information identification information of captured image information, and image information of in-place image information. The product identification information storage unit 31 stores the identification information and the face identification information in association with each other. Similarly, an input may be received for the correction processing of the specified product identification information.

  By performing the processing as described above, it is possible to specify the product identification information of the products displayed on the shelf of the display shelf shown in the captured image information.

  Thus, the product tag recognition result by the product tag recognition processing unit 28, the display result of the display product recognition by the display product recognition processing unit 29, and the possibility that the consistency determination processing unit 32 may be included in each shelf and each shelf. The consistency of whether the product is high is determined (S210). For example, the product tag recognition result or the product recognition result recognizes that the product has a rating of 350 ml, but the product display shelf or shelf has a 500 ml product displayed. In this case, the rating of the same product name is changed to “500 ml”.

  The recognition result matching processing unit 33 matches the product identification information of the product for each face recognized by the display product recognition processing unit 29 with the product information (product identification information) recognized by the product tag recognition processing unit 28, It is checked whether the recognition results match (S220).

  That is, the recognition result collation processing unit 33 specifies the number of groups on one shelf as a group of sections in which faces having high similarity are arranged as a result of the recognition process by the display merchandise recognition processing unit 29. Also, specify where the left and right positions of the shelves of each group are. And the information of the recognition result of the product tag by the product tag recognition process part 28 by which the left-right position corresponds with each group is matched with a group.

  In this way, after associating the group with the product tag by the face, the product identification information (product name) corresponding to the face or group stored in the product identification information storage unit 31 and the product tag recognition by the product tag recognition processing unit 28. Compare the results with each other, identify whether the recognition results match, and determine the read price.

  Through the processing described above, the result of recognizing the product displayed on the display shelf by the image recognition processing can be compared with the result recognized by the character recognition processing by the product tag.

  In the first and second embodiments described above, the trapezoid correction process is executed by designating four points. However, it is burdensome to designate and input a vertex as a reference every time. Therefore, a configuration may be adopted in which a vertex serving as a reference for trapezoid correction processing is automatically specified. Processing in this case will be described.

  In this case, the captured image information normalization processing unit 22 is divided into an initial trapezoid correction process and a second and subsequent trapezoid correction processes. In addition to the first time, the first time includes a case where it is manually performed at an arbitrary timing in order to correct a deviation when automatically specifying the vertex. The second and subsequent times are other than the first time.

  The first trapezoid correction process in the captured image information normalization processing unit 22 accepts input of designation of four vertices of the rectangular area of the display shelf, as in the first embodiment. The four vertices of the rectangular area of the display shelf may be the four vertices of the shelf position of the display shelf, the four vertices of the shelf stage, or the four vertices of the area to which the product tag is attached. Further, it may be the four vertices of a group of two or three shelves. The four vertices that have received the designation are stored in association with the image information identification information of the shooting date / time information, the store information, and the shot image information. Then, the captured image information normalization processing unit 22 performs trapezoidal correction processing on the captured image information based on the coordinates of the four vertices that have received the designation, thereby obtaining the in-place image information.

  It is desirable that the photographed image information is photographed in the same region at the same angle for every fixed period. However, it is not possible to photograph the same area at the same angle. Therefore, the captured image information normalization processing unit 22 executes the second and subsequent trapezoidal correction processes as follows.

  First, the captured image information normalization processing unit 22 uses the vertex coordinates of the (N-1) th captured image information obtained by capturing the same (substantially the same) area corresponding to the Nth captured image information in the previous processing. Specify from the stored information. The vertex coordinates of the (N-1) th imaged image information obtained by imaging the same (substantially the same) area corresponding to the Nth imaged image information are stored in store identification information, image identification information, imaged date / time information, etc. corresponding to the imaged image information. Identify based on. Then, with respect to the (N-1) th captured image information, a rectangular area having a predetermined size including the identified vertex coordinates of the four vertices, for example, a square having a width of about 1/5 of the shelf width is used as the feature amount collection area 220. Set as. An example of a state in which the feature amount collection area 220 is set for the (N-1) th captured image information is shown in FIG. The feature amount collection region 220 may be a rectangular region including vertex coordinates. On the other hand, if the backgrounds of the display shelves match each other, the background will be greatly displaced by only a slight shift in the shooting position. Therefore, it is preferable to set the feature amount collection region 220 at a position including as much inside the display shelf as possible. That is, it is preferable that the vertex coordinates are located in the outer side of the display shelf in the feature amount collection region 220 than the center point of the feature amount collection region 220.

  For example, the four vertex coordinates are located at the upper left, upper right, lower left, and lower right. Then, when the rectangular area of the feature quantity collection area 220 is divided into a total of 4 areas each divided into two at the vertical and horizontal centers, in the feature quantity collection area 220 including the upper left vertex coordinates, the vertex coordinates are the upper left areas of the rectangular areas. The feature amount collection area 220 is set so as to be located at. Similarly, in the feature quantity collection area 220 including the upper right vertex coordinates, the feature quantity collection area 220 is set so that the vertex coordinates are located in the upper right area of the rectangular area, and the feature quantity collection including the lower left vertex coordinates is performed. In the area 220, the feature quantity collection area 220 is set so that the vertex coordinates are located in the lower left area of the rectangular area. In the feature quantity collection area 220 including the lower right vertex coordinates, the vertex coordinates of the rectangular area are The feature amount collection area 220 is set so as to be located in the lower right area. As a result, the vertex coordinates are located outside the display shelf in the feature amount collection region 220 from the center point of the feature amount collection region 220.

  Next, the captured image information normalization processing unit 22 includes the feature amount collection region 220 set in the N−1th captured image information in the Nth captured image information, and the N−1th captured image information. The feature amount collection region 221 having a size larger than the feature amount collection region 220 is set. The feature amount collection area 221 set in the Nth captured image information does not exceed half the short side. Further, if the image information is outside the captured image information, the range is trimmed. FIG. 32 shows an example of a state in which the feature amount collection area 221 is set for the Nth captured image information.

  Then, the captured image information normalization processing unit 22 collects local feature amounts in each feature amount collection region 220 set for the (N-1) th captured image information, and extracts feature points and their coordinates based on the local feature amounts. Memorize the set. In addition, in each feature amount collection area 221 set for the Nth captured image information, a local feature amount is collected, and a feature point based on the local feature amount and a set of coordinates thereof are stored.

  The captured image information normalization processing unit 22 corresponds to the local feature amount of the feature point in the feature amount collection region 220 of the (N-1) th captured image information and the feature amount collection region 220 of the N-1th captured image information. The local feature amount of the feature point in the feature amount collection region 221 of the Nth captured image information at the position to be compared is compared. Then, the feature point of each local feature amount of the N-th captured image information that is closest to each local feature amount of each feature point of the N-1th captured image information is specified. Then, the feature points of the nearest local feature values are paired, and the feature point coordinates by the paired local feature values are associated. At this time, pairs whose local feature quantities are close (similarity) less than a predetermined threshold are excluded. This makes it possible to identify a pair of feature points of the local feature amount in the (N-1) th feature amount collection region 220 and a feature point of the closest local feature amount in the Nth feature amount collection region 221. FIG. 33 shows a pair relationship between the feature points of the local feature amount of the N-1th feature amount collection region 220 and the feature points of the local feature amount of the Nth feature amount collection region 221. In FIG. 33, a point group of feature points based on local feature amounts in the N-1th feature amount collection region 220 is A, and a point group of feature points based on local feature amounts in the Nth feature amount collection region 221 is B, N−. The vertex used for the first trapezoid correction process is indicated by C.

  The coordinates of the point group A of feature points based on local feature amounts in the (N-1) th feature amount collection region 220 and the point group B of feature points based on local feature amounts in the Nth feature amount collection region 221 corresponding to the point group A And a function F (affine transformation) for projecting the point group A to the point group B is obtained. The function F includes, but is not limited to, a method of repeating sampling estimation and using Ransac of OpenCV which is a kind of robust estimation. It should be noted that a pair having a large deviation from the relationship line in the projection relationship is excluded from the processing target.

  After obtaining the function F in the captured image information normalization processing unit 22, the captured image information normalization processing unit 22 determines the coordinates of the vertex C used in the N−1th trapezoid correction processing based on the function F. The image information is projected onto the Nth captured image information and specified as the coordinates of the vertex D for the Nth trapezoidal correction process. This is schematically shown in FIG.

  By performing the above processing on the feature amount collection regions 220 and 221, the four vertices of the shelf position for the trapezoid correction processing in the Nth captured image information are specified. Then, the captured image information normalization processing unit 22 performs trapezoidal correction processing on the Nth captured image information based on the identified four vertices to generate the correct image information and store it. At this time, the photographed image information normalization processing unit 22 corresponds to the photographed date and time information, the store information, the image information identification information of the photographed image information, and the image identification information of the regular image information in association with the regular image information. Let me remember. The coordinates of the vertex corresponding to the specified Nth captured image information are stored in association with the image information identification information of the captured date / time information, the store information, and the captured image information.

  Note that the vertex specifying process used in the trapezoidal correction process in the captured image information normalization processing unit 22 is not limited to the case of specifying the product from the image information obtained by capturing the display shelf as in the present invention, but the same object to be captured. The present invention can also be applied to the image recognition system 1 that corrects a plurality of pieces of image information obtained by photographing the image and generates the in-place image information. As a result, a plurality of pieces of image information obtained by photographing the same object to be photographed can be arranged in order to generate in-place image information of the object to be photographed.

  In addition, in order to specify the four vertices of the shelf position for the trapezoid correction process in the N-th captured image information, the feature points based on the local feature amounts in the feature amount collection region 220 in the N-1th captured image information are described above. The function F is obtained by using the point group A and the point group B of the feature points based on the local feature amounts in the feature amount collection area 221 in the N-th captured image information, and used in the N-1th trapezoid correction process. The process of projecting the coordinates of the vertex C onto the Nth captured image information by the function F and specifying the coordinates as the vertex D for the Nth trapezoid correction process has been described. However, in this process, it is only necessary to find the coordinates (positions) of corresponding points of similar image information in the (N-1) th captured image information and the Nth captured image information. However, any type of feature amount that identifies a location in the image information may be used. For example, there are sharp points in image information, highlight points, and the like. In this specification, a feature quantity that identifies a location in image information, such as a local feature quantity, is referred to as an image feature quantity (position-specific image feature quantity). In the description of the present specification, a case where the local feature amount is used as the image feature amount as described above will be described.

  Next, a vertex specifying process for performing the trapezoidal correction process in the third embodiment will be described. In this case, the vertex for performing the trapezoidal correction process has already been specified in the captured image information obtained by photographing an arbitrary display shelf, and a similar area is defined for the same display shelf after a predetermined period (for example, one week) has elapsed. A case will be described in which captured image information is captured at the same angle.

  The photographed image information obtained by photographing the display shelf of the store is input from the photographed image information input terminal 4, and the input is received by the photographed image information input reception processing unit 20 of the management terminal 2. FIG. 35 shows an example of photographed image information. Also, input of image information identification information such as shooting date / time, store identification information, and shooting image information is accepted. Then, the photographed image information input acceptance processing unit 20 stores the photographed image information, the photographing date and time, the store identification information, and the image information identification information that have received the input in association with each other in the photographed image information storage unit 21.

  When the management terminal 2 accepts a predetermined operation input, the photographed image information normalization processing unit 22 extracts the photographed image information stored in the photographed image information storage unit 21 and sets the shelf position for executing the trapezoid correction process. A process of specifying the vertex D (D1 to D4) is executed.

  Vertex coordinates (for example, vertex coordinates C1 to C4) of the previous (N-1) th imaged image information (FIG. 36) in which the same or substantially the same area corresponding to the current (Nth) imaged image information (FIG. 35) is imaged. Identify). The vertex coordinates of the previous photographed image information may be specified based on the photographing date / time, the store identification information, the image identification information of the photographed image information, and the like.

  The captured image information normalization processing unit 22 extracts the (N-1) th captured image information from the captured image information storage unit 21, and for each vertex C1 to C4, a rectangle having a predetermined size including one vertex. The region is set as the feature amount collection region 220 in the (N-1) th captured image information. FIG. 36 shows a state where the feature amount collection area 220 is set in the (N-1) th captured image information.

  The captured image information normalization processing unit 22 extracts the Nth captured image information (FIG. 35) from the captured image information storage unit 21, and features in a wider range than the N−1th feature amount collection region 220. The quantity collection area 221 is set to the Nth captured image information. FIG. 37 shows a state in which the feature amount collection area 221 is set in the Nth captured image information. Each feature amount collection region 221 in the Nth captured image information includes one (N−1) th feature amount collection region 220. In FIG. 37, the inclusion relationship is shown by indicating the (N-1) th feature value collection region 220 in the Nth feature value collection region 221.

  The captured image information normalization processing unit 22 collects local feature amounts in each feature amount collection region 220 set for the (N-1) th captured image information, and sets feature points and coordinates based on the local feature amounts. Remember. In addition, local feature amounts are collected in each feature amount collection region 221 set for the Nth captured image information, and feature points and coordinate sets based on the local feature amounts are stored.

  The captured image information normalization processing unit 22 includes the feature amount collection region 221 of the Nth captured image information that is closest to each local feature amount of each feature point in the feature amount collection region 220 of the (N-1) th captured image information. The feature points of the local feature quantity in are identified, and the coordinates are associated with them as the feature points of the local feature quantity as a pair. FIG. 33 shows a pair relationship between the (N−1) th (N−1) th feature amount collection region 220 and the Nth feature amount collection region 221.

  A point group of feature points based on the local feature amount in the N-1th feature amount collection region 220 is A, a point group of feature points based on the local feature amount in the Nth feature amount collection region 221 is B, and the N-1th time point. If the vertex used in the trapezoidal correction process is C (C1 to C4), the captured image information normalization processing unit 22 converts the point group A into the point group B based on the coordinates of the point group A and the point group B. A function F (affine transformation) to be projected onto is obtained.

  Then, the captured image information normalization processing unit 22 projects the coordinates of the vertex C (C1 to C4) used in the (N-1) th trapezoid correction process based on the obtained function F, and the Nth trapezoid correction process. Is specified as the coordinates of the vertex D (D1 to D4).

  By performing the above processing on the feature amount collection regions 220 and 221, the four vertices D (D1 to D4) for the Nth trapezoid correction processing can be automatically specified. The coordinates of the identified Nth vertex D (D1 to D4) are stored in association with the image information identification information of the shooting date / time, the store identification information, and the captured image information. FIG. 38 shows the identified vertices D1 to D4.

  As described above, when the vertex D (D1 to D4) of the shelf position for the trapezoidal correction process for the Nth captured image information is specified, the captured image information normalization processing unit 22 determines the vertex D (D1 to D4). ) To perform trapezoidal correction processing on the Nth captured image information.

  By executing the above process, the corresponding vertices can be automatically identified without specifying the four vertices used in the trapezoid correction process for the second and subsequent trapezoid correction processes. The burden on the user can be reduced.

  Furthermore, a modification of the face specifying process in the face specifying processor 292 of the second embodiment will be described. In the present embodiment, the face specifying process of the second embodiment may be the first process, and the face specifying process may be automatically performed as the second and subsequent face specifying processes. Processing in this case will be described.

  In addition to the first time, the first time includes a case where the processing of the second embodiment is performed at an arbitrary timing in order to correct a shift in automatic identification. The second and subsequent times are other than the first time.

  The face identification processing unit 292 executes the same process as that of the second embodiment as the first face identification process. Then, the face identification processing for the second and subsequent times in the face identification processing unit 292 is performed by determining the coordinates of the area of the face identified by the previous (N−1) th in-place image information for the same shelf of the same display shelf. The extracted coordinates are used as the coordinates of the face area specified by the current (Nth) in-place image information.

  The coordinates of the face area are relative coordinates based on a predetermined position (for example, the vertex C1 at the upper left of the display shelf) in the display shelf in the in-place image information, similarly to the coordinates of the position of the shelf.

  Further, as a modification of the second embodiment, the processing of the second embodiment is performed for the first time as processing for specifying the product identification information of the product displayed on the face for each shelf of the display shelf in the product identification information specifying processing unit 293. The following processing is executed as the product identification information specifying process for the second and subsequent product identification information specifying processes.

  The product identification information identification processing unit 293 first identifies the face identification information of the face to be processed in the identification processing of the product identification information of the face in the N-th in-place image information. Let X denote the identified face identification information. Then, the image information of the area of the face identification information X in the Nth position image information is compared with the image information of the area in the (N−1) th position image information at a position corresponding to the face identification information X. Regarding the determination of similarity, it is preferable to obtain EMD between color histograms and use it as a measure of similarity, but it is not limited thereto. If the similarity is equal to or greater than a certain threshold, the product identification information corresponding to the face of the area in the N−1th in-place image information is extracted from the product identification information storage unit 31, and the N-th in-place image information Product identification information of the face identification information X. Thereby, the product identification information of the face identification information X in the Nth position image information to be processed can be specified. If the similarity is less than a certain threshold, the image information of the area of the face identification information X in the N-th in-place image information and the sample image information stored in the sample image information storage unit 30 are the same as in the first case. And the product identification information having the highest similarity and having the highest similarity is specified as the product identification information of the face of the face identification information X in the Nth position image information.

  In comparison with the face area image information in the (N-1) th in-place image information, not only a comparison with the corresponding face position but also a face within a predetermined range may be included as a comparison target. For example, when comparing with the image information of the area of face identification information X in the N-th position image information, the comparison object includes the area of face identification information X in the N−1-th position image information and the area. An area of a face within a predetermined range, for example, a face located at a position spaced one or more in the left-right direction, and an area of a face located on upper and lower shelves may be included. Further, in addition to the area of face identification information X in the N-1th in-place image information, a plurality of adjacent face areas such as face identification information X-2, X-1, X, X + 1, and X + 2 are included. May be.

  In this case, the image information of the area of the face identification information X in the Nth position image information, and the image information of the area of the face range of the range to be compared in the N−1th position image information, And the face identification information of the N-1th in-place image information with the highest similarity is specified. Note that the similarity may be a condition that the similarity is equal to or greater than a certain threshold. Then, the product identification information corresponding to the identified face identification information of the N-1th in-place image information is extracted from the product identification information storage unit 31, and the product identification information of the face identification information X in the N-th in-position image information To do. This process is schematically shown in FIG. FIG. 27A shows the previous (N-1th) in-place image information, and FIG. 27B shows the current (N-th) in-place image information. Then, the image information of each face area of the shelf 1 of the N-th position image information is compared with the image information of each face area of the shelf 1 of the N-1th position image information, respectively. And the product identification information of the face of shelf 1 in the N-1th in-place image information with the highest similarity is the product identification information of the face in shelf 1 of the N-th in-place image information. To identify as. FIG. 27 shows a case where comparison is made between two faces on the left and right sides in addition to the face of the N−1th position image information at the position corresponding to the face of the Nth position image information. . In addition, you may compare with the image information of the face position of not only the same shelf but an upper and lower shelf. For example, in the case of FIG. 27, when specifying the product identification information of the face position at the center of the shelf 2 of the N-th position image information, the center of the shelf 2 of the N-th position image information and its left and right In addition to comparison with the image information of the face area of each two faces, the image information of the center area of the shelf 1 of the N-1th in-place image information and the face areas of the two left and right sides, the (N-1) th time. Similarity may be compared with the image information of the center of the shelf 3 of the in-place image information and the two face areas on the left and right.

  If the product identification information cannot be specified due to the similarity not satisfying the threshold value as a result of the comparison with the face image information of the N-1th position image information, the face in the Nth position image information The image information in the region of the identification information X is compared with the sample image information stored in the sample image information storage unit 30, and the product identification information having the highest similarity that is equal to or higher than a predetermined threshold is represented by N It is specified as the product identification information of the face of the face identification information X in the second in-place image information. The similarity determination process in this case can be performed in the same manner as the initial product identification information specifying process.

  The product identification information specified as described above is associated with shooting date / time information, store information, image information identification information of captured image information, image identification information of in-place image information, and face identification information for identifying a face. Store in the product identification information storage unit 31.

  In the first to fifth embodiments of the image recognition system 1 according to the present invention, the accuracy of the image information to be processed in each processing unit is improved by executing a normalization process or the like as a pre-process. In the above-described embodiments, such a case has been described. However, it is not always necessary to perform the normalization process on the image information to be processed. In this case, the processing target in each processing unit is an image when the normalization process is not performed. Information. For example, the position specification processing unit 23, the product tag placement area extraction processing unit 24, the product tag placement area correct processing unit 25, the product tag specification processing unit 26, the product tag internal information specification processing unit 27, and the product tag recognition processing unit 28. In the display product recognition processing unit 29, the shelf area extraction processing unit 291, the face identification processing unit 292, the product identification information identification processing unit 293, and the shelf image matching processing unit 294, the alignment processing is not executed. Image information may be processed. In this case, it is only necessary to execute processing for image information in which display shelves displaying products are displayed even if the normalization processing is not executed. Note that the image information showing the display shelves displaying the merchandise includes both the image information subjected to the normalization process and the image information not subjected to the normalization process.

  By using the image recognition system 1 of the present invention, it is possible to accurately specify the products displayed on the display shelf.

1: Image recognition system 2: Management terminal 4: Captured image information input terminal 20: Captured image information input reception processing unit 21: Captured image information storage unit 22: Captured image information alignment processing unit 23: Position specifying processing unit 24: Product tag placement area cutout processing unit 25: Product tag placement region alignment processing unit 26: Product tag specification processing unit 27: Product tag information specification processing unit 28: Product tag recognition processing unit 29: Displayed product recognition processing unit 30 : Sample image information storage unit 31: Product identification information storage unit 32: Consistency determination processing unit 33: Recognition result collation processing unit 220: Feature amount collection region 221 in the N-1th image information Feature in the Nth image information Quantity collection area 291: shelf area extraction processing unit 292: face identification processing section 293: product identification information identification processing section 294: shelf image matching processing section 70: arithmetic device 71: storage device 7 : Display device 73: input device 74: communication device

Claims (19)

A product tag identification processing unit for identifying a product tag area in image information in which display shelves displaying products are shown;
An in-product tag information specifying processing unit for specifying information described in the product tag by performing OCR recognition processing in the specified product tag area,
The product tag identification processing unit
Identify the product tag area using the brightness information of the image information,
The product tag information identification processing unit
The box is specified by binarizing the specified product tag area and performing a labeling process,
Among the identified boxes, identify the boxes that satisfy the specified condition as blocks,
Performing the OCR recognition process on the identified block;
The tax-excluded price and the tax-included price are specified by executing a logical consistency determination process for the two price information specified by the OCR recognition process .
An image recognition system characterized by that. A product tag identification processing unit for identifying a product tag area in image information in which display shelves displaying products are shown;
An in-product tag information specifying processing unit for specifying information described in the product tag by performing OCR recognition processing in a part or all of the specified product tag area,
The product tag identification processing unit
By specifying the rise and fall of the histogram obtained by integrating the lightness information of the image information in the horizontal direction, the upper side position and the lower side position of the product tag are specified,
By specifying the rise and fall of the histogram obtained by integrating the brightness information of the image information in the vertical direction, the left side position and the right side position of the product tag are specified as a pair of the fall within a predetermined distance from the rise.
An image recognition system characterized by that. A product tag identification processing unit for identifying a product tag area in image information in which display shelves displaying products are shown;
An in-product tag information specifying processing unit for specifying information described in the product tag by performing OCR recognition processing in a part or all of the specified product tag area,
The product tag identification processing unit
Among the rises of the histogram obtained by integrating the brightness information of the image information in the vertical direction, if there is no tag area within a predetermined distance on the right side of the rise that could not form a pair, the product information is specified as a product tag area.
Among the falling edges of the histogram obtained by integrating the brightness information of the image information in the vertical direction, if there is no tag area within a predetermined distance on the left side of the falling edge that could not form a pair, it is specified as a product tag area.
An image recognition system characterized by that. A product tag identification processing unit for identifying a product tag area in image information in which display shelves displaying products are shown;
An in-product tag information specifying processing unit for specifying information described in the product tag by performing OCR recognition processing in the specified product tag area,
The product tag information identification processing unit
Execute binarization processing in the specified product tag area,
By performing a labeling process on the binarized image information, a box is specified by grouping consecutive pixels,
Among the specified boxes, adjacent boxes having a predetermined similarity are merged and specified as a block,
Of the identified blocks, the block with the highest height is estimated as a price region, and OCR recognition processing is executed.
An image recognition system characterized by that. A product tag identification processing unit for identifying a product tag area in image information in which display shelves displaying products are shown;
An in-product tag information specifying processing unit for specifying information described in the product tag by performing OCR recognition processing in the specified product tag area,
The product tag information identification processing unit
By executing the OCR recognition process for the price information recognized as the price excluding tax and the price information recognized as the price including tax, the process of determining the size of the price and / or the consistency of the tax rate is executed, thereby the price excluding tax and tax included. Identify the price,
An image recognition system characterized by that. A product tag identification processing unit for identifying a product tag area in image information in which display shelves displaying products are shown;
An in-product tag information specifying processing unit for specifying information described in the product tag by performing OCR recognition processing in a part or all of the specified product tag area,
The product tag information identification processing unit
The tax-excluded price and the tax-included price are specified by executing a logical consistency determination process for the two price information specified by the OCR recognition process .
An image recognition system characterized by that. The product tag information identification processing unit
Execute binarization processing in the specified product tag area,
By performing the labeling process in the binarization processing image information, to identify the box by grouping pixels to the consecutive
Among the specified boxes, adjacent boxes having a predetermined similarity are merged and specified as a block,
Of the identified blocks, the block with the highest height is estimated as a price region, and OCR recognition processing is executed.
The image recognition system according to claim 1 , wherein the image recognition system is an image recognition system. The product tag information identification processing unit
By executing the OCR recognition process for the price information recognized as the price excluding tax and the price information recognized as the price including tax, the process of determining the size of the price and / or the consistency of the tax rate is executed, thereby the price excluding tax and tax included. Identify the price,
The image recognition system according to claim 1, wherein the image recognition system is one of the following. A product tag recognition processing unit for specifying product identification information in a product tag area in image information in which display shelves displaying products are shown;
A display product recognition processing unit for specifying product identification information of a product displayed in the image information;
Comprises a product identification information in the product tag regions above specified, by comparing the product identification information of merchandise displayed and the specific, the recognition result collation processing section for collating whether recognition result matches, the And
The recognition result matching processing unit
By the recognition processing in the display product recognition processing unit, a group of highly similar faces are grouped together, and the product tag recognition result by the product tag recognition processing unit is associated with each group in each shelf.
An image recognition system characterized by that. The image recognition system further includes:
A captured image information alignment processing unit that performs trapezoidal correction processing on image information obtained by capturing the display shelf,
The photographed image information normalization processing unit
A first image information, based on the correspondence between the second image information, the vertices used in the keystone correction process in the first image information, projected to the second image information,
Executing a trapezoidal correction process of the second image information based on the projected vertex;
The image recognition system according to any one of claims 1 to 9 , wherein The image recognition system further includes:
A position specifying processing unit for specifying a product tag placement area in the image information in which the trapezoidal correction processing is performed in the captured image information normalization processing unit;
A product tag placement area alignment processing unit for performing trapezoidal correction processing in the specified product tag placement area;
The image recognition system according to claim 10 , further comprising: The product tag placement area alignment processing unit
Edge detection is performed in the product tag placement area, and a group of straight lines satisfying a predetermined condition of the angle in the vertical direction is detected,
Identify at least two straight lines from the detected straight line group,
The trapezoid correction process is executed using the two identified straight lines.
The image recognition system according to claim 11 . Computer
A product tag specifying processing unit for specifying a product tag area in image information showing a display shelf for displaying products,
An image recognition program that functions as an in-product tag information identification processing unit that identifies information described in a product tag by performing OCR recognition processing in the identified product tag area,
The product tag identification processing unit
Identify the product tag area using the brightness information of the image information,
The product tag information identification processing unit
The box is specified by binarizing the specified product tag area and performing a labeling process,
Among the identified boxes, identify the boxes that satisfy the specified condition as blocks,
Performing the OCR recognition process on the identified block;
The tax-excluded price and the tax-included price are specified by executing a logical consistency determination process for the two price information specified by the OCR recognition process .
An image recognition program characterized by that. Computer
A product tag specifying processing unit for specifying a product tag area in image information showing a display shelf displaying a product,
An image recognition program that functions as an in-product tag information specifying processing unit that specifies information described in a product tag by performing OCR recognition processing in a part or all of the specified product tag area,
The product tag identification processing unit
By specifying the rise and fall of the histogram obtained by integrating the lightness information of the image information in the horizontal direction, the upper side position and the lower side position of the product tag are specified,
By specifying the rise and fall of the histogram obtained by integrating the brightness information of the image information in the vertical direction, the left side position and the right side position of the product tag are specified as a pair of the fall within a predetermined distance from the rise.
An image recognition program characterized by that. Computer
A product tag specifying processing unit for specifying a product tag area in image information showing a display shelf displaying a product,
An image recognition program that functions as an in-product tag information specifying processing unit that specifies information described in a product tag by performing OCR recognition processing in a part or all of the specified product tag area,
The product tag identification processing unit
Among the rises of the histogram obtained by integrating the brightness information of the image information in the vertical direction, if there is no tag area within a predetermined distance on the right side of the rise that could not form a pair, the product information is specified as a product tag area.
Among the falling edges of the histogram obtained by integrating the brightness information of the image information in the vertical direction, if there is no tag area within a predetermined distance on the left side of the falling edge that could not form a pair, it is specified as a product tag area.
An image recognition program characterized by that. Computer
A product tag specifying processing unit for specifying a product tag area in image information showing a display shelf displaying a product,
An image recognition program that functions as an in-product tag information identification processing unit that identifies information described in a product tag by performing OCR recognition processing in the identified product tag area,
The product tag information identification processing unit
Execute binarization processing in the specified product tag area,
By performing a labeling process on the binarized image information, a box is specified by grouping consecutive pixels,
Among the specified boxes, adjacent boxes having a predetermined similarity are merged and specified as a block,
Of the identified blocks, the block with the highest height is estimated as a price region, and OCR recognition processing is executed.
An image recognition program characterized by that. Computer
A product tag specifying processing unit for specifying a product tag area in image information showing a display shelf displaying a product,
An image recognition program that functions as an in-product tag information identification processing unit that identifies information described in a product tag by performing OCR recognition processing in the identified product tag area,
The product tag information identification processing unit
By executing the OCR recognition process for the price information recognized as the price excluding tax and the price information recognized as the price including tax, the process of determining the size of the price and / or the consistency of the tax rate is executed, thereby the price excluding tax and tax included. Identify the price,
An image recognition program characterized by that. Computer
A product tag specifying processing unit for specifying a product tag area in image information showing a display shelf displaying a product,
An image recognition program that functions as an in-product tag information specifying processing unit that specifies information described in a product tag by performing OCR recognition processing in a part or all of the specified product tag area,
The product tag information identification processing unit
The tax-excluded price and the tax-included price are specified by executing a logical consistency determination process for the two price information specified by the OCR recognition process .
An image recognition program characterized by that. Computer
A product tag recognition processing unit for specifying product identification information in a product tag area in image information in which display shelves displaying products are shown;
A display product recognition processing unit for specifying product identification information of a product displayed in the image information;
An image that functions as a recognition result matching processing unit that compares the product identification information in the region of the specified product tag with the product identification information of the specified displayed product and checks whether the recognition results match. A recognition program,
The recognition result matching processing unit
By the recognition processing in the display product recognition processing unit, a group of highly similar faces are grouped together, and the product tag recognition result by the product tag recognition processing unit is associated with each group in each shelf.
An image recognition program characterized by that.