CN113627415B - Method and device for determining placement information of target object - Google Patents
Method and device for determining placement information of target object Download PDFInfo
- Publication number
- CN113627415B CN113627415B CN202110990009.1A CN202110990009A CN113627415B CN 113627415 B CN113627415 B CN 113627415B CN 202110990009 A CN202110990009 A CN 202110990009A CN 113627415 B CN113627415 B CN 113627415B
- Authority
- CN
- China
- Prior art keywords
- target
- image
- information
- target object
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a method and a device for determining target object placement information. The method comprises the steps of processing a target image containing a target object, determining the target price tag image, wherein the target price tag image is a price tag image comprising the target object, determining a plurality of object frames from the target image, and determining the target object frames according to the plurality of object frames and the relative position relationship between the plurality of object frames and the target price tag image, wherein the object frames except the target object frames in the plurality of object frames are determined to be a plurality of comparison object frames. The method and the device are applied to the field of supermarket markets, and the technical problem that detection efficiency is low due to the fact that the condition of commodities on a shelf is checked by manpower in the related art is solved.
Description
Technical Field
The invention relates to the field of image processing, in particular to a method and a device for determining target object placement information.
Background
In the related art, the current super goods shelf management mainly relies on manual management, and related information of goods placed on a goods shelf can be identified only after the goods shelf is checked by people, so that the efficiency is low, the situation that the goods on the goods shelf cannot be timely supplemented and the like exist, and an effective solution is not proposed at present based on the existing problems.
Disclosure of Invention
The invention mainly aims to provide a method and a device for determining target object placement information so as to improve the efficiency of detecting commodity conditions on a goods shelf.
In order to achieve the above object, according to one aspect of the present invention, there is provided a method of determining target object placement information. The method comprises the steps of processing a target image containing a target object, determining the target price image, determining a plurality of object frames from the target image, determining the target object frames according to the plurality of object frames and the relative position relationship between the plurality of object frames and the target price image, wherein a region corresponding to one object frame is used for placing one target object, the target object frame is one object frame in the plurality of object frames, and determining object placement information of the target object based on the similarity between the target object frame and the plurality of comparison object frames, wherein the object frames except the target object frame in the plurality of object frames are determined to be the comparison object frames.
Further, the object placement information comprises at least one of placement quantity information of the target objects, placement position information of the target objects, vacancy position information of the target objects and position misplacement information of the target objects.
Further, before processing a target image containing a target object to determine the target price tag image, the method comprises the steps of calibrating a shelf area, obtaining a target image corresponding to the calibrated shelf area, extracting edges of the target image, conducting linear detection on the extracted edges to obtain a plurality of lines, identifying the plurality of lines, determining an upper boundary and a lower boundary of a price tag column corresponding to the target price tag, and identifying the price tag column through the upper boundary and the lower boundary.
Further, identifying a plurality of straight lines, determining the upper boundary and the lower boundary of a price tag column corresponding to a target price tag comprises counting a plurality of slopes corresponding to the plurality of straight lines, eliminating straight lines with absolute values of the slopes being greater than or equal to 1 to determine a plurality of first target straight lines, calculating slope differences between every two of the plurality of first target straight lines, eliminating invalid straight lines to obtain a plurality of second target straight lines, wherein the invalid straight lines are straight lines with slope differences being greater than a preset value, determining distance values between every two of the plurality of second target straight lines, and determining two straight lines with distance values within a preset range as the upper boundary and the lower boundary corresponding to the price tag column respectively.
Further, processing a target image containing a target object to determine the target price tag image, wherein the processing comprises the steps of performing enlarged scanning on an image corresponding to a price tag column through a zoom ball machine, performing quadrilateral detection on the image to obtain a quadrilateral image, extracting text information in the quadrilateral image, and determining the quadrilateral image as the target price tag image if the text information is text related to the target object.
Further, after determining that the quadrangular image is the target price tag image if the text information is text related to the target object, the method further includes storing coordinates of the text information and a center point of the quadrangular image corresponding to an absolute coordinate system of the zoom ball machine.
Further, determining the plurality of object frames from the target image comprises performing enlarged scanning on the image corresponding to the target object placement area by using a zoom ball machine, wherein the area between every two price tag columns in the target image is determined to be the target object placement area, detecting the image corresponding to the target object placement area by using a target object detection model, and determining the plurality of object frames.
Further, before object placement information of a target object is determined based on the similarity between the target object frame and the plurality of comparison object frames, the method further comprises the steps of extracting first text information and first color texture information in the target object frame, extracting a plurality of second text information and a plurality of second color texture information in the plurality of comparison object frames, determining the object placement information of the target object based on the similarity between the target object frame and the plurality of comparison object frames, wherein the step of comparing the plurality of second text information and the first text information in sequence, determining that an object corresponding to the comparison object frame is a homogeneous object with the target object if the second text information is matched with the first text information, sequentially comparing the second color texture information with the first color texture information if the second text information is not matched with the first color texture information, determining that an object corresponding to the comparison object frame is a homogeneous object with the target object if any one of the second text information and the second color texture information in the comparison object frame is a frame, determining that the object corresponding to the comparison object is a blank object according to the blank position of the target object, determining that the object corresponding to the second text information is a blank position of the target object, and determining that the object placement position is at least is different from the blank position of the object.
In order to achieve the above object, according to another aspect of the present invention, there is provided an apparatus for determining target object placement information. The device comprises a first determining unit, a second determining unit and a third determining unit, wherein the first determining unit is used for processing a target image containing a target object, determining the target price image, the target price image is a price image comprising the target object, the second determining unit is used for determining a plurality of object frames from the target image and determining the target object frames according to the plurality of object frames and the relative position relationship between the plurality of object frames and the target price image, the region corresponding to one object frame is used for placing one target object, the target object frame is one object frame in the plurality of object frames, and the third determining unit is used for determining object placement information of the target object based on the similarity between the target object frame and the plurality of comparison object frames, wherein the object frames except the target object frame are determined to be the plurality of comparison object frames.
Further, the object placement information comprises at least one of placement quantity information of the target objects, placement position information of the target objects, vacancy position information of the target objects and position misplacement information of the target objects.
The method comprises the steps of processing a target image containing a target object, determining the target price tag image, determining a plurality of object frames from the target image and determining the target object frame according to the plurality of object frames and the relative position relation between the plurality of object frames and the target price tag image, wherein a region corresponding to one object frame is used for placing one target object, the target object frame is one object frame in the plurality of object frames, determining object placement information of the target object based on the similarity between the target object frame and the plurality of comparison object frames, and determining the object frames except the target object frame in the plurality of object frames as a plurality of comparison object frames.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention. In the drawings:
FIG. 1 is a flow chart of a method for determining target object placement information according to an embodiment of the invention, and
FIG. 2 is a schematic diagram of three relative positional relationships between a target price tag and a commodity frame according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of determining an area corresponding to a target price tag through quadrilateral detection according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of commodity detection using a detection model according to an embodiment of the present invention;
FIG. 5 is a schematic illustration of a comparison area determined from the location of a target price tag provided in accordance with an alternative embodiment of the present invention;
FIG. 6 is a flow chart of contrast of feature vectors between a target commodity frame and a contrast object frame provided according to an embodiment of the present invention;
FIG. 7 is a diagram of a comparison result of feature vectors between a target commodity frame and a comparison object frame according to an embodiment of the present invention;
FIG. 8 is a schematic diagram showing the presence of a target commodity in a stock out condition after comparison by feature vectors according to an embodiment of the present invention;
FIG. 9 is a diagram showing the situation that there is a misplaced merchandise at the placement location of the target merchandise after feature vector comparison;
Fig. 10 is a schematic diagram of an apparatus for determining placement information of a target object according to an embodiment of the present invention.
Detailed Description
It should be noted that, without conflict, the embodiments of the present invention and features of the embodiments may be combined with each other. The invention will be described in detail below with reference to the drawings in connection with embodiments.
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate in order to describe the embodiments of the invention herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
According to an embodiment of the invention, a method for determining target object placement information is provided.
Fig. 1 is a flowchart of a method for determining object placement information according to an embodiment of the present invention. As shown in fig. 1, the invention comprises the following steps:
Step S101, processing a target image containing a target object, and determining a target price tag image, wherein the target price tag image is a price tag image comprising the target object;
the method for determining the placement information of the target object provided in the embodiment of the application can be applied to supermarket commodity identification, wherein in one implementation mode, the object in the embodiment of the application can be a commodity placed on a supermarket shelf for sale.
In the embodiment provided by the application, the management efficiency can be greatly improved by collecting the images and processing the collected images to monitor the commodities and identify the attributes, and the commodity information database can be established to help the manufacturer to optimize the types of the objects.
In the embodiment of the present application, only the commodity is taken as an example of the object, and those skilled in the art may flexibly set the object as other objects.
Specifically, the collected target image is processed to determine a target price tag image, where the target price tag image is an image including a price tag corresponding to a target object, that is, the target object to be processed can be determined by the determined price tag image of the target object, in this embodiment, a target commodity, for example, the target commodity is determined to be cola and a price tag image corresponding to cola by processing the target image.
Step S102, determining a plurality of object frames from the target image, and determining the target object frame according to the plurality of object frames and the relative position relationship between the plurality of object frames and the target price tag image, wherein the area corresponding to one object frame is used for placing one target object, and the target object frame is one object frame in the plurality of object frames.
Specifically, by determining the relative positional relationship between a plurality of commodity frames in the target commodity placement area and the target price, determining the target commodity frame corresponding to the anchor sample, it should be noted that in the embodiment of the present application, the commodity image included in the target commodity frame is matched with the target price by default, wherein in the embodiment, the size of the image of the commodity frame is basically consistent with the size of the image of the target commodity, and in the present application, the sequentially determined commodity frames default to sequentially arrange the commodities placed on the shelf, for example, the target commodity takes the cola as an example, and a can of cola is placed in the first determined commodity frame corresponding to the cola by default, and by processing the shelf image, whether other comparative commodity frames have the corresponding cola placed in order is determined.
Further, the relative positional relationship between the target price tag and the target commodity is generally classified into three types, as shown in fig. 2, and according to these relative positional relationships, the commodity frame closest to the top of the fix the price tag coordinates is regarded as the target commodity frame (also referred to as the comparison anchor sample), and this target commodity frame can be considered to be matched with the commodity price tag type.
It should be further noted that, in the embodiment of the present application, the relative positional relationship is actually set according to the actual situation, preferably, the uppermost commodity frame of the target price tag is determined to be the target commodity frame matched with the target price tag, alternatively, if the uppermost commodity frame of the target price tag is detected to be an empty commodity frame, alternatively, the commodity frame adjacent to the empty commodity frame may be determined to be the target commodity frame.
And step S103, determining object placement information of the target object based on the similarity between the target object frame and the plurality of comparison object frames, wherein the object frames except for the target object frame in the plurality of object frames are determined to be the plurality of comparison object frames.
Further, after the target commodity frame is determined, other commodity frames in the determined target commodity placement area are determined as comparison object frames for comparison with the target commodity frame.
Specifically, the object placement information of the target commodity is determined by sequentially comparing the feature vector in the comparison object frame with the feature vector in the target commodity frame (in this embodiment, the object placement information is commodity placement information, and it should be noted that the object in the present application changes with the application of different scenes).
The object placement information comprises at least one of the following information of the placement number of the target objects, the placement position information of the target objects, the vacancy position information of the target objects and the position misplacement information of the target objects.
Specifically, whether the commodity in the comparison commodity frame is the same type of the target commodity or whether the comparison frame is an empty commodity frame can be determined by comparing the characteristic vector in the comparison commodity frame with the characteristic vector in the target commodity frame, and commodity placement information of the target commodity is determined by determining the condition of the commodity in the comparison commodity frame.
In this embodiment, the target object is a target commodity, and the comparison object frame is a comparison commodity frame.
Further, when the commodities in the commodity frame are compared with the target commodities to be similar, the arrangement quantity of the target commodities is determined to be 1, namely the arrangement quantity information of the target commodities is determined.
Still further, the placement position information of the target commodity can be determined by comparing the position information of the commodity frame.
Meanwhile, if the commodity in the comparison commodity frame is empty, determining that the position corresponding to the comparison commodity frame is the empty position of the target commodity, so that the empty position information of the target commodity can be determined through the empty commodity frame.
On the other hand, if the commodity in the comparison commodity frame is not the same type of commodity as the target commodity, determining the position corresponding to the comparison commodity frame as the misplaced position of the target commodity, namely determining the misplaced position information of the target commodity.
Optionally, before processing the target image containing the target object to determine the target price label image, the method comprises:
calibrating the goods shelf area and acquiring a target image corresponding to the calibrated goods shelf area;
Specifically, before determining the target price tag image, the shelf area needs to be calibrated at first, and the camera is fixed in position, so that the shelf area needs to be calibrated only once.
Performing linear detection on the extracted edges to obtain a plurality of straight lines;
and secondly, identifying the shelf price tag column, wherein the shelf price tag column comprises the steps of processing a target image acquired by a camera, processing the target image, extracting the edge of the target image, and detecting the edge to obtain a plurality of straight lines.
Identifying a plurality of straight lines, and determining the upper boundary and the lower boundary of a price tag column corresponding to a target price tag;
Then, a plurality of straight lines are identified, the upper boundary and the lower boundary of the price tag column corresponding to the target price tag are determined, and the price tag column is identified through the upper boundary and the lower boundary.
The price tag field is identified by an upper boundary and a lower boundary.
Finally, the corresponding area of the price tag field in the target image is determined by identifying the upper and lower boundaries of the price tag field.
The method comprises the steps of identifying a plurality of straight lines, determining the upper boundary and the lower boundary of a price tag column corresponding to a target price tag, counting a plurality of slopes corresponding to the plurality of straight lines, eliminating the straight lines with absolute values larger than or equal to 1 to determine a plurality of first target straight lines, calculating slope differences between every two of the plurality of first target straight lines, eliminating invalid straight lines to obtain a plurality of second target straight lines, determining the distance value between every two of the plurality of second target straight lines, and determining the two straight lines corresponding to the distance value with the distance value within a preset range as the upper boundary and the lower boundary corresponding to the price tag column.
In the above-mentioned way, because the angle of the shooting shelf is generally horizontal, a plurality of straight lines are obtained by processing the target image containing the target commodity, outliers, that is, straight lines with obvious slopes and other excessively large differences are removed, and straight lines with absolute values of the slopes of the straight lines |k| <1 are counted, and meanwhile, straight lines outside the calibration area of the shelf are removed.
In an alternative embodiment, since the difference between the slopes of the two straight lines corresponding to the upper and lower boundaries of one price tag column is the smallest, the slope difference between every two straight lines in the plurality of straight lines is calculated, and when the slope difference is within a small preset range, the two straight lines corresponding to the slopes of the two straight lines corresponding to the price tag column are determined as the two upper boundaries and the lower boundary.
In another alternative embodiment, because the spacing between the lines of the price tag columns conforms to the rule of alternating width, the second target lines are sorted from top to bottom, and the upper and lower boundaries of the tag columns are determined according to the rule of alternating width.
Optionally, processing a target image containing a target object to determine the target price tag image, wherein the processing comprises the steps of performing enlarged scanning on an image corresponding to a price tag column through a zoom ball machine, performing quadrilateral detection on the image to obtain a quadrilateral image, extracting text information in the quadrilateral image, and determining the quadrilateral image as the target price tag image if the text information is text related to the target object.
Optionally, after determining that the quadrilateral image is the target price tag image if the text information is text related to the target object, the method further comprises storing the text information and coordinates of a center point of the quadrilateral image corresponding in an absolute coordinate system of the zoom ball machine.
In the above-described manner, since the information of the price tag and the commodity corresponding to the price tag is determined by the image processing means in the present application, in the present embodiment, the image corresponding to the price tag column is scanned in an enlarged manner by the zoom ball machine, and since the price tag in the supermarket or the sales place is generally a quadrangle, the quadrangle image is determined by the quadrangle detection, and the quadrangle image including the text related to the target commodity in the quadrangle image is determined as the image of the target price tag corresponding to the target commodity.
Further, since the price tag is smaller, in order to ensure the recognition effect in a large scene, the adopted image acquisition device is a zoom ball machine, when the price tag column is scanned, the zoom center of the ball machine is the center line of the price tag column, the price tag column is scanned in an amplified manner from left to right along the center line, quadrilateral detection is performed by using a hough conversion mode and the like, specifically, as shown in fig. 3, a quadrilateral range is obtained, text character detection and recognition are performed in the range, and text information in the price tag is extracted. If the text information related to the commodity is not extracted, the price tag is considered to be false detection, and the price tag column is discarded and continuously scanned. If the related text information of the commodity is extracted, the information is stored, and the coordinates of the quadrangular center point of the price tag in the absolute coordinate system of the ball machine are stored for backtracking.
Optionally, determining the plurality of object frames from the target image includes performing enlarged scanning on an image corresponding to the target object placement area by using a zoom ball machine, wherein an area between every two price tag columns in the target image is determined to be the target object placement area, detecting the image corresponding to the target object placement area by using a target object detection model, and determining the plurality of object frames.
In this way, the boundary of the commodity placement area can be determined by identifying the upper and lower boundaries of the bid fields, and therefore, the area between the lower boundary of the upper field and the upper boundary of the lower field of two adjacent upper and lower fields is determined as the target commodity placement area.
Further, the zoom ball machine is used for scanning from left to right along the center line of the commodity area after being enlarged, the common detection models such as yolo and frcnn are used for detecting the commodity, the positions of commodity frames in the absolute coordinate system of the ball machine are stored after the commodity is detected, and a schematic diagram of commodity detection by using the detection models is shown in fig. 4.
Optionally, before determining the object placement information of the target object based on the similarity between the target object frame and the plurality of comparison object frames, the method further comprises:
extracting first text information and first color texture information in a target object frame;
Extracting a plurality of second text information and a plurality of second color texture information in a plurality of comparison object boxes;
Based on the similarity between the target object frame and the plurality of comparison object frames, determining object placement information of the target object includes:
sequentially comparing the plurality of second text messages with the first text message;
If the second text information is matched with the first text information, determining that the object corresponding to the comparison object box and the target object are similar objects;
if the second text information is not matched with the first text information, comparing the second color texture information with the first color texture information in sequence;
If the second color texture information is matched with the first color texture information, determining that the object corresponding to the comparison object frame and the target object are similar objects;
if any one of the second text information and the second color texture information in the comparison object frame is empty, determining that the comparison object frame is an empty object frame;
determining at least one of the placement number information of the target objects, the placement position information of the target objects and the position misplacement information of the target objects according to the type difference condition between the objects corresponding to the comparison object frame and the target objects;
And determining the vacancy position information of the target object according to the vacancy object frame.
And the information of commodity placement in the commodity placement area is obtained by extracting the target commodity frame and comparing the image feature vectors in the commodity frame and comparing the image feature vectors in sequence.
In the embodiment provided by the application, the comparison range of the commodity is determined, and a plurality of commodity frames existing in the commodity comparison range are taken as the comparison commodity frames.
Optionally, in an alternative embodiment, a target commodity placement area corresponding to an area between two adjacent tags on the left and right of the target tag is taken as a comparison area, and a specific schematic diagram is shown in fig. 5.
Optionally, comparing the feature vector of the target commodity frame with the feature vectors of the plurality of comparison commodity frames in sequence to obtain a comparison result, and obtaining the placement information of the target commodity according to the comparison result includes:
Comparing the plurality of second text messages with the first text messages in sequence, and if the second text messages are matched with the first text messages, determining that the commodity corresponding to the compared commodity frame and the target commodity are similar commodities;
if the second text information is not matched with the first text information, comparing the second color texture information with the first color texture information in sequence;
if the second color texture information is matched with the first color texture information, determining that the commodity corresponding to the commodity frame is the same type of commodity as the target commodity;
In the above description, a specific feature vector comparison flowchart is shown in fig. 6, and a general comparison result diagram is shown in fig. 7, and as shown in fig. 7, the indicated commodity representation is consistent with the type of the target commodity, and the indicated commodity representation is inconsistent with the type of the target commodity.
If any one of the second text information and the second color texture information in the comparison commodity frame is empty, determining that the comparison commodity frame is an empty commodity frame;
Determining the placement quantity information of the target commodity, the placement position information of the target commodity and the misplacement information of the target commodity according to the type difference condition between the commodity corresponding to the commodity frame and the target commodity;
and determining the vacancy position information of the target commodity according to the vacancy commodity frame.
Above-mentioned, can confirm the empty commodity frame through extracting the eigenvector in the comparison commodity frame, namely there is not commodity in the position that corresponds to commodity frame.
Further, when the situation shown in fig. 8 occurs, fig. 8 is a schematic diagram showing that the target commodity is out of stock after the feature vectors are compared, the blank part in fig. 8 is a part corresponding to the empty commodity frame, and the blank part shows that the commodity is out of stock, that is, the current commodity is considered to be out of stock, and the out-of-stock position is reported.
Further, if the situation shown in fig. 9 occurs, fig. 9 is a schematic diagram showing the situation that the misplaced commodity exists at the placement position of the target commodity after the feature vector comparison, where the commodity of the picture yoke is the misplaced commodity, that is, the situation that there is no match in the matching range of the commodity, it is possible that the customer places other commodities there, and a misplaced notification needs to be reported to inform the supermarket staff to put the other commodities back to the correct position.
Therefore, the placement number, placement position, misplacement position and vacant position of the target commodity can be determined according to the comparison condition of whether the target commodity frame belongs to the same type of commodity.
The method for determining the object placement information comprises the steps of processing an object image containing an object, determining the object price tag image as a price tag image containing the object, determining the object frame from a plurality of object frames according to the plurality of object frames determined in the object image and the relative position relation between the plurality of object frames and the object price tag image, wherein a region corresponding to one object frame is used for placing one object, determining the object placement information of the object based on the similarity between the object frame and the plurality of object frames, wherein the object placement information is at least one of placement number information of the object, placement position information of the object, vacancy position information of the object and position misplacement information of the object, and solves the technical problem that detection efficiency is low due to the fact that inspection is carried out on object conditions on a goods shelf through manpower in the related technology, and achieves the technical effect of improving goods management efficiency.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is illustrated in the flowcharts, in some cases the steps illustrated or described may be performed in an order other than that illustrated herein.
The embodiment of the invention also provides a device for determining the target object placement information, and the device for determining the target object placement information can be used for executing the method for determining the target object placement information provided by the embodiment of the invention. The following describes a device for determining placement information of a target object provided by an embodiment of the present invention.
Fig. 10 is a schematic diagram of an apparatus for determining placement information of a target object according to an embodiment of the present invention. As shown in fig. 10, the apparatus includes:
A first determining unit 101, configured to process a target image including a target object, and determine a target price tag image, where the target price tag image is a price tag image including the target object;
A second determining unit 102, configured to determine a plurality of object frames from the target image, and determine a target object frame according to the plurality of object frames and a relative positional relationship between the plurality of object frames and the target price tag image, where an area corresponding to one object frame is used for placing one target object, and the target object frame is one object frame of the plurality of object frames;
And a third determining unit 103 for determining object placement information of the target object based on the similarity between the target object frame and the plurality of comparison object frames, wherein object frames other than the target object frame among the plurality of object frames are determined as the plurality of comparison object frames.
Optionally, the object placement information comprises at least one of placement quantity information of the target objects, placement position information of the target objects, vacancy position information of the target objects and position misplacement information of the target objects.
Optionally, the device comprises a calibration unit, a first extraction unit, a detection unit, a first identification unit and a second identification unit, wherein the calibration unit is used for calibrating a goods shelf area and acquiring a target image corresponding to the calibrated goods shelf area before processing a target image containing a target object and determining the target price tag image, the first extraction unit is used for extracting edges of the target image, the detection unit is used for conducting straight line detection on the extracted edges to obtain a plurality of straight lines, the first identification unit is used for identifying the plurality of straight lines and determining an upper boundary and a lower boundary of a price tag column corresponding to the target price tag, and the second identification unit is used for identifying the price tag column through the upper boundary and the lower boundary.
Optionally, the first identifying unit comprises a first eliminating module, a calculating module, a second eliminating module and a first determining unit, wherein the first eliminating module is used for counting a plurality of slopes corresponding to a plurality of straight lines and eliminating straight lines with absolute values larger than or equal to 1 to determine a plurality of first target straight lines, the calculating module is used for calculating slope differences between every two of the plurality of first target straight lines, the second eliminating module is used for eliminating invalid straight lines to obtain a plurality of second target straight lines, the invalid straight lines are straight lines with slope differences larger than a preset value, the first determining unit 101 is used for determining distance values between every two of the plurality of second target straight lines, and the second determining unit 102 is used for determining two straight lines with distance values within a preset range as an upper boundary and a lower boundary corresponding to a price tag column respectively.
Alternatively, the first determining unit 101 includes a first scanning module for performing enlarged scanning on an image corresponding to the price tag column through the zoom ball machine, and performing quadrilateral detection on the image to obtain a quadrilateral image, an extracting module for extracting text information in the quadrilateral image, and a first determining module for determining that the quadrilateral image is the target price tag image when the text information is text related to the target object.
Optionally, the device further comprises a storage unit, which is used for storing the text information and the corresponding coordinates of the center point of the quadrilateral image in the absolute coordinate system of the zoom dome camera after determining that the quadrilateral image is the target price tag image if the text information is the text related to the target object.
Optionally, the second determining unit 102 includes a second scanning module, configured to perform enlarged scanning on an image corresponding to the target object placement area by using the zoom ball machine, where an area between two price tag columns in the target image is determined to be the target object placement area, and a second determining module, configured to detect, by using the target object detection model, the image corresponding to the target object placement area, and determine a plurality of object frames.
Optionally, the device further comprises a second extraction unit for extracting first text information and first color texture information in the target object box before determining object placement information of the target object based on the similarity between the target object box and the plurality of comparison object boxes, a third extraction unit for extracting a plurality of second text information and a plurality of second color texture information in the plurality of comparison object boxes, a third determination unit 103 for determining object placement information of the target object based on the similarity between the target object box and the plurality of comparison object boxes, the object placement information of the target object comprising a first comparison unit for comparing the plurality of second text information with the first text information in sequence, a fourth determination unit for determining that an object corresponding to the comparison object box is a homogeneous object with the target object in case of matching the second text information with the first text information, a second comparison unit for comparing the second color texture information with the first color texture information in sequence in case of non-matching the second text information with the first text information, a fifth determination unit for determining that the object corresponding to the second color texture information is a homogeneous object in sequence in case of non-matching the second text information with the first text information, a position of the second texture information is a homogeneous object in case of determining that the object corresponding to the first color information is a homogeneous object, a position is a homogeneous object in case of the second contrast object, a position is a contrast element is a homogeneous object in the first contrast element is a contrast object, a contrast object is a contrast object in case of the second color is different than the first color, a contrast element is a contrast object, a contrast object is a contrast object, and a contrast object is a contrast object is in a first contrast object, and is contrast object is, and determining the vacancy position information of the target object according to the vacancy object frame.
The device for determining target object placement information provided by the embodiment of the invention is used for processing a target image containing a target object through a first determining unit 101 to determine the target object image, wherein the target object image is a price tag image comprising the target object, a second determining unit 102 is used for determining a plurality of target object frames from the target image and determining the target object frames according to the plurality of target object frames and the relative position relation between the plurality of target object frames and the target price tag image, wherein a region corresponding to one target object frame is used for placing one target object, the target object frame is one of the plurality of target object frames, and a third determining unit 103 is used for determining the object placement information of the target object based on the similarity between the target object frame and the plurality of comparison object frames, wherein the target object frames except the target object frame are determined to be the comparison object frames, and the technical problem of low detection efficiency caused by the fact that the object condition on a shelf is checked through manpower in a supermarket and other sales sites in related technologies is solved, and the technical effect of improving goods management efficiency is achieved.
An apparatus for determining target object placement information includes a processor and a memory, the first determining unit and the like are stored as program units in the memory, and the processor executes the program units stored in the memory to realize corresponding functions.
The processor includes a kernel, and the kernel fetches the corresponding program unit from the memory. The kernel can be provided with one or more than one kernel, and the technical problem that detection efficiency is low due to the fact that the condition of objects on the goods shelf is checked by manpower in the related art is solved by adjusting kernel parameters.
The memory may include volatile memory, random Access Memory (RAM), and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM), among other forms in computer readable media, the memory including at least one memory chip.
The embodiment of the invention provides a computer storage medium, on which a program is stored, which when executed by a processor implements a method of determining target object placement information.
The embodiment of the invention provides a processor, which is used for running a program, wherein the program runs to execute a method for determining target object placement information.
The embodiment of the invention provides equipment, which comprises a processor, a memory and a program which is stored in the memory and can run on the processor, wherein the processor processes a target image containing a target object, determines the target price image, wherein the target price image is a price image comprising the target object, determines a plurality of object frames from the target image, and determines the target object frames according to the plurality of object frames and the relative position relation between the plurality of object frames and the target price image, wherein the area corresponding to one object frame is used for placing one target object, the target object frame is one object frame in the plurality of object frames, and determines the object placement information of the target object based on the similarity between the target object frame and the plurality of comparison object frames, wherein the object frames except the target object frame in the plurality of object frames are determined to be the plurality of comparison object frames.
Optionally, the object placement information comprises at least one of placement quantity information of the target objects, placement position information of the target objects, vacancy position information of the target objects and position misplacement information of the target objects.
Optionally, before processing a target image containing a target object to determine a target price tag image, the method comprises the steps of calibrating a shelf area and obtaining a target image corresponding to the calibrated shelf area, extracting edges of the target image, detecting the extracted edges in a straight line to obtain a plurality of straight lines, identifying the plurality of straight lines, determining an upper boundary and a lower boundary of a price tag column corresponding to the target price tag, and identifying the price tag column through the upper boundary and the lower boundary.
The method comprises the steps of identifying a plurality of straight lines, determining the upper boundary and the lower boundary of a price tag column corresponding to a target price tag, counting a plurality of slopes corresponding to the plurality of straight lines, eliminating the straight lines with absolute values larger than or equal to 1 to determine a plurality of first target straight lines, calculating slope differences between every two of the plurality of first target straight lines, eliminating invalid straight lines to obtain a plurality of second target straight lines, determining the distance value between every two of the plurality of second target straight lines, and determining the two straight lines corresponding to the distance value with the distance value within a preset range as the upper boundary and the lower boundary corresponding to the price tag column.
Optionally, processing a target image containing a target object to determine the target price tag image, wherein the processing comprises the steps of performing enlarged scanning on an image corresponding to a price tag column through a zoom ball machine, performing quadrilateral detection on the image to obtain a quadrilateral image, extracting text information in the quadrilateral image, and determining the quadrilateral image as the target price tag image if the text information is text related to the target object.
Optionally, after determining that the quadrilateral image is the target price tag image if the text information is text related to the target object, the method further comprises storing the text information and coordinates of a center point of the quadrilateral image corresponding in an absolute coordinate system of the zoom ball machine.
Optionally, determining the plurality of object frames from the target image includes performing enlarged scanning on an image corresponding to the target object placement area by using a zoom ball machine, wherein an area between every two price tag columns in the target image is determined to be the target object placement area, detecting the image corresponding to the target object placement area by using a target object detection model, and determining the plurality of object frames.
Optionally, before determining object placement information of the target object based on the similarity between the target object frame and the plurality of comparison object frames, the method further comprises extracting first text information and first color texture information in the target object frame, extracting a plurality of second text information and a plurality of second color texture information in the plurality of comparison object frames, determining object placement information of the target object based on the similarity between the target object frame and the plurality of comparison object frames, wherein determining object placement information of the target object comprises comparing the plurality of second text information with the first text information in sequence, determining that an object corresponding to the comparison object frame is a homogeneous object with the target object if the second text information is matched with the first text information, comparing the second color texture information with the first color texture information in sequence if the second text information is not matched with the first color texture information, determining that an object corresponding to the comparison object frame is a homogeneous object with the target object if any one of the second text information and the second color texture information in the target object frame is an empty object, determining that the position of the object corresponding to the target object is an empty object according to at least, determining that the position of the object corresponding to the empty object is an empty object, and determining the position of the object is at least according to the empty position of the empty object. The device herein may be a server, PC, PAD, cell phone, etc.
The invention also provides a computer program product which is suitable for executing a program initialized with the following method steps when being executed on data processing equipment, wherein the program is used for processing a target image containing a target object, determining the target price label image, wherein the target price label image is a price label image comprising the target object, determining a plurality of object frames from the target image, and determining the target object frames according to the plurality of object frames and the relative position relation between the plurality of object frames and the target price label image, wherein the area corresponding to one object frame is used for placing one target object, the target object frame is one object frame in the plurality of object frames, and determining the object placement information of the target object based on the similarity between the target object frame and a plurality of comparison object frames, wherein the object frames except the target object frame are determined to be the comparison object frames.
Optionally, the object placement information comprises at least one of placement quantity information of the target objects, placement position information of the target objects, vacancy position information of the target objects and position misplacement information of the target objects.
Optionally, before processing a target image containing a target object to determine a target price tag image, the method comprises the steps of calibrating a shelf area and obtaining a target image corresponding to the calibrated shelf area, extracting edges of the target image, detecting the extracted edges in a straight line to obtain a plurality of straight lines, identifying the plurality of straight lines, determining an upper boundary and a lower boundary of a price tag column corresponding to the target price tag, and identifying the price tag column through the upper boundary and the lower boundary.
The method comprises the steps of identifying a plurality of straight lines, determining the upper boundary and the lower boundary of a price tag column corresponding to a target price tag, counting a plurality of slopes corresponding to the plurality of straight lines, eliminating the straight lines with absolute values larger than or equal to 1 to determine a plurality of first target straight lines, calculating slope differences between every two of the plurality of first target straight lines, eliminating invalid straight lines to obtain a plurality of second target straight lines, determining the distance value between every two of the plurality of second target straight lines, and determining the two straight lines corresponding to the distance value with the distance value within a preset range as the upper boundary and the lower boundary corresponding to the price tag column.
Optionally, processing a target image containing a target object to determine the target price tag image, wherein the processing comprises the steps of performing enlarged scanning on an image corresponding to a price tag column through a zoom ball machine, performing quadrilateral detection on the image to obtain a quadrilateral image, extracting text information in the quadrilateral image, and determining the quadrilateral image as the target price tag image if the text information is text related to the target object.
Optionally, after determining that the quadrilateral image is the target price tag image if the text information is text related to the target object, the method further comprises storing the text information and coordinates of a center point of the quadrilateral image corresponding in an absolute coordinate system of the zoom ball machine.
Optionally, determining the plurality of object frames from the target image includes performing enlarged scanning on an image corresponding to the target object placement area by using a zoom ball machine, wherein an area between every two price tag columns in the target image is determined to be the target object placement area, detecting the image corresponding to the target object placement area by using a target object detection model, and determining the plurality of object frames.
Optionally, before determining object placement information of the target object based on the similarity between the target object frame and the plurality of comparison object frames, the method further comprises extracting first text information and first color texture information in the target object frame, extracting a plurality of second text information and a plurality of second color texture information in the plurality of comparison object frames, determining object placement information of the target object based on the similarity between the target object frame and the plurality of comparison object frames, wherein determining object placement information of the target object comprises comparing the plurality of second text information with the first text information in sequence, determining that an object corresponding to the comparison object frame is a homogeneous object with the target object if the second text information is matched with the first text information, comparing the second color texture information with the first color texture information in sequence if the second text information is not matched with the first color texture information, determining that an object corresponding to the comparison object frame is a homogeneous object with the target object if any one of the second text information and the second color texture information in the target object frame is an empty object, determining that the position of the object corresponding to the target object is an empty object according to at least, determining that the position of the object corresponding to the empty object is an empty object, and determining the position of the object is at least according to the empty position of the empty object.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, etc., such as Read Only Memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, object, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, object, or apparatus. Without further limitation, an element defined by the phrase "comprising one does not exclude the presence of additional like elements in a process, method, object or apparatus comprising the element.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The foregoing is merely exemplary of the present invention and is not intended to limit the present invention. Various modifications and variations of the present invention will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the invention are to be included in the scope of the claims of the present invention.
Claims (11)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110990009.1A CN113627415B (en) | 2021-08-26 | 2021-08-26 | Method and device for determining placement information of target object |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110990009.1A CN113627415B (en) | 2021-08-26 | 2021-08-26 | Method and device for determining placement information of target object |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113627415A CN113627415A (en) | 2021-11-09 |
| CN113627415B true CN113627415B (en) | 2025-02-28 |
Family
ID=78387950
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110990009.1A Active CN113627415B (en) | 2021-08-26 | 2021-08-26 | Method and device for determining placement information of target object |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113627415B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114170204B (en) * | 2021-12-13 | 2024-06-21 | 国网重庆市电力公司电力科学研究院 | Image metering pricing method |
| CN114494921A (en) * | 2022-01-25 | 2022-05-13 | 上海汉时信息科技有限公司 | Goods shelf commodity identification method and system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110826359A (en) * | 2018-08-09 | 2020-02-21 | 北京京东尚科信息技术有限公司 | Method and device for detecting article display |
| CN110889419A (en) * | 2018-09-07 | 2020-03-17 | 杭州海康威视数字技术股份有限公司 | Shelf analysis method, device and system and electronic equipment |
| CN111783627A (en) * | 2020-06-29 | 2020-10-16 | 杭州海康威视数字技术股份有限公司 | Commodity stock determining method, device and equipment |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP6938169B2 (en) * | 2017-03-01 | 2021-09-22 | 東芝テック株式会社 | Label generator and program |
| CN108647553B (en) * | 2018-05-10 | 2022-01-25 | 上海扩博智能技术有限公司 | Method, system, device and storage medium for rapidly expanding images for model training |
| CN111080695B (en) * | 2019-12-31 | 2023-10-20 | 合肥美的智能科技有限公司 | Vending apparatus, monitoring method and monitoring device thereof |
| CN112215142B (en) * | 2020-10-12 | 2021-08-13 | 上海汉时信息科技有限公司 | Method, device and equipment for detecting goods shelf stock shortage rate based on depth image information |
| CN112990095B (en) * | 2021-04-13 | 2021-09-14 | 广州市玄武无线科技股份有限公司 | Commodity display analysis method, commodity display analysis device, commodity display analysis equipment and storage medium |
| CN112949785B (en) * | 2021-05-14 | 2021-08-20 | 长沙智能驾驶研究院有限公司 | Object detection method, device, equipment and computer storage medium |
-
2021
- 2021-08-26 CN CN202110990009.1A patent/CN113627415B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110826359A (en) * | 2018-08-09 | 2020-02-21 | 北京京东尚科信息技术有限公司 | Method and device for detecting article display |
| CN110889419A (en) * | 2018-09-07 | 2020-03-17 | 杭州海康威视数字技术股份有限公司 | Shelf analysis method, device and system and electronic equipment |
| CN111783627A (en) * | 2020-06-29 | 2020-10-16 | 杭州海康威视数字技术股份有限公司 | Commodity stock determining method, device and equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113627415A (en) | 2021-11-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107292248B (en) | Commodity management method and system based on image recognition technology | |
| EP3910608B1 (en) | Article identification method and system, and electronic device | |
| US9697429B2 (en) | Method and apparatus for image processing to avoid counting shelf edge promotional labels when counting product labels | |
| US20180253674A1 (en) | System and method for identifying retail products and determining retail product arrangements | |
| US20150262116A1 (en) | Machine vision technology for shelf inventory management | |
| US20140003655A1 (en) | Method, apparatus and system for providing image data to represent inventory | |
| CN113627415B (en) | Method and device for determining placement information of target object | |
| US20210166028A1 (en) | Automated product recognition, analysis and management | |
| US11748787B2 (en) | Analysis method and system for the item on the supermarket shelf | |
| US20210272170A1 (en) | Method, Device, Electronic Apparatus and Storage Medium for Generating Order | |
| JP2016201105A (en) | Information processor and information processing method | |
| CN115587769A (en) | Method and device for detecting commodity out-of-stock state, computer equipment and storage medium | |
| US11704787B2 (en) | Method and system for determining stock in an inventory | |
| CN111783627A (en) | Commodity stock determining method, device and equipment | |
| US20250086585A1 (en) | Retail shelf image processing and inventory tracking system | |
| CN117197420A (en) | Method and device for generating shelf display grid area | |
| CN113269053A (en) | Commodity shelf-level acquisition method, terminal and storage device | |
| CN115661624A (en) | Digital method and device for goods shelf and electronic equipment | |
| CN115601672A (en) | VR intelligent shop patrol method and device based on deep learning | |
| Pietrini et al. | Embedded vision system for real-time shelves rows detection for planogram compliance check | |
| US12136061B2 (en) | Retail shelf image processing and inventory tracking system | |
| US11875457B2 (en) | 3D product reconstruction from multiple images collected at checkout lanes | |
| CN116580390A (en) | Price tag content acquisition method, price tag content acquisition device, storage medium and computer equipment | |
| US20230419702A1 (en) | System, method and apparatus for price label modeling tool | |
| CN115953767A (en) | Array analysis method, array analysis apparatus, electronic device, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |