CN118097519B

CN118097519B - Intelligent shopping cart shopping behavior analysis method and system based on commodity track analysis

Info

Publication number: CN118097519B
Application number: CN202410499341.1A
Authority: CN
Inventors: 王庆刚; 徐步兵; 李晨; 余佳; 马晓慧
Original assignee: Nanjing Yimao Information Technology Co ltd
Current assignee: Nanjing Yimao Information Technology Co ltd
Priority date: 2024-04-24
Filing date: 2024-04-24
Publication date: 2024-06-25
Anticipated expiration: 2044-04-24
Also published as: CN118097519A

Abstract

The application discloses an intelligent shopping cart shopping behavior analysis method and system based on commodity track analysis, and belongs to the field of intelligent shopping carts. In the prior art, the intelligent shopping cart only relies on weight change to estimate the behavior of a customer as an antitheft means, and the contradiction between the antitheft requirement of a supermarket and smooth shopping of customers cannot be solved. According to the application, shopping video images are acquired through a camera arranged on the intelligent shopping cart; applying a target detection and identification model to obtain the position and category information of the commodity target in each frame of image; inputting the result into a target tracking model to obtain commodity track data; dividing the track into a static track and a moving track after calculating the motion measurement index of the target track; and analyzing the motion trail and corresponding the motion trail to shopping behaviors of commodity placement, commodity taking out and commodity arrangement. The application can rapidly and accurately identify the shopping behavior of the user under the condition that the consumer is completely free of sense, thereby improving the user experience and effectively reducing the risk of goods loss.

Description

Intelligent shopping cart shopping behavior analysis method and system based on commodity track analysis

Technical Field

The application relates to the field of intelligent shopping carts, in particular to aspects of intelligent shopping cart scene perception, shopping behavior analysis, intelligent shopping cart supermarket theft prevention and the like.

Background

Artificial intelligence technology is rapidly changing our production and lifestyle. The conventional shopping cart commonly adopted in the retail field such as supermarkets has a history of more than one hundred years, the shopping cart only provides a commodity temporary storage function, and customers need to push the shopping cart to a designated place for queuing and settlement after shopping is finished, so that time is consumed, certain congestion is possibly caused, and the shopping experience of the users is reduced. Many supermarkets currently employ intelligent shopping carts with customer self-checkout functionality. The intelligent shopping cart improves the shopping experience of users and simultaneously provides higher requirements for commodity theft prevention of supermarkets. In order to effectively catch the actions of missing sweeping, entrainment, goods replacement and the like. The intelligent shopping cart is generally provided with a high-precision gravity electronic scale at the bottom of the shopping cart, and theft prevention is carried out in a mode of comparing the actual weight of commodities with the standard weight of commodities, for example, patent document CN106981150A discloses an intelligent anti-theft system and method for a supermarket, and patent CN106408369A discloses an algorithm for intelligently identifying information of the commodities in the shopping cart.

In order to distinguish weight differences between different types of merchandise, weight sensors employed in intelligent shopping carts typically have high accuracy and sensitivity. In the supermarket shopping process, the shopping behaviors of users have randomness and complexity, and the purchased commodities have diversity. For example, the weight value can be offset due to the touch of a user, the behavior of a shopping frame being held by hand, and the like, the actual weight of a commodity with a larger size can be smaller than the standard weight of the commodity due to the leaning of the frame, the actual weight of the commodity can be inconsistent with the weight of the commodity marked in a database due to the change of the commodity packaging, and a plurality of commodities with different prices and the same commodity quality exist in a supermarket. These conditions often result in consumers shopping in normal operating procedures and not settling accounts.

Although the loss rate of the supermarket is reduced to a certain extent, the intelligent shopping cart based on strict gravity loss prevention often causes the situation that consumers can not settle accounts due to normal purchase, and the willingness of the consumers to use the intelligent shopping cart is reduced. In order to effectively solve the contradiction between effective loss prevention of the intelligent shopping cart and fluency, convenience and the like of consumer operation, the behavior of putting in or taking out commodities from the shopping cart and arranging the commodities in the shopping cart by a consumer needs to be accurately identified, the fluency of the consumer shopping process is ensured on the premise of meeting the effective anti-theft requirement of the supermarket, and the user experience is improved.

Disclosure of Invention

1. Technical problem to be solved

Aiming at the contradiction between the fluency and convenience requirements of consumer operation and effective loss prevention of the supermarket in the existing intelligent shopping cart, the invention provides a user shopping behavior analysis method based on analysis of the movement track of commodities in videos, which can decompose one-time shopping behavior of a customer into three basic types: putting in, taking out and finishing. The invention can rapidly and accurately identify the shopping behavior of the user under the condition that the consumer is completely free of sense, thereby improving the user experience and effectively reducing the risk of goods loss.

2. Technical proposal

The aim of the invention is achieved by the following technical scheme.

The intelligent shopping cart is provided with at least one shopping cart body provided with a shopping basket, and an image acquisition device on the cart body is used for monitoring the movement of a target in the shopping cart, and a gravity scale is arranged at the bottom of the shopping cart basket. In some embodiments, a processor is disposed on the vehicle body.

The invention provides an intelligent shopping cart shopping behavior analysis method based on commodity track analysis, which comprises the following steps:

(1) And acquiring videos in the shopping process of the user to obtain an image sequence. Preferably, the RGB camera installed on the intelligent shopping cart is used for collecting real-time video of the user in the shopping process.

(2) And inputting the image sequence into a target detection and identification model to obtain the coordinate information of all commodity targets in all N frames of images. Firstly, inputting an image sequence into a target detection recognition model to obtain coordinate information and category information (x 0, y0, w, h, cls) of all targets in all images, wherein x0 and y0 are the centers of target detection frames, w is the width of the target detection frames, h is the height of the target detection frames, and cls is the target category. The targets include both commodity targets and non-commodity targets.

According to the target class information cls output by the target detection and identification model, excluding non-commodity targets, and only analyzing the commodity targets. And outputting the coordinate information and the category information of the commodity target.

(3) And inputting the commodity target coordinate information into a target tracking model to obtain commodity tracks of all commodity targets in the image sequence. Specifically, the output (x 0, y0, w, h, cls) of the object detection recognition model is input into the object tracking model, so as to obtain track data of K objects in N frames of images of the image sequence: { track_k= [ (x0_i_k, y0_i_k, w_i_k, h_i_k, cls_k) ] }, where track_k is Track data where ID (target identification number) obtained by tracking the model is K, k=1, …, K, i=1, …, N, where (x0_i_k, y0_i_k, w_i_k, h_i_k, cls_k) represents detection frame coordinates and category information in the i-th frame image of the Track of id=k.

(4) And calculating a motion measurement index of each commodity track, and judging that each commodity track is one of a motion track or a static track. The method comprises the following steps: calculating a motion measurement index of the actual motion condition of the commodity in the shopping process, setting one or more thresholds, and dividing the commodity track into a motion track and a rest track by comparing the motion measurement index with the thresholds.

Let (xi, yi) be the i-th frame commodity track point coordinates, wherein the commodity track point is any one of the commodity target detection frame center point or the commodity target detection frame four corner points, and the motion measurement index is one or more of the following three combinations:

1) Track length: representing the actual movement track length of the commodity in the shopping process; the method is defined as the accumulation sum of Euclidean distances between the coordinates of the same commodity track point in all adjacent frames of the commodity track, wherein the unit is pixel number, and the calculation method is as follows:

，

dist is an algorithm for calculating the euclidean distance between two coordinate points, and i and i+1 are frame index numbers of two adjacent frames.

2) Maximum track distance: characterizing the magnitude of movement of a commodity in space during shopping; the maximum value of the coordinate distance of the same commodity track point in any two frames is defined, and the unit is the pixel number; the calculation method comprises the following steps:

，

max is an algorithm for maximizing; i and j are frame index numbers of any two different frames.

3) The track relative maximum distance is defined as the relative movement distance of the track maximum distance relative to the commodity size, the unit is pixel number, and the calculation method is as follows:

，

wi is the width of the commodity detection frame in the i frame and hi is the height of the commodity detection frame in the i frame.

Further, a motion measurement index of each commodity track is calculated, and each commodity track is judged to be one of a motion track or a static track. The specific method comprises the following steps:

1) For each commodity track, calculating coordinates of four corner points of the target detection frame by using the target detection frame position information (x 0, y0, w, h and cls) of the track in the image:

x0 and y0 are the centers of the target detection frames, w and h are the width and height of the target detection frames, and cls is the target class;

2) The track length of four corner points of each target detection frame is calculated:

3) And obtaining the corner point with the minimum track length of each commodity target.

4) Track length Trajlen, track maximum distance Distmax and track relative maximum distance Distref with the minimum track length angle point are calculated and used as track motion measurement indexes of the commodity track.

5) The following conditions were set:

condition 1: trajlen < THRESH1;

Condition 2: distmax < THRESH2;

condition 3: distref < THRESH3;

Condition 4: trajlen < THRESH4 and Distref < THRESH5;

condition 5: distmax < THRESH1 and Distref < THRESH5;

THRESH1, THRESH2, THRESH3, THRESH4 and THRESH5 are a set of thresholds;

judging one or more of the conditions 1,2, 3, 4 and 5, and marking the commodity track as stationary as long as one of the judging results is true; and finally, marking all the commodity tracks which are not marked as static as motion.

(5) And analyzing all the commodity movement tracks, and further identifying the commodity movement tracks as one of putting the commodity into a shopping cart, taking the commodity out of the shopping cart or sorting the commodity. The specific method comprises the following steps:

1) Frame index numbers of moving parts of the merchandise object in the image sequence are precisely located.

The target is detected by common M frames of images in the image sequence, and the corresponding image frame index numbers are as follows:

，

The coordinates of the corner point with the minimum track length in the four corner points of the target detection frame are (x_i, y_i), wherein i=1, … and M-1, and the Euclidean distance between the adjacent i frames and i+1 frames in the corner point track data is as follows:

，

The element in neighbor_ trajdist represents the euclidean distance of the motion of the ith+1st Frame and the ith Frame compared with the angular point, the unit is pixel number, a threshold value thresh_nb is set, and the condition is satisfied in the calculated Frame index number frame_id:

，

Video frame index number sequence of (c):

，

frame_id_move is a subset of frame_id, I _m1, …, I_mK is a Frame index sequence number value, and if the distance between the minimum track length corner points between adjacent frames is greater than the threshold thresh_nb, it indicates that the commodity moves between the two frames.

2) A sequence of consecutive motion Frame indices in frame_id_move is calculated.

Adjacent elements I _m（i+1） and I _mi in the Frame index sequence frame_id_move are differenced, and I _m（i+1）-I_mi =1 represents a continuous Frame index sequence in which I _m（i+1）、I_mi is commodity motion in video.

The video segment corresponding to the set of frame_id_move_flag of the continuous Frame index number sequence is the precise Frame index of the continuous motion segment of the commodity in the whole video sequence.

Judging the behavior of the commodity according to the time period of the continuous motion segment of the commodity in the commodity track and the change of the weighing value of the gravity scale:

if the continuous motion segment corresponds to the initial part in the Frame index number frame_ID, the commodity is in a motion state when the commodity is detected and tracked, and if the gravity weighing value is increased and reaches a stable state, the commodity track indicates that the commodity is placed in the shopping cart.

If the continuous motion segment corresponds to the end part in the Frame index number frame_ID, the commodity target is in a static state when the commodity target is detected and tracked, the state of the commodity is changed from static to motion, and if the gravity weighing value is reduced and reaches a stable state at the moment, the commodity track is taken out of the commodity.

If the continuous segment corresponds to the middle part in the Frame index number frame_ID, the commodity is in a static state in the starting stage and the ending stage, and if the gravity weighing value is consistent when the commodity is in the static state, the commodity track is the sorted commodity.

The second aspect of the invention provides an intelligent shopping cart shopping behavior analysis system based on commodity track analysis, and the intelligent shopping cart shopping behavior analysis method based on commodity track analysis provided by the first aspect of the invention comprises the following steps:

And an image acquisition module: is responsible for video and image acquisition.

An image detection and tracking module: and the image acquisition module is responsible for receiving and storing the video stream information acquired by the image acquisition module, obtaining an image sequence, performing target detection and target tracking processing on images in the image sequence, identifying all targets in the images, and outputting track information of commodity targets.

And an analysis decision module: the method is responsible for receiving commodity target track information of the image detection tracking module and eliminating static targets; and accurately positioning the time period that the continuous motion segment in the commodity track is in the commodity track, and comprehensively judging whether the commodity is one of three behaviors of putting in, taking out and arranging by combining the gravity weighing value change data.

Preferably, the method for accurately positioning the time period of the continuous motion segment in the commodity track by the intelligent shopping cart shopping behavior analysis system based on commodity track analysis comprises the following steps:

，

The coordinates of the corner point with the minimum track length in the four corner points of the target detection frame are (x_i, y_i), wherein i=1, … and M-1, and the Euclidean distance between adjacent frames in the corner point track data is as follows:

，

The element in neighbor_ trajdist represents the euclidean distance of the motion of the ith+1st Frame and the ith Frame compared with the angular point, the unit is the pixel number, a threshold value thresh_nb is set, and the condition is satisfied in the calculation frame_id:

，

Video frame index number sequence of (c):

，

Differencing adjacent elements I _m（i+1） and I _mi in the Frame index sequence frame_id_move, I _m（i+1）-I_mi =1 representing that I _m（i+1）、I_mi is a continuous Frame index sequence of commodity motion in video;

Preferably, the algorithm model of the present invention is deployed on a processor disposed on the body of the intelligent shopping cart.

3. Advantageous effects

Compared with the prior art, the invention has the advantages that:

(1) The invention is based on the original intelligent shopping cart equipment, does not need to add extra parts, and saves the cost.

(2) According to the method provided by the invention, the actions of entraining, replacing goods and the like are accurately identified by detecting, tracking, track analysis and commodity image comparison and the like of the moving commodities in the video through the artificial intelligence technology, so that the risk of the loss of goods in the supermarket is effectively reduced.

(3) The whole processing flow does not need user participation, and can effectively eliminate the behavior of abnormal change of the weighing value caused by touching, holding frames, commodity pressing frames and the like under the condition that the user is completely free of sense, so that the fluency of the user in using the intelligent shopping cart is improved.

(4) The algorithm model is deployed on a processor arranged on the intelligent shopping cart body, real-time detection, tracking and comparison analysis of moving goods can be realized by utilizing the self-contained operation chip, the execution efficiency of the algorithm can be improved to the greatest extent, and the execution time of the algorithm is shortened.

Drawings

FIG. 1 is a schematic flow chart of the present invention;

FIG. 2 is a schematic diagram of a shopping cart;

FIG. 3 is a graph showing the output of the merchandise detection and identification model before the merchandise is placed in the shopping cart (frame 24);

FIG. 4 is a graph showing the output of the merchandise detection identification model during the merchandise placement process (frame 44);

FIG. 5 is a plot of the center point of a tracking target during placement of merchandise in a shopping cart;

FIG. 6 is a graph of the trajectories of four corner points and center points of the object ID_1;

FIG. 7 is a graph of the trajectories of four corner points and center points of the object ID_7;

FIG. 8 is a graph of the trajectories of four corner points and a center point of the object ID_3;

FIG. 9 is a graph of the trajectories of four corner points and center points of the object ID_10;

Fig. 10 shows the pixel distance between adjacent frames of the minimum target track length corner.

Detailed Description

The invention will now be described in detail with reference to the drawings and the accompanying specific examples. For the purpose of the present invention, an intelligent shopping cart needs to accurately sense whether a user is putting in or taking out goods from the shopping cart, or simply sort the goods in the shopping cart. In some embodiments, the category and number of articles to be placed or removed are further identified. According to the invention, through a target detection tracking algorithm in the field of computer vision, a moving target of interest in a video image is effectively captured, and a unique ID number is assigned to the commodity target. The analysis of the purchasing behavior of the user is realized by analyzing the motion trail characteristics of the commodity, and the image of the motion commodity is compared with the commodity image in the database to accurately identify the commodity category.

Example 1

In a first aspect of the present invention, an intelligent shopping cart shopping behavior analysis method based on commodity track analysis is provided, as shown in fig. 1, wherein one implementation method is as follows:

Real-time video of a user in the shopping process is collected by using an RGB camera installed on the intelligent shopping cart shown in FIG. 2, and a series of images are generated by the video, wherein the total number of the images is N. The installation mode of the camera aims at obtaining clear commodity images in the shopping process as far as possible, and comprises the steps of adopting a plurality of cameras with different angles, auxiliary light sources and the like. Inputting an image sequence formed by N frames of images into a target detection and identification model to obtain the position and commodity category information of all targets in all images, wherein the j-th detection frame information of the i-th frame image output by the model is (x0_i_j, y0_i_j, w_i_j, h_i_j, cls_i_j), wherein (x0_i_j, y0_i_j) is the center of the detection frame of the target, and (w_i_j, h_i_j) is the width and height of the target detection frame, and cls_i_j is the category to which the target belongs.

All targets herein include commodity targets and non-commodity targets in supermarkets. There are two types of category definition of the commodity targets: 1) Defining a classification method of commodity types in a supermarket; 2) The products in the supermarket are classified into several large categories according to the apparent similarity of the products. Non-merchandise classes include hands and children that appear more often in images. According to the target class information cls output by the target detection and identification model, excluding non-commodity targets, and only analyzing the commodity targets.

The input target detection and identification model video sequence can be real-time video stream acquired by a camera on the shopping cart, or can be a video segment intercepted according to the change of the weighing value of a gravity electronic scale arranged on the shopping cart and the code scanning operation of a user.

The target detection and recognition algorithm comprises, but is not limited to, a moving target detection algorithm based on optical flow, background modeling and the like, a mode classification algorithm such as a support vector machine, a decision tree and the like; and various target detection and recognition algorithms based on deep learning, such as FASTER RCNN, yolo series, SWIN and the like. The algorithm deployment platform comprises edge equipment and an algorithm server, and the operation chips comprise, but are not limited to, various GPU, NPU, TPU, CPU chips and the like.

Fig. 3 is an output result of the commodity detection and identification model before the commodity is put into the purchase vehicle (24 th frame) in one shopping process, and fig. 4 is an output result of the target detection and identification model in the commodity putting process (44 th frame).

The output [ (x0_i_j, y0_i_j, w_i_j, h_i_j, cls_i_j) ] of the object detection recognition model is input into the object tracking model, so as to obtain Track data { track_k= [ (x0_i_k, y0_i_k, w_i_k, h_i_k, cls_k) ] of K objects in the video sequence composed of N frames of images, wherein track_k is Track data of which ID number obtained by the tracking model is K, i is image sequence number, i=1, …, N, K is identification sequence number of the objects, k=1, …, K, (x0_i_k, y0_i_k, w_i_k, h_i_k, cls_k) represents detection frame coordinates and category information in the ith frame image of the commodity Track with the sequence number K.

The target tracking model includes, but is not limited to, target tracking algorithm models such as DeepSort, byteSORT and BotSORT. FIG. 4 is a plot of the center point of a tracking target during the placement of merchandise in a shopping cart.

The track data (x0_i_k, y0_i_k, w_i_k, h_i_k) output by the target tracking model are utilized to define motion metric indexes of the target track, the indexes can effectively describe actual motion conditions of commodities in the shopping process, and specific index definitions can be designed according to specific conditions of the embodiment.

Let (xi, yi) be the coordinates of the target track point, i=1, … …, N be the total frame number of the image in the video, where the track point may be the center point of the detection frame, or may be one of four corner points of the detection frame, and the corner point coordinates may be calculated according to the coordinates (x 0, y 0) of the center point of the detection frame and the width and height (w, h) of the detection frame.

In this embodiment, the following three indexes are preferable:

，

dist is an algorithm for calculating Euclidean distance between two coordinate points, and i and i+1 are frame index numbers of two adjacent frames;

，

max is an algorithm for maximizing; i and j are frame index numbers of any two different frames;

，

Classifying the target track according to the motion measurement index of the track, and dividing the commodity track into a motion track and a stationary track. The specific method comprises the following steps:

1. For each commodity track, calculating coordinates of four corner points of the detection frame by using the position information (x 0, y0, w, h, cls) of the detection frame of the track in the image:

2. the track length of four corner points of each target detection frame is calculated:

3. Obtaining corner points with minimum track length of each commodity target;

4. calculating the track length Trajlen, the track maximum distance Distmax and the track relative maximum distance Distref with the minimum track length angle point as track motion measurement indexes of the commodity track; in this example, min (Trajlen), min (Distmax) and Min (Distref) are shown.

In a continuous shopping process, the commodity track of commodities put in or taken out from a shopping cart by a user is shown to have a large distribution range in space.

In this embodiment, id_7 is a commodity of the shopping cart placed in the current shopping behavior, and fig. 7 is a trace diagram of four corner points and a center point of the target. Original commodities in the shopping cart are in a relatively static state in the whole shopping process, and the track distribution of commodity targets is concentrated. FIG. 6 is a plot of four corner and center points of a stationary object ID_1 within a shopping cart. In addition, for stationary commodities, the detection frame of the stationary commodities is changed due to shielding of hands, moving commodities and the like in the shopping process, but the track distribution of one corner point is concentrated in four corner points of the detection frame, as shown in fig. 8 and 9. Therefore, for each track motion measurement index, the track motion measurement index of the commodity track is calculated by selecting the corner point with the minimum track length Trajlen of the four corner points (the upper left, the upper right, the lower left and the lower right) of the detection frame, and the commodity track is classified in a static and motion mode according to the track length Trajlen, the track maximum distance Distmax and the track relative maximum distance Distref of the corner points.

5. A series of thresholds and criteria are set to determine whether the product track is stationary or moving, and in particular embodiments, different thresholds, combinations of thresholds, and conditions or combinations of conditions may be set. In this embodiment, the following conditions are preferably set up:

condition 1: trajlen < THRESH1;

Condition 2: distmax < THRESH2;

condition 3: distref < THRESH3;

Condition 4: trajlen < THRESH4 and Distref < THRESH5;

condition 5: distmax < THRESH1 and Distref < THRESH5;

For all target tracks output by the target tracking model, firstly filtering uninteresting targets such as hands, children, pets and the like according to category information; then, the conditions 1 to 5 are calculated in turn, and the target track is marked as stationary as long as one of the conditions is true. And finally, marking the rest track which is not marked as stationary as a motion track.

Let the width and height of the image acquired by the camera be WI and HI, and the values of a group of typical WI and HI are 1024 and 1280. Preferably, in this embodiment, a set of typical thresholds is taken as:

THRESH1 = min（WI，HI）/10；

THRESH2 = min（WI，HI）/15；

THRESH3 = 0.12；

THRESH4 = min（WI，HI）/8；

THRESH5 = 0.2；

In fig. 5, a total of 5 tracks, id_1, id_2, id_3, id_7, and d_10, are shown as target tracks of shopping video in this embodiment. Wherein id_1 and id_2 are existing static commodities in a shopping cart, id_3 and id_10 are the same commodity in the shopping cart, id_3 is lost due to shielding in the process of purchasing by a user, a new ID number is given to the commodity after re-detection, id_7 is the commodity purchased in the video, track motion measurement indexes of 5 tracks are calculated, and the minimum value of each angular point motion measurement index of each track is shown in table 1:

table 1: minimum value of motion measurement index of each angular point of 5 tracks

And marking the track ID_7 as motion according to the condition 1-5.

Analyzing the target track marked as movement in the previous step, and judging one of the following three shopping behaviors according to the shopping behavior corresponding to the track by combining the change of the weighing value of the gravity scale: the method comprises the specific steps of putting commodities into a shopping cart, taking out or arranging the commodities from the shopping cart:

(1) Frame index numbers in the video sequence of moving parts of the merchandise object are precisely located.

，

The corner coordinates with the minimum track length in the four corners of the target detection frame are (x_i, y_i), wherein i=1, …, M-1 and i _1……I_M are frame index sequence number values. The euclidean distance between adjacent frames in the corner trace data is as follows:

，

Setting a threshold value THRESH_Nb, and calculating the satisfaction condition in the Frame index number frame_ID

，

Video frame index number sequence of (2) to obtain:

，

Neighbor_ trajdist represents the euclidean distance of the motion of the corner point between adjacent frames, the unit is the pixel number, for a moving object, the corner point of the same object detection frame is larger between adjacent frames, the typical value of the threshold value thresh_nb can be min (WI, HI)/10, and if the distance of the corner point between adjacent frames is larger than the threshold value, the motion of the commodity between the two frames is indicated. Frame_id_move is a subset of frame_id and I _m1,…,I_mK is a Frame index sequence number value.

(2) Successive Frame indexes in frame_id_move are calculated.

This step differends adjacent elements I _m（i+1） and I _mi in the Frame index sequence frame_id_move, I _m（i+1）-I_mi =1 representing a sequence of consecutive Frame indices where I _m（i+1）、I_mi is the motion of a commodity in video.

，

Index_frag represents that elements with adjacent Index numbers interpolated to 1 are taken from the Frame Index number sequence of frame_ID_move to obtain the sequence of the adjacent Index numbers in the frame_ID_move subset, and the set frame_ID_move_frag of the continuous Frame Index number sequence obtains the corresponding Frame Index number from the original frame_ID_move through the index_frag sequence number. The corresponding video clip is the exact frame index of the motion of the commodity throughout the video sequence.

Judging the behavior of the commodity according to the time period of the motion part of the commodity target in the commodity track and the change of the weighing value of the gravity scale:

if the continuous motion video segment corresponds to the Frame index number frame_ID, namely the initial part in all Frame index number values, the commodity is in a motion state when the commodity is detected and tracked, and if the gravity weighing value is increased and reaches a stable state at the moment, the commodity is placed in a shopping cart.

If the continuous motion video segment corresponds to the end part in the Frame index number frame_ID, the commodity target is in a static state when the commodity target is detected and tracked, the state of the commodity is changed from static to motion, and if the gravity weighing value is reduced and reaches a stable state at the moment, the track is taken out of the marked commodity.

If the continuous video segment corresponds to the middle part of the Frame index number frame_ID, the commodity is in a static state in the starting stage and the ending stage, and if the gravity weighing values are consistent when the commodity is in the static state, the track is marked and tidied.

And according to the conditions 1-5, marking the commodity track corresponding to the track ID_7 as motion. The track has the following frame index numbers corresponding to the video sequence:

，

Illustratively, the target is detected in 29, 30, … …, 74 frames of images.

Fig. 10 shows that the euclidean distance neighbor_ trajdist (unit: pixel number) between adjacent frames of the minimum target track length corner points is 10 video frame index numbers satisfying the commodity motion condition, taking the threshold thresh_nb=13:

，

The Frame index in frame_id_move satisfies the commodity continuous motion continuous Frame index sequence of 8, and their sequence number in frame_id_move is:

，

corresponds to index number 1, 29, in Frame_ID_move, due to

，

The video frame corresponding to the frame index 29 is also a motion frame, so the exact continuous motion segment index of the commodity in the entire video sequence is:

，

Frame_id_move_flag is at the beginning of the index sequence frame_id= [29, 30, …,74] indicating that the item has a large amplitude of continuous motion when initially detected, and tends to rest after stepping after Frame 40, corresponding shopping behavior is to place the item into the shopping cart.

Further, comparing the detected images in the track of the commodity with the images in the database corresponding to the code scanning commodity to judge whether the commodity is the code scanning commodity; and comparing the detected images in the commodity track with the commodity images in the shopping list for judging the commodity taken out from the shopping cart, and judging which commodity the user takes out.

The comparison method comprises two steps of feature extraction and feature matching. The feature extraction model includes but is not limited to the feature extraction model ResNet, mobileNet, vit and the feature matching algorithm includes but is not limited to the feature matching algorithm ArcFace, cosFace, sphereFace.

Example 2

The invention also provides an intelligent shopping cart shopping behavior analysis system based on commodity track analysis, which is provided by the embodiment 1, and comprises the following steps:

，

Video frame index number sequence of (c):

，

Frame_id_move is a subset of frame_id, I _m1,…,I_mK is a Frame index sequence number value, and if the distance between the minimum track length corner points between adjacent frames is greater than the threshold thresh_nb, it indicates that the commodity moves between the two frames.

Adjacent elements I _m（i+1） and I _mi in the Frame index sequence frame_id_move are differenced, and I _m（i+1）-I_mi =1 represents a continuous Frame index sequence in which I _{m（i+1）、}I_mi is commodity motion in video.

The foregoing has been described schematically the invention and embodiments thereof, which are not limiting, but are capable of other specific forms of implementing the invention without departing from its spirit or essential characteristics. The drawings are also intended to depict only one embodiment of the invention, and therefore the actual construction is not intended to limit the claims, any reference number in the claims not being intended to limit the claims. Therefore, if one of ordinary skill in the art is informed by this disclosure, a structural manner and an embodiment similar to the technical scheme are not creatively designed without departing from the gist of the present invention. In addition, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" preceding an element does not exclude the inclusion of a plurality of such elements. The various elements recited in the product claims may also be embodied in software or hardware. The terms first, second, etc. are used to denote a name, but not any particular order.

Claims

1. The intelligent shopping cart shopping behavior analysis method based on commodity track analysis is characterized by comprising the following steps of:

acquiring videos in the shopping process of a user to obtain an image sequence;

Inputting the image sequence into a target detection and identification model to obtain the coordinate information of all commodity targets in all N frames of images;

Inputting commodity target coordinate information into a target tracking model to obtain commodity tracks of all commodity targets in an image sequence;

Calculating a motion measurement index of each commodity track, and judging that each commodity track is one of a motion track or a static track;

Analyzing all the commodity movement tracks, and further identifying the commodity movement tracks as one of putting the commodity into a shopping cart, taking the commodity out of the shopping cart or sorting the commodity:

1) Accurately positioning a frame index number of a moving part of a commodity target in an image sequence;

，

Video frame index number sequence of (c):

，

frame_ID_move is a subset of frame_ID, I _m1, …,I_mK is a Frame index sequence number value, and if the distance between the minimum track length corner points between adjacent frames is greater than a threshold value THRESH_Nb, the commodity moves between the two frames;

2) Calculating a continuous motion Frame index number sequence in the frame_ID_move;

The video segment corresponding to the set frame_ID_move_frag of the continuous Frame index number sequence is the accurate Frame index of the continuous motion segment of the commodity in the whole video sequence;

If the continuous motion segment of the commodity corresponds to the initial part in the Frame index number frame_ID, the commodity is in a motion state when the commodity is detected and tracked, and if the gravity weighing value is increased and reaches a stable state at the moment, the commodity track indicates that the commodity is placed in the shopping cart;

If the commodity continuous motion segment corresponds to the tail end part in the Frame index number frame_ID, the commodity target is in a static state when the commodity target is detected and tracked, the state of the commodity is changed from static to motion, and if the gravity weighing value is reduced and reaches a stable state at the moment, the commodity track is taken out of the commodity;

If the continuous motion segment of the commodity corresponds to the middle part in the Frame index number frame_ID, the commodity is in a static state in the starting stage and the ending stage, and if the gravity weighing value is consistent when the commodity is in the static state, the commodity track is the sorted commodity.

2. The method for analyzing shopping behavior of an intelligent shopping cart based on commodity track analysis according to claim 1, wherein the specific method for obtaining the coordinate information of all commodity targets in all images is as follows:

inputting an image sequence into a target detection and identification model to obtain coordinate information and category information of all targets in all images, wherein the targets comprise commodity targets and non-commodity targets; and outputting the coordinate information and the category information of the commodity target.

3. The intelligent shopping cart shopping behavior analysis method based on commodity track analysis according to claim 1, wherein the specific method for judging each commodity track as a motion track or a stationary track is as follows: inputting coordinate information of commodity targets output by the target detection and identification model into a target tracking model to obtain track data of K commodity targets in an image sequence formed by N frames of images; calculating a motion measurement index of the actual motion condition of the commodity in the shopping process, setting one or more thresholds, and dividing the commodity track into a motion track and a rest track by comparing the motion measurement index with the thresholds.

4. The intelligent shopping cart shopping behavior analysis method based on commodity track analysis according to claim 1, wherein all commodity movement tracks are analyzed by combining weighing data of a gravity scale, and identified as one of commodity placement into a shopping cart, commodity removal from the shopping cart or commodity arrangement.

5. The intelligent shopping cart shopping behavior analysis method based on commodity track analysis according to claim 3, wherein (xi, yi) is set to be the coordinates of a commodity track point of an ith frame, the commodity track point is any one of a commodity target detection frame center point or four corner points of a commodity target detection frame, and the motion measurement index is one or a combination of three of the following three types:

1) Track length: representing the actual movement track length of the commodity in the shopping process; the method is defined as the accumulation sum of Euclidean distances between the coordinates of the same commodity track point in all adjacent frames of the commodity track, the unit is pixel number, and the calculation method is as follows:

，

2) Maximum track distance: characterizing the magnitude of movement of a commodity in space during shopping; the method is defined as the maximum value of the coordinate distance of the same commodity track point in any two frames, the unit is the pixel number, and the calculation method is as follows:

，

3) Track relative maximum distance: the relative movement distance of the maximum distance of the track relative to the commodity size is defined as the unit is the pixel number, and the calculation method is as follows:

，

6. The intelligent shopping cart shopping behavior analysis method based on commodity track analysis according to claim 5, wherein the motion metric index of each commodity track is calculated, and each commodity track is determined to be one of a motion track or a stationary track, and the specific method is as follows:

3) Obtaining corner points with minimum track length of each commodity target;

4) Calculating the track length Trajlen, the track maximum distance Distmax and the track relative maximum distance Distref with the minimum track length angle point as track motion measurement indexes of the commodity track;

5) The following conditions were set:

condition 1: trajlen < THRESH1;

condition 2: distmax < THRESH2;

condition 3: distref < THRESH3;

condition 4: trajlen < THRESH4 and Distref < THRESH5;

Condition 5: distmax < THRESH1 and Distref < THRESH5;

THRESH1, THRESH2, THRESH3, THRESH4 and THRESH5 are a set of thresholds;

7. The method for analyzing shopping behavior of an intelligent shopping cart based on commodity trajectory analysis according to claim 6, wherein the width of the image obtained by the camera is WI, the height is HI, thresh1=min (WI, HI)/10, thresh2=min (WI, HI)/15, thresh3=0.12, thresh4=min (WI, HI)/8, thresh5=0.2, and min is a minimum calculation.

8. An intelligent shopping cart shopping behavior analysis system based on commodity track analysis, applying the intelligent shopping cart shopping behavior analysis method based on commodity track analysis as set forth in any one of claims 1-7, comprising:

and an image acquisition module: is responsible for video and image acquisition;

An image detection and tracking module: the system is responsible for receiving and storing video stream information acquired by an image acquisition module, obtaining an image sequence, performing target detection and target tracking processing on images in the image sequence, identifying all targets in the images, and outputting track information of commodity targets;

9. The intelligent shopping cart shopping behavior analysis system based on commodity track analysis according to claim 8, wherein the method for precisely positioning the continuous motion segment in the commodity track in the time period of the commodity track is as follows:

，

Video frame index number sequence of (c):

，

frame_ID_move is a subset of frame_ID, I _m1, …, I_mK is a Frame index sequence number value, and if the distance between the minimum track length corner points between adjacent frames is greater than a threshold value THRESH_Nb, the commodity moves between the two frames;