CN114972170B - A fisheye camera-based object detection method for dense scenes with anti-occlusion - Google Patents

Abstract

The invention provides an anti-shielding object detection method based on a fisheye camera in a dense scene, which comprises the following steps: collecting an original image in an original scene and marking to obtain a mask of a single object; based on the object mask, synthesizing an image according to a shielding relation based on a spatial position relation between objects and a preset shielding priority, and obtaining a synthesized data set; dividing the synthetic data set into a plurality of sub data sets with different recognition difficulties according to the recognition difficulties of the objects; performing data enhancement on the synthesized data set; training an object detection network by adopting a synthetic data set after data enhancement; inputting the image to be detected into a trained object detection network to obtain an object detection result. The invention can greatly reduce the acquisition cost of data, can randomly increase different types, is used for simulating various possible shielding, and improves the object detection accuracy in complex scenes.

Description

A fisheye camera-based object detection method for dense scenes with anti-occlusion

Technical Field

The present invention belongs to the technical field of computer vision and relates to an anti-occlusion object detection method in a dense scene based on a fisheye camera.

Background technique

Benefiting from the rapid development of deep learning and the powerful learning ability of convolutional neural networks, as long as enough training data and corresponding sample labels are provided, the network model can achieve good performance in the process of continuous iterative learning and can be used in a variety of practical applications, such as target detection, semantic segmentation, face recognition, etc. In contrast, network models are often also limited by limited data sets. For some special scenarios, data collection may be very difficult, or data labeling is also very difficult, resulting in only a small amount of data being collected. A small amount of data cannot provide sufficient supervisory information to train the model, resulting in poor results in actual scenarios.

Due to the wide-angle field of view of fisheye cameras, the images they capture often have varying degrees of image distortion radiating from the center, which increases the difficulty of image recognition. In addition, if there are complex and severe occlusion relationships in the fisheye camera images, coupled with image distortion, it will further increase the difficulty of image recognition and object detection. However, in real scenes, complex occlusion relationships and severe image distortions contain a large number of changes, making it difficult to simulate all possible situations by simply collecting a large number of data sets as previous methods did.

Summary of the invention

In view of the problems existing in the prior art, the present invention provides an anti-occlusion object detection method in dense scenes based on a fisheye camera.

In order to achieve the purpose of the present invention, the present invention provides an anti-occlusion object detection method in a dense scene based on a fisheye camera, comprising the following steps:

Collect and annotate the original image in the original scene to obtain the mask of a single object;

Based on the object masks, synthesizing the images according to the occlusion relationship based on the spatial position relationship between the objects and the preset occlusion priority to obtain a synthetic data set;

According to the recognition difficulty of the object, the synthetic dataset is divided into multiple sub-datasets with different recognition difficulties;

Perform data augmentation on synthetic datasets;

Use data-augmented synthetic datasets to train object detection networks;

Input the image to be detected into the trained object detection network to obtain the object detection result.

Furthermore, the collecting of the original image in the original scene includes:

The original scene is divided into M rows and N columns, and the original image is collected at a specified position of the divided original scene;

The divided original scene is randomly rotated T times, and images are collected separately, and a total of M·N·T·K single object images are collected.

Furthermore, the occlusion relationship based on the spatial position relationship between objects is to determine the occlusion relationship between different objects based on the relative position relationship between the mask position and the imaging center of the fisheye camera, and objects farther from the center position will be occluded by objects closer to the center.

Furthermore, in the preset occlusion priority, a numerical value is used to represent the occlusion priority, and the numbers represent the occlusion priority from small to large, and the higher the occlusion priority, the more likely it is to occlude surrounding objects.

Furthermore, according to the difficulty of object recognition, the synthetic data set is divided into multiple sub-data sets with different recognition difficulties, and the recognition difficulty is measured according to the number of objects. The number of objects is used to measure the complexity of the occlusion relationship and the difficulty of identifying the object. Thus, it can be divided into difficult-to-distinguish samples and easy-to-distinguish samples. For easy-to-distinguish samples, the number of objects that appear is small, the occlusion relationship is simple, and it is easier to fit in training; for difficult-to-distinguish samples, the number of objects that appear is large, the occlusion relationship is complex, and it is more difficult to fit in training.

Furthermore, the dataset is divided into four levels of difficulty, and the ratio of each sub-dataset is: sub-dataset A: sub-dataset B: sub-dataset C: sub-dataset D = 0.15:0.35:0.35:0.1.

Furthermore, the object detection network is a cascadeCNN.

Furthermore, the data enhancement method is based on random brightness adjustment and random displacement perturbation of the mask.

Furthermore, the random brightness adjustment inside and outside the mask is performed as follows:

The illumination disturbance is adjusted by random gamma correction inside the mask. For the pixel point M={(x _m , y _m )} inside the mask, random illumination disturbance is performed using formula (1):

in, represents the image after data enhancement; α represents the linear coefficient, which is used to adjust the gamma correction scale; β obeys Gaussian distribution and represents random noise interference; γ represents the gamma correction illumination intensity adjustment coefficient, x _m , y _m represent the horizontal and vertical coordinates of the pixels in the mask respectively, and I (x _m , y _m ) represents the set of all pixels in the entire mask;

Outside the mask, a Gaussian kernel is used to calculate the interference attenuation coefficient to adjust the illumination disturbance:

Where ω represents the attenuation coefficient, x _o and _yo represent the horizontal and vertical coordinates of the pixel outside the mask, respectively. I(x _o , _yo ) represents the image after enhancement and the image before enhancement, respectively. x _i , y _i , and σ ² represent the coordinates and variance of a point in the mask, respectively.

Furthermore, the random displacement perturbation is to first perform distortion correction on the original fisheye image, then perform random displacement on the mask within a predetermined range when synthesizing the image, and then restore the original image using the fisheye distortion model.

Compared with the prior art, the present invention can achieve at least the following beneficial effects:

1. The present invention is particularly suitable for scenes with a narrow spatial range, close positions of target objects, and severe occlusion. Since there are a lot of changes when different objects are placed in different positions in actual scenes, it brings great difficulties to the collection of data sets. The solution adopted by the present invention is to infer possible occlusion relationships based on the spatial position relationship between objects, artificially simulate occlusions, and randomly synthesize a large amount of data to construct occlusion samples with rich variations for training models. This method can greatly reduce the cost of data collection, and can arbitrarily add different types to simulate various possible occlusions and improve the accuracy of object detection in complex scenes.

2. Input a large number of pictures with objects of interest and corresponding label data to the convolutional network model. In the process of continuous iterative learning, train the network model to determine whether there are objects of interest in the input pictures and output the corresponding confidence. The robustness of the network model is largely affected by the richness of the training data. Benefiting from the data generation scheme proposed in the present invention, the quantity and quality of the training data are ensured under the condition of limited manpower and material resources.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a dense scene image captured by a fisheye camera.

FIG2 is a schematic diagram of raw data collection and annotation.

FIG. 3 is a schematic diagram of occlusion priority and data synthesis rules.

FIG4 is a schematic diagram of recognition and anti-occlusion and distortion effects in a real scene.

FIG5 is a flowchart of anti-occlusion object detection.

Detailed ways

In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

In the prior art, due to the wide-angle field of view of the fisheye camera, the images it captures often have severe distortion. In addition, the occlusion relationship between different objects in dense scenes is intricate and coupled with distortion, making it very difficult to detect objects. In real scenes, the different arrangements of different objects and even changes in their own rotation will also lead to huge difficulties in data collection itself. As shown in Figures 1 and 2, the target area of the fisheye camera image is divided into M rows and N columns (such as M = 5, N = 7). As shown in Figure 2(b), each grid represents an area where an object may appear. Considering the singleness of the monocular camera's perspective, in order to ensure the realism of the scene and the richness of the data, when taking pictures of different object arrangements, in addition to considering the translation of the object in the horizontal plane, it is also necessary to consider the possible different rotations of the object (3 degrees of freedom). Assuming there are K objects, each object has T rotations, then for each object at each position, there are (K·T+1) ^M·N possible situations. It can be seen that all possible arrangements grow exponentially, limited by the expensive data collection and manual labeling costs. Therefore, in order to solve the problems existing in the prior art, the present invention proposes an anti-occlusion object detection method in dense scenes based on a fisheye camera.

Specifically, the present invention provides an anti-occlusion object detection method in a dense scene based on a fisheye camera, comprising the following steps:

Step 1: Collect and label original samples.

Please refer to Figure 2, step 1 specifically includes the following steps:

Step 1.1: Divide the original scene, such as dividing the original scene into M rows and N columns, and then collect the original image at a specified position of the divided original scene.

In some embodiments of the present invention, considering that there are T possible posture changes, random rotation is performed T times and images are collected.

Step 1.2: Use annotation software to manually annotate the image collected in step 1.1 and obtain the mask of the object.

In some embodiments of the present invention, labelme software is used for labeling to obtain a mask of the object.

In some embodiments of the present invention, only a single object needs to be labeled during the labeling process, and there is no need to process complex multi-object scenes.

In some embodiments of the present invention, a total of M·N·T·K single object images are obtained, which is far less than the (K·T+1) ^M·N possible situations required by existing solutions.

In one embodiment of the present invention, M=5, N=7. It is understandable that in other embodiments, they can be set to other values as needed.

Step 2: Artificially synthesize data based on occlusion relationships.

After the image data preparation work in step 1, a single object mask can be obtained, and the single object mask contains information such as color, size, and position. According to the positional relationship of the mask and the wide-angle imaging characteristics of the fisheye camera, the present invention can determine the occlusion relationship between different objects from the relative positional relationship between the mask position and the imaging center (i.e., the optical center) of the fisheye camera. As shown in FIG3(c), assuming that the target objects are all placed on the same plane, objects farther from the center position will be occluded by objects closer to the center. Therefore, as shown in FIG3(d), in some embodiments of the present invention, a relative occlusion priority can be specified to determine the occlusion relationship between objects: numbers from small to large represent the occlusion priority, and the higher the occlusion priority, the more likely it is to occlude the surrounding objects. According to the occlusion relationship, when synthesizing an image, in the order of occlusion priority from small to large, the object mask is cut out and pasted to the corresponding position of the synthesized image, and finally an image with a natural occlusion relationship (occlusion that conforms to physical rules) can be synthesized, as shown in FIG3(e), to obtain a synthetic data set.

Step 3: Data ratio allocation: According to the difficulty of object recognition, the synthetic dataset is divided into multiple sub-datasets with different recognition difficulties.

In actual scenes, the more objects that appear, the more complex the potential occlusion relationship, and the greater the difficulty of identification. In some embodiments of the present invention, the number of objects is used to measure the complexity of the occlusion relationship and the difficulty of identifying objects. It can be divided into difficult-to-distinguish samples and easy-to-distinguish samples. For easy-to-distinguish samples, the number of objects that appear is small, the occlusion relationship is simple, and it is easier to fit in training; for difficult-to-distinguish samples, the number of objects that appear is large, the occlusion relationship is intricate, and it is more difficult to fit in training. In addition, the distortion characteristics of the fisheye camera will further bring greater changes and uncertainties to the occlusion relationship.

In order to obtain better training results and simulate the situation in real scenes, according to the ratio of difficult samples and easy samples, the present invention divides the collected data sets into 4 levels of difficulty: sub-dataset A-each picture contains 1 to 10 objects; sub-dataset B-each picture contains 11 to 20 objects; sub-dataset C-each picture contains 20-28 objects; sub-dataset D-each picture contains 29 to 35 objects. In order to simulate the real scene as much as possible, the ratio of each sub-dataset is proposed as follows: sub-dataset A: sub-dataset B: sub-dataset C: sub-dataset D = 0.15:0.35:0.35:0.15.

Step 4: Perform random data augmentation on the synthetic data.

In order to enrich the distribution of the synthetic data set as much as possible and improve the robustness of the subsequent recognition model, in some embodiments of the present invention, a series of data enhancement schemes are used to process the synthetic data set, including mask-based random brightness adjustment and random displacement perturbation.

For mask-based random brightness adjustment:

In dense scenes, the light reflected from each other between objects may cause changes in the light field, affecting the image brightness during imaging. In order to simulate this irregular brightness fluctuation, an embodiment of the present invention performs random brightness adjustments on the space within a preset range (determined by the image size) around the mask. Inside the mask area, the illumination is adjusted by random gamma correction color disturbance; outside the mask area, in order to fit the interaction relationship between different objects reflecting each other, the present invention uses a Gaussian kernel to calculate the interference attenuation coefficient to adjust the illumination disturbance. That is:

For the pixel point M={(x _m , y _m )} in the mask, the following formula can be used to simulate random illumination disturbance:

in, represents the image after data enhancement; α represents the linear coefficient used to adjust the gamma correction scale; β∈N(0,1), obeys Gaussian distribution, and represents random noise interference; γ represents the gamma correction illumination intensity adjustment coefficient, _xm , _ym represent the horizontal and vertical coordinates of the pixels in the mask respectively, and I( _xm , _ym ) represents the set of all pixels in the entire mask.

For pixels outside the mask In addition to the random illumination perturbations, the attenuation coefficient must also be calculated. The farther away from the mask, the smaller the impact on the illumination;

Where ω represents the attenuation coefficient, x _o and _yo represent the horizontal and vertical coordinates of the pixel outside the mask, I(x _o , _yo ) represents the image after enhancement and the image before enhancement, respectively. x _i , y _i , and σ ² represent the coordinates and variance of a point in the mask, respectively.

For random displacement perturbations:

In the process of collecting raw data, objects are often placed within a pre-set specified position range. Although this strategy can greatly reduce the workload, it cannot fully cover the possible placement of any position in the real scene. In order to enhance the randomness of the placement position, enhance the realism of the data set, and take into account the distortion characteristics of the fisheye camera, in some embodiments of the present invention, the original fisheye image is first distorted and corrected, and then the mask is randomly displaced within a predetermined range (determined by the image size) when synthesizing the image, and then the fisheye distortion model is used to restore the original image.

Step 5: Use the data-enhanced synthetic dataset to train the object detection network and model deployment.

All existing detection networks are applicable. In some embodiments of the present invention, the network used is cascadeCNN. Model deployment refers to applying the model in actual scenarios, such as on mobile devices.

Step 6: Input the image to be detected into the trained object detection network to obtain the object detection result.

Due to the powerful feature learning ability of convolutional neural networks, the current target detection algorithms based on deep learning mostly adopt an end-to-end mechanism. A large number of pictures with objects of interest and corresponding label data are input to the convolutional network model. In the process of continuous iterative learning, the network model is trained to determine whether there are object categories of interest in the input pictures, and output the corresponding confidence. The robustness of the network model is greatly affected by the richness of the training data. Benefiting from the data generation scheme proposed in the present invention, the quantity and quality of the training data are ensured under the condition of limited manpower and material resources. As shown in Figure 4, in a real scene, by adopting the detection method provided by the present invention, the objects in the scene can be well identified.

In summary, the method provided by the embodiment of the present invention is mainly used to solve the problem of image distortion of fisheye panoramic images in dense scenes and the problem of reduced object detection performance caused by complex occlusion. It is suitable for scenes with a narrow spatial range, close positions of target objects and severe occlusion. Since there are a lot of changes when different objects are placed in different positions in actual scenes, it brings great difficulties to the collection of data sets. The solution adopted by the embodiment of the present invention is to artificially simulate occlusion and randomly synthesize a large amount of data based on the spatial position relationship between objects to infer possible occlusion relationships, and to construct occlusion samples with rich variations for training models. This method can greatly reduce the cost of data collection, and can synthesize new data without restriction to simulate various possible occlusions and improve the accuracy of object detection in complex scenes.

The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims (7)

1. An anti-shielding object detection method based on a fisheye camera in a dense scene is characterized by comprising the following steps:

collecting an original image in an original scene and marking to obtain a mask of a single object;

based on the object mask, synthesizing an image according to a shielding relation based on a spatial position relation between objects and a preset shielding priority, and obtaining a synthesized data set;

Dividing the synthetic data set into a plurality of sub data sets with different recognition difficulties according to the recognition difficulties of the objects;

Performing data enhancement on the synthesized data set;

Training an object detection network by adopting a synthetic data set after data enhancement;

Inputting the image to be detected into a trained object detection network to obtain an object detection result;

wherein the data enhancement mode is based on random brightness adjustment and random displacement disturbance of the mask;

The method for randomly adjusting the brightness of the inside and the outside of the mask comprises the following steps:

Adjusting the disturbance of the illumination by random gamma correction in the mask for the pixel points in the mask Carrying out random illumination disturbance by adopting a formula (1):

（1）

Wherein, Representing the image after the data enhancement; /(I)Representing a linear coefficient for adjusting the gamma correction scale; /(I)Obeying Gaussian distribution to represent random noise interference; /(I)Representing gamma corrected illumination intensity adjustment coefficient,/>Respectively represent the horizontal and vertical coordinates of pixels in the mask/>Representing a set of all pixels within the entire mask;

outside the mask, the disturbance attenuation coefficient is calculated by Gaussian kernel to adjust the illumination disturbance:

Wherein the method comprises the steps of Representing the attenuation coefficient,/>Respectively represent the abscissa and the ordinate of the pixel points outside the mask,/>、/>Representing an enhanced image and a pre-enhanced image, respectively,/>、/>、/>Representing the coordinates and variance of one of the points within the mask, respectively;

the random displacement disturbance is that the original fisheye image is subjected to distortion correction, then the mask is subjected to random displacement within a preset range when the image is synthesized, and then the fisheye distortion model is used for restoring the original image.

2. The method for detecting an anti-occlusion object in a dense scene based on a fisheye camera according to claim 1, wherein the capturing an original image in an original scene comprises:

dividing an original scene into M rows and N columns, and collecting an original image at a designated position of the divided original scene;

randomly rotating the divided original scene for T times, respectively collecting images, and collecting the total Zhang Shan object image,/>K objects are represented.

3. The method for detecting the anti-occlusion object in the dense scene based on the fisheye camera according to claim 1, wherein the occlusion relation between different objects is determined according to the relative position relation between the mask position and the imaging center of the fisheye camera according to the occlusion relation based on the spatial position relation between the objects, and the object far from the center position is occluded by the object near the center.

4. The method for detecting the anti-occlusion object in the dense scene based on the fisheye camera according to claim 1, wherein the preset occlusion priority is represented by a numerical value, the occlusion priority is represented by a number from small to large, and the higher the occlusion priority is, the more likely surrounding objects are occluded.

5. The method for detecting the anti-occlusion object in the dense scene based on the fisheye camera according to claim 1, wherein the synthetic dataset is divided into a plurality of sub-datasets with different recognition difficulties according to the recognition difficulty of the object, and the recognition difficulty is measured according to the number of the objects.

6. The method for detecting an anti-occlusion object in a dense scene based on a fisheye camera according to claim 5, wherein the data set is divided into 4 kinds of difficulties, and the proportions of the sub data sets are as follows: sub-data set a, sub-data set B, sub-data set C, sub-data set d=0.15:0.35:0.35:0.1.

7. The method for detecting an anti-occlusion object in a dense scene based on a fisheye camera according to claim 1, wherein the object detection network is CASCADECNN.