CN104851089A - Static scene foreground segmentation method and device based on three-dimensional light field - Google Patents

Abstract

The invention discloses a static scene foreground segmentation method and device based on a three-dimensional light field. The method includes the steps of: taking a sequence of images of a scene at equal intervals on a one-dimensional straight line with a camera to construct a three-dimensional light field, and generating the scene the epipolar plane; using a straight line detection algorithm to extract the straight line features in the epipolar plane and calculating slope information, recovering the depth information of different objects in the scene from the slope information, and using a fast interpolation algorithm to generate a depth image of the entire scene; Set corresponding depth thresholds for different objects in the depth image, and quickly segment different objects according to the depth thresholds; in the segmentation of complex outdoor scenes, the present invention can accurately restore the distance between multiple objects in the scene. Spatial relationship, which overcomes the over-segmentation problem existing in complex scene applications based on regional clustering and mathematical morphology methods, and has high segmentation efficiency when extracting specific targets.

Description

A static scene foreground segmentation method and device based on three-dimensional light field

technical field

The present invention relates to the technical field of image processing, in particular to a static scene foreground segmentation method and device based on a three-dimensional light field.

Background technique

Image segmentation refers to distinguishing different regions with special meaning in the image. Each region generated by the segmentation is connected, and the pixels in the region satisfy a specific consistency criterion, while different image regions do not intersect each other, and there are differences between them. Based on the concept of segmentation, images can be abstracted into two categories, "foreground" and "background". Wherein, the concept of image "foreground" is relative to "background", and usually refers to an area of interest to people in a picture scene. Effective foreground segmentation is the premise and basis for high-level applications such as virtual scene construction, intelligent video surveillance, and natural human-computer interaction.

The existing foreground segmentation methods in static scenes are mainly divided into threshold-based, edge-based, region-based and methods combined with other specific theories. Among them, the watershed algorithm proposed by L.Vincent regards the image as a topological landform, constructs a watershed area centered on the local minimum point of the gray level, and can obtain continuous segmentation boundaries, but over-segmentation problems often occur in practical applications; Felzenszwalb proposed a graph segmentation method using a greedy search strategy, which can obtain the global optimal solution by using a paired region comparison method; Chan et al. designed a geometric active contour model based on level sets, which achieved a good single-target contour extraction, but in Edge positioning is still imprecise.

Contents of the invention

In view of this, the object of the present invention is to propose a static scene foreground segmentation method and device based on a three-dimensional light field that can effectively and efficiently realize image segmentation.

A kind of static scene foreground segmentation method based on three-dimensional light field that the present invention provides based on above-mentioned purpose, comprises the following steps:

A sequence of images of a scene is taken at equal intervals on a one-dimensional straight line by the camera to construct a three-dimensional light field and generate an epipolar plan view of the scene;

Using a straight line detection algorithm to extract the straight line features in the epipolar plan and calculating slope information, recovering the depth information of different objects in the scene from the slope information, and using a fast interpolation algorithm to generate a depth image of the entire scene;

Corresponding depth thresholds are set for different objects in the depth image, and different objects are quickly segmented according to the depth thresholds.

Preferably, in the three-dimensional light field, any light L is expressed as:

L=LF(x,y,t)

Wherein, t is the starting point of the ray, that is, the coordinates of the camera on the one-dimensional straight line; (x, y) represents the direction of the ray, corresponding to the two-dimensional coordinate value in the image;

The epipolar plan is the stack of horizontal pixels of the sequence image under the same y value condition, that is, the (x, t) section perpendicular to the y coordinate; the pixels of the same object in the scene form a line in the epipolar plan The linear trajectory, and the spatial distance between the object and the camera linear motion trajectory is proportional to the slope of the corresponding straight line of the object in the epipolar plan view.

Preferably, the step of generating the depth image further includes:

Select one of the sequence images as the target image for depth recovery and foreground segmentation;

extracting straight lines from said epipolar plan using a straight line detection algorithm and determining all straight line regions;

Generating a slope distribution of straight line feature points in the target image according to the straight line area;

According to the slope distribution of the feature points of the straight line, an interpolation algorithm is used to generate the slope distribution of all pixels of the target image;

The slope distribution of all pixels of the target image is transformed into a depth distribution, and then linearly mapped to a gray scale interval, to finally generate the depth image.

Preferably, before using the straight line detection algorithm to extract the straight line, Gaussian scaling is performed on the epipolar plan with a scaling ratio of 0.9.

Preferably, the step of determining the straight line area includes:

For each pixel point in the epipolar plan view, calculate the angle between the adjacent point direction and the horizontal direction with the same relative color, and the pixel points with similar angles constitute a straight line candidate area;

Covering each of the straight line candidate areas with an approximate rectangle, constructing a noise model to perform verification on the straight line candidate areas, and obtaining the probability that the straight line candidate areas form a straight line;

A probability threshold for straight line determination is set, and the straight line area is finally determined.

Preferably, after the step of extracting straight lines from the epipolar plan view, the step of screening the extraction results is also included:

Only extracting the straight lines whose endpoints fall within the first ten pixels of the epipolar plane in the y-axis direction;

Extending the straight line that does not directly intersect with the upper boundary of the epipolar plan, and calculating the estimated intersection point;

Eliminate straight lines whose inferred intersection point exceeds the boundary of the image, and merge two coincident straight lines that appear due to extension into a single straight line.

The present invention also provides a static scene foreground segmentation device based on a three-dimensional light field, including:

The building block is used to take a sequence of images of a scene at equal intervals on a one-dimensional straight line through the camera to construct a three-dimensional light field, and generate an epipolar plan view of the scene;

The depth recovery module is used to extract the straight line features in the epipolar plane using a straight line detection algorithm and calculate slope information, restore the depth information of different objects in the scene from the slope information, and use a fast interpolation algorithm to generate a depth image of the entire scene ;

The segmentation module is configured to set corresponding depth thresholds for different objects in the depth image, and quickly segment different objects according to the depth thresholds.

Preferably, in the three-dimensional light field generated by the building block, any light L is expressed as:

L=LF(x,y,t)

Wherein, t is the starting point of the ray, that is, the coordinates of the camera on the one-dimensional straight line; (x, y) represents the direction of the ray, corresponding to the two-dimensional coordinate value in the image;

The epipolar plan is the stack of horizontal pixels of the sequence image under the same y value condition, that is, the (x, t) section perpendicular to the y coordinate; the pixels of the same object in the scene form a line in the epipolar plan The linear trajectory, and the spatial distance between the object and the camera linear motion trajectory is proportional to the slope of the corresponding straight line of the object in the epipolar plan view.

Preferably, the depth recovery module is further used to: select one of the sequence images as a target image for depth recovery and foreground segmentation; use a line detection algorithm to extract lines from the epipolar plan and determine all line areas; In the straight line area, the slope distribution of the straight line feature points is generated in the target image; according to the slope distribution of the straight line feature points, an interpolation algorithm is used to generate the slope distribution of all pixels of the target image; all pixels of the target image are The slope distribution of the points is transformed into the depth distribution, and then linearly mapped to the gray scale interval to finally generate the depth image.

Preferably, the depth restoration module further includes a scaling module for performing Gaussian scaling on the epipolar plan before using a straight line detection algorithm to extract straight lines, and the scaling ratio of the scaling module is 0.9.

As can be seen from the above, the static scene foreground segmentation method and device based on a three-dimensional light field provided by the present invention constructs a three-dimensional light field by shooting sequence images of the scene at different viewpoints at equal intervals on a straight line, and uses a straight line detection algorithm from The edge of the scene and its depth information are extracted by analysis in the epipolar plane; the depth information of the entire scene is restored with the help of a fast interpolation algorithm, and finally the segmentation of foreground objects at different depths is realized through the threshold method. The present invention can more accurately restore the spatial relationship between multiple objects in the scene, and the result of foreground segmentation can better overcome the over-segmentation problem existing in complex scene applications based on methods such as area clustering and mathematical morphology. It has high segmentation efficiency when extracting specific objects.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 is the flowchart of the static scene foreground segmentation method based on the three-dimensional light field of the preferred embodiment of the present invention;

FIG. 2 is a schematic diagram of a three-dimensional light field description in a preferred embodiment of the present invention;

Fig. 3 is a schematic diagram of the geometric relationship between the object, the imaging plane, and the imaging center path in a preferred embodiment of the present invention;

Fig. 4 (a) determines the original image pixel in the straight line region step in the preferred embodiment of the present invention;

Fig. 4 (b) is the pixel point of the calculated included angle in the step of determining the straight line area in the preferred embodiment of the present invention;

Fig. 4 (c) is the straight line candidate area in the step of determining the straight line area in the preferred embodiment of the present invention;

Fig. 5 (a) is the straight line detection result when the Gaussian scaling ratio is 0.5 in the preferred embodiment of the present invention;

Fig. 5 (b) is the straight line detection result when the Gaussian scaling ratio is 0.9 in the preferred embodiment of the present invention;

Fig. 5 (c) is the straight line detection result when Gaussian scaling ratio is 1.5 in the preferred embodiment of the present invention;

Fig. 6 (a) is the original scene image in the preferred embodiment of the present invention;

Fig. 6 (b) is the EPI in the preferred embodiment of the present invention;

Fig. 6 (c) is the depth image in the preferred embodiment of the present invention;

Figure 7 shows the histogram statistics of the depth image of the "Mansion" scene;

Fig. 8 (a) is the segmentation result of applying the method of the present invention to the "Church" scene;

Fig. 8 (b) is the segmentation result of applying the method of the present invention to the "Mansion" scene;

Fig. 8 (c) is the segmentation result of applying the method of the present invention to the "Statue" scene;

Figure 9(a) shows the segmentation results of the three scenes of "Church", "Mansion" and "Statue" using the watershed segmentation algorithm;

Figure 9(b) shows the segmentation results using the Graph Cut segmentation algorithm for the three scenes of "Church", "Mansion", and "Statue";

Figure 9(c) of the segmentation algorithm based on K-means clustering) is the segmentation result for the three scenes of "Church", "Mansion" and "Statue";

FIG. 10 is a schematic structural diagram of a static scene foreground segmentation device based on a three-dimensional light field according to an embodiment of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in further detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

An embodiment of the present invention provides a static scene foreground segmentation method based on a three-dimensional light field. The method constructs a three-dimensional light field by taking sequential images of the scene at different viewpoints at equal intervals on a straight line, and uses a straight line detection algorithm to obtain Analyze and extract the edge of the scene and its depth information; restore the depth information of the entire scene with the help of a fast interpolation algorithm, and finally realize the segmentation of foreground objects at different depths through the threshold method. Among them, the light field (Light Fields) is a description of all the light in the space. Scenes in the real world contain rich 3D information, and a single 2D image cannot fully describe the spatial relationship between objects. Using a dense sequence of images can be used to characterize static scenes. The temporal continuity in the fast-sampling sequence images is roughly equivalent to the spatial continuity of the scene, and the description method of "Epipolar Plane Image (EPI)" is used to analyze the three-dimensional scene. Hereinafter, the method for segmenting the foreground of a static scene based on a three-dimensional light field will be further described through a preferred embodiment of the present invention.

Referring to FIG. 1 , it is a flowchart of a static scene foreground segmentation method based on a three-dimensional light field according to a preferred embodiment of the present invention.

The static scene foreground segmentation method based on the three-dimensional light field of the preferred embodiment of the present invention comprises the following steps:

Step 1: Take a sequence of images of a scene at equal intervals on a one-dimensional straight line through the camera to construct a three-dimensional light field, and generate an epipolar plan view of the scene;

Step 2: using a straight line detection algorithm to extract the straight line features in the epipolar plan and calculating slope information, recovering the depth information of different objects in the scene from the slope information, and using a fast interpolation algorithm to generate a depth image of the entire scene;

Step 3: Set corresponding depth thresholds for different objects in the depth image, and quickly segment different objects according to the depth thresholds.

(1) The specific implementation of step 1

In step 1, a three-dimensional light field is established first. A typical light field construction process usually needs to take a large number of images of the same scene under different viewing angles, and then use a suitable geometric model to describe the distribution of light in space. Aiming at the application background of the scene foreground segmentation in this embodiment, the case where the imaging center of the image sequence is on a horizontal straight line is studied, that is, the three-dimensional light field is established by acquiring ordered two-dimensional images on a one-dimensional straight line.

Referring to FIG. 2 , it is a schematic diagram illustrating a three-dimensional light field in a preferred embodiment of the present invention.

Image generation can be regarded as the projection process of light on the imaging plane. The description of any ray L in space is as follows:

L=LF(x,y,t)

Wherein, t is the starting point of the ray, that is, the coordinates of the camera on the one-dimensional straight line; (x, y) represents the direction of the ray, corresponding to the two-dimensional coordinate value in the image. In the three-dimensional light field described by LF, the (x, y) section perpendicular to the t coordinate corresponds to the scene image under different viewing angles, and the (x, t) section perpendicular to the y coordinate corresponds to the epipolar plane, that is epi.

Intuitively, EPI corresponds to the stacking of horizontal pixels of sequence images under the same y value. For the convenience of subsequent processing, this embodiment assumes that the distance between the optical centers (optical centers, that is, camera positions) of any two adjacent sequential images is the same, the optical center is the center of imaging, and the equal distance between the camera positions can ensure the equal distance between the optical centers, that is Ensure the continuity of pixels corresponding to the same object in the formed EPI. When the number of sampled images is large enough, the pixels of the same object in the scene form a straight line trajectory in the EPI. The spatial information of objects in the scene can be obtained by analyzing the geometric features of straight lines.

Referring to FIG. 3 , it is a schematic diagram of the geometric relationship among the object, the imaging plane, and the imaging center path in a preferred embodiment of the present invention.

As shown in Figure 3, assume that the distance from the imaging plane to the imaging center is h, the distance from the object P to the center of the lens is D, x ₁ and x ₂ are the abscissas of P’s pixels in the image, and Δt is the moving distance of the imaging center.

From the triangular similarity relation, we can get:

$\frac{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{\mathrm{<span\; class="notranslate">\; \&Delta;t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; h</span>}}{\mathrm{<span\; class="notranslate">\; D.</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; t</span>}}$

Eliminate t to get:

$\mathrm{<span\; class="notranslate">\; D.</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; \&times;</span>\frac{\mathrm{<span\; class="notranslate">\; \&Delta;t</span>}}{\mathrm{<span\; class="notranslate">\; \&Delta;x</span>}}$

Wherein, Δx=|x ₁ −x ₂ |.

In EPI, Proportional to the slope k of the straight line, it can be seen that the spatial distance D between the object and the camera is also proportional to the slope k of the straight line in the EPI. On the basis of obtaining the corresponding slope of each pixel in the scene, the depth information in it can be further estimated.

(2) The specific implementation of step 2

As a preferred embodiment, the step of generating the depth image specifically includes:

Select one of the sequence images as the target image for depth restoration and foreground segmentation;

extracting straight lines from said epipolar plan using a straight line detection algorithm and determining all straight line regions;

Generating a slope distribution of straight line feature points in the target image according to the straight line area;

According to the slope distribution of the feature points of the straight line, an interpolation algorithm is used to generate the slope distribution of all pixels of the target image;

The slope distribution of all pixels of the target image is transformed into a depth distribution, and then linearly mapped to a gray scale interval, to finally generate the depth image.

Specifically, line detection is performed first. The number and quality of line detections in EPI are critical for estimating the depth of scene objects. There are usually rapid changes in color at the boundary of scene objects, which are manifested as obvious straight line features in EPI. The line detection in the present embodiment adopts the LSD (Line Segment Detection) method, and screens and processes the specific features of the EPI image.

For each pixel, this embodiment calculates the angle α between the direction of the adjacent point with the same relative color and the horizontal direction. Pixels with a similar angle α constitute a line candidate region (Line Support Regions), that is, a possible line area. Referring to Fig. 4(a), Fig. 4(b), and Fig. 4(c), they are the original image pixels, the pixel points with calculated included angles, and the straight line candidate areas in this preferred embodiment, respectively.

For each line candidate area, an approximate rectangle is used to cover the area pixels. Select the mode α' of the horizontal angle α of the pixels in the rectangle as the main direction of the rectangle, and perform a straight line verification process for each rectangle. Given a deviation tolerance τ (angle, value 0~π), the pixel whose angle β=|α-α'| with the main direction of the rectangle is smaller than τ is used as a straight line approximation point. Then for any pixel in the rectangular area, the probability that it belongs to a straight line All pixel angles β are independent and satisfy the binomial distribution:

β～B(n,s,p)

Among them, n is the total number of pixels in the rectangular area, and s is the number of linear approximation points. The verification process is performed on the rectangular area by constructing a noise model (Noise Model), and the probability Q that the rectangular area forms a straight line is obtained. Set the probability threshold for straight line judgment, and finally determine the straight line area.

Further, since the scaling ratio in the image processing process can significantly affect the effect of the EPI straight line detection, tests and comparisons are made in different Gaussian scaling ratios in this embodiment. Referring to FIG. 5(a), FIG. 5(b), and FIG. 5(c), they are the straight line detection results when the Gaussian scaling ratio is 0.5, 0.9, and 1.5, respectively. It can be seen from the above figures that when the Gaussian zoom ratio is low, the line detection results are relatively sparse, and part of the straight line features are lost; when the zoom ratio is high, more interfering straight lines appear. Considering the depth information restoration effect in the actual test, the scaling ratio selected in this embodiment is 0.9.

It should be noted that, in other embodiments of the present invention, when performing line detection on EPI, you can also use: line extraction based on Hough transform, line extraction based on Canny edge detection, line extraction based on chain code or other A method that enables accurate straight line detection.

Based on the straight line area obtained by the above straight line detection for EPI, the depth image is restored next. In this embodiment, the first sequence image is selected as the target image for depth restoration and foreground segmentation. EPI collects the scene information of all image sequences. Due to the change of perspective, new objects may appear in subsequent scenes; due to the inconsistency of image acquisition time, objects (such as pedestrians, etc.) that flash in a frame of subsequent scenes will form noise in EPI. . In addition, LSD line detection still has limitations, such as line breaks and unextractable situations. In view of the above problems, this embodiment further screens and processes the straight line extraction results, and performs the following steps:

1. Only extract the straight lines whose endpoints fall within the first ten pixels of the epipolar plane in the y-axis direction;

② Extend the straight line that does not directly intersect with the upper boundary of the epipolar plan, and calculate the estimated intersection point;

③ Eliminate the straight lines whose estimated intersection points exceed the image boundary, and merge the two overlapping straight lines that appear due to the extension into a single straight line.

For a given value of y, the corresponding EPI _y can generate an ordered binary array (u, k) after the above processing. Among them, u represents the coordinates of the intersection point of the EPI line and the upper boundary, that is, the corresponding abscissa value of the target image under the ordinate y, and k represents the corresponding slope on this abscissa. By traversing all the y values, the slope value of the target image at the coordinates (u, y, k) can be obtained, that is, the slope distribution of the straight line feature points generated in the target image; the straight line feature points are (u, y), which is the coordinate position of the pixel point in the target image (original image) that appears as a straight line feature in the EPI; in the target image, the straight line feature point is displayed as a number of discretely distributed pixel points.

In order to obtain dense depth information, this embodiment obtains a complete slope representation in the x-axis direction through interpolation mapping. The rationale for using the interpolation method is due to the following points:

1) The color value of pixels with large depth changes in the original image changes significantly, and can form obvious straight line features in EPI;

2) The straight lines in the EPI can be detected relatively completely by the LSD algorithm, and more accurate depth information can be estimated for the pixels corresponding to the detected straight lines;

3) The depth value changes between the corresponding points of the two EPI straight lines in the original image are continuous and smooth.

Therefore, the choice of interpolation method should satisfy:

1) In order to ensure the accuracy of the detected depth value, the interpolation function f must pass through a given control point, that is, satisfy f(u)=k;

2) The interpolation function f can obtain the estimated depth values at all pixel points, that is, f is continuous on the domain of definition [0, X _max ];

3) The interpolation function f makes a smooth transition between two control points, that is, for a given u ₁ , u ₂ , x ₂ ∈[u ₁ ,u ₂ ], the following formula holds:

f(x ₁ )′×f(x ₂ )′＞0

Based on the above considerations, this embodiment uses Pchip (Piecewise Cubic Hermite Interpolating Polynomial) as an interpolation method to obtain the estimated slope value k at any pixel point (x, y) in the original image.

Due to the characteristics of the _Pchip interpolation method itself, the estimated value of the slope of the pixel near the two ends of [0, X _max ] is often too large or too small. group value. According to the correspondence between the spatial distance between the object and the camera is proportional to the slope of the corresponding straight line of the object in the epipolar plan view, linear mapping can be used to transform the slope distribution (x, y, k) into a depth distribution (x, y ,d), which is linearly mapped to the gray-scale interval again, and finally the intuitive output of the depth image is obtained. The intuitive process of extracting the depth image is shown in Fig. 6(a), 6(b) and 6(c). The red lines in the above figures indicate the corresponding position of EPI in the original figure.

In other embodiments of the present invention, when carrying out above-mentioned depth recovery, can also use: linear interpolation method, Lagrangian interpolation method, spline interpolation method, Newton interpolation method or other interpolation methods.

(3) The specific implementation of step 3

The distinction of objects in real scenes often manifests as discontinuity in spatial position. The scene depth information obtained in this paper provides the support of third-dimensional data for the foreground segmentation process. Based on the depth map image, for objects of different spatial levels, setting the corresponding depth threshold can realize the rapid extraction of objects.

Fig. 7 is the image histogram obtained from the statistics of the depth image of the "Mansion" scene in Fig. 8(b). The horizontal axis in the grayscale histogram represents the number of grayscale levels (generally 0 to 255, a total of 256 levels), and the vertical axis represents the number of pixels whose grayscale value is in the current grayscale. By making statistics on all the pixels in the grayscale image, the grayscale histogram distribution shown in Figure 7 can be formed. Observation shows that the histogram shows a more obvious depth distribution feature, and selecting an appropriate threshold point on the slope axis can distinguish different objects in the scene.

Based on the method of the above-mentioned preferred embodiment of the present invention, the beneficial effect of the present invention will be further described through the evaluation of the segmentation results and the comparison with the segmentation results of the existing common segmentation methods.

The applicable situation studied by the embodiment of the present invention is object-based segmentation in a static scene, and the data requires the use of a large number of photo sequences taken at different angles on the same horizontal line. The experiment uses image segmentation to process commonly used data sets constructed in the prior art for evaluation and comparison. The present invention mainly selects three scenes of "Church", "Mansion" and "Statue" as the experimental objects, wherein each scene data set includes images taken at different imaging points of the same static scene on a sliding platform controlled by a computer. 101 images, all of which have been rectified and pre-aligned.

Foreground segmentation tests are performed on the three datasets separately. Set corresponding grayscale thresholds for different target objects, and draw the segmentation results, as shown in Figure 8(a), 8(b), and 8(c), respectively for "Church", "Mansion", and "Statue" Three scenes are segmented by applying the method of the present invention. The three pictures in Figure 8(a) are the segmentation results of utility poles, flower bushes, towers, and trees in the "Church" scene; the three pictures in Figure 8(b) are the trees, fences, and houses in the "Mansion" scene in order The segmentation results of ; the three pictures in 8(c) are the segmentation results of statues, bushes, statues and cars in the "Statue" scene in turn. Intuitively, the method of the present invention has a better segmentation effect on scene details. And in the Church dataset, objects with similar color characteristics such as trees, towers and houses in the scene are also better segmented.

The segmentation result is analyzed by using two quantitative indicators of recall rate and precision rate defined in the prior art. The recall rate represents the ratio of the number of correctly segmented pixels to the number of standard segmented pixels, and the precision rate represents the ratio of the number of correctly segmented pixels in the segmented image to the total number of segmented pixels. Among them, the standard segmented images used for comparison are obtained by manual calibration. Table 1 shows the calculations for the tower in "Church", the tree in "Mansion" and the statue in "Statue".

Table 1 uses the quantitative evaluation of the segmentation result of the method of the present invention

In order to further verify the effectiveness of the method foreground segmentation of the present invention in complex static scenes, the watershed segmentation algorithm, the Graph Cut segmentation algorithm and the segmentation algorithm based on K-means clustering are selected for visual comparison with the method of the present invention.

Referring to Figures 9(a), 9(b), and 9(c), the watershed segmentation algorithm, Graph Cut segmentation algorithm, and K-means-based clustering are used for the three scenes of "Church", "Mansion", and "Statue" respectively The segmentation result of the segmentation algorithm. Due to the complex and changeable color (grayscale) features in complex scenes, the same object may have several colors with high contrast (such as trees in the Mansion scene), and different objects may also have similar colors (such as trees in the Church scene). ), the above-mentioned commonly used methods will produce obvious over-segmentation problems, and usually require subsequent processing to form segmentation results for specific target objects. In contrast, the method of the present invention is more convenient and effective.

The embodiment of the present invention also provides a static scene foreground segmentation device based on a three-dimensional light field. Referring to FIG. 10 , it is a schematic structural diagram of a static scene foreground segmentation device based on a three-dimensional light field according to an embodiment of the present invention. The devices include:

The construction module 101 is used to take a sequence of images of a scene at equal intervals on a one-dimensional straight line by the camera to construct a three-dimensional light field, and generate an epipolar plan view of the scene;

The depth restoration module 102 is used to extract the straight line features in the epipolar plane using a straight line detection algorithm and calculate slope information, restore the depth information of different objects in the scene from the slope information, and generate the depth of the entire scene using a fast interpolation algorithm image;

The segmentation module 103 is configured to set corresponding depth thresholds for different objects in the depth image, and quickly segment different objects according to the depth thresholds.

Specifically, in the three-dimensional light field generated by the building block 101, any light L is expressed as:

L=LF(x,y,t)

Wherein, t is the starting point of the ray, that is, the coordinates of the camera on the one-dimensional straight line; (x, y) represents the direction of the ray, corresponding to the two-dimensional coordinate value in the image;

The epipolar plan is the stack of horizontal pixels of the sequence image under the same y value condition, that is, the (x, t) section perpendicular to the y coordinate; the pixels of the same object in the scene form a line in the epipolar plan The linear trajectory, and the spatial distance between the object and the camera linear motion trajectory is proportional to the slope of the corresponding straight line of the object in the epipolar plan view.

As a preferred embodiment, the depth recovery module 102 is further configured to: select one of the sequence images as a target image for depth recovery and foreground segmentation; use a line detection algorithm to extract lines from the epipolar plan and determine all lines area; according to the straight line area, the slope distribution of the straight line feature points is generated in the target image; according to the slope distribution of the straight line feature points, an interpolation algorithm is used to generate the slope distribution of all pixels of the target image; the target The slope distribution of all pixels in the image is transformed into the depth distribution, and then linearly mapped to the gray scale interval, and finally the depth image is generated.

In a preferred embodiment, the depth recovery module 102 further includes a scaling module for performing Gaussian scaling on the epipolar plan before using a straight line detection algorithm to extract straight lines, and the scaling ratio of the scaling module is 0.9.

In a preferred embodiment, when the depth recovery module 102 determines the straight line region, firstly, for each pixel in the epipolar plan view, it calculates the angle between the direction of the adjacent point with the same relative color and the horizontal direction, the Pixels with close angles form a line candidate area; then cover each line candidate area with an approximate rectangle, construct a noise model to perform verification on the line candidate area, and obtain the probability that the line candidate area forms a line; set Determine the probability threshold for straight line determination, and finally determine the straight line area.

In a preferred embodiment, the depth recovery module 102 is also used to filter the result of extracting straight lines in the polar plan view, specifically including:

Only extracting the straight lines whose endpoints fall within the first ten pixels of the epipolar plane in the y-axis direction;

Extending the straight line that does not directly intersect with the upper boundary of the epipolar plan, and calculating the estimated intersection point;

Eliminate straight lines whose inferred intersection point exceeds the boundary of the image, and merge two coincident straight lines that appear due to extension into a single straight line.

In summary, the present invention constructs a three-dimensional light field by shooting sequence images at equal intervals on a one-dimensional linear trajectory with a camera, estimates scene object depth information in EPI analysis, restores the entire depth image through a fast interpolation method, and based on this Implemented a segmentation method for foreground objects. In the segmentation of complex outdoor scenes, it effectively overcomes the over-segmentation problem of traditional methods, and has higher segmentation efficiency when extracting specific objects.

Those of ordinary skill in the art should understand that: the above descriptions are only specific embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present invention etc., should be included within the protection scope of the present invention.

Claims (10)

1. A static scene foreground segmentation method based on three-dimensional light field, is characterized in that, comprises the following steps: A sequence of images of a scene is taken at equal intervals on a one-dimensional straight line by the camera to construct a three-dimensional light field and generate an epipolar plan view of the scene; Using a straight line detection algorithm to extract the straight line features in the epipolar plan and calculating slope information, recovering the depth information of different objects in the scene from the slope information, and using a fast interpolation algorithm to generate a depth image of the entire scene; Corresponding depth thresholds are set for different objects in the depth image, and different objects are quickly segmented according to the depth thresholds. 2. The method according to claim 1, characterized in that, in the three-dimensional light field, any light L is expressed as: L=LF(x,y,t) Wherein, t is the starting point of the ray, that is, the coordinates of the camera on the one-dimensional straight line; (x, y) represents the direction of the ray, corresponding to the two-dimensional coordinate value in the image; The epipolar plan is the stack of horizontal pixels of the sequence image under the same y value condition, that is, the (x, t) section perpendicular to the y coordinate; the pixels of the same object in the scene form a line in the epipolar plan The linear trajectory, and the spatial distance between the object and the camera linear motion trajectory is proportional to the slope of the corresponding straight line of the object in the epipolar plan view. 3. The method according to claim 2, wherein the step of generating the depth image further comprises: Select one of the sequence images as the target image for depth restoration and foreground segmentation; extracting straight lines from said epipolar plan using a straight line detection algorithm and determining all straight line regions; Generating a slope distribution of straight line feature points in the target image according to the straight line area; According to the slope distribution of the feature points of the straight line, an interpolation algorithm is used to generate the slope distribution of all pixels of the target image; The slope distribution of all pixels of the target image is transformed into a depth distribution, and then linearly mapped to a gray scale interval, to finally generate the depth image. 4. The method according to claim 3, characterized in that, before using the straight line detection algorithm to extract the straight line, Gaussian scaling is carried out to the epipolar plan, and the scaling ratio is 0.9. 5. The method according to claim 3, wherein the step of determining the straight line region comprises: For each pixel point in the epipolar plan view, calculate the angle between the adjacent point direction and the horizontal direction with the same relative color, and the pixel points with similar angles constitute a straight line candidate area; Covering each of the straight line candidate areas with an approximate rectangle, constructing a noise model to perform verification on the straight line candidate areas, and obtaining the probability that the straight line candidate areas form a straight line; A probability threshold for straight line determination is set, and the straight line area is finally determined. 6. method according to claim 3, is characterized in that, after the step of extracting straight line from described epipolar plan view, also comprises the screening processing step to extraction result: Only extracting the straight lines whose endpoints fall within the first ten pixels of the epipolar plane in the y-axis direction; Extending the straight line that does not directly intersect with the upper boundary of the epipolar plan, and calculating the estimated intersection point; Eliminate straight lines whose inferred intersection point exceeds the boundary of the image, and merge two coincident straight lines that appear due to extension into a single straight line. 7. A static scene foreground segmentation device based on a three-dimensional light field, characterized in that it comprises: The building block is used to take a sequence of images of a scene at equal intervals on a one-dimensional straight line through the camera to construct a three-dimensional light field, and generate an epipolar plan view of the scene; The depth recovery module is used to extract the straight line features in the epipolar plane using a straight line detection algorithm and calculate slope information, restore the depth information of different objects in the scene from the slope information, and use a fast interpolation algorithm to generate a depth image of the entire scene ; The segmentation module is configured to set corresponding depth thresholds for different objects in the depth image, and quickly segment different objects according to the depth thresholds. 8. The device according to claim 7, characterized in that, in the three-dimensional light field generated by the building block, any light L is expressed as: L=LF(x,y,t) Wherein, t is the starting point of the ray, that is, the coordinates of the camera on the one-dimensional straight line; (x, y) represents the direction of the ray, corresponding to the two-dimensional coordinate value in the image; The epipolar plan is the stack of horizontal pixels of the sequence image under the same y value condition, that is, the (x, t) section perpendicular to the y coordinate; the pixels of the same object in the scene form a line in the epipolar plan The linear trajectory, and the spatial distance between the object and the camera linear motion trajectory is proportional to the slope of the corresponding straight line of the object in the epipolar plan view. 9. The device according to claim 7, wherein the depth restoration module is further used for: selecting one of the sequence images as a target image for depth restoration and foreground segmentation; Extracting straight lines in the polar plane and determining all straight line regions; generating the slope distribution of straight line feature points in the target image according to the straight line areas; generating all pixels of the target image using an interpolation algorithm according to the slope distribution of the straight line feature points The slope distribution of points; the slope distribution of all pixels of the target image is transformed into a depth distribution, and then linearly mapped to a gray scale interval, and finally the depth image is generated. 10. The device according to claim 9, wherein the depth restoration module further comprises a scaling module for performing Gaussian scaling on the epipolar plan before using a straight line detection algorithm to extract a straight line, the scaling module Scaling is done with a zoom ratio of 0.9.