CN105488812A

CN105488812A - Motion-feature-fused space-time significance detection method

Info

Publication number: CN105488812A
Application number: CN201510823908.7A
Authority: CN
Inventors: 于凤芹
Original assignee: Jiangnan University
Current assignee: Jiangnan University
Priority date: 2015-11-24
Filing date: 2015-11-24
Publication date: 2016-04-13

Abstract

The invention belongs to the field of image and video processing, in particular to a space-time significance detection method which fuses space-time significance and motion features. The space-time significance detection method comprises the following steps: firstly, utilizing a superpixel partitioning algorithm to express each frame of image as one series of superpixels, and extracting a superpixel-level color histogram as features; then, obtaining a spatial salient map through the calculation of the global comparison and the spatial distribution of colors; thirdly, through optical flow estimation and block matching methods, obtaining a temporal salient map; and finally, using a dynamic fusion strategy to fuse the spatial salient map and the temporal salient map to obtain a final space-time salient map. The method fuses the space significance and the motion features to carry out significance detection, and the algorithm can be simultaneously applied to the significance detection in dynamic and static scenes.

Description

A spatio-temporal saliency detection method integrating motion features

1、技术领域1. Technical field

本发明属于图像与视频处理领域，具体是一种融合运动特征的时空显著性检测方法。本发明以基于区域的显著性检测模型为基础，首先，用超像素分割算法将每一帧图像表示为一系列超像素并且提取超像素级的颜色直方图作为特征；然后，通过光流估计和块匹配的方法得到运动显著图，根据颜色的全局对比和空间分布得到空间显著图；最后，使用一种动态融合策略将运动显著图和空间显著图融合成为最终的时空显著图。该方法融合了运动特征进行显著性检测，能够同时运用于静态和动态场景中。The invention belongs to the field of image and video processing, and in particular relates to a spatio-temporal saliency detection method integrating motion features. The present invention is based on a region-based saliency detection model. First, each frame of image is represented as a series of superpixels with a superpixel segmentation algorithm and the superpixel-level color histogram is extracted as a feature; then, through optical flow estimation and The block matching method is used to obtain the motion saliency map, and the spatial saliency map is obtained according to the global contrast and spatial distribution of colors; finally, a dynamic fusion strategy is used to fuse the motion saliency map and the spatial saliency map into the final spatio-temporal saliency map. The method combines motion features for saliency detection and can be applied to both static and dynamic scenes.

2、背景技术2. Background technology

显著性检测是指准确地定位并提取在视频和图像中包含较大的信息量、能吸引人的注意力的区域。在图像处理过程中，将较高的处理优先级赋予图像显著区域，既可以降低计算过程的复杂度，又可以提高图像处理的效率。因此，显著性检测在目标识别、图像检索、视频图像编码等领域具有广泛的应用前景。Saliency detection refers to accurately locating and extracting areas in videos and images that contain a large amount of information and can attract people's attention. In the process of image processing, assigning higher processing priority to salient regions of the image can not only reduce the complexity of the calculation process, but also improve the efficiency of image processing. Therefore, saliency detection has broad application prospects in object recognition, image retrieval, video image coding and other fields.

根据处理的信息源图像的不同，显著性检测模型可分为空域显著性检测模型和时空显著性检测模型。针对静态场景的空域显著性检测，主要有基于生物启发模型和特征整合理论的显著性检测方法，该算法通过中心-周围算子计算不同尺度上特征的差别来得到显著度。有通过计算像素与其周边像素的特征距离的基于局部对比度分析的图像显著性检测。也有将图像分割为若干区域，结合图像的空间特性和颜色对比度来计算全局对比的显著区域检测方法。According to the different information source images processed, saliency detection models can be divided into spatial-domain saliency detection models and spatio-temporal saliency detection models. For the spatial saliency detection of static scenes, there are mainly saliency detection methods based on biologically inspired models and feature integration theory. This algorithm calculates the difference of features at different scales through the center-surrounding operator to obtain the saliency. There is image saliency detection based on local contrast analysis by computing the feature distance of a pixel to its surrounding pixels. There is also a salient area detection method that divides the image into several areas, and combines the spatial characteristics and color contrast of the image to calculate the global contrast.

动态场景下的时空显著性检测算法不仅要考虑检测空间显著性区域，还要考虑时间轴上显著性的影响因素，如物体运动、自然条件变化、相机移动等等。根据计算方式的不同，时空显著性检测算法主要分为四类：(1)基于特征融合的模型：在图像显著性检测模型的基础上加入运动特征，计算连续两帧图像的差异来得到运动产生的显著性；(2)基于空间平面的模型：谱残余方法的基础上，认为在时间轴上，帧序列构成的像素点在X-T，Y-T平面都满足谱残余的结论，将X-T，Y-T平面看作二维矩阵，分别进行低秩-稀疏分解，最后融合成为最终的显著图；(3)基于频域分析的模型：从频域分析角度出发，将亮度、颜色、运动三类特征组合四元数组，然后通过四元数傅里叶变换获取视频在时空域上的相位谱，并利用该相位谱检测视频的显著性；(4)基于背景检测的模型：高斯混合模型也被用于时空显著性检测，在单一场景中，将高斯混合模型计算出的前景区域作为显著区域。以上方法将静态显著性和动态显著性简单的线性融合或者只突出了运动特征，忽略了场景中的动态特性和空间特性，难以获取准确的显著区域。Spatio-temporal saliency detection algorithms in dynamic scenes should not only consider the detection of spatial saliency regions, but also consider factors affecting saliency on the time axis, such as object movement, changes in natural conditions, camera movement, and so on. According to different calculation methods, spatiotemporal saliency detection algorithms are mainly divided into four categories: (1) Models based on feature fusion: motion features are added to the image saliency detection model, and the difference between two consecutive frames of images is calculated to obtain motion generation. (2) The model based on the spatial plane: based on the spectral residual method, it is considered that on the time axis, the pixels formed by the frame sequence satisfy the conclusion of the spectral residual in the X-T and Y-T planes. Viewing the X-T and Y-T planes Make a two-dimensional matrix, perform low-rank-sparse decomposition respectively, and finally fuse into the final saliency map; (3) A model based on frequency domain analysis: from the perspective of frequency domain analysis, combine the three types of features of brightness, color, and motion into four elements Array, and then obtain the phase spectrum of the video in the time-space domain through the quaternion Fourier transform, and use the phase spectrum to detect the saliency of the video; (4) Model based on background detection: Gaussian mixture model is also used for spatio-temporal saliency In a single scene, the foreground area calculated by the Gaussian mixture model is regarded as the salient area. The above methods simply linearly fuse static saliency and dynamic saliency or only highlight motion features, ignoring the dynamic and spatial characteristics of the scene, making it difficult to obtain accurate salient regions.

3、发明内容3. Contents of the invention

本发明以基于区域的显著性检测模型为基础，融入光流向量块对比的方法提取图像序列的运动特征，并提出了一种将空域显著性和时域显著性动态融合的策略，能够同时运用于静态和动态的自然场景中。Based on the region-based saliency detection model, the invention incorporates the optical flow vector block comparison method to extract the motion features of the image sequence, and proposes a strategy to dynamically integrate spatial saliency and temporal saliency, which can simultaneously use in static and dynamic natural scenes.

(1)超像素分割及特征提取(1) Superpixel segmentation and feature extraction

使用简单线性迭代聚类将每一帧图像F_t表示为一系列超像素将超像素作为基本的处理单元，可以减少处理单元的数量，而且可以保证最终的检测结果能够均匀地突出显著对象。颜色特征的提取采用颜色直方图，将Lab颜色空间的每一个通道量化得到12个不同的值，将颜色的数量减少到qc＝12³＝1728，对于每一个超像素计算超像素中所有像素在Lab空间的均值，并且进行量化得到颜色直方图CH_t，最后将颜色直方图归一化，使得 $Σ_{k = 1}^{q c} {CH}_{t} (k) = 1.$ Represent each frame of image F _t as a sequence of superpixels using simple linear iterative clustering Using superpixels as the basic processing unit can reduce the number of processing units and ensure that the final detection results can evenly highlight salient objects. The extraction of color features uses a color histogram, quantizes each channel of the Lab color space to obtain 12 different values, and reduces the number of colors to qc=12 ³ =1728, for each superpixel Calculate the mean value of all pixels in the superpixel in the Lab space, and perform quantization to obtain the color histogram CH _t , and finally normalize the color histogram, so that $Σ_{k = 1}^{q c} {CH}_{t} (k) = 1.$

(2)基于运动特征的时间显著性(2) Temporal salience based on motion features

本发明采用光流运动估计和块匹配的方法来提取图像序列的运动特征。光流运动估计法的基本思想是将运动图像函数f(x，y，t)作为基本函数，根据图像强度守恒原理建立光流约束方程，通过求解光流约束方程，计算运动参数。对于当前帧F_t，使用其前一帧F_t-1作为参考帧，通过光流运动估计法计算得到F_t的运动向量场(u_(x，y)，v_(x，y))，对于当前帧F_t中的每一个超像素计算其平均运动矢量大小 The invention adopts the method of optical flow motion estimation and block matching to extract the motion feature of the image sequence. The basic idea of the optical flow motion estimation method is to use the moving image function f(x, y, t) as the basic function, establish the optical flow constraint equation according to the image intensity conservation principle, and calculate the motion parameters by solving the optical flow constraint equation. For the current frame F _t , using its previous frame F _t-1 as a reference frame, the motion vector field (u _{(x, y)} , v _{(x, y)} ) of F _t is calculated by the optical flow motion estimation method. For Each superpixel in the current frame F _t Calculate its average motion vector magnitude

为了克服背景运动和相机抖动的影响，本发明通过块匹配的方法在前一帧中找到与当前帧最匹配的超像素，并且计算该超像素与其背景超像素的相对运动值作为其显著值。具体的实现方法：在帧F_t-1中选择与其最匹配的超像素把相连接的超像素与其一起作为相关联超像素集合ψ_i，分别表示超像素i，j的中心位置。因此，超像素的时间显著性定义为：In order to overcome the influence of background motion and camera shake, the present invention uses a block matching method to find the superpixel that best matches the current frame in the previous frame, and calculates the relative motion value of the superpixel and its background superpixel as its salient value. The specific implementation method: select the superpixel that best matches it in the frame F _t-1 Bundle The connected superpixels are together with it as the set of associated superpixels ψ _i , Denote the center positions of superpixels i and j, respectively. Therefore, superpixels The temporal significance of is defined as:

${S S}_{m m o o t t i i o o n no} (({sp sp}_{t t}^{i i})) = = \frac{\underset{{sp sp}_{t t - - 11}^{j j} &Element; &Element; {ψ ψ}_{i i}}{Σ Σ} λ λ (({sp sp}_{t t}^{i i},, {sp sp}_{t t - - 11}^{j j})) \cdot \cdot {S S}_{m m o o t t i i o o n no} (({sp sp}_{t t - - 11}^{j j}))}{\underset{{sp sp}_{t t - - 11}^{j j} &Element; &Element; {ψ ψ}_{i i}}{Σ Σ} λ λ (({sp sp}_{t t}^{i i},, {sp sp}_{t t - - 11}^{j j}))} - - - - - - ((11))$

其中，帧间相关性 $λ ({sp}_{t}^{i}, {sp}_{t - 1}^{j}) = \sqrt{c (i) \cdot c (j)} \cdot [1 - \frac{| | μ_{t}^{i} - μ_{t - 1}^{j} + {mmv}_{t}^{i} | |}{d}],$ c(i)和c(j)分别表示超像素i和j在Lab颜色空间量化后的颜色值。Among them, the inter-frame correlation $λ ({sp}_{t}^{i}, {sp}_{t - 1}^{j}) = \sqrt{c (i) &Center Dot; c (j)} &Center Dot; [1 - \frac{| | μ_{t}^{i} - μ_{t - 1}^{j} + {mmv}_{t}^{i} | |}{d}],$ c(i) and c(j) denote the quantized color values of superpixels i and j in the Lab color space, respectively.

(3)空间显著性(3) Spatial salience

本发明根据颜色的全局对比和空间分布来计算每一帧图像的空间显著性。对于当前帧中的超像素其颜色全局对比显著值定义为：The present invention calculates the spatial salience of each frame image according to the global contrast and spatial distribution of colors. For the superpixels in the current frame Its color global contrast salient value is defined as:

${S S}_{G G C C} ((i i)) = = {Σ Σ}_{j j = = 11}^{{n no}_{t t}} {f f}_{j j} \cdot \cdot | | | | c c ((i i)) - - c c ((j j)) | | | | - - - - - - ((22))$

其中，f_j超像素的直方图在整幅图像中出现的概率，c(i)和c(j)分别表示超像素i和j在Lab颜色空间量化后的颜色值。where f _j superpixel The probability that the histogram of is present in the entire image, c(i) and c(j) represent the color values of superpixels i and j quantized in the Lab color space, respectively.

颜色的空间分布也会影响图像的显著性，颜色分布越紧密，则显著性越高，因此颜色的空间分布显著性定义为：The spatial distribution of colors also affects the salience of the image, the tighter the color distribution, the higher the salience, so the salience of the spatial distribution of colors is defined as:

${S S}_{s the s c c} ((i i)) = = \frac{{Σ Σ}_{j j = = 11}^{{n no}_{t t}} λ λ (({sp sp}_{t t}^{j j},, {sp sp}_{t t}^{j j})) \cdot \cdot D D. (({sp sp}_{t t}^{j j}))}{{Σ Σ}_{j j = = 11}^{{n no}_{t t}} λ λ (({sp sp}_{t t}^{i i},, {sp sp}_{t t}^{j j}))} - - - - - - ((33))$

其中，为超像素j的中心与图像中心的距离， $λ ({sp}_{t}^{i}, {sp}_{t}^{j}) = \sqrt{c (i) \cdot c (j)} \cdot [1 - \frac{| | μ_{t}^{i} - μ_{t}^{j} | |}{d}],$ 当超像素i和j的颜色越相近并且距离越小，的值越大。in, is the distance between the center of superpixel j and the image center, $λ ({sp}_{t}^{i}, {sp}_{t}^{j}) = \sqrt{c (i) \cdot c (j)} &Center Dot; [1 - \frac{| | μ_{t}^{i} - μ_{t}^{j} | |}{d}],$ When the colors of superpixels i and j are closer and the distance is smaller, The larger the value is.

最后将颜色全局分布显著性与空间分布显著性融合得到图像的空间显著性：Finally, the spatial saliency of the image is obtained by combining the saliency of the global color distribution and the saliency of the spatial distribution:

${S S}_{s the s p p a a t t i i a a l l} (({sp sp}_{t t}^{i i})) = = {S S}_{G G C C} (({sp sp}_{t t}^{i i})) \cdot \cdot {S S}_{S S C C} (({sp sp}_{t t}^{i i})) - - - - - - ((44))$

(4)时空显著性(4) Spatiotemporal salience

将时间显著性和空间显著性自适应线性融合得到时空显著图，即：The temporal saliency and spatial saliency are adaptively fused linearly to obtain a spatiotemporal saliency map, namely:

$S S a a l l i i e e n no c c y the y (({sp sp}_{t t}^{i i})) = = α α \cdot &Center Dot; {S S}_{s the s p p a a t t i i a a l l} (({sp sp}_{t t}^{i i})) + + ((11 - - α α)) \cdot &Center Dot; {S S}_{n no m m p p o o r r a a l l} (({sp sp}_{t t}^{i i})) - - - - - - ((55))$

权值α定义为：The weight α is defined as:

$α α = = m m a a x x [[\frac{m m e e d d (({MS MS}_{t t - - f f},, ... ...,, {MS MS}_{t t - - 11}))}{m m e e d d (({MS MS}_{t t - - f f},, ... ...,, {MS MS}_{t t - - 11})) + + {MS MS}_{t t}},, 0.5 0.5]] - - - - - - ((66))$

其中，表示超像素i中的像素个数。动态场景下当运动越明显时，时间显著性的权值越大，静态场景下，α直接设置为1。in, Indicates the number of pixels in superpixel i. In a dynamic scene, when the movement is more obvious, the weight of time salience is larger. In a static scene, α is directly set to 1.

4、附图说明4. Description of drawings

附图是本发明的原理和实现步骤。Accompanying drawing is the principle and implementation steps of the present invention.

5、具体实施方式5. Specific implementation

附图1为该发明的实施流程图，其具体步骤为：Accompanying drawing 1 is the implementation flowchart of this invention, and its concrete steps are:

(1)图像预处理：将输入的每一帧图像通过SLIC超像素分割算法分割为一系列大小均匀、紧凑的超像素作为显著性检测的基本处理单元。(1) Image preprocessing: Each frame of input image is divided into a series of uniform and compact superpixels by SLIC superpixel segmentation algorithm as the basic processing unit of saliency detection.

(2)颜色特征的提取：对于每一帧图像，以超像素为单位，计算超像素中所有像素在Lab空间的均值，进行量化得到颜色直方图CH_t并且归一化使得 (2) Extraction of color features: For each frame of image, in units of superpixels, calculate the mean value of all pixels in the superpixel in the Lab space, quantify to obtain the color histogram CH _t and normalize so that

(3)时间显著性计算：通过光流运动估计法计算得到F_t相对于前一帧F_t-1的运动向量场(u_(x，y)，v_(x，y))，并且计算得到超像素内平均矢量场大小然后通过块匹配的方法在前一帧中找到与当前帧最匹配的超像素及其相关超像素集合，使用公式(1)得到图像的时间显著图。(3) Temporal saliency calculation: Calculate the motion vector field (u _{(x, y)} , v _{(x, y)} ) of F _t relative to the previous frame F _t-1 by optical flow motion estimation method, and calculate super pixel Inner mean vector field size Then find the superpixel and its related superpixel set that best match the current frame in the previous frame through the block matching method, and use formula (1) to get the temporal saliency map of the image.

(4)空间显著性计算：根据基于区域的静态显著性检测模型为基础，使用公式(2)和公式(3)分别得到以超像素为基本单位的颜色全局对比显著性和颜色空间分布显著性，并通过公式(4)将其融合成为最终的空间显著图。(4) Spatial saliency calculation: Based on the region-based static saliency detection model, formula (2) and formula (3) are used to obtain the color global contrast saliency and color space distribution saliency with superpixels as the basic unit , and fuse them into the final spatial saliency map by formula (4).

(5)时空显著性计算：根据公式(5)将时间显著性和空间显著性自适应线性融合得到时空显著图。(5) Calculation of spatio-temporal saliency: According to formula (5), the spatio-temporal saliency and spatial saliency are adaptively fused linearly to obtain a spatio-temporal saliency map.

Claims

1. A spatio-temporal saliency detection method of fusion motion feature, characterized in that:

Superpixel segmentation is performed on each frame of image, and the color histogram is extracted with superpixel as the basic unit. The spatial saliency of each frame of image is calculated according to the global contrast and spatial distribution of the color, and the method of optical flow motion estimation and block matching is used. To extract the motion features of the image sequence, the spatial saliency and temporal saliency are dynamically fused, so that the algorithm can be applied to the detection of motion features in static and dynamic natural scenes at the same time.

2. a kind of space-time saliency detection method of fusion motion feature as claimed in claim 1, it is characterized in that described superpixel segmentation and superpixel region color histogram feature extraction method are:

Represent each frame of image F _t as a series of superpixels using a linear iterative clustering method Using superpixels as the basic processing unit can reduce the number of processing units and ensure that the final detection results can evenly highlight salient objects.

A color histogram is calculated for the color features of the segmented superpixel regions. Each channel of the Lab color space is quantized to obtain 12 different values, reducing the number of colors to qc=12 ³ =1728. For each superpixel Calculate the mean value of all pixels in the superpixel in the Lab space, and perform quantization to obtain the color histogram CH _t , and finally normalize the color histogram, so that

3. A kind of spatio-temporal saliency detection method of fusion motion feature as claimed in claim 1, it is characterized in that described spatial saliency calculation method is:

The present invention calculates the spatial salience of each frame image according to the global contrast and spatial distribution of colors. For the superpixels in the current frame Its color global contrast salient value is defined as:

where f _j superpixel The probability that the histogram of is present in the entire image, c(i) and c(j) represent the color values of superpixels i and j quantized in the Lab color space, respectively.

The spatial distribution of colors also affects the salience of the image, the tighter the color distribution, the higher the salience, so the salience of the spatial distribution of colors is defined as:

in, is the distance between the center of superpixel j and the image center, When the colors of superpixels i and j are closer and the distance is smaller, The larger the value is.

Finally, the spatial saliency of the image is obtained by combining the saliency of the global color distribution and the saliency of the spatial distribution:

.

4. a kind of spatio-temporal saliency detection method of fusion motion feature as claimed in claim 1, it is characterized in that described employing optical flow motion estimation and the method for block matching to extract the calculation method of the motion feature of image sequence is:

The basic idea of the optical flow motion estimation method is to use the moving image function f(x, y, t) as the basic function, establish the optical flow constraint equation according to the image intensity conservation principle, and calculate the motion parameters by solving the optical flow constraint equation. For the current frame F _t , using its previous frame F _t-1 as a reference frame, the motion vector field (u _{(x, y)} , v _{(x, y)} ) of F _t is calculated by the optical flow motion estimation method. For Each superpixel in the current frame F _t Calculate its average motion vector magnitude

In order to overcome the influence of background motion and camera shake, the present invention uses a block matching method to find the superpixel that best matches the current frame in the previous frame, and calculates the relative motion value of the superpixel and its background superpixel as its salient value. The specific implementation method: select the superpixel that best matches it in the frame F _t-1 Bundle The connected superpixels are together with it as the set of associated superpixels ψ _i , Denote the center positions of superpixels i and j, respectively. Therefore, superpixels The temporal significance of is defined as:

Among them, the inter-frame correlation c(i) and c(j) denote the quantized color values of superpixels i and j in the Lab color space, respectively.

5. A kind of spatio-temporal saliency detection method of fusion motion feature as claimed in claim 1, it is characterized in that described temporal saliency and spatial saliency adaptive linear fusion obtains the calculation method of spatio-temporal saliency map is:

The weight α is defined as:

in, Indicates the number of pixels in superpixel i. In a dynamic scene, when the movement is more obvious, the weight of time salience is larger. In a static scene, α is directly set to 1.