CN104506852B - A kind of objective quality assessment method towards video conference coding - Google Patents

Abstract

The invention discloses an objective quality evaluation method for video conference coding, which includes two parts: training and evaluation; the training part includes step 1: extracting the face and face area; step 2: obtaining the attention degree of a single pixel point; step Step 3: Calibrate and normalize the face area; Step 4: Obtain a Gaussian mixture model; the evaluation part includes Step 1: For a set of videos, automatically extract the background, face, left eye, right eye, mouth, and nose area The number of inner pixels; Step 2: Calibrate and normalize the face area; Step 3: Obtain the weight map; Step 4: Calculate the peak signal-to-noise ratio based on the Gaussian mixture model, and evaluate the encoded image quality of the video conference system. The invention avoids the deficiency that the traditional method does not take into account the video content, and can improve the accuracy of image quality evaluation by giving more weight to the face of the video image, so that it can better reflect the result of subjective quality evaluation.

Description

An Objective Quality Assessment Method for Video Conference Coding

technical field

The invention relates to an objective quality evaluation method for video conference coding, and belongs to the technical field of perceptual visual quality evaluation of video conference coding.

Background technique

Visual quality metrics are essential when evaluating the efficiency of different video encoding schemes. Visual quality assessment for perceptual video coding can be divided into two categories: subjective assessment and objective assessment. Since humans are the most immediate recipients when viewing video, subjective visual quality assessment is the most accurate and reliable method of evaluating video encoding. But its low efficiency and high cost have promoted the development of objective visual quality evaluation metrics. The purpose of objective assessment is to improve its correlation with subjective visual quality to accurately measure visual quality. The most widely used objective metrics include peak signal-to-noise ratio (PSNR), structural similarity (SSIM), visual signal-to-noise ratio (VSNR), visual quality Metrics (video quality metrics, VQM) and motion-based video integrity evaluation (MOtion-based Video Integrity Evaluation, MOVIE).

Perceptual video coding for video conferencing has been extensively studied because the face is a ROI (Region-of-Interest) for video conferencing. However, there is currently no objective visual quality assessment method developed specifically for videoconferencing.

Contents of the invention

The purpose of the present invention is to solve the deficiency of the existing objective evaluation method of video quality, and provide an objective index for video conference coding, aiming at improving the correlation with the viewer's subjective perception quality.

An objective quality assessment method for video conference coding, including training and evaluation;

The training part consists of the following steps:

Step 1: Face and face area extraction;

Step 2: Carry out an eye tracker experiment to obtain the coordinate position of the attention point of each frame of the image when the tester watches the video, and obtain the degree of attention of a single pixel;

Step 3: Calibrate and normalize the face area;

Step 4: Obtain the Gaussian mixture model;

The evaluation section consists of the following steps:

Step 1: For a group of videos, repeat step 1 of the training part to automatically extract the number of pixels in the background, face, left eye, right eye, mouth, and nose areas;

Step 2: Repeat step 3 of the training process to calibrate and normalize the face area;

Step 3: Based on the Gaussian mixture model obtained during the training phase, calculate the weights of the right eye, left eye, mouth, nose, other areas of the face, background area, and the Gaussian distribution weights around the above areas to obtain a weight map;

Step 4: On the basis of the weight map, calculate the peak signal-to-noise ratio based on the Gaussian mixture model, and evaluate the encoded image quality of the video conference system.

The advantages of the present invention are:

(1) The present invention is aimed at the image quality evaluation method after video conferencing system encoding, which avoids the lack of video content that is not considered in the traditional method, and can improve the accuracy of image quality evaluation by giving more weight to the face of the video image, making it More reflect the results of subjective quality assessment;

(2) On the basis of extracting various areas of the face (such as nose and mouth), the present invention assigns greater weights to some key areas of the face, so as to meet the needs of current and future video conferencing systems with continuous improvement in resolution and increasing display size. greater development trend;

(3) The present invention can excavate the rule of people's visual attention when video conferencing by introducing the experimental data of the eye tracker, in conjunction with the calculation tool of statistical learning, and further apply it to the image quality evaluation after video conferencing system coding, greatly Improve its correlation with subjective quality assessment.

Description of drawings

Fig. 1 is method flowchart of the present invention;

Figure 2 facial features automatic calibration algorithm;

Figure 3 Automatic extraction of key areas of the face;

Figure 4 Calibration and normalization methods;

The drawing method of Fig. 5 weight map;

Fig. 6 Schematic diagram of calculation of GMM-PSNR.

detailed description

The present invention will be further described in detail with reference to the accompanying drawings and embodiments.

The present invention adopts a real-time facial feature automatic calibration method to track key feature points of the face. After face detection, a Point Distribution Model (PDM) of key features is generated on video frames by combining local detection (texture information) and global optimization (facial structure). In the present invention, a 66-point PDM is used to extract the face and the contour of the face. The 66-point PDM can well sample key points of faces and facial features, and thus these points can be connected to accurately extract contours and regions of faces and facial features. Therefore, a 66-point PDM is used in the method to extract face and facial key features. Finally, faces and face key regions are extracted according to their contours.

In the video of dialogue scenes, it is found through experiments that the face content in the video can attract most of the attention of the observer. Therefore, according to the observer's focus, the importance of background, face and facial features is further quantified, thereby improving the accuracy of objective quality assessment of video conferencing. To obtain such unequal importance values, some eye-tracking experiments were performed on conference-related videos.

In the experiment, an eye tracker was used to record the gaze point of the observer's eyes on the video frame when watching the video. Eye gaze points represent the focus points of the observer, therefore, the results of eye tracking can be used to generate subjective attention models. After the eye tracker experiment, the number of eye fixations belonging to the right eye, left eye, mouth, nose, other areas of the face, and the background were recorded. According to the number of eye fixation points falling on different areas, a new concept, eye fixation point/pixel (EFP/P), is introduced to reflect the pixel level of attention in these areas. Here, there are the following EFP/P values.

Once the results of the above eye tracker experiments are obtained, they are used to train a GMM to produce an importance weight map for each video frame. Therefore, GMM-PSNR can be computed by combining the corresponding weight maps. Before training the GMM, preprocessing is performed to calibrate and normalize the eye fixations obtained in the previous section. Subsequently, the GMM is trained with the Expectation-Maximization (EM) algorithm on the calibrated and normalized eye fixations. GMM can be obtained by running several EM iterations until convergence. Given the parameters of the obtained GMM, a weight map can be calculated to establish an objective measure of GMM-PSNR.

The present invention is an objective quality evaluation method oriented to video conference coding, the process flow is shown in Figure 1, including two parts of training and evaluation;

The training part consists of the following steps:

Step 1: Face and face area extraction;

Using the facial feature automatic calibration algorithm to automatically extract the number of pixels in the background, face, left eye, right eye, mouth, and nose area in a given video conference sequence.

Specifically: first, obtain the key points of the face area in each frame image in the video conference sequence through the facial feature automatic calibration algorithm; second, use the average value drift technology to locally search the face area The key points of the left eye, right eye, mouth, and nose area in the image of the inner region, and match these key points with the key point distribution model (PDM) in the database to realize the key points of the left eye, right eye, mouth, and nose area Optimization, thirdly, get the key points of the face, left eye, right eye, mouth, and nose in each frame of image after optimization, as shown in Figure 2, a total of 66 key points have been obtained, and fourthly, the face The key points of the face, left eye, right eye, mouth, and nose are connected to obtain the contours of the face, left eye, right eye, mouth, and nose, as shown in Figure 3. Fifth, obtain the face, left eye, and right The number of pixels in the eye, mouth, and nose areas is subtracted from the number of image pixels by the number of face pixels to obtain the number of background pixels, and finally the automatic extraction of key areas of the face is realized.

Among them, the point distribution model adopts the mean shift technique and is trained on a set of standard test images.

Among them, the key points of the face, left eye, right eye, mouth, and nose area in different face images can be extracted.

Step 2: Carry out an eye tracker experiment to obtain the coordinate position of the attention point of each frame of the image when the tester watches the video, and obtain the degree of attention of a single pixel;

Let the degree of attention of a single area (left eye, right eye, mouth, nose, other areas of the face, background) be the number of eye attention points / the number of pixels in this area (efp/p):

$\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}\\ {\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}\\ {\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}\\ {\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}\\ {\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}}<span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}}\\ {\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; b</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; b</span>}}<span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; b</span>}}\end{array}\right.$

Among them: c _r , c _l , _{cm , c n ,} c _o , c _b respectively represent the degree of attention of a single pixel in the right eye, left eye, mouth, nose, other areas of the face, and background area, f _r , f _l , f _m , f _n , f _o , and f _b respectively represent the number of focus points of the tester on the right eye, left eye, mouth, nose, other areas of the face, and background areas in the eye tracker experiment, p _r , p _l , p _m , p _n , p _o , p _b respectively represent the number of pixels in the right eye, left eye, mouth, nose, other areas of the face, and the background area;

Step 3: Calibrate and normalize the face area;

The calibration can avoid the uncertainty caused by different positions of the face in the image, and the normalization method can make the present invention adapt to the situation that the number of pixels in the face area is not equal in the video conference.

The specific method is:

As shown in Figure 4(a), a frame of image is randomly selected, and the leftmost point in the key points of the face area of the image is used as the original point B for calibration, and the leftmost point A in the key points of the face area in other images is obtained. Obtain the coordinate transformation relationship between A and B, convert the points of interest in other images according to the coordinate transformation relationship, and complete the calibration.

As shown in Figure 4(b), a frame of image is randomly selected, and the abscissa length of the right eye of the person in the image (the distance between the point on the right side of the right eye and the point on the left side of the right eye among the 66 points) is used as The normalization unit is used to normalize the points of interest in other images according to the normalization unit.

Step 4: Obtain a Gaussian mixture model;

Assuming that the gaze point of the eyes obeys the Gaussian mixture model, on the basis of the normalized and calibrated eye tracker data, the linear superposition of Gaussian components is written through the Gaussian mixture model as follows:

$\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; K</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; \&pi;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}{<span\; class="notranslate">\; \&aleph;</span>}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; )</span>$

${<span\; class="notranslate">\; \&aleph;</span>}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;</span>}}<span\; class="notranslate">\; \&Center\; Dot;</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; |</span>}^{\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}}}<span\; class="notranslate">\; \&Center\; Dot;</span>\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; \{</span><span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; T</span>}}<span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; \&Center\; Dot;</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \}</span>$

in: represents a Gaussian component, π _k , μ _k and Σ _k are the mixing coefficients, mean and variance of the kth Gaussian component, and x ^* represents the eye gaze point after two-dimensional calibration and normalization. K represents the number of Gaussian components of the GMM. Since the number of eye fixations of the nose is much less than that of the eyes and mouth, the number K of Gaussian components is set to 3 here, which correspond to the right eye, left eye and mouth respectively. Meanwhile, _μk is set as the normalized centroid of each facial feature.

In the above steps, in an offline situation, aiming at a set of training videos, the Gaussian mixture model used to evaluate the objective quality of the video conference system is obtained by designing an eye tracker experiment and its data analysis.

(2) The evaluation part includes the following steps

Step 1: Same as Step 1 of the training process, the background, face, left eye, right eye, mouth, and nose regions are automatically extracted.

Step 2: Same as step 3 of the training process, calibrate and normalize the face area of the video. See Figure 4 for details.

Step 3: Based on the Gaussian mixture model obtained during the training phase, calculate the weights of the right eye, left eye, mouth, nose, other areas of the face, background area, and the weights of the Gaussian distribution around the above areas. See Figure 5 for details.

Figure 5 shows the drawing method of the weight map. In the implementation of this example, the weight map can be used to quantify the importance of each pixel of the face and the background in the video conferencing system. The input of this example is a frame image of a video conference. First, according to the method in Fig. 3, the face and key areas of the face are automatically extracted. Secondly, the key points in the video are calibrated and normalized according to the method in Figure 4. Finally, using the parameters obtained from the GMM training in steps 2 and 4 of the training part in Figure 1, according to the area to which each pixel belongs (mainly background, face, left eye, right eye, nose, and mouth), calculate the value of each pixel by the following formula: weight, and output the weight map of the video conference image, and set the importance of each pixel of the image in quality evaluation through the weight.

in,

$\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; max</span>}}{\mathrm{<span\; class="notranslate">\; \&pi;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}{<span\; class="notranslate">\; \&aleph;</span>}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span>}{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; others</span>}}\underset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; max</span>}}{\mathrm{<span\; class="notranslate">\; \&pi;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}{<span\; class="notranslate">\; \&aleph;</span>}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}}$

The present invention is not limited to adopting this method to set the weight of image pixels.

Step 4: On the basis of the weight map, calculate the peak signal-to-noise ratio (GMM-PSNR) based on the Gaussian mixture model, and evaluate the encoded image quality of the video conference system. See Figure 4 for details.

Figure 6 is a schematic diagram of the calculation of GMM-PSNR. In this implementation example, the calculation of the GMM-PSNR can output the GMM-PSNR which measures the quality of the encoded image of the video conferencing system. First, as with traditional measurement methods (such as PSNR), by calculating the root mean square error between the original video image and the video image to be evaluated, the residual error of the image before and after encoding is obtained. Then, the root mean square error is multiplied by the weight of the weight map to obtain the value of GMM-MSE. Finally, the GMM-PSNR is calculated by taking the logarithm. The specific calculation method and its calculation formula are in the description of Figure 1. The invention is not limited to improvements over conventional PSNR. Other measurement methods (such as SSIM=Structural SIMilarity, SSIM) can also be improved by multiplying them with the weights in the weight map.

The specific calculation formula is as follows:

${\mathrm{<span\; class="notranslate">\; MSE</span>}}_{\mathrm{<span\; class="notranslate">\; GMM</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; \&CenterDot;</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}$

${\mathrm{<span\; class="notranslate">\; PSNR</span>}}_{\mathrm{<span\; class="notranslate">\; GMM</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 10</span>}<span\; class="notranslate">\; \&CenterDot;</span>\mathrm{<span\; class="notranslate">\; log</span>}\frac{{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{{\mathrm{<span\; class="notranslate">\; MSE</span>}}_{\mathrm{<span\; class="notranslate">\; GMM</span>}}}$

where I'x and _Ix are the values of pixel _x on the processed video and the original video frame, respectively. M and N are the number of pixels along the vertical direction and the horizontal direction, respectively. n (=8) is the bit depth.

Finally, the present invention can output the peak signal-to-noise ratio (GMM-PSNR) based on the Gaussian mixture model (GMM-PSNR) of the video conference system after encoding, which is used to measure the degradation of image quality before and after video encoding. Like traditional peak signal-to-noise ratio (PSNR), GMM-PSNR is measured in dB. However, because people pay different attention to each area in the image when watching the video, GMM-PSNR assigns different weights to the face areas of different importance in the video conferencing system, thereby greatly improving its correlation with subjective quality assessment. Spend.

The present invention can provide a more effective evaluation method for the quality of video transmission in the video conference. After testing, compared with traditional objective video evaluation methods, such as VQM, MOVIE, PSNR, GMM-PSNR significantly improves the correlation with subjective test standards, such as MOS, DMOS, which shows that GMM-PSNR can be used as A More Efficient Objective Metric for Videoconferencing Coding. This is very beneficial for video processing, compression and video communication of video conferencing. It can monitor the performance of the video system and give feedback to adjust the codec or channel parameters to ensure that the video quality is within an acceptable range. Video quality assessment criteria can also be used for the design, evaluation and optimization of codec performance. It can also be used to design and optimize digital video systems that conform to the human visual model.

The invention relates to an objective quality assessment method for video sequences, which is used for perceptual visual quality assessment of video conference coding. The present invention employs eye tracker experiments and real-time techniques for face and face feature extraction. In the experiments, the importance of the background, face, and facial feature regions is determined based on the observer's attention to each part. Using the gaze points collected by the eye tracker, and assuming that the distribution is a Gaussian mixture model, an importance weight map can be generated, so that the observer's attention to each area in the conference video can be measured. According to this generated weight map, different weights can be assigned to each pixel on the video frame, thereby improving the existing objective video quality assessment methods. More specifically, the present invention relates to a perceptual video quality assessment for video conference coding based on existing video quality assessment methods.

Claims (3)

1. An objective quality assessment method for video conference coding, including training and evaluation; The training part consists of the following steps: Step 1: Face and face area extraction; Use the facial feature automatic calibration algorithm to automatically extract the number of pixels in the background, face, left eye, right eye, mouth, and nose area in a given video conference sequence; Step 2: Carry out an eye tracker experiment to obtain the coordinates of the key points of each frame of the image when the tester watches the video, and obtain the degree of attention of a single pixel; Let the degree of attention of a single area be the number of eye key points/the number of pixels in this area efp/p, where a single area is the left eye, right eye, mouth, nose, other areas of the face or the background, then: $\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}\\ {\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}\\ {\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}\\ {\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}\\ {\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}}<span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}}\\ {\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; b</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; b</span>}}<span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; b</span>}}\end{array}\right.$ Among them: c _r , c _l , _{cm , c n ,} c _o , c _b respectively represent the degree of attention of a single pixel in the right eye, left eye, mouth, nose, other areas of the face, and background area, f _r , f _l , f _m , f _n , f _o , and f _b represent the number of key points of the tester's eyes falling on the right eye, left eye, mouth, nose, other areas of the face, and background areas in the eye tracker experiment, p _r , p _l , p _m , p _n , p _o , p _b respectively represent the number of pixels in the right eye, left eye, mouth, nose, other areas of the face, and the background area; Step 3: Calibrate and normalize the face area; The specific method is: Randomly select a frame of image, use the leftmost point in the key points of the face area of the image as the calibration original point B, obtain the leftmost point A in the key points of the face area in other images, and obtain the coordinate transformation relationship between A and B , transform the key points in other images according to the coordinate transformation relationship, and complete the calibration; Randomly select a frame of image, use the abscissa length of the right eye of the person in the image as the normalization unit, and normalize the key points in other images according to the normalization unit; Step 4: Obtain the Gaussian mixture model; Assuming that the gaze point of the eyes obeys the Gaussian mixture model, on the basis of the normalized and calibrated eye tracker data, the linear superposition of Gaussian components is written through the Gaussian mixture model as follows: in: Represents a Gaussian component, π _k , μ _k and Σ _k are the mixing coefficients, mean and variance of the kth Gaussian component, and x ^* represents the eye gaze point after two-dimensional calibration and normalization; K represents the Gaussian component of GMM quantity; The above steps 1-4 are offline to obtain a Gaussian mixture model for evaluating the objective quality of the video conferencing system for a set of training videos; The evaluation section consists of the following steps: Step 1: For a group of videos, repeat step 1 of the training part to automatically extract the number of pixels in the background, face, left eye, right eye, mouth, and nose areas; Step 2: Repeat step 3 of the training process to calibrate and normalize the face area; Step 3: Based on the Gaussian mixture model obtained during the training phase, calculate the weights of the right eye, left eye, mouth, nose, other areas of the face, background area, and the Gaussian distribution weights around the above areas to obtain a weight map; Step 4: On the basis of the weight map, calculate the peak signal-to-noise ratio based on the Gaussian mixture model, and evaluate the encoded image quality of the video conference system. 2. A kind of objective quality assessment method for video conference coding according to claim 1, step one of the described training part is specifically: First, obtain the key points of the face area in each frame of the image in the video conference sequence through the facial feature automatic calibration algorithm; Second, use the average value drift technology to locally search the key points of the left eye, right eye, mouth, and nose area in the face area image on the extracted face area, and distribute these key points with the key points in the database The model is matched to realize the optimization of key points in the left eye, right eye, mouth, and nose areas; Third, get the key points of the face, left eye, right eye, mouth, and nose area in each frame of image after optimization; Fourth, connect the key points of the face, left eye, right eye, mouth, and nose respectively to obtain the contours of the face, left eye, right eye, mouth, and nose; Fifth, get the number of pixels in the face, left eye, right eye, mouth, and nose area respectively, subtract the number of face pixels from the number of image pixels, get the number of background pixels, and finally realize the automatic detection of key areas of the face extract. 3. a kind of objective quality assessment method facing video conference coding according to claim 1, in the step 4 of described training part, set K=3, corresponding to right eye, left eye and mouth respectively; Set μ _k Normalized centroid for each facial feature.