CN103618900A - Video region-of-interest extraction method based on encoding information - Google Patents

Abstract

The invention discloses a video interest region extraction method based on visual perception features and coding information, and relates to the field of video coding. The present invention includes the following steps: first extracting the brightness information of the current coded macroblock from the original video stream; then, using the inter-frame prediction mode type of the current coded macroblock to identify the saliency region of the spatial domain visual feature; and then encoding the macroblock of the previous frame The average motion vectors in the horizontal and vertical directions respectively are dynamic double thresholds. According to the comparison results of the horizontal and vertical motion vectors of the current coded macroblock and the dynamic double thresholds, the time-domain visual feature salience area is identified; finally, combined with the spatial domain and The identification result of the saliency region of temporal visual features defines the priority of video interest, and realizes the automatic extraction of video interest. The method of the invention can provide an important coding basis for ROI (Region of Interest, ROI)-based video coding technology.

Description

Extraction method of video region of interest based on coding information

technical field

The invention belongs to the field of video information processing. Using video coding technology and the principle of human visual perception to realize a fast extraction method of video interest region. The method can automatically analyze the input video stream, and use the encoding information to mark and output the region of interest in the video.

Background technique

The latest video coding standard H.264/AVC uses a variety of advanced coding technologies. While improving the coding performance, its coding complexity also increases sharply, which limits its wide application in multimedia information processing and real-time communication services. People have conducted in-depth research on how to improve the encoding speed of H.264/AVC, and proposed a large number of fast encoding optimization algorithms, but most of the algorithms do not distinguish the visual importance of each area in the video image, and use the same method for all encoding content. The encoding scheme ignores the difference in the human visual system HVS (Human Visual System, HVS) perception of video scenes.

Visual neuroscience research has demonstrated that HVS is selective in the perception of video scenes, giving different visual importance to different regions. Therefore, it is of great theoretical significance and application value to improve the real-time performance of video coding algorithms and reduce computational complexity by using existing coding information to analyze visual perception features, and then preferentially allocate computing resources to regions of interest based on visual perception features. Fast and effective visual feature analysis, especially effective detection of visual regions of interest is an important basis for optimizing coding resources and writing efficient video coding schemes.

Contents of the invention

The present invention is different from existing video moving object extraction methods such as optical flow method, frame difference method, motion energy detection method, background difference method, etc., and is based on coding information such as prediction mode and motion vector in video code stream, and according to coding The correlation between information and visual regions of interest, identifying spatial visual feature salience regions and temporal feature visual salience regions in video coding content, so as to realize automatic identification and acquisition of video regions of interest.

According to the HVS feature, human eyes are more sensitive to luminance information than chrominance information, and the method of the present invention aims at the encoding information of luminance components in video sequences to automatically identify and acquire video regions of interest.

The inventive method specifically comprises the following steps:

Step 1: Input a video sequence in YUV format and GOP (Group of Picture, GOP) structure as IPPP, read the luminance component Y of the encoded macroblock, and configure the encoding parameters and initialize the parameters;

Step 2: performing intra-frame predictive coding on the first frame of the video sequence, i.e. the I frame;

In the video coding standard, the I frame is used as a reference point for random access and contains a large amount of information. Since it cannot be coded using the temporal correlation between adjacent frames, the intra-frame predictive coding method is used to utilize the coded data in the current frame. The coding information of the reconstructed macroblock is used to predict the current macroblock to eliminate spatial redundancy. Performing intra-frame predictive coding on the first frame of a video sequence, that is, an I frame, is a conventional coding method commonly used in video coding.

Step 3: Perform inter-frame predictive coding on the current p frame, and eliminate time redundancy by utilizing the correlation of video content of adjacent frames. Record the inter prediction mode types of all coded macroblocks in the current frame, denoted as Mode _pn ;

Among them, p=1,2,3,...,L-1, represents the p-th video frame for inter-frame encoding, L is the total number of frames encoded in the entire video sequence; n indicates the nth in the current encoding frame The sequence number of a coded macroblock.

Step 4: Identify the spatial visual feature saliency area of the current p frame, specifically: if the inter-frame prediction mode Mode _pn of the current coded macroblock belongs to the sub-segmentation mode set or the intra-frame prediction mode set, that is, Mode _pn ∈{8×8 ,8×4,4×8,4×4}or{Intra16×16,Intra4×4}, then mark the macroblock as S _Yp (x,y,Mode _pn )=1, which belongs to the saliency of spatial visual features area, otherwise mark S _Yp (x, y, Mode _pn )=0; wherein, Y represents the luminance component of the coded macroblock, (x, y) represents the position coordinates of the coded macroblock, and the definitions of p and Mode _pn are the same as above, Traverse all coded macroblocks in the current p frame;

FIG. 1 shows a schematic diagram of the H.264 standard inter-frame prediction mode selection process.

After experiments, it is found that in the H.264/AVC standard coding, there is a strong correlation between the predictive coding results and the area of interest to the human eye: for the moving area or texture-rich area with high human attention, Mode _pn mostly chooses A set of sub-segmentation modes {8×8, 8×4, 4×8, 4×4}; when the camera is switched, the video content changes suddenly, or there is a moving object with a large range of motion, the human eye pays the most attention. At this time Mode _pn will select the intra prediction mode set {Intra16×16, Intra4×4}; for the background smooth area with low human attention, Mode _pn mostly selects the macroblock segmentation mode set {Skip,16×16,16× 8,8×16}. Figure 2 takes the Claire sequence as an example, and shows the distribution of inter-frame prediction modes in the 50th frame of the Claire sequence. From the figure, it can be found that in areas with high human attention, most of the coded macroblocks choose inter-frame sub-segmentation prediction Pattern collection.

以及垂直方向上的平均运动矢量

Step 5: Record the motion vector V _xpn in the horizontal direction and the motion vector V _ypn in the vertical direction of each coded macroblock in the pth frame; and calculate the average of all coded macroblocks in the horizontal direction in the previous coded frame motion vector

and the average motion vector in the vertical direction

in, ${\stackrel{\‾}{V}}_{x(p-1)\mathrm{the\; th}}=\frac{\underset{\mathrm{no}=1}{\overset{\mathrm{Num}}{\mathrm{\Σ}}}{V}_{x(p-1)\mathrm{no}}}{\mathrm{Num}},{\stackrel{\‾}{V}}_{\mathrm{the\; y}(p-1)\mathrm{the\; th}}=\frac{\underset{\mathrm{no}=1}{\overset{\mathrm{Num}}{\mathrm{\Σ}}}{V}_{\mathrm{the\; y}(p-1)\mathrm{no}}}{\mathrm{Num}};$ V _x(p-1)n and V _y(p-1)n represent the horizontal and vertical motion vectors of each coded macroblock in the previous coded frame, and the definitions of p and n are the same as in step 3; Num represents The number of macroblocks included in a coded frame, that is, the number of times of accumulation. Figure 3 takes the video in QCIF format (176×144) as an example, and shows the positions and serial numbers n of all coded macroblocks (16×16) in a coded frame. At this time, $\mathrm{Num}=\frac{176}{16}\×\frac{144}{16}=11\×9=99.$

或者当前编码宏块的垂直方向运动矢量V_ypn大于前一帧编码宏块在垂直方向运动矢量平均值

则该宏块属于时域视觉特征显著度区域，标记T_Yp(x,y,V_xpn,V_ypn)=1，否则标记T_Yp(x,y,V_xpn,V_ypn)=0，遍历当前p帧中的所有编码宏块；Step 6: Identify the temporal visual feature salience area of the current p frame, specifically: if the horizontal motion vector V _xpn of the current coded macroblock is greater than the average value of the horizontal motion vector of the coded macroblock in the previous frame

Or the vertical motion vector V _ypn of the current coded macroblock is greater than the average value of the vertical motion vector of the coded macroblock in the previous frame

Then the macroblock belongs to the time-domain visual feature saliency area, mark T _Yp (x,y,V _xpn ,V _ypn )=1, otherwise mark T _Yp (x,y,V _xpn ,V _ypn )=0, traverse the current all coded macroblocks in the p frame;

Wherein, Y represents the luminance component of the coded macroblock, (x, y) represents the position coordinates of the coded macroblock, and the definition of p is the same as that in Step 3.

Motion perception is one of the most important visual processing mechanisms in the human visual system. After experiments, it is found that the coded content with a large motion vector is just the moving area of interest to the human eye (such as the head, arm, person, etc.); Low static background area. Figure 4, taking the Akiyo sequence as an example, shows the motion vector distribution diagram of the 50th frame of the Akiyo sequence. From the figure, it can be found that in the face and head and shoulder areas where the human eye pays more attention, the coded macroblock usually has a larger Motion vector.

和

表示前一帧中所有编码宏块在水平方向上的平均运动矢量，

表示前一帧中所有编码宏块在垂直方向上的平均运动矢量。本发明中动态阈值的设定，充分考虑了视频序列的时间相关性，使阈值能够随前一帧编码宏块运动矢量平均值的变化而改变，有效减少了误判，能够快速、准确地获得时域视觉特征显著度区域。Whether the degree of motion of the currently coded macroblock is severe or not, the setting of the decision threshold has a great influence on the result. In order to reduce the misjudgment rate, the present invention records the thresholds for judging the degree of movement in the horizontal direction and the vertical direction as

and

Indicates the average motion vector in the horizontal direction of all coded macroblocks in the previous frame,

Indicates the average motion vector in the vertical direction of all coded macroblocks in the previous frame. The setting of the dynamic threshold in the present invention fully considers the time correlation of the video sequence, so that the threshold can be changed with the change of the average value of the motion vector of the coded macroblock in the previous frame, effectively reducing misjudgment, and can quickly and accurately obtain Temporal visual feature saliency regions.

Step 7: mark the video region of interest of the current p frame, specifically: traverse all the coded macroblocks in the current p frame, mark according to the spatial domain feature saliency and time domain visual feature saliency of each coded macro block, specifically mark The formula is as follows:

${\mathrm{<span\; class="notranslate">\; ROI</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{c}\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; mode</span>}}_{\mathrm{<span\; class="notranslate">\; pn</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; xpn</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; ypn</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}\\ \mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; mode</span>}}_{\mathrm{<span\; class="notranslate">\; pn</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; xpn</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; ypn</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}\\ \mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; mode</span>}}_{\mathrm{<span\; class="notranslate">\; pn</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; xpn</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; ypn</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; mode</span>}}_{\mathrm{<span\; class="notranslate">\; pn</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; xpn</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; ypn</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}\end{array}\right.$

Mark video regions of interest, divided into the following categories:

If the current coded macroblock has both spatial and temporal visual feature salience, that is, S _Yp (x,y,Mode _pn )=1 and T _Yp (x,y,V _xpn ,V _ypn )=1, it means that the current coded macroblock The block is not only rich in texture details, but also produces a large motion vector, the human eye is most interested, and the mark ROI _Yp (x,y)=3;

If it only has the saliency of temporal visual features, but does not have the saliency of spatial visual features, that is, T _Yp (x,y,V _xpn ,V _ypn )=1 and S _Yp (x,y,Mode _pn )=0, it means that the current The coded macroblock produces a large motion vector. According to the perceptual characteristics of HVS, the human eye is highly sensitive to the motion of the object, and the human eye is less interested. Mark ROI _Yp (x,y)=2;

If the motion degree of the macroblock is low, it does not have temporal visual feature saliency, but it has rich texture information, and only has spatial domain visual feature salience, that is, S _Yp (x,y,Mode _pn )=1 and T _Yp (x ,y,V _xpn ,V _ypn )=0, the degree of human interest is again, mark ROI _Yp (x,y)=1;

If there is neither spatial visual feature salience nor temporal visual feature saliency, that is, S _Yp (x,y,Mode _pn )=0 and T _Yp (x,y,V _xpn ,V _ypn )=0, it means The currently coded macroblock has flat texture, smooth motion, or stillness, usually a static background area, which is not an area of interest to the human eye, with the lowest degree of interest to the human eye, and the mark ROI _Yp (x,y)=0;

Among them, ROI _Yp (x, y) represents the visual interest priority of the current coded macroblock; T _Yp (x, y, V _xpn , V _ypn ) represents the temporal visual feature salience of the current coded macroblock; S _Yp (x , y, Mode _pn ) represents the spatial visual feature saliency of the current coded macro block; (x, y) represents the position coordinates of the current coded macro block; Y represents the brightness component of the macro block; p represents the pth inter-frame coding Video frame; n represents the sequence number of the nth coded macroblock in the current coded frame.

Step 8: output the video coded stream, specifically: according to the ROI _Yp (x, y) marked ROI Yp (x, y) interest priority level, do the following processing to the luminance component Y of all macroblocks in the current p frame, and output the marked video flow,

${\mathrm{<span\; class="notranslate">\; Y</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{c}\mathrm{<span\; class="notranslate">\; 255</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; ROI</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 3</span>}\\ \mathrm{<span\; class="notranslate">\; 150</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; ROI</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 2</span>}\\ \mathrm{<span\; class="notranslate">\; 100</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; ROI</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; ROI</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}\end{array}\right.$

Since the value range of the luminance component of the coded macroblock is Y∈[0,255], from 0 to 255 represents 256 levels of the luminance component of the macroblock from all black to all white. According to the interest priority level of the marked ROI _Yp (x, y), the present invention performs the following processing on the luminance component Y of the macroblock, and outputs the marked video stream.

If ROI _Yp (x, y)=3, the degree of interest is the highest, and the degree of human eye attention is the highest. The brightness component of the coded macroblock is set to 255, and the value of the brightness component of the output macroblock is the highest, that is, Y _p (x, y) =255;

If ROI _Yp (x, y)=2, the degree of interest is second, and the human eye is more concerned, and the luminance component of the coded macroblock is set to 150, and the value of the luminance component of the output macroblock is higher, that is, Y _p (x ,y)=150;

If ROI _Yp (x, y)=1, the degree of interest is again, and the attention of the human eye is low. The luminance component of the coded macroblock is set to 100, and the value of the luminance component of the output macroblock is low, that is, Y _p (x, y)=100;

If ROI _Yp (x, y)=0, it is a non-interest region, and the human eye has the lowest degree of attention. The brightness component of the coded macroblock is set to 0, and the output macroblock brightness component value is the lowest, that is, Y _p (x, y) =0.

Step 9: Return to Step 3 to process the next frame until the entire video sequence is traversed.

Fig. 5 shows the flow chart of the video ROI identification and extraction method.

Figure 6 shows the output results of the labeled video ROIs for a typical video sequence. Beneficial effect

This method realizes the rapid extraction of the region of interest in the video according to the basic coding information. This method uses the correlation between the basic coding information and the human visual interest area to identify the spatial visual feature salience area and the temporal feature visual saliency area in the video coding content respectively, and then combines the spatial and temporal visual feature salience The identification results of the degree area define the priority of video interest, and finally realize the automatic extraction of video interest. The method of the invention can provide an important coding basis for the video coding technology based on the ROI (Region of Interest, ROI).

Description of drawings

Figure 1. Schematic diagram of the H.264 standard inter prediction mode selection process;

Figure 2. Distribution of inter-frame prediction modes in the 50th frame of the Claire sequence;

Figure 3. A schematic diagram of the position and sequence number of each coded macroblock in a video frame;

Figure 4. Motion vector distribution diagram of the 50th frame of the Akiyo sequence;

Fig. 5. flow chart of the method of the present invention;

Fig. 6. Schematic diagram of the output result of marking a region of interest in a video using the method of the present invention.

Detailed ways

In view of the fact that human eyes are more sensitive to luminance information than chrominance information, the method of the present invention encodes the luminance component of the video frame. First read in the video sequence, extract its luminance component, and call the video region of interest extraction module of the present invention to complete the automatic identification and extraction of the region of interest.

In the implementation of the present invention, a video capture device (such as a digital video camera, etc.) is used to collect video images, and the images are transmitted to a computer, and the automatic identification of the video region of interest is realized in the computer according to the coding information in the video code stream. According to the predictive coding mode of the current coded macroblock, the spatial domain visual feature salience area is identified; then according to the motion vector of the current coded macro block in the horizontal or vertical direction, the time domain visual feature salience area is identified, and the dynamic motion vector judgment threshold is set Reduce the impact of different video motion types on the accuracy of ROI extraction; finally, the classification results of video interest are obtained according to the salience of spatial/temporal visual features, and the automatic extraction of ROI in videos is realized.

In the specific implementation, the following procedures are completed in the computer:

Step 1: Read in the video sequence according to the encoding configuration file encoder.cfg, and configure the encoder according to the parameters in the configuration file. For example: complete the video code stream structure GOP=IPPP...; number of encoded frames FramesToBeEncoded=100; frame rate FrameRate=30f/s; video file width SourceWidth=176, height SourceHeight=144; output file name OutputFile=ROI.264; quantization step Long value QPISlice=28, QPPSlice=28; motion estimation search range SearchRange=±16; reference frame number NumberReferenceFrames=5; activation rate distortion cost function RDOptimization=on; set entropy coding type SymbolMode=CAVLC and other parameter configurations, initialization parameters L= Number of encoded frames, p=1;

The second step: read the coded macroblock luminance component value Y frame by frame from the input video sequence in sequence;

Step 3: carry out intra-frame predictive encoding to the first frame of the video sequence, i.e. the I frame;

Step 4: Perform inter-frame predictive coding on the current p frame; record the inter-frame prediction mode type Mode _pn of the current coded macroblock; where, p=1,2,3,...,L-1, representing the p-th frame L is the total number of coded frames of the entire video sequence; n represents the sequence number of the nth coded macroblock in the current coded frame.

Step 5: Identify the spatial visual feature salience area. If the inter prediction mode Mode _pn of the current coded macroblock belongs to the sub-segmentation mode set or the intra prediction mode set, Mode _pn ∈ {8×8,8×4,4× 8,4×4}or{Intra16×16,Intra4×4}, then mark the macroblock as S _Yp (x,y,Mode _pn )=1, which belongs to the spatial visual feature saliency area, otherwise mark S _Yp ( x,y,Mode _pn )=0;

$\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; mode</span>}}_{\mathrm{<span\; class="notranslate">\; pn</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>& {\mathrm{<span\; class="notranslate">\; mode</span>}}_{\mathrm{<span\; class="notranslate">\; pn</span>}}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; 8</span>}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; 8,8</span>}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; 4,4</span>}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; 8,4</span>}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; \}</span>\mathrm{<span\; class="notranslate">\; or</span>}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; Intra</span>}\mathrm{<span\; class="notranslate">\; 16</span>}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; 16</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; Intra</span>}\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; \}</span>\\ \mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>& \mathrm{<span\; class="notranslate">\; else</span>}\end{array}\right.$

以及垂直方向上的平均运动矢量

否则，跳转至第十步；Step 6: If p≠1, record the motion vector V _xpn in the horizontal direction and the motion vector V _ypn in the vertical direction of each coded macroblock in the pth frame; and calculate all coded macroblocks in the previous coded frame Average motion vector in the horizontal direction

and the average motion vector in the vertical direction

Otherwise, skip to the tenth step;

或者当前编码宏块的垂直方向运动矢量V_ypn大于前一帧编码宏块在垂直方向运动矢量平均值

满足其中任何一个判别条件，则该宏块属于时域视觉特征显著度区域，标记T_Yp(x,y,V_xpn,V_ypn)=1，否则标记T_Yp(x,y,V_xpn,V_ypn)=0；Step 7: Identify the time-domain visual feature salience area, if the horizontal motion vector V _xpn of the current coded macroblock is greater than the average value of the horizontal motion vector of the coded macroblock in the previous frame

Or the vertical motion vector V _ypn of the current coded macroblock is greater than the average value of the vertical motion vector of the coded macroblock in the previous frame

If any one of the discriminant conditions is met, the macroblock belongs to the time-domain visual feature saliency area, marked T _Yp (x,y,V _xpn ,V _ypn )=1, otherwise marked T _Yp (x,y,V _xpn ,V _ypn )=0;

${\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; xpn</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; ypn</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>& {\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; xpn</span>}}<span\; class="notranslate">\; ></span>{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; V</span>}}}_{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; the\; th</span>}}\mathrm{<span\; class="notranslate">\; or</span>}{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; ypn</span>}}<span\; class="notranslate">\; ></span>{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; V</span>}}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; the\; <figure-callout\; id="0"\; label="th"\; filenames="DEST\_PATH\_EDA0000419026730000011.png"\; state="\{\{state\}\}">th</figure-callout></span>}}\\ \mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>& \mathrm{<span\; class="notranslate">\; else</span>}\end{array}\right.$

Step 8: Mark the region of interest in the video.

${\mathrm{<span\; class="notranslate">\; ROI</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{c}\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; mode</span>}}_{\mathrm{<span\; class="notranslate">\; pn</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; xpn</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; ypn</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}\\ \mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; mode</span>}}_{\mathrm{<span\; class="notranslate">\; pn</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; xpn</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; ypn</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}\\ \mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; mode</span>}}_{\mathrm{<span\; class="notranslate">\; pn</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; xpn</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; ypn</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; mode</span>}}_{\mathrm{<span\; class="notranslate">\; pn</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; xpn</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; ypn</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}\end{array}\right.$

If the current coded macroblock has both spatial and temporal visual feature salience, that is, S _Yp (x,y,Mode _pn )=1 and T _Yp (x,y,V _xpn ,V _ypn )=1, the human eye perception The highest degree of interest, marked ROI _Yp (x,y)=3;

If it only has the salience of time-domain visual features, that is, T _Yp (x,y,V _xpn ,V _ypn )=1 and S _Yp (x,y,Mode _pn )=0, the degree of interest of human eyes is next, mark ROI _Yp (x,y)=2;

If it only has the salience of spatial visual features, that is, S _Yp (x,y,Mode _pn )=1 and T _Yp (x,y,V _xpn ,V _ypn )=0, the degree of human interest is again, and the ROI _Yp ( x,y)=1;

If there is neither spatial visual feature salience nor temporal visual feature saliency, that is, S _Yp (x,y,Mode _pn )=0 and T _Yp (x,y,V _xpn ,V _ypn )=0, then For the non-interest area of the human eye, mark the ROI _Yp (x,y)=0;

Step 9: Output video encoding stream.

${\mathrm{<span\; class="notranslate">\; Y</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{c}\mathrm{<span\; class="notranslate">\; 255</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; ROI</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 3</span>}\\ \mathrm{<span\; class="notranslate">\; 150</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; ROI</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 2</span>}\\ \mathrm{<span\; class="notranslate">\; 100</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; ROI</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; ROI</span>}}_{\mathrm{<span\; class="notranslate">\; Yp</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}\end{array}\right.$

If ROI _Yp (x, y)=3, the degree of interest is the highest, and the degree of human eye attention is the highest. The brightness component of the coded macroblock is set to 255, and the value of the brightness component of the output macroblock is the highest, that is, Y _p (x, y) =255;

If ROI _Yp (x, y)=2, the degree of interest is second, and the human eye is more concerned, and the luminance component of the coded macroblock is set to 150, and the value of the luminance component of the output macroblock is higher, that is, Y _p (x ,y)=150;

If ROI _Yp (x, y)=1, the degree of interest is again, and the attention of the human eye is low. The luminance component of the coded macroblock is set to 100, and the value of the luminance component of the output macroblock is low, that is, Y _p (x, y)=100;

If ROI _Yp (x, y)=0, it is a non-interest region, and the human eye has the lowest degree of attention. The brightness component of the coded macroblock is set to 0, and the output macroblock brightness component value is the lowest, that is, Y _p (x, y) =0.

Step 10: If p≠L-1, p=p+1, go to step 3; otherwise, end coding.

A schematic diagram of the output result of marking a region of interest in a video using the method of the present invention is shown in FIG. 6 . Taking a typical video surveillance sequence (Hall) and an indoor activity video sequence (Salesman) as examples, use the motion vector distribution results and the inter-frame prediction mode selection results to mark the video interest area. If it is high, the brightness value at this position in the output video is higher, otherwise the brightness value is lower. From the marking results in the rightmost column in Figure 6, it can be found that the shape of the video region of interest obtained by the method of the present invention is irregular, compared with the region of interest obtained by the traditional moving object detection method using a fixed shape template , the marking result of the method of the present invention is closer to the shape of the object of interest that human eyes focus on, and can mark the region of interest more accurately.

The method of the present invention can also be combined with other fast coding techniques, under the premise of ensuring the coding quality of the region of interest to the human eye, it can reduce the coding complexity of the background region that is not of interest to the human eye, and further reduce the coding time. It can also be used in H.264-based In the scalable coding of , the selective enhancement coding of the region of interest is realized.

Claims (1)

1. The method for extracting the video region of interest based on coding information is characterized in that comprising the following steps: Step 1: Input a video sequence in YUV format and GOP (Group of Picture, GOP) structure as IPPP, read the luminance component Y of the encoded macroblock, and configure the encoding parameters; Step 2: performing intra-frame predictive coding on the first frame of the video sequence, i.e. the I frame; Step 3: Perform interframe predictive coding on the current p frame, and record the interframe prediction mode types of all coded macroblocks in the current p frame, denoted as Mode _pn ; p=1,2,3,...,L-1, representing The pth video frame for inter-frame encoding, L is the total number of frames encoded in the entire video sequence; n represents the sequence number of the nth encoded macroblock in the current encoded frame; Step 4: Identify the spatial visual feature saliency area of the current p frame, specifically: if the inter-frame prediction mode Mode _pn of the current coded macroblock belongs to the sub-segmentation mode set or the intra-frame prediction mode set, that is, Mode _pn ∈{8×8 ,8×4,4×8,4×4}or{Intra16×16,Intra4×4}, then mark the macroblock as S _Yp (x,y,Mode _pn )=1, which belongs to the saliency of spatial visual features area, otherwise mark S _Yp (x, y, Mode _pn )=0; Y represents the luminance component of the coded macroblock, (x, y) represents the position coordinates of the coded macroblock, and traverses all coded macroblocks in the current p frame ; Step 5: Record the motion vector V _xpn in the horizontal direction and the motion vector V _ypn in the vertical direction of each coded macroblock in the pth frame; and calculate the average of all coded macroblocks in the horizontal direction in the previous coded frame motion vector

and the average motion vector in the vertical direction

Num represents the number of macroblocks contained in a coded frame, that is, the number of times of accumulation; Step 6: identify the temporal visual feature salience region of the current p frame, specifically: if the horizontal direction motion vector V _xpn of the current coded macroblock is greater than the previous The average value of the motion vector in the horizontal direction of a coded macroblock in one frame

Or the vertical motion vector V _ypn of the current coded macroblock is greater than the average value of the vertical motion vector of the coded macroblock in the previous frame

Then the macroblock belongs to the time-domain visual feature saliency area, mark T _Yp (x,y,V _xpn ,V _ypn )=1, otherwise mark T _Yp (x,y,V _xpn ,V _ypn )=0, traverse the current All coded macroblocks in the p frame; Step 7: mark the video region of interest of the current p frame, specifically: traverse all the coded macroblocks in the current p frame, according to the spatial domain feature saliency and time domain of each coded macroblock The visual feature salience is marked, and the specific marking formula is as follows: ${\mathrm{ROI}}_{\mathrm{Yp}}(x,\mathrm{the\; y})=\left\{\begin{array}{c}3,S_{\mathrm{Yp}}(x,\mathrm{the\; y},{\mathrm{mode}}_{\mathrm{pn}})=1\left|\right|T_{\mathrm{Yp}}(x,\mathrm{the\; y},V_{\mathrm{xpn}},V_{\mathrm{ypn}})=1\\ 2,S_{\mathrm{Yp}}(x,\mathrm{the\; y},{\mathrm{mode}}_{\mathrm{pn}})=0\left|\right|T_{\mathrm{Yp}}(x,\mathrm{the\; y},V_{\mathrm{xpn}},V_{\mathrm{ypn}})=1\\ 1,S_{\mathrm{Yp}}(x,\mathrm{the\; y},{\mathrm{mode}}_{\mathrm{pn}})=1\left|\right|T_{\mathrm{Yp}}(x,\mathrm{the\; y},V_{\mathrm{xpn}},V_{\mathrm{ypn}})=0\\ 0,S_{\mathrm{Yp}}(x,\mathrm{the\; y},{\mathrm{mode}}_{\mathrm{pn}})=0\left|\right|T_{\mathrm{Yp}}(x,\mathrm{the\; y},V_{\mathrm{xpn}},V_{\mathrm{ypn}})=0\end{array}\right.$ If the current coded macroblock has both spatial and temporal visual feature salience, that is, S _Yp (x,y,Mode _pn )=1 and T _Yp (x,y,V _xpn ,V _ypn )=1, mark ROI _Yp (x,y)=3; if the current coded macroblock only has temporal visual feature saliency, but does not have spatial domain visual feature saliency, that is, T _Yp (x,y,V _xpn ,V _ypn )=1 and S _Yp ( x,y,Mode _pn )=0, mark ROI _Yp (x,y)=2; if the current coded macroblock does not have temporal visual feature salience, it only has spatial visual feature saliency, that is, S _Yp (x, y,Mode _pn )=1 and T _Yp (x,y,V _xpn ,V _ypn )=0, then mark ROI _Yp (x,y)=1; if the current coded macroblock has neither spatial visual feature salience nor Does not have temporal visual feature salience, that is, S _Yp (x,y,Mode _pn )=0 and T _Yp (x,y,V _xpn ,V _ypn )=0, then mark ROI _Yp (x,y)=0 ; Step 8: output video coded code stream, specifically: according to the ROI _Yp (x, y) of the mark ROI Yp (x, y) the level of interest priority is high or low, the luminance component Y of all macroblocks in the current p frame is processed as follows, and output the mark after video stream, $Y_p(x,\mathrm{the\; y})=\left\{\begin{array}{c}255,{\mathrm{ROI}}_{\mathrm{Yp}}(x,\mathrm{the\; y})=3\\ 150,{\mathrm{ROI}}_{\mathrm{Yp}}(x,\mathrm{the\; y})=2\\ 100,{\mathrm{ROI}}_{\mathrm{Yp}}(x,\mathrm{the\; y})=1\\ 0,{\mathrm{ROI}}_{\mathrm{Yp}}(x,\mathrm{the\; y})=0\end{array}\right.$ Step 9: Return to Step 3 to process the next frame until the entire video sequence is traversed.

Images