CN102208023B - Method for recognizing and designing video captions based on edge information and distribution entropy - Google Patents

Abstract

The invention discloses a video subtitle recognition method based on edge information and distribution entropy. It uses the edge detection method of corner point enhancement to get the edge information of the image, then connects the edge points and collects the connected domain, uses the segmentation algorithm to properly segment the connected domain, and then uses the refinement operation to get their accurate position, uses the trailing filter and The joint entropy filter filters out non-text regions, leaving only text regions. For the detected text domain, after unifying into black and white characters, use local threshold binarization, edge noise point expansion removal operation based on forbidden extension point constraint, and noise removal operation based on surrounding edge point counting to obtain binary value The image is sent to the OCR software for identification. This method can overcome the shortcomings of general methods that are sensitive to language, subtitle arrangement, and background complexity. By introducing segmentation algorithms and joint entropy filters, good detection results can be obtained, and the traditional binarization method is greatly improved. Improved recognition accuracy.

Description

Design method of video subtitle recognition based on edge information and distribution entropy

technical field

The invention relates to a video subtitle recognition method based on edge information and distribution entropy. The method is used to detect and extract subtitles in a video for OCR recognition, and belongs to the field of computer image processing.

Background technique

With the development of multimedia and electronic industries, more and more video information is produced. How to effectively organize and retrieve them becomes a difficult problem. Many video materials, such as TV news, sports games, movies, variety shows, etc., have subtitle information added in post-production, and these subtitle information is closely related to video content. If these subtitles can be effectively identified, they can be used to organize and retrieve video materials, which has strong practical value.

Video subtitle recognition is divided into four steps: subtitle detection, subtitle positioning, subtitle extraction and OCR recognition. Subtitle detection is used to determine the subtitle area; subtitle positioning is used to locate the precise position of each line of subtitles; subtitle extraction is used to binarize the subtitle area, and only stroke pixels are reserved; the last step is generally implemented by commercial OCR software. Subtitle detection can be divided into four methods: edge-based methods, connected domain-based methods, color clustering-based methods, and texture-based methods. Edge-based methods use edge filters to detect text edges and then use morphological operations to merge them. The method published in the 8th Conference on Document Analysis and Recognition (In Proceedings of 8 ^rd International Conference on Document Analysis and Recognition (ICDAR), 2005, 610-614) uses the edge detection method to obtain four edge maps, and then uses K-MEANS Algorithms detect candidate text regions, and finally use heuristic rules and projection analysis to identify and refine text regions. Edge-based methods work well if there is no complex background, but they do not work well when the background contains a lot of side information. Texture-based methods use Gabor filters, wavelet transforms, fast Fourier transforms, etc. to extract texture features, and then use machine learning methods such as neural networks and SVM classifiers to detect subtitle regions. A method published in Proceeding of IEEE International Conference on Communication Technology (ICCT), 2008, 722-725) uses HARR wavelet transform to locate by combining 4 small blocks of wavelet coefficients into a large block. large font text, and then use morphological dilation operations and neural meshes to enhance the effect. Connected domain-based methods segment a frame into multiple small connected domains, and then merge them into a larger connected domain for subtitle localization. A method published in Proceedings of the ACM International Multimedia Conference and Exhibition 2007 (MM), 847-850 uses credit-based clustering of colors to remove noise. Text contrast differences are adaptively selected relative to the best color panel to perform binarization. The method based on color clustering assumes that the text color in the video frame is uniform, but this assumption is not true in most cases, so its application is limited. Since the effect of subtitle detection using one feature is not ideal, many methods use the above features in combination. For subtitle positioning, the method of grayscale projection is generally used. Subtitle extraction methods can be divided into color-based methods and stroke-based methods. Many color-based methods use the Otsu method to binarize the grayscale image, but when the grayscale levels of the subtitle and the background are very similar, this method cannot distinguish them well, so it cannot denoise well. "IEEE Transactions on Circuits and Systems for Video Technology 2005, 15 (2): 243-255" 2005 No. 15 and "IEEE Transactions on Circuits and Systems for Video Technology" A method published in Issue 18 of 2009 (IEEE Transactions on Image Processing 2009, 18(2):401-411) uses a local adaptive threshold with better resolution, combined with dam point marking and inward padding , so that most of the noise points can be removed.

The above subtitle detection methods have made some useful attempts to detect video subtitles, but these methods are not very effective in distinguishing between subtitles and backgrounds. Only these methods are used to detect some videos with changing languages, fonts and text alignments. Handling is poor. In addition, although the existing subtitle extraction methods can remove most of the noise, because OCR software is very sensitive to noise points, the effect of text recognition in complex backgrounds is not good.

Contents of the invention

The purpose of the present invention is to overcome the deficiencies of the prior art and provide a video subtitle recognition method based on edge information and distribution entropy.

The steps of the video subtitle recognition method based on edge information and distribution entropy are as follows:

1) Detect the difference between the current frame and the previous processed frame, if the difference is large, perform the following subtitle recognition operation, otherwise continue to take the next frame for judgment;

2) Subtitle recognition First, perform subtitle detection. In subtitle detection, use edge detection, edge point connection, connected domain collection and segmentation methods, connected domain refinement and trailing filtering methods to obtain candidate text domains and their positions, and then use joint entropy filtering The filter removes the non-text field, leaving only the subtitle area;

3) Check the repeatability of the subtitle area. If the area is not repeated, the color is very uniform as black and white characters, and then the subtitle is extracted, otherwise the next subtitle area is processed;

4) In the subtitle extraction, binarize the subtitle area after the color is extremely uniform, remove the noise point and send it to the OCR software for recognition.

The difference between the described detection current frame and the previous processed frame, if the difference is large, then carry out the following subtitle recognition operation, otherwise continue to take the next frame and proceed to the judgment step as follows: set this frame as I _i , and its edge binary image is E _i , its previous processed frame is I _i-5 , its edge binary image is E _i-5 , let D _i,i-5 =E _i ⊕E _i-5 , let last time The detected subtitle area is Area _i-5,j , and the last time the minimum value of the cumulative sum of the edge binary images of each subtitle area is pMES , then the cumulative difference of the subtitle area in the current frame is calculated as follows:

                                                   1

1

If cFD is less than or equal to pMES × 0.5, then there is no need to perform subtitle recognition operation on this frame, and continue to take the fifth frame to judge, otherwise, it is necessary to perform subtitle recognition operation on this frame. In order to further prevent missing subtitles, another count value is set ck, every time cFD is less than or equal to pMES × 0.5, the value of ck is increased by 1, otherwise, ck is reset to 0, if ck is equal to 5, no matter what the previous judgment is, the subtitle recognition operation of this frame is required, and ck is reassigned to 0 .

Described subtitle recognition first carries out subtitle detection, uses edge detection, edge point connection, connected domain collection and segmentation method, connected domain refinement and trailing filter method to obtain candidate text domain and its position in subtitle detection, then uses joint entropy The filter removes non-text fields, leaving only subtitles. The steps are:

(1) Edge detection method

Given an image I, the Sobel operator is used to detect edges. The Sobel operator is composed of gradient templates in four directions: horizontal S _H , vertical S _V , diagonal S _LD , and anti-diagonal S _RD . The edge field is given by the following formula calculate:

                      2

2

表示在像素(x,y)处与最大梯度绝对值方向垂直的方向，k为一个调节系数，本文中其取值为1，S然后量化成16级，量化后表示为S’，然后用下式得到边缘映射图EdgeMap： in

Indicates the direction perpendicular to the direction of the maximum gradient absolute value at the pixel (x, y), k is an adjustment coefficient, its value is 1 in this paper, S is then quantized into 16 levels, and expressed as S' after quantization, and then used as follows The formula to get the edge map EdgeMap :

                                  3

3

(2) Edge point connection method

For the edge map EdgeMap , if the distance between two edge points in the same row is less than a certain threshold T _d , set the pixel values between these two pixels in the EdgeMap to 1, that is, fill the space between the two edge points pixel, T _d is determined by:

                                    4

4

Wherein height and width are the height and width of image I respectively;

(3) Connected domain collection and segmentation method

Collect the connected domains of the EdgeMap obtained in the previous step, remove those connected domains whose height or width is less than 1% of the entire image height or width, and remove those connected domains whose smallest enclosing rectangle is less than 0.2% of the entire image area, and then use the following The step is to segment each connected domain C:

，得到该行及以上部分的最小包围矩形的面积

和该行以下部分的最小包围矩形的面积，求出这两个面积的和，找出取得最小和的行号存储在bR中； a) for each row i in C

, to get the area of the minimum enclosing rectangle of the row and above

and the area of the smallest enclosing rectangle below the row , calculate the sum of these two areas, find out the row number that obtains the minimum sum and store it in bR;

，得到该列及左边部分的最小包围矩形的面积

和该列右边部分的最小包围矩形的面积

，求出这两个面积的和，找出取得最小和的列号存储在bC中； b) For each column j in C

, get the area of the minimum enclosing rectangle of the column and the left part

and the area of the smallest enclosing rectangle for the right part of the column

, calculate the sum of these two areas, find out the column number that obtains the minimum sum and store it in bC;

，

，若mRA<mCA，则将连通域C在行上以第bR行为界分成两个连通域，否则将连通域C在列上以第bC列为界分成两个连通域； c) orders

,

, if mRA<mCA, the connected domain C is divided into two connected domains on the row with the bRth line boundary, otherwise the connected domain C is divided into two connected domains on the column with the bCth column as the boundary;

Among them, t _c , b _c l _c and r _c are the upper boundary row number, lower boundary row number, left boundary column number and right boundary column number of area C respectively;

连通域填充率小于0.8；

分成的两个新连通域面积都大于整幅图像面积的0.2%； In order to prevent over-segmentation, only when the connected region C satisfies the following two conditions at the same time will it be segmented:

Connected domain filling rate is less than 0.8;

The areas of the two new connected domains are both larger than 0.2% of the entire image area;

(4) Connected domain refinement and trailing filtering method

Before performing area refinement, first remove those connected domains whose height is greater than twice the width, which may accidentally delete those vertical subtitles. In order to process vertical subtitles, you only need to rotate the image by 90 degrees, and the other operations are exactly the same;

For each connected region C obtained in the previous step, the steps to further refine its position are as follows:

Input: edge map edgeMap , the initial upper and lower boundary positions of the connected domain C

Output: refined upper and lower bound positions

，计算其在edgeMap中的左右非0像素跨距

，并存储在集合cSA中； d) For any row in the connected domain C

, calculate its left and right non-zero pixel spans in edgeMap

, and stored in the set cSA;

，计算其在edgeMap中的行像素点数，并存储在集合

中，即有

； e) For any row in the connected domain C

, calculate the number of row pixels in the edgeMap , and store it in the collection

in, that is

;

f) Take the maximum value in cSA to exist , and store its serial number in pSRN ;

中，并将其序号存在

中； Pick The maximum value in

, and store its sequence number in

middle;

范围内的所有行，取

的最大行序号

； g) for the

For all rows in the range, take

The maximum row number of

;

范围内的所有行，取

的最小行序号

； for in

For all rows in the range, take

The minimum row number of

;

范围内的所有行，取

的最大行序号

； for in

For all rows in the range, take

The maximum row number of

;

范围内的所有行，取

的最小行序号

； for in

For all rows in the range, take

The minimum row number of

;

，即得到精化后的上下界位置； h) orders

, that is, the upper and lower bound positions after refinement are obtained ;

 和 

通常取值为0.6和0.3； in

and

Usually the values are 0.6 and 0.3;

Use the following trailing filtering method to remove some non-subtitle connected domains:

，假设得到的行号分别为t _tail 和 b _tail ； i) After the above step g) is completed, continue to scan up and down in oPNA until the value at the current row is less than

, assuming that the obtained line numbers are t _tail and b _tail respectively;

j) Calculate the length of the tail using the following formula:

tl ₁ = t ₂ - t _tail , tl ₂ = b _tail - b ₂ , tl =max ( tl ₁ , tl ₂ )

k) Use the following formula to filter, if deleteFlag(C) is 1, it means that this connected domain is not a subtitle area and should be deleted;

                                             5  

5

where ub _c and ut _c denote the refined upper and lower bounds of connected domain C respectively, and and usually take values of 0.2 and 0.3;

(5) Joint entropy filter

Use the joint entropy filter of the joint foreground pixel distribution entropy and edge pixel distribution entropy to filter, leaving only the subtitle area;

For the foreground pixel distribution entropy, it is the smallest enclosing rectangle Rect [t _c , b _c , l _c , r _c ] for a connected domain C, where t _c , b _c are the upper and lower bounds respectively, and l _c , r _c are respectively For the left and right boundaries, use the Otsu threshold to binarize it, then divide it into 2 rows × 4 columns = 8 parts, and use the following formula to calculate the distribution entropy:

            6

6

Among them, p _i,j represents the ratio of non-zero pixels in the i-th row and j-th column;

For the edge pixel distribution entropy, the Sobel edge binary image in the smallest enclosing rectangle Rect [t _c , b _c , l _c , r _c ] of the connected domain C is divided into 2 rows × 4 columns = 8 parts, and calculated using the following formula Distribution entropy:

 7

7

Where e _ij represents the number of edge pixels in row i, column j, and e _r is the sum of the number of edge pixels in 8 parts,

且

，则认为其是字幕区域，否则就是非字幕区域，应该删除，实验得

和

分别取6.4和2.76时效果最好； For any refined connected domain C, if its

and

, it is considered to be a subtitle area, otherwise it is a non-subtitle area and should be deleted. The experiment shows that

and

The effect is best when taking 6.4 and 2.76 respectively;

For some images with both horizontal and vertical subtitles, subtitle detection is performed in the original image and the image obtained by rotating 90 degrees, and then the detection results of the two are combined to eliminate duplication.

Described repeatability detection is carried out to subtitle area, if this area does not repeat, then its color is extremely unified as black and white characters, then carry out subtitle extraction, otherwise process next subtitle area step as:

(6) Repeatability detection

Use the method of combining position and grayscale color histogram to deduplicate the detected subtitle area, the steps are as follows:

为第i个字幕区域灰度级为k的像素数目；提取并存储当前帧所有字幕区域位置Rect_j[t_j,b_j,l_j,r_j] 及灰度直方图GH_j{g_j,0,g_j,1,…g_j,255}； l) Extract and store all subtitle area positions Rect _i [t _i ,b _i ,l _i , _ri ] and grayscale histogram GH _i {g _i,0 ,g _i,1 ,…g _i of the previous processed frame _,255 }, where

is the number of pixels whose gray level is k in the i-th subtitle area; extract and store all subtitle area positions Rect _j [t _j ,b _j ,l _j ,r _j ] and gray histogram GH _j {g _{j, 0} , g _{j, 1} , ... g _{j, 255} };

和灰度直方图相似度

，其中

是它们的公共部分的面积，而

是它们中大的那个的面积，若与

有一个大于0.8，则是相同区域的重复检测，此时去掉一个，保留一个； m) Calculate their positional similarity

Similarity to grayscale histogram

,in

is the area of their common part, and

is the area of the larger of them, if and

If one is greater than 0.8, it means repeated detection in the same area. At this time, one is removed and one is kept;

(7) Extremely uniform color

To unify the grayscale image of the subtitle area into black and white characters, take the following steps:

和

对二值化后的字幕区域进行卷积操作，用下式确定每个像素处的边缘颜色： n) First binarize the grayscaled subtitle area with the Otsu method, and then use a 3×3 mask

and

Perform convolution operation on the binarized subtitle area, and use the following formula to determine the edge color at each pixel:

                                8

8

为它们的比； Let N _w and N _b denote the number of white edge pixels and black edge pixels respectively, define

for their ratio;

； o) For the edge map P obtained in the previous step 8, the edge pixels are projected by column, and the edge map is decomposed into {x ₀ , x ₁ ,…,x _n } on the column, where x _i is the edge map in The midpoint of a continuous interval whose projection is 0 on the column, establish a rectangle Rect _i [1, height, x _i , x _i+1 ] in sequence, scan from four sides inward in the edge map P within the range of the rectangle, Delete the first edge pixel encountered, re-count the number of white edge pixels and the number of black edge pixels, respectively set to and

;

为它们的比，定义 p) definition

For their ratio, define

                                           9

9

Use the following method to judge the color pole of the subtitle area:

(a) if , it is white;

，则当

时，字幕为白色，当

时字幕为黑色； (b) if

, then when

When , the subtitle is white, when

When the subtitle is black;

，则为白字； (c) if

, it is white;

，则当

时，字幕为白色，当

时字幕为黑色； (d) if

, then when

When , the subtitle is white, when

When the subtitle is black;

，则为黑字； (e) if

, it is in black;

； in

;

q) After judging that the color of the subtitle is extremely high, if it is black, then invert the grayscale image of the subtitle area, otherwise no operation is performed.

In the subtitle extraction, the subtitle area after the extremely unified color is binarized, and the OCR software recognition steps are sent after removing the noise points as follows:

r) Normalize the height of the grayscaled subtitle area to 24, and then expand 4 pixels up and down respectively, so that the height after expansion is 32, which is set as EI;

s) Initialize each pixel of the resulting binary image B to 1, and then perform step-by-step horizontal local threshold binarization on the EI, and use the Otsu method to binarize in a 16×32 local window, each level Step by 8 pixels, the same method is used to perform step-by-step vertical local threshold binarization on EI, and use the Otsu method to binarize in a local window of image_width×8, each vertical step by 4 pixels, in In each sub-window, if the gray value in EI is lower than the local threshold, the corresponding pixel value in B is set to 0; Pixels are also set to 0, defining dam points:

Where H_len(x,y) represents the length of the longest horizontally continuous 1 sequence where the pixel (x,y) is located, and V_len(x,y) represents the length of the longest vertically continuous 1 sequence where the pixel (x,y) is located, For dam points, it cannot be expanded into background pixels;

u) Use the Sobel operator to get the edge information of EI. For each connected domain with a value of 1 in B, count the number epn of the edge pixels falling in it or surrounding it. If epn<tepn , the connected domain Set all the pixels in the domain to 0, so as to remove the connected domain, tepn is determined by the following formula:

tepn =max( cheight,cwidth )

Among them, cheight and cwidth are the height and width of the connected domain, respectively.

v) Send binary image B to OCR software for recognition.

The present invention has the beneficial effect compared with prior art:

1) The subtitle detection algorithm in the present invention can overcome the shortcomings of common detection algorithms that are sensitive to language, subtitle alignment and background complexity. Obtain detection results that are robust to changes in language, subtitle alignment, and background complexity;

2) The subtitle extraction algorithm in the present invention can further remove noise pixels on the basis of general extraction algorithms, so that the subsequent OCR recognition accuracy has been improved to a certain extent;

3) The present invention can solve the problem of too many repeated subtitles in video frames to a certain extent, and at the same time prevent some subtitles from being missed, and achieve better results in continuous video frame sequences.

Description of drawings

Fig. 1 is a frame diagram of video subtitle recognition;

Fig. 2 is a frame diagram of video subtitle detection;

Fig. 3 is the flowchart example figure that a certain frame image is carried out video subtitle detection;

Fig. 4 is the example figure that a certain subtitle area is carried out video subtitle extraction;

Detailed ways

In order to better understand the technical solution of the present invention, the present invention will be further described below in conjunction with accompanying drawings 1 and 2 . Accompanying drawing 1 has described the frame diagram of the video subtitle recognition method of the present invention, and accompanying drawing 2 has described the frame diagram of the video subtitle detection method in the present invention.

The steps of the video subtitle recognition method based on edge information and distribution entropy are as follows:

1) Detect the difference between the current frame and the previous processed frame, if the difference is large, perform the following subtitle recognition operation, otherwise continue to take the next frame for judgment;

2) Subtitle recognition First, perform subtitle detection. In subtitle detection, use edge detection, edge point connection, connected domain collection and segmentation methods, connected domain refinement and trailing filtering methods to obtain candidate text domains and their positions, and then use joint entropy filtering The filter removes the non-text field, leaving only the subtitle area;

3) Check the repeatability of the subtitle area. If the area is not repeated, the color is very uniform as black and white characters, and then the subtitle is extracted, otherwise the next subtitle area is processed;

4) In the subtitle extraction, binarize the subtitle area after the color is extremely uniform, remove the noise point and send it to the OCR software for recognition.

The difference between the described detection current frame and the previous processed frame, if the difference is large, then carry out the following subtitle recognition operation, otherwise continue to take the next frame and proceed to the judgment step as follows: set this frame as I _i , and its edge binary image is E _i , its previous processed frame is I _i-5 , its edge binary image is E _i-5 , let D _i,i-5 =E _i ⊕E _i-5 , let last time The detected subtitle area is Area _i-5,j , and the last time the minimum value of the cumulative sum of the edge binary images of each subtitle area is pMES , then the cumulative difference of the subtitle area in the current frame is calculated as follows:

                                                   1

1

If cFD is less than or equal to pMES × 0.5, then there is no need to perform subtitle recognition operation on this frame, and continue to take the fifth frame to judge, otherwise, it is necessary to perform subtitle recognition operation on this frame. In order to further prevent missing subtitles, another count value is set ck, every time cFD is less than or equal to pMES × 0.5, the value of ck is increased by 1, otherwise, ck is reset to 0, if ck is equal to 5, no matter what the previous judgment is, the subtitle recognition operation of this frame is required, and ck is reassigned to 0 .

Described subtitle recognition first carries out subtitle detection, uses edge detection, edge point connection, connected domain collection and segmentation method, connected domain refinement and trailing filter method to obtain candidate text domain and its position in subtitle detection, then uses joint entropy The filter removes non-text fields, leaving only subtitles. The steps are:

(1) Edge detection method

Given an image I, the Sobel operator is used to detect edges. The Sobel operator is composed of gradient templates in four directions: horizontal S _H , vertical S _V , diagonal S _LD , and anti-diagonal S _RD . The edge field is given by the following formula calculate:

                      2

2

表示在像素(x,y)处与最大梯度绝对值方向垂直的方向，k为一个调节系数，本文中其取值为1，S然后量化成16级，量化后表示为S’，然后用下式得到边缘映射图EdgeMap： in

Indicates the direction perpendicular to the direction of the maximum gradient absolute value at the pixel (x, y), k is an adjustment coefficient, its value is 1 in this paper, S is then quantized into 16 levels, and expressed as S' after quantization, and then used as follows The formula to get the edge map EdgeMap :

                                  3

3

(2) Edge point connection method

For the edge map EdgeMap , if the distance between two edge points in the same row is less than a certain threshold T _d , set the pixel values between these two pixels in the EdgeMap to 1, that is, fill the space between the two edge points pixel, T _d is determined by:

                                    4

4

Wherein height and width are the height and width of image I respectively;

(3) Connected domain collection and segmentation method

Collect the connected domains of the EdgeMap obtained in the previous step, remove those connected domains whose height or width is less than 1% of the entire image height or width, and remove those connected domains whose smallest enclosing rectangle is less than 0.2% of the entire image area, and then use the following The step is to segment each connected domain C:

，得到该行及以上部分的最小包围矩形的面积

和该行以下部分的最小包围矩形的面积

，求出这两个面积的和，找出取得最小和的行号存储在bR中； a) for each row i in C

, to get the area of the minimum enclosing rectangle of the row and above

and the area of the smallest enclosing rectangle below the row

, calculate the sum of these two areas, find out the row number that obtains the minimum sum and store it in bR;

，得到该列及左边部分的最小包围矩形的面积

和该列右边部分的最小包围矩形的面积

，求出这两个面积的和，找出取得最小和的列号存储在bC中； b) For each column j in C

, get the area of the minimum enclosing rectangle of the column and the left part

and the area of the smallest enclosing rectangle for the right part of the column

, calculate the sum of these two areas, find out the column number that obtains the minimum sum and store it in bC;

，

，若mRA<mCA，则将连通域C在行上以第bR行为界分成两个连通域，否则将连通域C在列上以第bC列为界分成两个连通域； c) orders

,

, if mRA<mCA, the connected domain C is divided into two connected domains on the row with the bRth line boundary, otherwise the connected domain C is divided into two connected domains on the column with the bCth column as the boundary;

Among them, t _c , b _c l _c and r _c are the upper boundary row number, lower boundary row number, left boundary column number and right boundary column number of area C respectively;

连通域填充率小于0.8；

分成的两个新连通域面积都大于整幅图像面积的0.2%； In order to prevent over-segmentation, only when the connected region C satisfies the following two conditions at the same time will it be segmented:

Connected domain filling rate is less than 0.8;

The areas of the two new connected domains are both larger than 0.2% of the entire image area;

(4) Connected domain refinement and trailing filtering method

Before performing area refinement, first remove those connected domains whose height is greater than twice the width, which may accidentally delete those vertical subtitles. In order to process vertical subtitles, you only need to rotate the image by 90 degrees, and the other operations are exactly the same;

For each connected region C obtained in the previous step, the steps to further refine its position are as follows:

Input: edge map edgeMap , the initial upper and lower boundary positions of the connected domain C

Output: refined upper and lower bound positions

，并存储在集合cSA中； d) For any row in the connected domain C , calculate its left and right non-zero pixel spans in edgeMap

, and stored in the set cSA;

，计算其在edgeMap中的行像素点数，并存储在集合

中，即有

； e) For any row in the connected domain C

, calculate the number of row pixels in the edgeMap , and store it in the collection

in, that is

;

f) Take the maximum value in cSA to exist , and store its serial number in pSRN ;

中的最大值存在

中，并将其序号存在

中； Pick

The maximum value in

, and store its sequence number in

middle;

的最大行序号

； g) for the For all rows in the range, take

The maximum row number of

;

范围内的所有行，取的最小行序号

； for in

For all rows in the range, take The minimum row number of

;

范围内的所有行，取

的最大行序号

； for in

For all rows in the range, take

The maximum row number of

;

范围内的所有行，取

的最小行序号

； for in

For all rows in the range, take

The minimum row number of

;

，即得到精化后的上下界位置； h) orders

, that is, the upper and lower bound positions after refinement are obtained ;

通常取值为0.6和0.3； in and

Usually the values are 0.6 and 0.3;

Use the following trailing filtering method to remove some non-subtitle connected domains:

i) After the above step g) is completed, continue to scan up and down in oPNA until the value at the current row is less than , assuming that the obtained line numbers are t _tail and b _tail respectively;

j) Calculate the length of the tail using the following formula:

tl ₁ = t ₂ - t _tail , tl ₂ = b _tail - b ₂ , tl =max ( tl ₁ , tl ₂ )

k) Use the following formula to filter, if deleteFlag(C) is 1, it means that this connected domain is not a subtitle area and should be deleted;

5

 和 

通常取值为0.2和0.3； Among them, ub _c and ut _c represent the upper and lower bound positions of the connected domain C after refinement, and

and

Usually the values are 0.2 and 0.3;

(5) Joint entropy filter

Use the joint entropy filter of the joint foreground pixel distribution entropy and edge pixel distribution entropy to filter, leaving only the subtitle area;

For the foreground pixel distribution entropy, it is the smallest enclosing rectangle Rect [t _c , b _c , l _c , r _c ] for a connected domain C, where t _c , b _c are the upper and lower bounds respectively, and l _c , r _c are respectively For the left and right boundaries, use the Otsu threshold to binarize it, then divide it into 2 rows × 4 columns = 8 parts, and use the following formula to calculate the distribution entropy:

            6

6

Among them, p _i,j represents the ratio of non-zero pixels in the i-th row and j-th column;

For the edge pixel distribution entropy, the Sobel edge binary image in the smallest enclosing rectangle Rect [t _c , b _c , l _c , r _c ] of the connected domain C is divided into 2 rows × 4 columns = 8 parts, and calculated using the following formula Distribution entropy:

 7

7

Where e _ij represents the number of edge pixels in row i, column j, and e _r is the sum of the number of edge pixels in 8 parts,

且

，则认为其是字幕区域，否则就是非字幕区域，应该删除，实验得

和

分别取6.4和2.76时效果最好； For any refined connected domain C, if its

and

, it is considered to be a subtitle area, otherwise it is a non-subtitle area and should be deleted. The experiment shows that

and

The effect is best when taking 6.4 and 2.76 respectively;

For some images with both horizontal and vertical subtitles, subtitle detection is performed in the original image and the image obtained by rotating 90 degrees, and then the detection results of the two are combined to eliminate duplication.

Described repeatability detection is carried out to subtitle area, if this area does not repeat, then its color is extremely unified as black and white characters, then carry out subtitle extraction, otherwise process next subtitle area step as:

(6) Repeatability detection

Use the method of combining position and grayscale color histogram to deduplicate the detected subtitle area, the steps are as follows:

为第i个字幕区域灰度级为k的像素数目；提取并存储当前帧所有字幕区域位置Rect_j[t_j,b_j,l_j,r_j] 及灰度直方图GH_j{g_j,0,g_j,1,…g_j,255}； l) Extract and store all subtitle area positions Rect _i [t _i ,b _i ,l _i , _ri ] and grayscale histogram GH _i {g _i,0 ,g _i,1 ,…g _i of the previous processed frame _,255 }, where

is the number of pixels whose gray level is k in the i-th subtitle area; extract and store all subtitle area positions Rect _j [t _j ,b _j ,l _j ,r _j ] and gray histogram GH _j {g _{j, 0} , g _{j, 1} , ... g _{j, 255} };

和灰度直方图相似度

，其中是它们的公共部分的面积，而

是它们中大的那个的面积，若

与有一个大于0.8，则是相同区域的重复检测，此时去掉一个，保留一个； m) Calculate their positional similarity

Similarity to grayscale histogram

,in is the area of their common part, and

is the area of the larger of them, if

and If one is greater than 0.8, it means repeated detection in the same area. At this time, one is removed and one is kept;

(7) Extremely uniform color

To unify the grayscale image of the subtitle area into black and white characters, take the following steps:

和

对二值化后的字幕区域进行卷积操作，用下式确定每个像素处的边缘颜色： n) First binarize the grayscaled subtitle area with the Otsu method, and then use a 3×3 mask

and

Perform convolution operation on the binarized subtitle area, and use the following formula to determine the edge color at each pixel:

                                8

8

为它们的比； Let N _w and N _b denote the number of white edge pixels and black edge pixels respectively, define

for their ratio;

和

； o) For the edge map P obtained in the previous step 8, the edge pixels are projected by column, and the edge map is decomposed into {x ₀ , x ₁ ,…,x _n } on the column, where x _i is the edge map in The midpoint of a continuous interval whose projection is 0 on the column, establish a rectangle Rect _i [1, height, x _i , x _i+1 ] in sequence, scan from four sides inward in the edge map P within the range of the rectangle, Delete the first edge pixel encountered, re-count the number of white edge pixels and the number of black edge pixels, respectively set to

and

;

为它们的比，定义 p) definition

For their ratio, define

                                           9

9

Use the following method to judge the color pole of the subtitle area:

，则为白字； (a) if

, it is white;

，则当时，字幕为白色，当时字幕为黑色； (d) if

, then when When , the subtitle is white, when When the subtitle is black;

，则为白字； (e) if

, it is white;

，则当时，字幕为白色，当

时字幕为黑色； (d) if

, then when When , the subtitle is white, when

When the subtitle is black;

，则为黑字； (f) if

, it is in black;

； in

;

q) After judging that the color of the subtitle is extremely high, if it is black, then invert the grayscale image of the subtitle area, otherwise no operation is performed.

In the subtitle extraction, the subtitle area after the extremely unified color is binarized, and the OCR software recognition steps are sent after removing the noise points as follows:

r) Normalize the height of the grayscaled subtitle area to 24, and then expand 4 pixels up and down respectively, so that the height after expansion is 32, which is set as EI;

s) Initialize each pixel of the resulting binary image B to 1, and then perform step-by-step horizontal local threshold binarization on the EI, and use the Otsu method to binarize in a 16×32 local window, each level Step by 8 pixels, the same method is used to perform step-by-step vertical local threshold binarization on EI, and use the Otsu method to binarize in a local window of image_width×8, each vertical step by 4 pixels, in In each sub-window, if the gray value in EI is lower than the local threshold, the corresponding pixel value in B is set to 0; Pixels are also set to 0, defining dam points:

Where H_len(x,y) represents the length of the longest horizontally continuous 1 sequence where the pixel (x,y) is located, and V_len(x,y) represents the length of the longest vertically continuous 1 sequence where the pixel (x,y) is located, For dam points, it cannot be expanded into background pixels;

u) Use the Sobel operator to get the edge information of EI. For each connected domain with a value of 1 in B, count the number epn of the edge pixels falling in it or surrounding it. If epn<tepn , the connected domain Set all the pixels in the domain to 0, so as to remove the connected domain, tepn is determined by the following formula:

tepn =max( cheight,cwidth )

Among them, cheight and cwidth are the height and width of the connected domain, respectively.

v) Send binary image B to OCR software for recognition.

Example

As shown in Figures 3 and 4, for a certain frame image in the video, an example of the process of identifying the subtitles contained in it is given. Below in conjunction with the method of the present invention describe in detail the concrete steps that this example implements, as follows:

For a certain frame of image, as shown in Figure 3(a), use the edge detection method (1) in claim 3 to obtain an edge map with enhanced corner points, and the result is shown in Figure 3(b);

(1) The edge map obtained in the above step is used as input, and the edge points are connected by (2) edge point connection method in claim 3, and the result is shown in Figure 3 (c);

(2) Taking the map of the connected edge points as input, adopt (3) connected domain collection and segmentation algorithm in claim 3 to obtain a larger connected domain, and the result is shown in Figure 3(d);

(3) For the connected domain obtained in the previous step, use the (4) connected domain refinement and tailing filtering method in claim 3 to obtain a more accurate area position size and perform preliminary filtering. The result is shown in Figure 3 (e) Show;

(4) For the remaining connected domains after filtering, use the (5) joint entropy filter in claim 3 to remove the non-subtitle area, and the final detection result is shown in Figure 3 (f);

(5) For a specific subtitle area detected in the previous step, as shown in Figure 4(a), first use the (6) repeatability detection in claim 4 to determine whether it is repeated with the previously detected area, if not Repeatedly, use the (7) color pole unification method in claim 4 to unify the area into white characters on a black background;

(6) For the subtitle area behind the uniform color, use the binarization and denoising algorithm in claim 5 to obtain a better binary image, the result is shown in Figure 4 (b);

(7) Use commercial OCR software to identify the binary image, and the result is shown in Figure 4 (c).

It can be seen from the accompanying drawings that this method can better detect the subtitle area in the video image frame and binarize it, and the binarized result can achieve better recognition accuracy.

Claims (4)

1. A video subtitle recognition method based on edge information and distribution entropy, is characterized in that its steps are as follows: 1) Detect the difference between the current frame and the previous processed frame, if the difference is large, perform the following subtitle recognition operation, otherwise continue to take the next frame for judgment; 2) Subtitle recognition First, perform subtitle detection. In subtitle detection, use edge detection, edge point connection, connected domain collection and segmentation methods, connected domain refinement and trailing filtering methods to obtain candidate text domains and their positions, and then use joint entropy filtering The filter removes the non-text field, leaving only the subtitle area; 3) Check the repeatability of the subtitle area. If the area is not repeated, the color is very uniform as black and white characters, and then the subtitle is extracted, otherwise the next subtitle area is processed; 4) In the subtitle extraction, binarize the subtitle area after the color is extremely uniform, remove the noise point and send it to the OCR software for recognition; the difference between the current frame and the previous processed frame is detected, if the difference is large, the following is performed Subtitle recognition operation, otherwise continue to take the next frame to judge the steps as follows: set this frame as I _i , its edge binary image is E _i , its previous processed frame is I _i-5 , its edge The binary image is E _i－5 , let D _i,i-5 =E _i ⊕E _i－5 , let any subtitle area detected last time be Area _i-5,j , and the edge of each subtitle area last time The minimum value of the cumulative sum of the binary image is pMES, and the cumulative difference of the subtitle area in the current frame is calculated as follows: $\mathrm{cFD}=\underset{j}{\mathrm{\&Sigma;}}{\mathrm{D.}}_{i,i-5}\left({\mathrm{area}}_{i-5,j}\right)---1$ If cFD is less than or equal to pMES×0.5, then there is no need to perform subtitle recognition operation on this frame, and continue to take the next 5th frame for judgment, otherwise, it is necessary to perform subtitle recognition operation on this frame. In order to further prevent missing subtitles, another count value is set ck, every time cFD is less than or equal to pMES×0.5, the value of ck is increased by 1, otherwise, ck is reset to 0, if ck is equal to 5, no matter what the previous judgment is, the subtitle recognition operation of this frame is required, and ck is reassigned to 0 . 2. a kind of video subtitle recognition method based on edge information and distribution entropy according to claim 1, it is characterized in that described subtitle recognition first carries out subtitle detection, uses edge detection, edge point connection, connected region in subtitle detection The collection and segmentation method, connected region refinement and trailing filtering method obtain the candidate text domain and its position, and then use the joint entropy filter to remove the non-text domain, leaving only the subtitle area. The steps are: (1) Edge detection method Given an image I, the Sobel operator is used to detect edges. The Sobel operator is composed of gradient templates in four directions: horizontal S _H , vertical S _V , diagonal S _LD , and anti-diagonal S _RD . The edge field is given by the following formula calculate: S＝MAX(|S _H |,|S _V |,|S _LD |,|S _RD |)+k×|S _⊥-MAX | 2 Among them, ⊥-MAX represents the direction perpendicular to the direction of the maximum gradient absolute value at the pixel (x, y), k is an adjustment coefficient, and its value is 1 in this paper, and S is then quantized into 16 levels, and it is expressed as S' after quantization , and then use the following formula to get the edge map EdgeMap: $\mathrm{<span\; class="notranslate">\; EdgeMap</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; 1</span>}& {\mathrm{<span\; class="notranslate">\; S</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ></span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 15</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& {\mathrm{<span\; class="notranslate">\; S</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; 15</span>}\end{array}\right.<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 3</span>}$ (2) Edge point connection method For the edge map EdgeMap, if the distance between two edge points in the same line is less than a certain threshold T _d , then set the pixel values between these two pixels in the EdgeMap to 1, and T _d is determined by the following formula: ${\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; d</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; max</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; min</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; [</span>\frac{\mathrm{<span\; class="notranslate">\; max</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; height</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; width</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; 50</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 16</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 4</span>}$ Wherein height and width are the height and width of image I respectively; (3) Connected area collection and segmentation method Collect the connected regions of the EdgeMap obtained in the previous step, remove those connected regions whose height or width is less than 1% of the entire image height or width, and remove those connected regions whose smallest enclosing rectangle is less than 0.2% of the entire image area, and then use the following The step performs region segmentation on each connected region C: a) For each row i∈[t _c , b _c ] in the connected region C, get the area of the smallest enclosing rectangle Area _up (i) of the row and above and the area of the smallest enclosing rectangle Area _down of the part below the row (i), find out the sum of these two areas, find out the row number of the minimum sum and store it in bR; b) For each column j∈[l _c , r _c ] in the connected region C, obtain the area of the minimum enclosing rectangle Area _left (j) of the column and the left part and the area of the minimum enclosing rectangle Area _right of the right part of the column (j), calculate the sum of these two areas, find out the column number to obtain the minimum sum and store it in bC; c) Let mRA=Area _up (bR)+Area _down (bR), mCA=Area _left (bC)+Area _right (bC), if mRA<mCA, then divide the connected area C into Two connected areas, otherwise, the connected area C is divided into two connected areas on the column with the bC column as the boundary; Among them, t _c , b _c , l _c and r _c are the upper boundary row number, lower boundary row number, left boundary column number and right boundary column number of connected area C respectively; In order to prevent over-segmentation, the segmentation is only performed when the connected region C satisfies the following two conditions at the same time: ①The filling rate of the connected region is less than 0.8; ②The areas of the two new connected regions are both larger than 0.2% of the entire image area; (4) Connected area refinement and trailing filtering method Before performing area refinement, remove those connected areas whose height is greater than twice the width, which may mistakenly delete those vertical subtitles. In order to process vertical subtitles, you only need to rotate the image by 90 degrees, and the other operations are exactly the same; For each connected region C obtained in the previous step, the steps to further refine its position are as follows: Input: edge map edgeMap, the initial upper and lower bound line numbers t _c , b _c of the connected region C Output: refined upper and lower bound line numbers ut _c , ub _c d) For any row i∈[t _c , b _c ] of the connected area C, calculate its left and right non-zero pixel span r _i -l _i in the edgeMap, and store it in the set cSA; e) For any row i∈[t _c ,b _c ] of the connected domain C, calculate the number of row pixels in edgeMap and store it in the set oPNA

f) Take the maximum value in cSA and store it in pCS, and store its serial number in pSRN; Take the maximum value in oPNA and store it in pOPN, and store its serial number in pPRN; g) For all rows within the range of i∈[t _c , pSRN], take the maximum row number t ₁ with cSA[i]<pCS×η ₁ ; For all rows within the range of i∈[pSRN,b _c ], take the smallest row number b ₁ where cSA[i]<pCS×η ₁ ; For all rows within the range of i∈[t _c ,pPRN], take the maximum row number t ₂ of oPNA[i]<pOPN×η ₂ ; For all rows within the range of i∈[pPRN,b _c ], take the smallest row number b ₂ where oPNA[i]<pOPN×η ₂ ; h) Make ut _c ＝max(t ₁ , t ₂ ), ub _c ＝min(b ₁ , b ₂ ), that is, get the refined upper and lower boundary row numbers ut _c , ub _c ; Wherein η ₁ and η ₂ usually take a value of 0.6 and 0.3; Use the following trailing filtering method to remove some non-subtitle connected domains: i) after the above step g) is completed, continue to scan up and down in the oPNA until the value at the current row is less than pOPN × η ₃ , assuming that the row numbers obtained are respectively t _tail and b _tail ; j) Use the following formula to calculate the length of the tail: tl ₁ =t ₂ -t _tail ,tl ₂ =b _tail -b ₂ ,tl=max(tl ₁ ,tl ₂ ) k) Use the following formula to filter, if deleteFlag (C) is 1, it means that this connected area is not a subtitle area and should be deleted; $\mathrm{<span\; class="notranslate">\; deleteFlag</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; 1</span>}& \mathrm{<span\; class="notranslate">\; tl</span>}<span\; class="notranslate">\; ></span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; ub</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; out</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; \&eta;</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; others</span>}\end{array}\right.<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 5</span>}$ Among them, ut _c and ub _c respectively represent the upper and lower boundary line numbers of the connected region C after refinement, and η ₃ and η ₄ usually take values of 0.2 and 0.3; (5) Joint entropy filter Use the joint entropy filter of the joint foreground pixel distribution entropy and edge pixel distribution entropy to filter, leaving only the subtitle area; For the distribution entropy of foreground pixels, it is the smallest enclosing rectangle Rect[t _c , b _c , l _c , r _c ] for a connected region C, where t _c , b _c are the upper and lower boundary row numbers, l _c , r _c They are the column numbers of the left and right borders, and use the Otsu threshold to binarize them, and then divide them into 2 rows × 4 columns = 8 parts, and use the following formula to calculate the distribution entropy: ${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; FPD</span>}}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}\mathrm{<span\; class="notranslate">\; ln</span>}{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; ln</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; 1,2</span>}<span\; class="notranslate">\; \}</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; 1,2,3,4</span>}<span\; class="notranslate">\; \}</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 6</span>}$ Among them, p _i,j represents the ratio of non-zero pixels in the i-th row and j-th column; For the edge pixel distribution entropy, the Sobel edge binary image in the smallest enclosing rectangle Rect[t _c , b _c , l _c , r _c ] of the connected area C is divided into 2 rows × 4 columns = 8 parts, and calculated using the following formula Distribution entropy: ${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; E.</span>}}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; (</span>\frac{{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}}{{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}}\mathrm{<span\; class="notranslate">\; ln</span>}\frac{{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}}{{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}}<span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\frac{{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}}{{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; ln</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\frac{{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}}{{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; 1,2</span>}<span\; class="notranslate">\; \}</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; 1,2,3,4</span>}<span\; class="notranslate">\; \}</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 7</span>}$ Where e _ij represents the number of edge pixels in row i, column j, and e _r is the sum of the number of edge pixels in 8 parts, For any refined connected area C, if its E _FPD > E _T1 and E _E > E _T2 , it is considered to be a subtitle area, otherwise it is a non-subtitle area and should be deleted. Experimentally, E _T1 and E _T2 are respectively taken as 6.4 and 2.76 are the best; For some images with both horizontal and vertical subtitles, subtitle detection is performed in the original image and the image obtained by rotating 90 degrees, and then the detection results of the two are combined to eliminate duplication. 3. a kind of video subtitle recognition method based on edge information and distribution entropy according to claim 1, it is characterized in that described subtitle area is carried out repetition detection, if this area does not repeat, then its color is extremely unified as Black background and white text, and then perform subtitle extraction, otherwise, the steps to process the next subtitle area are: (6) Repeatability detection Use the method of combining position and grayscale color histogram to deduplicate the detected subtitle area, the steps are as follows: l) Extract and store all subtitle area positions Rect _i [t _i , b _i , l _i , r _i ] and grayscale histogram GH _i {g _i,0 ,g _i,1 ,…g _i of the previous processed frame _,k ,...g _i,255 }, where g _i,k is the number of pixels whose gray level is k in the i-th subtitle area; extract and store all subtitle area positions of the current frame Rect _j [t _j ,b _j ,l _j ,r _j ] and gray histogram GH _j {g _j,0 ,g _j,1 ,…g _j,255 }; m) Calculate their positional similarity

Similarity to grayscale histogram

Where Rect _i ∩ Rect _j is the area of their common part, and max(Rect _i ,Rect _j ) is the area of the larger of them, if Simi _Loc (i,j) and Simi _GHis (i,j) have a If it is greater than 0.8, it means repeated detection in the same area. At this time, one is removed and one is kept; (7) Extremely uniform color To unify the grayscale image of the subtitle area into black and white characters, take the following steps: n) First binarize the grayscaled subtitle area with the Otsu method, and then use a 3×3 mask $k_w(x,\mathrm{the\; y})=\left[\begin{array}{ccc}0& -1& 0\\ -1& 4& -1\\ 0& -1& 0\end{array}\right]$ and $k_b(x,\mathrm{the\; y})=\left[\begin{array}{ccc}0& 1& 0\\ 1& -4& 1\\ 0& 1& 0\end{array}\right]$ Perform convolution operation on the binarized subtitle area, and use the following formula to determine the edge color at each pixel: $\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; White</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; Edge</span>}<span\; class="notranslate">\; ,</span>& {\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; w</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ></span>\mathrm{<span\; class="notranslate">\; 0</span>}\\ \mathrm{<span\; class="notranslate">\; Black</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; Edge</span>}<span\; class="notranslate">\; ,</span>& {\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; b</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ></span>\mathrm{<span\; class="notranslate">\; 0</span>}\\ \mathrm{<span\; class="notranslate">\; No</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; Edge</span>}<span\; class="notranslate">\; ,</span>& {\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; w</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; 0</span>}\mathrm{<span\; class="notranslate">\; and</span>}{\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; b</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; 0</span>}\end{array}\right.<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 8</span>}$ Let N _w and N _b represent the number of white edge pixels and the number of black edge pixels respectively, define R ₁ =N _w /N _b as their ratio; o) For the edge map P obtained in the previous step 8, the edge pixels are projected by column, and the edge map is decomposed into {x ₀ , x ₁ ,…,x _n } on the column, where x _i is the edge map At the midpoint of a continuous interval whose projection is 0 on the column, a rectangle Rect _i [1, height, x _i , x _i+1 ] is sequentially established, and the edge map P within the range of the rectangle is scanned from the four sides inward , delete the first edge pixel encountered, recount the number of white edge pixels and the number of black edge pixels, and set them to N _w ' and N _b 'respectively; p) Define R ₂ =N _w '/N _b ' as their ratio, define $\mathrm{<span\; class="notranslate">\; \&Delta;R</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{\mathrm{<span\; class="notranslate">\; max</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; \&Delta;R</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 9</span>}$ Use the following method to judge the color pole of the subtitle area: (a) If ΔR>T _2h , it is white; If T _2l ≤ΔR≤T _2h , when R ₁ >T _2v , the subtitle is white, and when R ₁ ≤T _2v , the subtitle is black; If T _1h ≤ΔR≤T _2l , then it is white; (d) If T _1l ≤ΔR≤T _1h , when R ₁ >T _1v , the subtitle is white, and when R ₁ ≤T _1v , the subtitle is black; If ΔR<T _1l , it is in black; Where T _1l =-0.25, T _1h =-0.15, T _1v =1.2, T _2l =0, T _2h =0.35, T _2v =0.8; q) After judging the color of the subtitle, if it is black, then invert the grayscale image of the subtitle area, otherwise no operation is performed. 4. a kind of video subtitle recognition method based on edge information and distribution entropy according to claim 1, it is characterized in that in described subtitle extraction, the subtitle region after the extremely uniform color is carried out binarization, after removing noise point The steps to send OCR software recognition are: r) normalize the height of the grayscaled subtitle area to 24, then expand 4 pixels up and down respectively, so that the height after expansion is 32, and the subtitle area image after the expansion is set as EI; s) Initialize each pixel of the resulting binary image B to 1, then perform step-by-step horizontal local threshold binarization on EI, and use the Otsu method to binarize in a 16×32 local window, each time Stepping 8 pixels, the same method is used to perform step-by-step vertical local threshold binarization on EI, and use the Otsu method to binarize in a local window of image_width×8, each vertical step is 4 pixels, in In each sub-window, if the gray value in EI is lower than the local threshold value, the corresponding pixel value in B is set to 0; Pixels are also set to 0, defining dam points: Dam points＝{(x,y)|B(x,y)＝1∧1≤min(H_len(x,y),V_len(x,y)≤4 Where H_len(x,y) represents the length of the longest horizontally continuous 1 sequence where the pixel (x,y) is located, and V_len(x,y) represents the length of the longest vertically continuous 1 sequence where the pixel (x,y) is located, For dam points, it cannot be expanded into background pixels; u) Use the Sobel operator to obtain the edge information of EI. For each connected area with a value of 1 in B, count the number epn of the edge pixels falling in it or surrounding it. If epn<tepn, the connected area All pixels in the area are set to 0, thereby removing the connected area, and tepn is determined by the following formula: tepn=max(cheight,cwidth) where cheight and cwidth are the height and width of the connected area, respectively; v) Send binary image B to OCR software for recognition.

Images