CN115131648A - On-line handwritten Chinese text line recognition method based on real-time updating of local feature map - Google Patents

Abstract

The invention provides an online handwritten Chinese text line identification method based on real-time update of a local feature map, which comprises the following steps; step S1, initializing a real-time recognition model; step S2, when the stroke input is started, the handwriting surface of the handwriting equipment touched by the pen point is used as a starting mark; s3, collecting the track of the online handwritten Chinese text line, recording the coordinate point sequence of the stroke, and taking the pen as the end mark of the stroke; step S4, intercepting a local image corresponding to the newly input stroke, and preprocessing the local image; step S5, calculating the CNN characteristic of the local image corresponding to the new input stroke, and replacing the new characteristic with the previous characteristic graph to realize the real-time update of the local characteristic of the new stroke; step S6, using the language model to decode and update the recognition result, realizing the real-time recognition of the text line; the method has the advantages of greatly improved speed, lower image size dependence, stronger expansibility and robustness, capability of being used in various embedded devices with limited resources and higher application value.

Description

Online handwritten Chinese text line recognition method based on real-time update of local feature map

technical field

The invention relates to the technical field of computer vision and document image analysis, in particular to an online handwritten Chinese text line recognition method based on real-time updating of local feature maps.

Background technique

As an extremely important part of Optical Character Recognition (OCR), handwritten text recognition has been the focus of many researchers. Compared with document print text recognition, online handwritten Chinese text line recognition is more complex, and has problems such as large changes in font styles and a wide variety of types. Most of the currently used recognition methods are image-based codec recognition methods, which have achieved certain results. However, for Chinese texts, especially long text lines, it is difficult to realize real-time recognition of online handwritten texts smoothly because the generated images are too large. For users, if they wait for the entire sentence to be written and then click for recognition, the user needs a click operation and a short wait, which is unbearable. In actual use, in order not to be stuck in actual use, it is often limited to recognize single words or short texts in application, which limits the practicability of the system. Most of the current real-time recognition methods use the trajectory-based over-segmentation method for text line recognition. This type of method is mainly based on LSTM for recognition. However, due to the large number of long text lines in the online handwritten Chinese text lines, the trajectory sequence is very It is difficult to perform accurate over-segmentation using the LSTM method, and it is also difficult to achieve good recognition accuracy. With the development of deep learning, image-based non-segmentation recognition methods have gradually become dominant in text line recognition models in recent years, and have certain advantages for long text line recognition. However, in order to retain complete information, a large text line image needs to be generated in online handwritten text line recognition. If real-time recognition is performed using the image-based text line recognition method, it will take a lot of time, affecting the user's writing experience, and is not conducive to real-time recognition. Recognition is applied in practical scenarios such as embedded devices.

SUMMARY OF THE INVENTION

The invention proposes an online handwritten Chinese text line recognition method based on real-time update of local feature maps, which greatly improves the speed, has low image size dependence, strong scalability and robustness, and can be used for various resource constraints. It has high application value in embedded devices.

The present invention adopts the following technical solutions.

An online handwritten Chinese text line recognition method based on real-time update of local feature maps for real-time recognition of handwritten text, including the following steps;

Step S1, initialize the real-time recognition model;

Step S2, when starting the stroke input, use the pen tip to touch the handwriting surface of the handwriting device as a start sign;

Step S3, collects the online handwritten Chinese text line track, records the coordinate point sequence of each stroke, and takes the pen as the end mark of a stroke;

Step S4, intercepting the local image corresponding to the newly input stroke, and preprocessing it;

Step S5, calculate the CNN feature of the local image corresponding to the new input stroke, replace the new feature on the last feature map, and realize the real-time update of the local feature of the new stroke;

Step S6, using the CTC combined with the n-gram language model to decode and update the recognition result, so as to realize the real-time recognition of online handwritten Chinese text lines.

In step S1, the real-time recognition model includes a deep convolutional neural network encoder, an RNN decoder, and a CTC and language model transcription layer; when the real-time recognition model is initialized, a preset height and width are first initialized, and the pixel The text line image with all zero values is sent to the handwritten text line recognition model, and the global output feature map of each convolution layer and pooling layer in the handwritten text line recognition model is obtained for subsequent real-time feature updates. .

The height and width of the text line image match the actual handwriting scene, the writing order of the handwritten text is from left to right and allows the pen to be inserted backwards, and the size of the handwritten text matches the size of the text line image.

In step S1, the real-time recognition model performs image feature extraction on the basis of CRNN, in which the pooling layers in the network are all set to average pooling, except for the first two convolutional layers, all subsequent convolutional layers are followed by batches. normalization layer;

The real-time recognition model uses 7 convolutional layers and 4 pooling layers to extract features, and the corresponding number of convolution kernels is the number of output feature maps. }, and finally use a global average pooling layer to pool the image to a height of 1, and the global average pooling process width remains unchanged;

The RNN part of the real-time recognition model uses a two-layer stacked bidirectional LSTM, and the number of hidden layer nodes is 256. Finally, the language model is introduced through CTC and Beam Search for final decoding.

In step S3, the online handwritten Chinese text line trajectory is collected, the coordinate point sequence of each stroke is recorded and expressed as (x1, y1), (x2, y2), ..., (xn, yn), and the pen is taken as a stroke. end sign.

In step S4, the corresponding partial image of the newly inputted stroke is intercepted and preprocessed; specifically: when the user inputs a new stroke, first obtain the left and right x-axis boundaries of the stroke, denoted as L, R, and set the volume The size of the product kernel is 3×3, and 2 columns of pixels on both sides of the boundary need to be added, so the size of the intercepted local image M is:

Among them, H and W are the height and width of the image respectively; the first case and the second case corresponding to the upper two lines on the right side of formula 1 are the cases where the stroke boundary is on the left and right edges of the original image, and the first and second cases on the right side of formula 1 The third situation corresponding to the three lines is that the partial image captured after the stroke is written is a rectangular area formed by extending the left and right borders by 2 columns of pixels;

The first case is at the left edge of the original image (L≤1). At this time, the right border is expanded by 2 columns of pixels, and the left border is expanded to the first column on the left, so the width is ((W-1)-(L-2)) ;

The second case is at the right edge of the original image (R≥W-2). At this time, the left border is expanded by 2 columns of pixels, and the right border is expanded to the last column on the right, so the width is ((R+2)-0).

After the partial image is intercepted, padding is performed on the intercepted partial image M. The specific method is: padding 1 row of pixels on each of the upper and lower sides of the partial image M to simulate the padding required for the convolution of the original image; if the partial image M is on the left and right sides of the original image , then padding is performed according to different situations, which is expressed by the formula as

In the formula, P _L and P _R are the padding conditions on the left and right sides respectively; if the partial image is not on the left and right sides of the original image, at this time, the left and right sides of the partial image have been expanded by 2 columns of pixels, and no padding operation is required, that is, P _L =P _R = 0; if the partial image is on the left and right sides of the original image, such as the left side (L=0), at this time, it is necessary to padding a column of all 0 pixels on the left side to simulate the padding operation of the original image.

In step S5, only the CNN feature of the local image corresponding to the new input stroke is calculated, and the new feature is replaced on the previous feature map to realize the real-time update of the local feature of the new stroke. The specific method is:

The local convolution operation is performed on the intercepted local image, and the feature map of each network layer is iteratively convolved, pooled and updated. There is no padding in the pooling process. The pixels and padding to be expanded in the convolution process are as follows:

Wi represents the width of the feature map of the _i -th layer; E(L, R) represents the pixel to be expanded. When it is (-2,+2), that is, the left and right sides are expanded by 2 columns, then the corresponding pixel value of the original feature map is On the local feature map to be updated, continue the iterative convolution and feature map update of the next layer;

Consider whether it can be divisible by the step size of the pooling layer when expanding the feature map pixels, specifically: when the step size of the pooling layer is 2, if the current L cannot be divisible by the step size 2, the L%2 column should be expanded to the left Pixels; if the step size is S, counting the current L-th column, you need to expand L%S column pixels to the left; similarly, for the right side of the local feature, counting the current R-th column, you need to expand S-L to the right %S columns of pixels.

In step S6, use the CTC combined with the N-Gram language model to decode and update the recognition result, so as to realize the real-time recognition of online handwritten Chinese text lines.

The invention proposes a real-time recognition method for online handwritten Chinese text lines based on real-time updating of local feature maps. For the recognition of online handwritten Chinese text lines, the CRNN is improved to extract and decode image features. In the inference stage, real-time recognition of online handwritten Chinese text lines is realized. The method achieves excellent results in two different categories of handwriting scenes, and the speed is greatly improved, with low image size dependence, strong scalability and robustness, the method can be used in various It has high application value in embedded devices with limited resources.

The recognition speed of the invention is greatly improved, and the invention has lower image size dependence, stronger expandability and robustness, can be used in various embedded devices with limited resources, and has higher application value.

Compared with the prior art, the present invention also has the following beneficial effects:

(1) Realize online handwritten Chinese text line recognition.

(2) Real-time recognition of online handwritten Chinese text lines is realized.

(3) The method proposed in the present invention has high scalability and can be applied to other network structures and even other additional features such as path reintegration features.

Description of drawings

The present invention will be described in further detail below in conjunction with the accompanying drawings and specific embodiments:

Accompanying drawing 1 is the flow chart of the method of the present invention;

Accompanying drawing 2 is the overall network structure schematic diagram of online handwritten text line recognition;

Accompanying drawing 3 is the schematic diagram of partial update area;

FIG. 4 is a schematic diagram of an implementation example of the method.

Detailed ways

As shown in the figure, an online handwritten Chinese text line recognition method based on real-time update of local feature maps is used for real-time recognition of handwritten text, including the following steps;

Step S1, initialize the real-time recognition model;

Step S2, when starting the stroke input, use the pen tip to touch the handwriting surface of the handwriting device as a start sign;

Step S3, collects the online handwritten Chinese text line track, records the coordinate point sequence of each stroke, and takes the pen as the end mark of a stroke;

Step S4, intercepting the local image corresponding to the newly input stroke, and preprocessing it;

Step S5, calculate the CNN feature of the local image corresponding to the new input stroke, replace the new feature on the last feature map, and realize the real-time update of the local feature of the new stroke;

Step S6, using the CTC combined with the n-gram language model to decode and update the recognition result, so as to realize the real-time recognition of online handwritten Chinese text lines.

In step S1, the real-time recognition model includes a deep convolutional neural network encoder, an RNN decoder, and a CTC and language model transcription layer; when the real-time recognition model is initialized, a preset height and width are first initialized, and the pixel The text line image with all zero values is sent to the handwritten text line recognition model, and the global output feature map of each convolution layer and pooling layer in the handwritten text line recognition model is obtained for subsequent real-time feature updates. .

The height and width of the text line image match the actual handwriting scene, the writing order of the handwritten text is from left to right and allows the pen to be inserted backwards, and the size of the handwritten text matches the size of the text line image.

In step S1, the real-time recognition model performs image feature extraction on the basis of CRNN, in which the pooling layers in the network are all set to average pooling, except for the first two convolutional layers, all subsequent convolutional layers are followed by batches. normalization layer;

The real-time recognition model uses 7 convolution layers and 4 pooling layers to extract features. The corresponding number of convolution kernels is the number of output feature maps. The number of convolution kernels of the 7 convolution layers is {64, 128 respectively. , 256, 256, 256, 512, 512}, and finally use a global average pooling layer to pool the image to a height of 1, and the global average pooling process width remains unchanged;

The RNN part of the real-time recognition model uses a two-layer stacked bidirectional LSTM, and the number of hidden layer nodes is 256. Finally, the language model is introduced through CTC and Beam Search for final decoding.

In step S3, the online handwritten Chinese text line trajectory is collected, the coordinate point sequence of each stroke is recorded and expressed as (x1, y1), (x2, y2), ..., (xn, yn), and the pen is taken as a stroke. end sign.

In step S4, the corresponding partial image of the newly inputted stroke is intercepted and preprocessed; specifically: when the user inputs a new stroke, first obtain the left and right x-axis boundaries of the stroke, denoted as L, R, and set the volume The size of the product kernel is 3×3, and 2 columns of pixels on both sides of the boundary need to be added, so the size of the intercepted local image M is:

Among them, H and W are the height and width of the image respectively; the first case and the second case corresponding to the upper two lines on the right side of formula 1 are the cases where the stroke boundary is on the left and right edges of the original image, and the first and second cases on the right side of formula 1 The third situation corresponding to the three lines is that the partial image captured after the stroke is written is a rectangular area formed by extending the left and right borders by 2 columns of pixels;

The first case is at the left edge of the original image (L≤1). At this time, the right border is expanded by 2 columns of pixels, and the left border is expanded to the first column on the left, so the width is ((W-1)-(L-2)) ;

The second case is at the right edge of the original image (R≥W-2). At this time, the left border is expanded by 2 columns of pixels, and the right border is expanded to the last column on the right, so the width is ((R+2)-0).

After the partial image is intercepted, padding is performed on the intercepted partial image M. The specific method is: padding 1 row of pixels on each of the upper and lower sides of the partial image M to simulate the padding required for the convolution of the original image; if the partial image M is on the left and right sides of the original image , then padding is performed according to different situations, which is expressed by the formula as

In the formula, P _L and P _R are the padding conditions on the left and right sides respectively; if the partial image is not on the left and right sides of the original image, at this time, the left and right sides of the partial image have been expanded by 2 columns of pixels, and no padding operation is required, that is, P _L =P _R = 0; if the partial image is on the left and right sides of the original image, such as the left side (L=0), at this time, it is necessary to padding a column of all 0 pixels on the left side to simulate the padding operation of the original image.

In step S5, only the CNN feature of the local picture corresponding to the new input stroke is calculated, and the new feature is replaced on the previous feature map, so that the local feature of the new stroke is updated in real time. The specific method is:

The local convolution operation is performed on the intercepted local image, and the feature map of each network layer is iteratively convolved, pooled and updated. There is no padding in the pooling process. The pixels and padding to be expanded in the convolution process are as follows:

Wi represents the width of the feature map of the _i -th layer; E(L, R) represents the pixel to be expanded. When it is (-2,+2), that is, the left and right sides are expanded by 2 columns, then the corresponding pixel value of the original feature map is On the local feature map to be updated, continue the iterative convolution and feature map update of the next layer;

Consider whether it can be divisible by the step size of the pooling layer when expanding the feature map pixels, specifically: when the step size of the pooling layer is 2, if the current L cannot be divisible by the step size 2, the L%2 column should be expanded to the left Pixels; if the step size is S, counting the current L-th column, you need to expand L%S column pixels to the left; similarly, for the right side of the local feature, counting the current R-th column, you need to expand S-L to the right %S columns of pixels.

In step S6, use the CTC combined with the N-Gram language model to decode and update the recognition result, so as to realize the real-time recognition of online handwritten Chinese text lines.

In this example, a full black text line image with a height of 128 and a width of 2400 is initialized in step S1.

In this example, CTC is a text recognition algorithm, and N-Gram is a language model commonly used in large-vocabulary continuous speech recognition. For Chinese, it can be called a Chinese Language Model (CLM, Chinese Language Model).

Claims (9)

1. the on-line handwritten Chinese text line recognition method updated in real time based on local feature map, is used for the real-time recognition of handwritten text, it is characterized in that: comprise the following steps; Step S1, initialize the real-time recognition model; Step S2, when starting the stroke input, use the pen tip to touch the handwriting surface of the handwriting device as a start sign; Step S3, collects the online handwritten Chinese text line track, records the coordinate point sequence of each stroke, and takes the pen as the end mark of a stroke; Step S4, intercepting the local image corresponding to the newly input stroke, and preprocessing it; Step S5, calculate the CNN feature of the local image corresponding to the new input stroke, replace the new feature on the last feature map, and realize the real-time update of the local feature of the new stroke; Step S6, using the CTC combined with the n-gram language model to decode and update the recognition result, so as to realize the real-time recognition of online handwritten Chinese text lines. 2. the online handwritten Chinese text line recognition method based on local feature map real-time updating according to claim 1, is characterized in that: in step S1, real-time recognition model comprises a deep convolutional neural network encoder, an RNN decoder , and the CTC and language model transcription layer; when the real-time recognition model is initialized, a text line image with a preset height and width and all zero pixel values is initialized, and then the text line image is sent to the handwritten text line recognition model to obtain The global output feature map of each convolutional layer and pooling layer in the handwritten text line recognition model is used for subsequent real-time feature updates. 3. the online handwritten Chinese text line recognition method based on local feature map real-time update according to claim 2, is characterized in that: the height, width of described text line image match with actual handwriting scene, and the writing order of handwritten text is from Left-to-right and allow for inverted pen insertion, the size of the handwritten text matches the size of the text line image. 4. the online handwritten Chinese text line recognition method based on local feature map real-time update according to claim 2, is characterized in that: in step S1, real-time recognition model carries out the feature extraction of image on the basis of CRNN, wherein in the network The pooling layers are all set to average pooling, except for the first two convolutional layers, all subsequent convolutional layers are followed by batch normalization layers; The real-time recognition model uses 7 convolutional layers and 4 pooling layers to extract features, and the corresponding number of convolution kernels is the number of output feature maps. }, and finally use a global average pooling layer to pool the image to a height of 1, and the global average pooling process width remains unchanged; The RNN part of the real-time recognition model uses a two-layer stacked bidirectional LSTM, and the number of hidden layer nodes is 256. Finally, the language model is introduced through CTC and Beam Search for final decoding. 5. the on-line handwritten Chinese text line recognition method based on local feature map real-time update according to claim 2, is characterized in that: in step S3, collect on-line handwritten Chinese text line track, record the coordinate point sequence of each stroke and represent It is (x1, y1), (x2, y2), . 6. the online handwritten Chinese text line recognition method based on local feature map real-time update according to claim 2, is characterized in that: in step S4, intercept the corresponding local image of newly inputted strokes and carry out preprocessing; Be specially: when user After inputting a new stroke, first find the left and right x-axis boundaries of the stroke, denoted as L, R, set the size of the convolution kernel to 3 × 3, and add 2 columns of pixels on both sides of the boundary, thus obtaining the interception The size of the local image M is:

Among them, H and W are the height and width of the image respectively; the first case and the second case corresponding to the upper two lines on the right side of formula 1 are the cases where the stroke boundary is on the left and right edges of the original image, and the first and second cases on the right side of formula 1 The third situation corresponding to the three lines is that the partial image captured after the stroke is written is a rectangular area formed by extending the left and right borders by 2 columns of pixels; The first case is at the left edge of the original image (L≤1). At this time, the right border is expanded by 2 columns of pixels, and the left border is expanded to the first column on the left, so the width is ((W-1)-(L-2)) ; The second case is at the right edge of the original image (R≥W-2). At this time, the left border is expanded by 2 columns of pixels, and the right border is expanded to the last column on the right, so the width is ((R+2)-0). 7. the online handwritten Chinese text line recognition method based on local feature map real-time update according to claim 6, is characterized in that: after intercepting the partial image, the partial image M of interception is filled with padding, and the concrete method is: the partial image Padding 1 line of pixels on the upper and lower sides of M to simulate the padding required for the convolution of the original image; if the partial image M is on the left and right sides of the original image, padding is performed according to different situations, which is expressed by the formula as

In the formula, P _L and P _R are the padding conditions on the left and right sides respectively; if the partial image is not on the left and right sides of the original image, at this time, the left and right sides of the partial image have been expanded by 2 columns of pixels, and no padding operation is required, that is, P _L =P _R = 0; if the partial image is on the left and right sides of the original image, such as the left side (L=0), at this time, it is necessary to padding a column of all 0 pixels on the left side to simulate the padding operation of the original image. 8. the online handwritten Chinese text line recognition method based on local feature map real-time update according to claim 7, is characterized in that: in step S5, only calculate the CNN feature of the local picture corresponding to the new input stroke, the new feature is replaced to On the last feature map, the local features of the new strokes are updated in real time. The specific methods are as follows: The local convolution operation is performed on the intercepted local image, and the feature map of each network layer is iteratively convolved, pooled and updated. There is no padding in the pooling process. The pixels and padding to be expanded in the convolution process are as follows:

Wi represents the width of the feature map of the _i -th layer; E(L, R) represents the pixel to be expanded. When it is (-2, +2), that is, the left and right sides are expanded by 2 columns, the corresponding pixel value of the original feature map is On the local feature map to be updated, continue the iterative convolution and feature map update of the next layer; Consider whether it can be divisible by the step size of the pooling layer when expanding the feature map pixels. Specifically, when the step size of the pooling layer is 2, if the current L cannot be divisible by the step size 2, the column L%2 should be expanded to the left. Pixels; if the step size is S, counting the current L-th column, you need to expand L%S column pixels to the left; similarly, for the right side of the local feature, counting the current R-th column, you need to expand S-L to the right %S columns of pixels. 9. the online handwritten Chinese text line recognition method based on local feature map real-time update according to claim 1, is characterized in that: in step S6, use CTC in conjunction with N-Gram language model decoding update recognition result, realize online handwritten Chinese text real-time identification.

Images