CN110110715A - Text detection model training method, text filed, content determine method and apparatus - Google Patents

Abstract

The present invention provides a text detection model training method, text area, and content determination method and device; wherein, the text detection model training method includes: extracting multiple initial feature maps of a target training image through a first feature extraction network; The fusion network fuses multiple initial feature maps to obtain a fusion feature map; inputs the fusion feature map to the first output network, and outputs the candidate regions of the text region in the target training image and the probability value of each candidate region; The first loss value is determined by the detection loss function of , and the first initial model is trained according to the first loss value until the parameters in the first initial model converge, and a text detection model is obtained. The present invention can quickly, comprehensively and accurately detect various types of texts in images under the scenarios of various font sizes, various fonts, various shapes and various orientations, which is also beneficial to the accuracy of subsequent text recognition and improves the accuracy of text recognition. Effect.

Description

Text detection model training method, text area, content determination method and device

technical field

The present invention relates to the technical field of image processing, and in particular, to a text detection model training method, a text area, and a content determination method and device.

Background technique

In the related art, text detection and recognition can be realized by character segmentation or deep learning. However, these methods are usually suitable for simple scenarios such as single font size, simple background, and single text arrangement direction; in complex scenarios, such as multiple font sizes, multiple fonts, multiple shapes, multiple orientations, and changing backgrounds, etc. The above text detection and recognition methods are less effective.

SUMMARY OF THE INVENTION

In view of this, the purpose of the present invention is to provide a text detection model training method, text area, content determination method and device, so as to be fast, comprehensive and accurate in a variety of font sizes, fonts, shapes, and directions. It can detect all kinds of text in the image, which is also conducive to the accuracy of subsequent text recognition and improves the effect of text recognition.

In a first aspect, an embodiment of the present invention provides a method for training a text detection model. The method includes: determining a target training image based on a preset training set; inputting the target training image into a first initial model; the first initial model includes a first a feature extraction network, a feature fusion network and a first output network; multiple initial feature maps of the target training image are extracted through the first feature extraction network; the scales between the multiple initial feature maps are different; The feature map is fused to obtain a fused feature map; the fused feature map is input to the first output network, and the candidate region of the text region in the target training image and the probability value of each candidate region are output; the candidate region is determined by the preset detection loss function. region and the first loss value of the probability value of each candidate region; the first initial model is trained according to the first loss value, until the parameters in the first initial model converge, and the text detection model is obtained.

In some embodiments, the above-mentioned first feature extraction network includes a plurality of groups of first convolutional networks connected in sequence; each group of first convolutional networks includes a convolutional layer, a batch normalization layer, and an activation function layer that are connected in sequence.

In some embodiments, the above-mentioned step of fusing multiple initial feature maps through a feature fusion network to obtain a fused feature map includes: arranging the multiple initial feature maps in sequence according to the scale of the initial feature maps; The scale of the initial feature map of the bottom level is the smallest; the scale of the initial feature map of the bottom level is the largest; the initial feature map of the top level is determined as the fusion feature map of the top level; except for the top level, the initial feature map of the current level The image is fused with the fusion feature map of the previous level of the current level to obtain the fusion feature map of the current level; the fusion feature map of the lowest level is determined as the final fusion feature map.

In some embodiments, the above-mentioned first output network includes a first convolutional layer and a second convolutional layer; the above-mentioned inputting the fusion feature map to the first output network outputs the candidate regions of the text region in the target training image and each candidate region The step of obtaining the probability value of the region includes: inputting the fusion feature map into the first convolution layer and the second convolution layer respectively; performing a first convolution operation on the fusion feature map through the first convolution layer, and outputting a coordinate matrix; The matrix includes the vertex coordinates of the candidate regions of the text region in the target training image; the second convolution operation is performed on the fusion feature map through the second convolution layer to output a probability matrix; the probability matrix includes the probability value of each candidate region.

In some embodiments, the above-mentioned detection loss function includes a first function and a second function; the above-mentioned first function is L ₁ =|G ^* -G|; wherein, G ^* is the coordinate of the text area in the pre-labeled target training image matrix; G is the coordinate matrix of the candidate region of the text region in the target training image output by the first output network; the above-mentioned second function is L ₂ =-Y ^* logY-(1-Y ^* )log(1-Y); wherein , Y ^* is the probability matrix of the text region in the pre-labeled target training image; Y is the probability matrix of the candidate region of the text region in the target training image output by the first output network; log represents the logarithmic operation; the above candidate regions and each The first loss value L=L ₁ +L ₂ of the probability value of the candidate region.

In some embodiments, the above-mentioned step of training the first initial model according to the first loss value until the parameters in the first initial model converge to obtain the text detection model includes: updating the first initial model according to the first loss value determine whether the updated parameters are converged; if the updated parameters are converged, the first initial model after parameter update is determined as the detection model; if the updated parameters are not converged, continue to perform the training based on the preset Sets the steps to determine the target training image until the updated parameters converge.

In some embodiments, the above-mentioned step of updating parameters in the first initial model according to the first loss value includes: according to preset rules, determining parameters to be updated from the first initial model; calculating the first loss value for the first initial model the derivative of the parameter to be updated in Among them, L is the first loss value; W is the parameter to be updated; update the parameter to be updated to obtain the updated parameter to be updated Among them, α is a preset coefficient.

In a second aspect, an embodiment of the present invention provides a method for determining a text area, the method includes: acquiring an image to be detected; inputting the image to be detected into a pre-trained text detection model, and outputting a plurality of text areas in the image to be detected The candidate area, and the probability value of each candidate area; the text detection model is obtained by training the above-mentioned text detection model training method; according to the probability value of the candidate area and the degree of overlap between multiple candidate areas, it is determined from multiple candidate areas The text area in the image to be detected.

In some embodiments, the above-mentioned step of determining the text region in the image to be detected from the multiple candidate regions according to the probability value of the candidate region and the degree of overlap between the multiple candidate regions includes: according to the probability value of the candidate region, Arrange multiple candidate regions in order; among them, the probability value of the first candidate region is the largest, and the probability value of the last candidate region is the smallest; the first candidate region is used as the current candidate region, and the current candidate region and the current candidate region are calculated one by one. The degree of overlap of candidate regions other than the current candidate region; the candidate regions whose overlap degree is greater than the preset overlap threshold in the candidate regions other than the current candidate region are eliminated; the next candidate region of the current candidate region is regarded as the new current candidate region, and the execution continues Steps of calculating the degree of overlap between the current candidate region and the candidate regions other than the current candidate region one by one, until the last candidate region is reached; the remaining candidate regions after culling are determined as the text regions in the image to be detected.

In some embodiments, before the step of arranging the multiple candidate regions in sequence according to the probability values of the candidate regions, the method further includes: removing candidate regions whose probability value is lower than a preset probability threshold from the multiple candidate regions , to get the final multiple candidate regions.

In a third aspect, an embodiment of the present invention provides a method for determining text content. The method includes: obtaining a text area in an image through the above-mentioned text area determining method; inputting the text area into a pre-trained text recognition model, and outputting the text The recognition result of the area; the text content in the text area is determined according to the recognition result.

In some embodiments, before the above step of inputting the text area into the pre-trained recognition model, the above method further includes: normalizing the text area according to a preset size.

In some embodiments, the above-mentioned text recognition model is trained by the following methods: determining a target training text image based on a preset training set; inputting the target training text image into a second initial model; the second initial model includes a second feature extraction network, feature splitting network, second output network and classification function; extract feature map of target training text image through second feature extraction network; split feature map into at least one sub-feature map through feature splitting network; sub-feature The image is input to the second output network respectively, and the output matrix corresponding to each sub-feature map is output; the output matrix corresponding to each sub-feature map is input to the classification function respectively, and the probability matrix corresponding to each sub-feature map is output; through the preset recognition loss The function determines the second loss value of the probability matrix; the second initial model is trained according to the second loss value until the parameters in the second initial model converge, and a text recognition model is obtained.

In some embodiments, the above-mentioned second feature extraction network includes a plurality of groups of second convolution networks connected in sequence; each group of second convolution networks includes a convolution layer, a pooling layer, and an activation function layer connected in sequence.

In some embodiments, the above step of splitting the feature map into at least one sub-feature map through the feature splitting network includes: splitting the feature map into at least one sub-feature map along the column direction of the feature map; the feature map The column direction is the vertical direction of the text line direction.

In some embodiments, the above-mentioned second output network includes a plurality of fully connected layers; the number of fully connected layers corresponds to the number of sub-feature maps; the sub-feature maps are respectively input to the second output network, and the corresponding sub-feature maps are output. The step of outputting the matrix includes: inputting each sub-feature map into the corresponding fully-connected layer, so that each fully-connected layer outputs an output matrix corresponding to the sub-feature map.

In some embodiments, the above-mentioned classification function includes a Softmax function; the Softmax function is Wherein, e represents a natural constant; t represents the t-th probability matrix; K represents the number of different characters contained in the target training text image of the training set; m represents from 1 to K+1; ∑ represents a summation operation; is the i-th element in the output matrix; the is the ith element in the probability matrix p ^t .

In some embodiments, the above-mentioned recognition loss function includes L=-log p(y|{p _t } _t=1...T ); wherein, y is the pre-labeled probability matrix of the target training text image; t represents the first t probability matrices; p ^t is the probability matrix corresponding to each of the sub-feature maps output by the classification function; T is the total number of the probability matrices; p represents the calculation probability; log represents the logarithmic operation.

In some embodiments, the above-mentioned step of training the second initial model according to the second loss value until the parameters in the second initial model converge to obtain the text recognition model includes: updating the second initial model according to the second loss value determine whether the updated parameters are all converged; if the updated parameters are all converged, the second initial model after parameter updating is determined as the text recognition model; if the updated parameters are not all converged, continue to execute the The training set determines the steps of the target training text image until the updated parameters converge.

In some embodiments, the above step of updating each parameter in the second initial model according to the second loss value includes: determining the parameter to be updated from the second initial model according to a preset rule; calculating the derivative of the parameter to be updated by the second loss value Among them, L' is the loss value of the probability matrix; W' is the parameter to be updated; update the parameter to be updated to obtain the updated parameter to be updated Among them, α' is a preset coefficient.

In some embodiments, the recognition result of the text area includes a plurality of probability matrices corresponding to the text area; the step of determining the text content in the text area according to the recognition result includes: determining the position of the maximum probability value in each probability matrix; From the correspondence between each position and the character in the preset probability matrix, obtain the character corresponding to the position with the maximum probability value; arrange the obtained characters according to the arrangement order of the plurality of probability matrices; determine the characters in the text area according to the arranged characters text content.

In some embodiments, the above-mentioned step of determining the text content in the text area according to the arranged characters includes: according to a preset rule, deleting repeated characters and null characters in the arranged characters to obtain the text content in the text area.

In some embodiments, after the above step of determining the text content in the text area according to the recognition result, the method further includes: if the image contains multiple text areas, acquiring the text content in each text area; The thesaurus determines whether the textual content corresponding to the image contains sensitive information.

In some embodiments, the above-mentioned step of determining whether the text content corresponding to the image contains sensitive information by using a pre-established sensitive thesaurus includes: performing a word segmentation operation on the acquired text content; The pre-established sensitive thesaurus is matched; if at least one word segmentation is successfully matched, it is determined that the text content corresponding to the image contains sensitive information.

In some embodiments, after it is determined that the text content corresponding to the image contains sensitive information, the above method further includes: acquiring the text area to which the successfully matched word segmentation belongs, and identifying the acquired text area in the image, or the successfully matched text area. Participle.

In a fourth aspect, an embodiment of the present invention provides a text detection model training device, the device includes: a training image determination module for determining a target training image based on a preset training set; a training image input module for The image is input to the first initial model; the first initial model includes a first feature extraction network, a feature fusion network and a first output network; a feature extraction module is used to extract multiple initial feature maps of the target training image through the first feature extraction network ; The scales between multiple initial feature maps are different; the feature fusion module is used to fuse multiple initial feature maps through the feature fusion network to obtain a fusion feature map; the output module is used to input the fusion feature map to the first The output network outputs the candidate regions of the text region in the target training image and the probability value of each candidate region; the loss value determination and training module is used to determine the candidate region and the probability value of each candidate region through the preset detection loss function. The first loss value; the first initial model is trained according to the first loss value, until the parameters in the first initial model converge, and a text detection model is obtained.

In some embodiments, the above-mentioned first feature extraction network includes a plurality of groups of first convolutional networks connected in sequence; each group of first convolutional networks includes a convolutional layer, a batch normalization layer, and an activation function layer that are connected in sequence.

In some embodiments, the above-mentioned feature fusion module is further configured to: arrange multiple initial feature maps in sequence according to the scale of the initial feature map; wherein, the scale of the initial feature map of the topmost level is the smallest; the initial feature map of the bottommost level The scale is the largest; the initial feature map of the top level is determined as the fusion feature map of the top level; except for the top level, the initial feature map of the current level and the fusion feature map of the previous level of the current level are fused to obtain The fusion feature map of the current level; the fusion feature map of the lowest level is determined as the final fusion feature map.

In some embodiments, the above-mentioned first output network includes a first convolutional layer and a second convolutional layer; the above-mentioned output module is further configured to: input the fusion feature map to the first convolutional layer and the second convolutional layer respectively; The first convolution operation is performed on the fusion feature map through the first convolution layer, and a coordinate matrix is output; the coordinate matrix includes the vertex coordinates of the candidate region of the text region in the target training image; The convolution operation outputs a probability matrix; the probability matrix includes the probability value of each candidate region.

In some embodiments, the above-mentioned detection loss function includes a first function and a second function; the first function is L ₁ =|G ^* -G|; wherein, G ^* is the coordinates of the text area in the pre-labeled target training image matrix; G is the coordinate matrix of the candidate region of the text region in the target training image output by the first output network; the second function is L ₂ =-Y ^* logY-(1-Y ^* )log(1-Y); wherein , Y ^* is the probability matrix of the text region in the pre-labeled target training image; Y is the probability matrix of the candidate region of the text region in the target training image output by the first output network; log represents the logarithmic operation; the above candidate regions and each The first loss value L=L ₁ +L ₂ of the probability value of the candidate region.

In some embodiments, the above-mentioned loss value determination and training module is further used for: updating the parameters in the first initial model according to the first loss value; judging whether the updated parameters are all converged; The updated first initial model is determined as the detection model; if the updated parameters do not all converge, the step of determining the target training image based on the preset training set is continued until the updated parameters converge.

In some embodiments, the above-mentioned loss value determination and training module is further used to: determine the parameter to be updated from the first initial model according to a preset rule; calculate the derivative of the first loss value to the parameter to be updated in the first initial model Among them, L is the first loss value; W is the parameter to be updated; update the parameter to be updated to obtain the updated parameter to be updated Among them, α is a preset coefficient.

In a fifth aspect, an embodiment of the present invention provides an apparatus for determining a text area, the apparatus includes: an image acquisition module, used to acquire an image to be detected; a detection module, used to input the to-be-detected image into a pre-trained text detection model , output multiple candidate regions of the text region in the image to be detected, as well as the probability value of each candidate region; the text detection model is obtained by training the above-mentioned text detection model training method; the text region determination module is used to determine the probability value of the candidate region according to the and the degree of overlap between the multiple candidate regions, the text region in the image to be detected is determined from the multiple candidate regions.

In some embodiments, the above-mentioned text region determination module is further configured to: arrange multiple candidate regions in sequence according to the probability values of the candidate regions; wherein, the probability value of the first candidate region is the largest, and the probability value of the last candidate region is the smallest ; Take the first candidate region as the current candidate region, and calculate the degree of overlap between the current candidate region and the candidate regions other than the current candidate region one by one; Eliminate candidate regions; take the next candidate region of the current candidate region as the new current candidate region, and continue to perform the steps of calculating the degree of overlap between the current candidate region and the candidate regions other than the current candidate region one by one, until the last candidate region is reached; The remaining candidate regions after culling are determined as text regions in the image to be detected.

In some embodiments, the above-mentioned apparatus further includes: a region elimination module, configured to eliminate candidate regions whose probability value is lower than a preset probability threshold from the plurality of candidate regions to obtain the final plurality of candidate regions.

In a sixth aspect, an embodiment of the present invention provides an apparatus for determining text content, the apparatus comprising: an area acquisition module, configured to acquire a text area in an image through the above-mentioned text area determination method; a recognition module, used to input the text area into To the pre-trained text recognition model, the recognition result of the text area is output; the text content determination module is used to determine the text content in the text area according to the recognition result.

In some embodiments, the above apparatus further includes: a normalization module, configured to perform normalization processing on the text area according to a preset size.

In some embodiments, the above-mentioned apparatus further includes a text recognition model training module, configured to train the text recognition model by the following methods: determining a target training text image based on a preset training set; inputting the target training text image into a second an initial model; the second initial model includes a second feature extraction network, a second output network and a classification function; the feature map of the target training text image is extracted through the second feature extraction network; the feature map is split into at least one Sub-feature map; input the sub-feature maps to the second output network respectively, and output the output matrix corresponding to each sub-feature map; input the output matrix corresponding to each sub-feature map to the classification function respectively, and output the probability matrix corresponding to each sub-feature map ; Determine the second loss value of the probability matrix by the preset recognition loss function; train the second initial model according to the second loss value, until the parameters in the second initial model converge, and obtain the text recognition model.

In some embodiments, the above-mentioned second feature extraction network includes a plurality of groups of second convolution networks connected in sequence; each group of second convolution networks includes a convolution layer, a pooling layer, and an activation function layer connected in sequence.

In some embodiments, the above recognition model training module is further configured to: split the feature map into at least one sub-feature map along the column direction of the feature map; the column direction of the feature map is the vertical direction of the text line direction.

In some embodiments, the above-mentioned second output network includes multiple fully-connected layers; the number of fully-connected layers corresponds to the number of sub-feature maps; the recognition model training module is further configured to: input each sub-feature map to the corresponding full-feature map respectively. In the connection layer, so that each fully connected layer outputs the output matrix corresponding to the sub-feature map.

In some embodiments, the above-mentioned classification function includes a Softmax function; the Softmax function is Wherein, e represents a natural constant; t represents the t-th probability matrix; K represents the number of different characters contained in the target training text image of the training set; m represents from 1 to K+1; ∑ represents a summation operation; is the i-th element in the output matrix; the is the ith element in the probability matrix p ^t .

In some embodiments, the above-mentioned recognition loss function includes L=-log p(y|{p _t } _t=1...T ); wherein, y is the pre-labeled probability matrix of the target training text image; t represents the first t probability matrices; p ^t is the probability matrix corresponding to each of the sub-feature maps output by the classification function; T is the total number of the probability matrices; p represents the calculation probability; log represents the logarithmic operation.

In some embodiments, the above-mentioned recognition model training module is further used for: updating the parameters in the second initial model according to the second loss value; judging whether all the updated parameters converge; The updated second initial model is determined to be a text recognition model; if the updated parameters do not converge, continue to perform the step of determining the target training text image based on the preset training set until the updated parameters converge.

In some embodiments, the above-mentioned recognition model training module is further configured to: determine the parameter to be updated from the second initial model according to a preset rule; calculate the derivative of the parameter to be updated from the second loss value Among them, L' is the loss value of the probability matrix; W' is the parameter to be updated; update the parameter to be updated to obtain the updated parameter to be updated Among them, α' is a preset coefficient.

In some embodiments, the recognition result of the text area includes a plurality of probability matrices corresponding to the text area; the text content determination module is further configured to: determine the position of the maximum probability value in each probability matrix; In the correspondence between each position and character in the , obtain the character corresponding to the position with the maximum probability value; arrange the obtained characters according to the arrangement order of multiple probability matrices; determine the text content in the text area according to the arranged characters.

In some embodiments, the above-mentioned text content determination module is further configured to: delete repeated characters and null characters in the arranged characters according to a preset rule to obtain the text content in the text area.

In some embodiments, the above-mentioned apparatus further includes: an information acquisition module, configured to acquire the text content in each text area if the image contains multiple text areas; a sensitive information determination module, configured to obtain the text content in each text area through a pre-established sensitive word The library determines whether the textual content corresponding to the image contains sensitive information.

In some embodiments, the above-mentioned sensitive information determination module is further configured to: perform a word segmentation operation on the acquired text content; match the word segmentation obtained after the word segmentation operation with a pre-established sensitive word database one by one; if at least one word segmentation is successfully matched, Determine that the text content corresponding to the image contains sensitive information.

In some embodiments, the above-mentioned apparatus further includes: a region identification module, configured to acquire the text region to which the successfully matched word segmentation belongs, and identify the acquired text region in the image.

In a seventh aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory, the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the above text detection model training method , the steps of the above text area determination method, or the above text content determination method.

In an eighth aspect, embodiments of the present invention provide a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are invoked and executed by a processor, the machine-executable instructions cause the The processor implements the steps of the above text detection model training method, the above text area determination method, or the above text content determination method.

The embodiments of the present invention have brought the following beneficial effects:

The text detection model training method provided by the embodiment of the present invention firstly extracts multiple initial feature maps with different scales of the target training image; then fuses the multiple initial feature maps to obtain a fused feature map; and then inputs the fused feature map into To the first output network, output the candidate region of the text region in the target training image and the probability value of each candidate region; after determining the first loss value through the preset detection loss function, according to the first loss value to the first initial model Perform training to obtain a detection model. In this method, the feature extraction network can automatically extract features of different scales. Therefore, the text detection model only needs to input an image to obtain the candidate regions of text regions of various scales in the image, and no need to manually transform the image scale. The operation is convenient, especially in the scenarios of multiple font sizes, multiple fonts, multiple shapes, and multiple orientations, it can quickly, comprehensively and accurately detect various texts in the image, which is also conducive to the accuracy of subsequent text recognition, improving the The effect of text recognition.

Additional features and advantages of the present invention will be set forth in the description which follows, or some may be inferred or unambiguously determined from the description, or may be learned by practicing the above-described techniques of the present invention.

In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the preferred embodiments are exemplified below, and are described in detail as follows in conjunction with the accompanying drawings.

Description of drawings

In order to illustrate the specific embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the specific embodiments or the prior art. Obviously, the accompanying drawings in the following description The drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

1 is a flowchart of a method for training a text detection model according to an embodiment of the present invention;

2 is a schematic structural diagram of a first feature extraction network provided by an embodiment of the present invention;

3 is a schematic diagram of performing fusion processing on multiple initial feature maps according to an embodiment of the present invention;

4 is a flowchart of a method for determining a text area provided by an embodiment of the present invention;

5 is a flowchart of another method for determining a text area provided by an embodiment of the present invention;

6 is a flowchart of a method for determining text content according to an embodiment of the present invention;

7 is a flowchart of a method for training a text recognition model according to an embodiment of the present invention;

8 is a schematic structural diagram of a second feature extraction network provided by an embodiment of the present invention;

9 is a flowchart of another text content determination method provided by an embodiment of the present invention;

10 is a flowchart of another method for determining text content provided by an embodiment of the present invention;

11 is a schematic structural diagram of an apparatus for training a text detection model according to an embodiment of the present invention;

12 is a schematic structural diagram of an apparatus for determining a text area according to an embodiment of the present invention;

13 is a schematic structural diagram of an apparatus for determining text content according to an embodiment of the present invention;

FIG. 14 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of them. example. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In the traditional text recognition technology, the text area where the text may exist is detected from the picture through the rules set by humans, and then the detected text area is characterized to obtain the image block corresponding to each character. The classifier recognizes each image patch, and then obtains the final text recognition result. In this method, due to the limited number of artificially set rules, most of the detected text areas are regular-shaped areas, and the application range is limited, which is difficult to apply to text detection and recognition in complex scenarios, such as multiple font sizes, In addition, this method recognizes a single character without considering the correlation between characters, resulting in poor detection and recognition effect in complex scenes.

In addition, text recognition can also be achieved through deep learning; first, the recognition model needs to be trained through a recurrent neural network; then the images to be detected are transformed into various scales, and input into the recognition model one by one to detect the text area and recognize the text; this method It is necessary to manually transform the image scale, and input images of various scales into the recognition model, so that the recognition model can recognize texts of different sizes, which is cumbersome and difficult to meet the needs of real-time recognition. The recursive operation of time series is difficult to process in parallel, and the operation speed is slow. In addition, this recognition model usually uses a rectangular detection frame to detect text areas, so it can only detect and recognize text in the horizontal direction, and the recognition effect for text at any angle is poor, making it difficult to apply to text detection and recognition in complex scenes.

To sum up, the text detection and recognition methods in the related art have poor effects in complex scenarios; based on this, the embodiments of the present invention provide a text detection model training method, a text area, and a content determination method and device; the technology can be widely used in Text detection and text recognition in various scenarios, especially text detection and text recognition in complex scenarios such as online live broadcast, limited TV live broadcast, games, and videos.

In order to facilitate the understanding of this embodiment, a method for training a text detection model disclosed in the embodiment of the present invention is first introduced in detail. The text detection model can be used for text detection, and the file detection can be understood as: locating files containing Image area with text. As shown in Figure 1, the method includes the following steps:

Step S102, determining a target training image based on a preset training set.

The training set can contain multiple images. In order to improve the wide application of the detection model, the images in the training set can contain images in various scenarios, such as live scene images, game scene images, outdoor scene images, and indoor scenes. Images, etc.; the images in the training set can also contain text lines of various font sizes, shapes, fonts, and languages, so that the trained detection model can detect various text lines.

Each image contains a text area of manually annotated text lines, which can be marked by quadrilateral boxes such as rectangles, or by other polygonal boxes; the marked text area can usually completely cover the entire text line, and The text area and the text line can fit tightly. In addition, the plurality of images in the training set may also be divided into training subsets and test subsets according to a preset ratio. During training, target training images can be obtained from the training subset. After training, target test images can be obtained from the test subset to test the performance of the detection model.

Step S104, input the target training image into a first initial model; the first initial model includes a first feature extraction network, a feature fusion network and a first output network.

Before input to the first initial model, the target training image can be resized to a preset size, such as 512*512.

Step S106, extracting multiple initial feature maps of the target training image through the first feature extraction network; the scales between the multiple initial feature maps are different.

Among them, the first feature extraction network can be implemented by multi-layer convolutional layers. Generally, the multi-layered convolutional layers are connected in sequence, and each layer of convolutional layers is set with different convolution kernels to extract feature maps of different scales. Multiple initial feature maps of the target training image, each initial feature map can be calculated by convolution of the corresponding convolution layer. Taking four convolutional layers as an example, each convolutional layer can output an initial feature map; each convolutional layer can be set with convolution kernels of different sizes, so that the scales of the initial feature maps output by each convolutional layer are different. . In actual implementation, the scale of the initial feature map output by the convolutional layer of the input target training image can be set to be the largest, and the scale of the initial feature map output by each subsequent convolutional layer is gradually reduced.

Step S108, performing fusion processing on a plurality of the initial feature maps through the feature fusion network to obtain a fusion feature map.

Generally, smaller convolution kernels can sense high-frequency features in images, and the initial feature maps output by convolutional networks using smaller convolution kernels carry small-scale text line features; larger convolution kernels can Sensing the low-frequency features in the image, the initial feature map output by the convolutional layer of the larger convolutional network carries large-scale text line features; based on this, multiple initial feature maps of different scales carry various scales The fused feature map obtained by fusing multiple initial feature maps also carries text line features of various scales. In this way, the detection model can detect text lines of various scales without artificially performing image scale transformation before detection.

In actual implementation, since the scales of multiple initial feature maps are different, before the fusion, the initial feature maps of smaller scales can be interpolated to expand the initial feature maps of smaller scales to match the larger ones. match the initial feature map. During the fusion process, between different initial feature maps, the feature points at the same position can be multiplied or added to obtain the final fusion feature map.

Step S110, input the fusion feature map to the first output network, and output the candidate regions of the text region in the target training image and the probability value of each candidate region.

The first output network is used to extract the required features from the fusion feature map to obtain the output result; if the output result of the detection model is the only result, the first output network usually includes a group of networks; if the output result of the detection model is the only result For multiple results, the first output network usually includes multiple groups of networks, the multiple groups of networks are arranged in parallel, and each group of networks outputs a corresponding result. The first output network can be composed of convolutional layers or fully connected layers. In the above steps, the first output network needs to output two results, the candidate region and the probability value of the candidate region. Therefore, the first output network may include two groups of networks, and each group of networks may be a convolutional network or a fully connected network.

Step S112: Determine the first loss value of the above-mentioned candidate area and the probability value of each candidate area through a preset detection loss function; train the first initial model according to the first loss value until the parameters in the first initial model are Convergence to get the text detection model.

A standard text area is pre-marked in the target training image, and a coordinate matrix of the text area and a probability matrix of the text area can be generated based on the position of the marked text area; wherein, the coordinate matrix of the text area contains the vertices of the standard text area Coordinates; the probability matrix of the text area contains the probability value of the text area, which is usually 1.

The detection loss function can compare the difference between the coordinate matrix of the candidate area and the coordinate matrix of the standard text area, and the difference between the probability value of the candidate area and the probability value of the standard text area. Generally, the greater the difference, the greater the above-mentioned first loss value. . Based on the first loss value, the parameters of each part of the above-mentioned first initial model can be adjusted to achieve the purpose of training. When each parameter in the model converges, the training ends, and the detection model is obtained.

The text detection model training method provided by the embodiment of the present invention firstly extracts multiple initial feature maps with different scales of the target training image; then fuses the multiple initial feature maps to obtain a fused feature map; and then inputs the fused feature map into To the first output network, output the candidate region of the text region in the target training image and the probability value of each candidate region; after determining the first loss value through the preset detection loss function, according to the first loss value to the first initial model Perform training to obtain a detection model. In this method, the feature extraction network can automatically extract features of different scales. Therefore, the text detection model only needs to input an image to obtain the candidate regions of text regions of various scales in the image, and no need to manually transform the image scale. The operation is convenient, especially in the scenarios of multiple font sizes, multiple fonts, multiple shapes, and multiple orientations, it can quickly, comprehensively and accurately detect various texts in the image, which is also conducive to the accuracy of subsequent text recognition, improving the The effect of text recognition.

The embodiment of the present invention also provides another text detection model training method, which is implemented on the basis of the method described in the above-mentioned embodiment; the method focuses on describing the specific implementation process of each step in the above-mentioned training method; the method includes the following steps:

Step 202: Determine a target training image based on a preset training set.

Step 204 , input the target training image into a first initial model; the first initial model includes a first feature extraction network, a feature fusion network and a first output network.

Step 206: Extract multiple initial feature maps of the target training image through the first feature extraction network; the scales of the multiple initial feature maps are different.

In actual implementation, in order to improve the performance of the first feature extraction network, the first feature extraction network may include multiple groups of first convolutional networks connected in sequence; each group of first convolutional networks includes sequentially connected convolutional layers, batch Normalization layer and activation function layer. Fig. 2 shows a schematic structural diagram of a first feature extraction network; in Fig. 2, four groups of first convolutional networks are used as an example to illustrate, and the convolutional layers of the latter group of first convolutional networks are connected to the former group of first convolutional networks. The activation function layer of the convolutional network. In addition, the first feature extraction network may further include more or less groups of first convolutional networks.

The batch normalization layer in the first convolutional network is used to normalize the feature map output by the convolutional layer. This process can speed up the convergence speed of the first feature extraction network and the detection model, and can alleviate the multi-layer The problem of gradient dispersion in convolutional networks makes the first feature extraction network more stable. The activation function layer in the first convolutional network can perform functional transformation on the normalized feature map, and the transformation process breaks the linear combination of the input of the convolutional layer, which can improve the feature expression ability of the first convolutional network. The activation function layer may specifically be a Sigmoid function, a tanh function, a Relu function, or the like.

Step 208: Perform fusion processing on a plurality of the initial feature maps through the feature fusion network to obtain a fusion feature map.

The following steps 02-08 provide a specific implementation of step 208. In this method, the pyramid feature is used as an example to illustrate, that is, the scale of the initial feature map output by each convolutional layer is sequentially reduced:

Step 02, according to the scale of the initial feature map, arrange the multiple initial feature maps in sequence; wherein, the scale of the initial feature map of the topmost level is the smallest; the scale of the initial feature map of the bottommost level is the largest;

Step 04, the initial feature map of the topmost level is determined as the fusion feature map of the topmost level;

Step 06, except for the topmost level, the initial feature map of the current level and the fusion feature map of the previous level of the current level are fused to obtain the fusion feature map of the current level;

Since the scale of the fusion feature map of the previous level of the current level is smaller than the initial feature map of the current level, before the two are fused, the scale of the fusion feature map of the previous level of the current level can be extended to the same size as the current level through interpolation operation. The scale of the initial feature map of the level is the same, and then the fusion processing of point-by-point addition or point-by-point multiplication is performed to obtain the fusion feature map of the current level.

Step 08: Determine the fusion feature map of the lowest level as the final fusion feature map.

Figure 3 shows a schematic diagram of fusion processing of multiple initial feature maps; the target training image is convolved by the first feature extraction network to obtain four layers of initial feature maps; the top-level initial feature map is used as the top-level The fusion feature map of the first level; the fusion feature map of the top level is fused with the initial feature map of the second level to obtain the fusion feature map of the second level; the fusion feature map of the second level is fused with the initial feature map of the third level , the fusion feature map of the third level is obtained; the fusion feature map of the third level is fused with the initial feature map of the fourth level to obtain the fusion feature map of the fourth level; the fusion feature map of the fourth level is the final fusion feature picture.

Step 210: Input the fusion feature map to the first output network, and output the candidate regions of the text region in the target training image and the probability value of each candidate region.

Taking a convolutional network as an example, the above-mentioned first output network includes a first convolutional layer and a second convolutional layer; wherein, the first convolutional layer and the second convolutional layer are arranged side by side, and the first convolutional layer and the second convolutional layer are arranged in parallel. The product layer is respectively used to output the vertex coordinates of the selected area and the probability value of the candidate area. The above step 210 can also be implemented by the following steps 12-16:

Step 12, input the fusion feature map to the first convolutional layer and the second convolutional layer respectively;

Step 14, performing the first convolution operation on the fusion feature map through the first convolution layer, and outputting a coordinate matrix; the coordinate matrix includes the vertex coordinates of the candidate region of the text region in the target training image;

For example, the coordinate matrix can be expressed as n*H*W, where H and W are the height and width of the coordinate matrix, respectively, and n is the dimension of the coordinate matrix; for example, when the candidate area is a quadrilateral, a candidate area needs to pass through four The vertex coordinates are determined, so n is 8; when the candidate area is other polygons, the value of n is usually twice the number of sides of the candidate area.

In step 16, a second convolution operation is performed on the fusion feature map through the second convolution layer, and a probability matrix is output; the probability matrix includes the probability value of each candidate region.

The probability value of each candidate region can also be referred to as the score of each candidate region, and the probability value can be used to represent the probability that the candidate region can completely contain text lines.

Step 212: Determine the first loss value of the above-mentioned candidate regions and the probability value of each candidate region through a preset detection loss function; train the first initial model according to the first loss value until the parameters in the first initial model are Convergence to get the text detection model.

In actual implementation, the above-mentioned detection loss function includes a first function and a second function, which are respectively used to calculate the vertex coordinates of the candidate regions and the loss value of the probability value of each candidate region; wherein, the first function is L ₁ =|G ^* -G|; wherein, G ^* is the coordinate matrix of the text area in the pre-marked target training image; G is the coordinate matrix of the candidate area of the text area in the target training image output by the first output network; the second function is L ₂ =-Y ^* logY-(1-Y ^* )log(1-Y); wherein, Y ^* is the probability matrix of the text area in the pre-labeled target training image; Y is the text in the target training image output by the first output network Probability matrix of candidate regions for regions; log represents a logarithmic operation. The first loss value of the vertex coordinates of the candidate regions and the probability value of each candidate region is the sum of the first function and the second function, that is, L=L ₁ +L ₂ .

Based on the above description of the first loss value, in the above steps, the process of training the first initial model according to the first loss value can also be implemented through the following steps 22-28:

Step 22, update the parameters in the first initial model according to the first loss value;

In actual implementation, the function mapping relationship can be preset, and the original parameters and the first loss value are input into the function mapping relationship, and then the updated parameters can be calculated. The function mapping relationship of different parameters can be the same or different.

Specifically, the parameters to be updated can be determined first according to preset rules; the parameters to be updated can be all parameters in the first initial model, or some parameters randomly determined from the first initial model; and then the first loss is calculated Derivative of the value to the parameter to be updated in the first initial model Among them, L is the first loss value; W is the parameter to be updated; the parameter to be updated may also be called the weight of each neuron. This process can also be called a back-propagation algorithm; if the first loss value is large, it means that the output of the current first initial model does not match the expected output result, and then the above-mentioned first loss value is found to be the difference between the first initial model and the expected output. The derivative of the update parameter, which can be used as the basis for adjusting the parameter to be updated.

After obtaining the derivative of each parameter to be updated, update each parameter to be updated to obtain the updated parameter to be updated Among them, α is a preset coefficient. This process can also be called a stochastic gradient descent algorithm; the derivative of each parameter to be updated can also be understood as the direction in which the first loss value decreases the fastest relative to the current parameter. By adjusting the parameters in this direction, the first loss value can be quickly reduced. , so that the parameter converges. In addition, after the first initial model is trained once, a first loss value is obtained. At this time, one or more parameters can be randomly selected from the parameters in the first initial model to perform the above update process. The model training time of this method is Shorter, the algorithm is faster; of course, the above update process can also be performed on all parameters in the first initial model, and the model training in this way is more accurate.

Step 24, determine whether the updated parameters are all converged; if the updated parameters are converged, go to step 26; if the updated parameters are not converged, go to step 28;

Step 26: Determine the first initial model after parameter update as the detection model; end.

Step 28: Continue to perform the step of determining the target training image based on the preset training set until all the updated parameters converge.

Specifically, a new image can be re-acquired from the training set as the target training image, or the current target training image can be continued to be used as the target training image for training.

In the above method, the feature extraction network can automatically extract feature maps of different scales, and then fuse the feature maps of different scales, and obtain candidate regions of text regions of various scales in the image based on the obtained fused feature maps. The detection model only needs to input an image to obtain the candidate regions of the text regions of various scales in the image, no need to manually transform the image scale, and the operation is convenient, especially in the case of multiple font sizes, multiple fonts, multiple shapes, In a variety of orientation scenarios, various texts in the image can be detected quickly, comprehensively and accurately, which is also beneficial to the accuracy of subsequent text recognition and improves the effect of text recognition.

Based on the text detection model training method provided in the foregoing embodiment, the embodiment of the present invention further provides a text region determination method, which is implemented on the basis of the text detection model training method described in the foregoing embodiment; as shown in FIG. 4 , The method includes the following steps:

Step S402, acquiring an image to be detected; the image to be detected may be a picture, or a video frame intercepted from a video file or a live video, or the like.

Step S404, input the image to be detected into the pre-trained text detection model, output multiple candidate regions of the text region in the image to be detected, and the probability value of each candidate region; the text detection model passes the training of the above-mentioned text detection model method training;

Step S406, according to the probability value of the candidate area and the degree of overlap between the multiple candidate areas, determine the text area in the image to be detected from the multiple candidate areas.

Among the candidate regions output by the above text detection model, there may be multiple candidate regions corresponding to the same text line; in order to find the region that best matches the text line from the multiple candidate regions, it is necessary to screen the multiple candidate regions. In most cases, multiple candidate regions with a high degree of mutual overlap usually correspond to the same text line, and then according to the probability values of the multiple candidate regions with a high degree of mutual overlap, the text region corresponding to the text line can be determined. ; For example, among multiple candidate regions with a high degree of mutual overlap, the candidate region with the largest probability value is determined as the text region. If there are multiple lines of text in the image, then multiple text areas are usually finalized.

The above-mentioned text region determination method provided by the embodiment of the present invention inputs the acquired image to be detected into the text detection model, and outputs multiple candidate regions of the text region in the to-be-detected image and the probability value of each candidate region; and then according to the candidate region The probability value of , and the degree of overlap between multiple candidate regions, determine the text region in the image to be detected from the multiple candidate regions. In this method, the text detection model can automatically extract features of different scales, so it is only necessary to input an image to the model to obtain candidate regions of text regions of various scales in the image, without the need to manually transform the image scale, and the operation is convenient , especially in the scenes of multiple font sizes, multiple fonts, multiple shapes, and multiple orientations, it can quickly, comprehensively and accurately detect various texts in the image, which is also beneficial to the accuracy of subsequent text recognition and improves text recognition. Effect.

The embodiment of the present invention also provides another method for determining a text area, which is implemented on the basis of the method described in the above-mentioned embodiment; the method focuses on the determination of the to-be-to-be-to-be-determined according to the vertex coordinates of the candidate area output by the detection network and the probability value of the candidate area The specific process of detecting the text area in the image; as shown in Figure 5, the method includes the following steps:

Step S502, acquiring an image to be detected.

Step S504, input the image to be detected into the pre-trained text detection model, output multiple candidate regions of the text region in the image to be detected, and the probability value of each candidate region;

Step S506 , among the plurality of candidate regions, candidate regions whose probability value is lower than a preset probability threshold are eliminated to obtain a plurality of final candidate regions.

This step S506 is an optional step, that is, in the following step S508, each candidate region output by the detection model may be arranged, or candidates whose probability value is lower than a preset probability threshold in the candidate regions output by the detection model may be first Region culling, and then arranging the remaining candidate regions. The above-mentioned preset probability threshold can be preset, such as 0.2, 0.1, etc.; by eliminating candidate regions whose probability value is lower than the preset probability threshold, it is beneficial to reduce the calculation amount of subsequent determination of the text region in the image to be detected, and improve the calculation speed .

Step S508, according to the probability value of the candidate region, arrange the multiple candidate regions in sequence; wherein, the probability value of the first candidate region is the largest, and the probability value of the last candidate region is the smallest;

Step S510, taking the first candidate region as the current candidate region, and calculating the degree of overlap between the current candidate region and the candidate regions other than the current candidate region one by one;

Candidate regions other than the current candidate region can also be referred to as other candidate regions for short. When calculating the degree of overlap between the current candidate region and each other candidate region, the intersection ratio of the two candidate regions can be calculated specifically, and the intersection ratio is equal to two The area size of the intersection of the two candidate areas and the area size of the union of the two candidate areas. It can be understood that the larger the intersection and union ratio, the greater the degree of overlap between the two candidate regions. For the current candidate area, other candidate areas with a large degree of overlap with the current candidate area usually represent the same text line as the current candidate area, and because the probability value of other candidate areas is smaller than the current candidate area, the other candidate areas can be Candidate region culling to characterize the text line by the current candidate region.

Step S512: Eliminate candidate regions whose overlap degree is greater than a preset overlap threshold in candidate regions other than the current candidate region; the overlap threshold may be preset, such as 0.5, 0.6, and the like.

Step S514, taking the next candidate region of the current candidate region as the new current candidate region, and continuing to perform the step of calculating the overlap degree between the current candidate region and the candidate regions other than the current candidate region one by one, until the last candidate region is reached.

The above steps S510-S514 include a loop process, and some candidate regions will be eliminated in each round of loops. When the last candidate region is traversed, the loop ends, and the final remaining candidate regions are determined as the text regions in the image to be detected. If there are multiple candidate regions remaining in the end, it can be determined that there are multiple text regions in the image to be detected.

In step S516, the remaining candidate regions after the elimination are determined as the text regions in the image to be detected.

In the above manner, multiple candidate regions and the probability value of each candidate region can be obtained through the text detection model, and then the text region is determined from the multiple candidate regions by means of non-maximum suppression. In this method, the text detection model can automatically extract features of different scales, so it is only necessary to input an image to the model to obtain candidate regions of text regions of various scales in the image, without the need to manually transform the image scale, and the operation is convenient , especially in the scenes of multiple font sizes, multiple fonts, multiple shapes, and multiple orientations, it can quickly, comprehensively and accurately detect various texts in the image, which is also beneficial to the accuracy of subsequent text recognition and improves text recognition. Effect.

Based on the text area determination method provided by the foregoing embodiment, the embodiment of the present invention further provides a text content determination method, which is implemented on the basis of the text area determination method described in the foregoing embodiment; as shown in FIG. 6 , this method It includes the following steps:

Step S602, obtaining the text area in the image by the above-mentioned text area determination method;

Step S604, input the text area into the pre-trained text recognition model, and output the recognition result of the text area;

Step S606: Determine the text content in the text area according to the recognition result.

The above text recognition model can be trained in various ways, such as recurrent neural network, convolutional neural network, and of course, the recognition result of text area can also be obtained by means of optical character recognition. The recognition result output by the text recognition model can be determined as the text content in the text area, or the recognition result output by the text recognition model can be optimized first, such as deleting repeated characters, null characters, null characters, etc. The recognition result is determined as the text content in the text area.

In the text content determination method provided by the embodiment of the present invention, firstly, the text area in the image is obtained by the above-mentioned text area determination method; then the text area is input into the pre-trained text recognition model, and the recognition result of the text area is output; finally, according to the text area The recognition result determines text information in the text area. In this way, since the above-mentioned text area determination method can obtain text areas of various scales through the text detection model, it can quickly, comprehensively and accurately detect the scene of various font sizes, fonts, shapes and directions. Various types of text in the image are also beneficial to the accuracy of text recognition and improve the effect of text recognition.

The embodiments of the present invention also provide another method for determining text content, which is implemented on the basis of the methods described in the above embodiments; the method focuses on describing the training method of the text recognition model; the text recognition model can be used for text recognition, and the Text recognition can be understood as: detecting the text area in the picture, so as to locate the picture area containing the text, and then identify the specific content of the text in the picture area. As shown in Figure 7, the detection model is trained in the following ways:

Step S702, determining a target training text image based on a preset training set;

The target training text image can be a separate image or an image area marked on the image. The training set can contain multiple images. In order to improve the applicability of the text recognition model, the images in the training set can contain images from various scenes, such as live broadcast scene images, game scene images, outdoor scene images, indoor scene images, etc. Scene images, etc.; the images in the training set can also contain text lines of various font sizes, shapes, fonts, and languages, so that the trained text recognition model can detect various text lines. Each target training text image corresponds to the text content of the manually annotated text line, such as "Hello", "Awesome" and so on. Each target training text image corresponds to an annotated text content.

After the annotation is completed, a character library can also be established through the text content of all text lines corresponding to all images in the training set; specifically, the text content of all text lines corresponding to all images in the training set is obtained, and different characters, and group characters different from each other into a character library. In addition, the plurality of images in the training set may also be divided into training subsets and test subsets according to a preset ratio. During training, target training images can be obtained from the training subset. After training, target test images can be obtained from the test subset to test the performance of the text recognition model.

Step S704, input the target training text image into the second initial model; the second initial model includes a second feature extraction network, a feature splitting network, a second output network and a classification function;

Step S706, extracting the feature map of the target training text image through the second feature extraction network;

The second feature extraction network can be implemented by multi-layer convolution layers. Usually, the multi-layer convolution layers are connected in sequence. Each convolution layer sets the corresponding convolution kernel to perform convolution calculation on the input data. The last layer The data output by the convolutional layer can be used as the feature map of the target training text image.

Step S708, splitting the feature map into at least one sub-feature map through a feature splitting network;

Based on the purpose of recognizing text content, the text recognition model needs to split the feature map corresponding to the text line, so that each sub-feature map contains one or a small amount of words or symbols, which is convenient for text content recognition. In the splitting process, the scale of the sub-feature map can be preset, and the feature map can be split based on the scale of the sub-feature map; the number of sub-feature maps can also be preset, and the feature map can be split based on the number of the sub-feature maps. Of course, if the text line is inherently short, such as only one character, the feature map may also be split into only one sub-feature map.

Step S710, input the above-mentioned sub-feature maps to the second output network respectively, and output the output matrix corresponding to each sub-feature map;

The second output network is used to recalculate the sub-feature map; in the output matrix corresponding to each output sub-feature map, each position corresponds to a preset character; the value at the position can represent the relationship between the sub-feature map and the sub-feature map. The matching degree of the character corresponding to this position. The second output network can be a convolutional network or a fully connected network.

Step S712, input the output matrix corresponding to each sub-feature map to the classification function respectively, and output the probability matrix corresponding to each sub-feature map;

The classification function can map each value in the output matrix to a probability value to obtain a probability matrix. The probability value at each position in the probability matrix can be used to characterize the probability that the sub-feature map matches the character corresponding to the position.

Step S714: Determine the second loss value of the probability matrix by using a preset recognition loss function; train the above-mentioned second initial model according to the second loss value until the parameters in the second initial model converge to obtain a text recognition model.

The target training text image is pre-marked with standard text content, which consists of one or more standard characters; a probability matrix can be generated based on the text content; in the probability matrix, the position corresponding to the standard character corresponding to the sub-feature map is The probability value is 1, and the probability value for other positions is 0. The recognition loss function can compare the difference between the probability matrix output by the classification function and the probability matrix of standard text content. Generally, the greater the difference, the greater the above-mentioned second loss value. Based on the second loss value, the parameters of each part of the second initial model can be adjusted to achieve the purpose of training. When each parameter in the model converges, the training ends, and a text recognition model is obtained.

In the training method of the above-mentioned text recognition model, first extract the feature map of the target training text image; then split the feature map into at least one sub-feature map; and then input the sub-feature map to the second output network respectively, and output each sub-feature map. The output matrix corresponding to the feature map; then the probability matrix corresponding to each sub-feature map is obtained through the classification function; after the second loss value of the probability matrix is determined through the preset recognition loss function, the second initial model is performed according to the second loss value. Train a text recognition model. In this way, the model can automatically segment the feature map of the image, so the text recognition model only needs to input the image containing the text line to obtain the text content in the image, and there is no need to segment the text line. The text content of the text line can be directly obtained, the operation editing is performed, the operation speed is fast, and the recognition accuracy of the text is high.

The embodiment of the present invention also provides another training method for a text recognition model, which is implemented on the basis of the method described in the above-mentioned embodiment; the method focuses on describing the specific implementation process of each step in the above-mentioned training method; the method includes the following steps :

Step 802, determining a target training text image based on a preset training set;

Step 804, input the target training text image into a second initial model; the second initial model includes a second feature extraction network, a feature splitting network, a second output network and a classification function;

Step 806, extract the feature map of the target training text image through the second feature extraction network;

In order to improve the performance of the second feature extraction network, the second feature extraction network may include multiple groups of second convolutional networks connected in sequence; each group of second convolutional networks includes sequentially connected convolutional layers, pooling layers and activation functions Floor. Fig. 8 shows a schematic diagram of the structure of a second feature extraction network; in Fig. 8, four groups of second convolutional networks are used as examples for illustration, and the convolutional layers of the latter group of second convolutional networks are connected to the former group of second convolutional networks. The activation function layer of the convolutional network. The second feature extraction network may further include more or less groups of second convolutional networks.

It can be understood that the convolutional layer in the second convolutional network is used to extract features and generate feature maps; the pooling layer can be an average pooling layer (Average Pooling or mean-pooling), a global average pooling layer (Global Average Pooling) , max-pooling, etc.; the pooling layer can be used to compress the feature map output by the convolutional layer, retain the main features in the feature map, and delete non-main features to reduce the dimension of the feature map. Taking the average pooling layer as an example, the average pooling layer can average the feature point values in the neighborhood of the current feature point with a preset range size, and use the average value as the new feature point value of the current feature point. In addition, the pooling layer can also help the feature map to maintain some invariance, such as rotation invariance, translation invariance, scaling invariance, etc. The activation function layer can perform functional transformation on the feature map processed by the pooling layer. The transformation process breaks the linear combination of the input of the convolution layer, which can improve the feature expression ability of the second convolution network. The activation function layer may specifically be a Sigmoid function, a tanh function, a Relu function, or the like.

Step 808, splitting the feature map into at least one sub-feature map through a feature splitting network;

Considering that most text behaviors are arranged horizontally, in order to make the split sub-feature map contain one or a small number of features corresponding to characters, the feature map can be split into at least one sub-feature along the column direction of the feature map. Figure; the column direction of the feature map can be understood as the vertical direction of the text row direction. In actual implementation, the width of the sub-feature map is set according to the width of most characters, and the above-mentioned feature map is split according to the width. For example, the above feature map is H*W*C, and the preset width of the sub-feature map is k, then each sub-feature map is H*(W/k)*C. In addition, the number of sub-feature maps can also be preset, such as T, and each sub-feature map is H*(W/T)*C.

Step 810, respectively input the above-mentioned sub-feature maps to the second output network, and output the output matrix corresponding to each sub-feature map;

Taking the convolutional network as an example, the second output network includes multiple fully connected layers; multiple fully connected layers are arranged in parallel; the number of the fully connected layers corresponds to the number of sub-feature maps, and each sub-feature map is input to the corresponding In the fully connected layer of , so that each fully connected layer outputs the output matrix corresponding to the sub-feature map.

Step 812, input the output matrix corresponding to each sub-feature map to the classification function respectively, and output the probability matrix corresponding to each sub-feature map;

The classification function can be a Softmax function; the Softmax function can be identified as Wherein, e represents a natural constant; t represents the t-th probability matrix; K represents the number of different characters contained in the target training text image of the training set; m represents from 1 to K+1; ∑ represents a summation operation; is the i-th element in the output matrix; the is the ith element in the probability matrix p ^t .

relative to the elements in the output matrix itself, the exponential function value of the element The difference between each element can be enlarged. For example, the output matrix is [3,1,-3]. After calculating the exponential function value of each element, the exponential function value matrix corresponding to the output matrix is [20,2.7,0.05] . Using the exponential function value of the elements to calculate the probability of each element can increase the probability gap between them, so that the probability of the correct identification result is higher, which is beneficial to the accuracy of the identification result.

Step 814: Determine the second loss value of the probability matrix by using the preset recognition loss function; train the second initial model according to the second loss value until the parameters in the second initial model converge to obtain a text recognition model.

The recognition loss function includes L=-log p(y|{p ^t } _t=1...T ); wherein, y is the pre-labeled probability matrix of the target training text image; t represents the t-th probability matrix; p ^t is the probability matrix corresponding to each of the sub-feature maps output by the classification function; T is the total number of the probability matrices; p is the calculated probability; log is the logarithmic operation. Based on the recognition loss function, in the above steps, the process of training the second initial model according to the second loss value can also be implemented through the following steps 32-38:

Step 32, update the parameters in the second initial model according to the second loss value;

In actual implementation, the function mapping relationship can be preset, and the original parameters and the second loss value are input into the function mapping relationship, and then the updated parameters can be calculated. The function mapping relationship of different parameters can be the same or different.

Specifically, the parameters to be updated can be determined from the second initial model according to preset rules; the parameters to be updated can be all parameters in the second initial model, or some parameters can be randomly determined from the second initial model; and then calculate The second loss value treats the derivative of the updated parameter Among them, L' is the loss value of the probability matrix; W' is the parameter to be updated; the parameter to be updated may also be called the weight of each neuron. This process can also be called a back-propagation algorithm; if the second loss value is large, it means that the output of the current second initial model does not match the expected output result, then the above-mentioned second loss value is calculated for the second initial model to be in the second initial model. The derivative of the update parameter, which can be used as the basis for adjusting the parameter to be updated.

After obtaining the derivative of each parameter to be updated, update the parameter to be updated to obtain the updated parameter to be updated Among them, α' is a preset coefficient. This process can also be called a stochastic gradient descent algorithm; the derivative of each parameter to be updated can also be understood as the direction in which the first loss value decreases the fastest based on the current parameter to be updated. By adjusting the parameters in this direction, the first loss value can be reduced Decrease quickly to bring this parameter to convergence. In addition, after the second initial model is trained once, a second loss value is obtained. At this time, one or more parameters can be randomly selected from the parameters in the second initial model to perform the above updating process. The model training time of this method is Shorter, the algorithm is faster; of course, the above update process can also be performed on all parameters in the first initial model, and the model training in this way is more accurate.

Step 34, determine whether the updated parameters are all converged; if the updated parameters are all converged, go to step 36; if the updated parameters are not all converged, go to step 38;

Step 36, the second initial model after the parameter update is determined as the identification model;

Step 38: Continue to perform the step of determining the target training text image based on the preset training set until all the updated parameters converge.

Specifically, a new image may be re-acquired from the training set as the target training text image, or the current target training text image may be continued to be used as the target training text image for training.

In the above method, the model can automatically segment the feature map of the image, so the text recognition model only needs to input the image containing the text line to obtain the text content in the image, and there is no need to segment the text line. The text content of the text line can be directly obtained, the operation editing is performed, the operation speed is fast, and the recognition accuracy of the text is high.

Based on the text content determination method provided by the above embodiment, the embodiment of the present invention further provides another text content determination method, which is implemented on the basis of the text content determination method or the training method of the text recognition model described in the above embodiment; The method focuses on describing the process of obtaining the text content of the text area based on the recognition result after the text recognition model outputs the recognition result; as shown in Figure 9, the method includes the following steps:

Step S902, obtaining the text area in the image by the above-mentioned text area determination method;

Step S904, normalize the text area according to the preset size.

The preset size can include preset length and width. If the text area does not meet the preset size, the text area can be scaled, or the text area can be cut or filled in the blank area, so that the The processed text area satisfies the above preset size.

Step S906, input the processed text area into the pre-trained text recognition model, and output the recognition result of the text area; the recognition result of the text area includes a plurality of probability matrices corresponding to the text area;

During the recognition process of the text recognition model, the feature map corresponding to the text area needs to be segmented, and the segmented sub-feature maps are output through the corresponding output network to output the output matrix, and then the classification function is used to obtain the corresponding output matrix of each output matrix. The probability matrix, and thus the recognition result of the text area includes multiple probability matrices, each probability matrix usually corresponds to one or a small number of characters.

Step S908, determining the position of the maximum probability value in each probability matrix;

Step S910, from the correspondence between each position and the character in the preset probability matrix, obtain the character corresponding to the position of the maximum probability value;

As described in the above embodiment, the probability value at each position in the probability matrix can be used to represent the probability that the sub-feature map matches the character corresponding to the position. Therefore, the character corresponding to the position of the maximum probability value can be determined as the recognition result of the corresponding sub-feature map. In most cases, the character corresponding to the position of the maximum probability value can be one character or multiple characters. The corresponding relationship between the above-mentioned positions and characters can be established by the following methods: firstly collecting characters, which can include words, punctuation marks, mathematical symbols, network emoticons, etc. in multiple languages; specifically, in the process of establishing a training set To collect characters, you can also collect them through dictionaries, character libraries, and symbol libraries.

Step S912, arranging the acquired characters according to the arrangement order of the plurality of probability matrices;

The multiple probability matrices output by the text recognition model are usually arranged according to the position of the sub-feature map corresponding to each probability matrix in the feature map. Therefore, the arrangement order of multiple probability matrices is usually the same as that contained in the sub-feature map corresponding to each probability matrix. The arrangement order of the characters is consistent; based on this, according to the arrangement order of multiple probability matrices, arranging the acquired characters can make the arranged characters consistent with the character arrangement of the original text line, so it can be determined according to the arranged characters The text content in the text area.

Step S914: Determine the text content in the text area according to the arranged characters.

In actual implementation, the arranged characters can directly determine the text content in the text area; however, considering that the font sizes of the characters in the text are different, in the text recognition model, when segmenting the feature map, it may not be completely according to a Characters are implemented in a way that corresponds to a sub-feature map. Therefore, there may be repeated characters in the final arranged characters. In order to further optimize the text recognition effect, the repeated characters and empty spaces in the arranged characters can be deleted according to preset rules. character to get the text content in the text area.

Specifically, a word repetition database can be established in advance. If there is a repeated character in the arranged characters, it can be searched from the repetition word database to see if the repeated character exists. If not, the repeated character is deleted, and only the repeated characters are retained. In addition, it can also be combined with the semantics of other characters to determine whether there should be repeated characters in the current context. For null characters, you can also judge whether to delete them in combination with the current context. If the null characters are located between two English words, you don't need to delete them and can keep them. For example, the above arrangement of characters is "--hh-e-l-ll-oo-", where "-" represents a null character; after deleting the repeated characters and null characters, the obtained text content is "hello".

In the above method, first normalize the acquired text area, and then obtain the recognition result of the text area through the text recognition model; then determine the recognized characters through each probability matrix in the recognition result, and then obtain the text of the text area. content. Since the text recognition model can automatically segment the feature map of the image, in this method, only the image containing the text line needs to be input to obtain the recognition result of the image, and then the text content can be obtained, and there is no need to segment the text line. points, the text content of the text line can be directly obtained, the operation editing, the operation speed is fast, and the text recognition accuracy is high.

Based on the text content determination method provided by the above embodiment, the embodiment of the present invention further provides another text content determination method, which is implemented on the basis of the above method; Content The process of judging whether an image contains sensitive words.

Usually, a sensitive thesaurus needs to be established in advance, through which it is determined whether the text content corresponding to the image contains sensitive information; the sensitive thesaurus contains sensitive words, such as sensitive words involving pornography, reaction, and terrorism; You can match the words in the text content to the sensitive word database one by one. If the match is successful, it means that the current word is a sensitive word. Based on this, the method for determining text content in this embodiment includes the following steps, as shown in FIG. 10 :

Step S1002, obtaining the text area in the image by the above-mentioned text area determination method;

Step S1004, normalize the text area according to the preset size.

The preset size can include preset length and width. If the text area does not meet the preset size, the text area can be scaled, or the text area can be cut or filled in the blank area, so that the The processed text area satisfies the above preset size.

Step S1006, input the processed text area into the pre-trained text recognition model, and output the recognition result of the text area; the recognition result of the text area includes a plurality of probability matrices corresponding to the text area;

Step S1008, determining the position of the maximum probability value in each probability matrix;

Step S1010, from the correspondence between each position and the character in the preset probability matrix, obtain the character corresponding to the position of the maximum probability value;

Step S1012, arranging the acquired characters according to the arrangement order of the plurality of probability matrices;

Step S1014: Determine the text content in the text area according to the arranged characters.

Step S1016, if the image contains multiple text regions, obtain the text content in each text region;

Step S1018, performing a word segmentation operation on the acquired text content;

The word segmentation operation can also be called the word segmentation operation; in actual implementation, a thesaurus can be established, and the word segmentation operation can be performed based on the thesaurus; specifically, starting from the first character in the text content, the first character The character and the second character are used as a combination to search from the thesaurus. If the corresponding word containing the combination is not found, the first character is divided into a separate word; if the corresponding word containing the combination can be found, Then add the third character to the combination, and continue to search from the thesaurus; until the word corresponding to the combination cannot be found, divide the characters in the combination except the last character into a word, and so on. Until the word segmentation operation of the text content is completed.

Step S1020, one by one, the word segmentations obtained after the word segmentation operation are matched with the pre-established sensitive thesaurus;

Step S1022, if at least one word segmentation is successfully matched, it is determined that the text content corresponding to the image contains sensitive information.

Step S1024: Acquire the text area to which the successfully matched word segmentation belongs, and identify the acquired text area in the image, or the successfully matched word segmentation.

In actual implementation, the acquired text area can be identified by means of an identification box, or the successfully matched word segmentation; if it is real-time detection in video playback or real-time live broadcast scenarios, the acquired text area can be identified by mosaic or fuzzification. Text area, or matching successful word segmentation, in order to achieve the purpose of filtering sensitive words.

In the above method, after the text content in the text area is obtained, the sensitive words are identified from the text content through the sensitive thesaurus, so as to realize the purpose of speech supervision; this method can obtain the content in real time and identify the sensitive words, which is conducive to the realization of live broadcast on the Internet. , video live broadcast and other scenarios, and restrict the purpose of dissemination of sensitive words.

It should be noted that the above method embodiments are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts of the various embodiments may be referred to each other.

Corresponding to the above method embodiment, refer to the schematic structural diagram of a text detection model training device shown in FIG. 11 , the device includes:

A training image determination module 110, configured to determine a target training image based on a preset training set;

The training image input module 111 is used to input the target training image into the first initial model; the first initial model includes a first feature extraction network, a feature fusion network and a first output network;

The feature extraction module 112 is used for extracting multiple initial feature maps of the target training image through the first feature extraction network; the scales between the multiple initial feature maps are different;

The feature fusion module 113 is configured to perform fusion processing on a plurality of initial feature maps through a feature fusion network to obtain a fusion feature map;

The output module 114 is used to input the fusion feature map to the first output network, and output the candidate regions of the text region in the target training image and the probability value of each candidate region;

The loss value determination and training module 115 is used to determine the candidate area and the first loss value of the probability value of each candidate area through the preset detection loss function; according to the first loss value, the first initial model is trained until the first The parameters in the initial model converge, resulting in a text detection model.

The text detection model training device provided by the embodiment of the present invention first extracts multiple initial feature maps with different scales of the target training image; then fuses the multiple initial feature maps to obtain a fused feature map; and then inputs the fused feature map into To the first output network, output the candidate region of the text region in the target training image and the probability value of each candidate region; after determining the first loss value through the preset detection loss function, according to the first loss value to the first initial model Perform training to obtain a detection model. In this method, the feature extraction network can automatically extract features of different scales. Therefore, the text detection model only needs to input an image to obtain the candidate regions of text regions of various scales in the image, and no need to manually transform the image scale. The operation is convenient, especially in the scenarios of multiple font sizes, multiple fonts, multiple shapes, and multiple orientations, it can quickly, comprehensively and accurately detect various texts in the image, which is also conducive to the accuracy of subsequent text recognition, improving the The effect of text recognition.

In some embodiments, the above-mentioned first feature extraction network includes a plurality of groups of first convolutional networks connected in sequence; each group of first convolutional networks includes a convolutional layer, a batch normalization layer, and an activation function layer that are connected in sequence.

In some embodiments, the above-mentioned feature fusion module is further configured to: arrange multiple initial feature maps in sequence according to the scale of the initial feature map; wherein, the scale of the initial feature map of the topmost level is the smallest; the initial feature map of the bottommost level The scale is the largest; the initial feature map of the top level is determined as the fusion feature map of the top level; except for the top level, the initial feature map of the current level and the fusion feature map of the previous level of the current level are fused to obtain The fusion feature map of the current level; the fusion feature map of the lowest level is determined as the final fusion feature map.

In some embodiments, the above-mentioned first output network includes a first convolutional layer and a second convolutional layer; the above-mentioned output module is further configured to: input the fusion feature map to the first convolutional layer and the second convolutional layer respectively; The first convolution operation is performed on the fusion feature map through the first convolution layer, and a coordinate matrix is output; the coordinate matrix includes the vertex coordinates of the candidate region of the text region in the target training image; The convolution operation outputs a probability matrix; the probability matrix includes the probability value of each candidate region.

In some embodiments, the above-mentioned detection loss function includes a first function and a second function; the first function is L ₁ =|G ^* -G|; wherein, G ^* is the coordinate matrix of the text area in the pre-labeled target training image ; G is the coordinate matrix of the candidate region of the text region in the target training image output by the first output network; The second function is L ₂ =-Y ^* logY-(1-Y ^* )log(1-Y); Wherein, Y ^* is the probability matrix of the text area in the pre-labeled target training image; Y is the probability matrix of the candidate area of the text area in the target training image output by the first output network; the first loss of the candidate area and the probability value of each candidate area The value L=L ₁ +L ₂ .

In some embodiments, the above-mentioned loss value determination and training module is further used for: updating the parameters in the first initial model according to the first loss value; judging whether the updated parameters are all converged; The updated first initial model is determined as the detection model; if the updated parameters do not all converge, the step of determining the target training image based on the preset training set is continued until the updated parameters converge.

In some embodiments, the above-mentioned loss value determination and training module is further used to: determine the parameter to be updated from the first initial model according to a preset rule; calculate the derivative of the first loss value to the parameter to be updated in the first initial model Among them, L is the first loss value; W is the parameter to be updated; update the parameter to be updated to obtain the updated parameter to be updated Among them, α is a preset coefficient.

Referring to the schematic structural diagram of a text area determination device shown in FIG. 12 ; the device includes:

an image acquisition module 120, configured to acquire an image to be detected;

The detection module 122 is used to input the image to be detected into the pre-trained text detection model, and output multiple candidate regions of the text region in the image to be detected, as well as the probability value of each candidate region; the text detection model passes the above-mentioned text detection model. The training method is trained;

The text area determination module 124 is configured to determine the text area in the image to be detected from the multiple candidate areas according to the probability value of the candidate area and the degree of overlap between the multiple candidate areas.

The above-mentioned text region determination device provided in the embodiment of the present invention inputs the acquired image to be detected into the text detection model, and outputs multiple candidate regions of the text region in the to-be-detected image and the probability value of each candidate region; and then according to the candidate region The probability value of , and the degree of overlap between multiple candidate regions, determine the text region in the image to be detected from the multiple candidate regions. In this method, the text detection model can automatically extract features of different scales, so it is only necessary to input an image to the model to obtain candidate regions of text regions of various scales in the image, without the need to manually transform the image scale, and the operation is convenient , especially in the scenes of multiple font sizes, multiple fonts, multiple shapes, and multiple orientations, it can quickly, comprehensively and accurately detect various texts in the image, which is also beneficial to the accuracy of subsequent text recognition and improves text recognition. Effect.

In some embodiments, the above-mentioned text region determination module is further configured to: arrange multiple candidate regions in sequence according to the probability values of the candidate regions; wherein, the probability value of the first candidate region is the largest, and the probability value of the last candidate region is the smallest ; Take the first candidate region as the current candidate region, and calculate the degree of overlap between the current candidate region and the candidate regions other than the current candidate region one by one; Eliminate candidate regions; take the next candidate region of the current candidate region as the new current candidate region, and continue to perform the steps of calculating the degree of overlap between the current candidate region and the candidate regions other than the current candidate region one by one, until the last candidate region is reached; The remaining candidate regions after culling are determined as text regions in the image to be detected.

In some embodiments, the above-mentioned apparatus further includes: a region elimination module, configured to eliminate candidate regions whose probability value is lower than a preset probability threshold from the plurality of candidate regions to obtain the final plurality of candidate regions.

Referring to the schematic structural diagram of a text content determination device shown in FIG. 13; the device includes:

The area acquisition module 130 is used to acquire the text area in the image by the text area determination method of any one of claims 8-10;

The recognition module 132 is used to input the text area into the pre-trained text recognition model, and output the recognition result of the text area;

The text content determination module 134 is configured to determine the text content in the text area according to the recognition result.

The text content determination device provided by the embodiment of the present invention first obtains the text area in the image through the above-mentioned text area determination method; then inputs the text area into the pre-trained text recognition model, and outputs the recognition result of the text area; finally, according to the text area The recognition result determines text information in the text area. In this way, since the above-mentioned text area determination method can obtain text areas of various scales through the text detection model, it can quickly, comprehensively and accurately detect the scene of various font sizes, fonts, shapes and directions. Various types of text in the image are also beneficial to the accuracy of text recognition and improve the effect of text recognition.

In some embodiments, the above apparatus further includes: a normalization module, configured to perform normalization processing on the text area according to a preset size.

In some embodiments, the above-mentioned apparatus further includes a text recognition model training module, configured to train the text recognition model by the following methods: determining a target training text image based on a preset training set; inputting the target training text image into a second an initial model; the second initial model includes a second feature extraction network, a second output network and a classification function; the feature map of the target training text image is extracted through the second feature extraction network; the feature map is split into at least one Sub-feature map; input the sub-feature maps to the second output network respectively, and output the output matrix corresponding to each sub-feature map; input the output matrix corresponding to each sub-feature map to the classification function respectively, and output the probability matrix corresponding to each sub-feature map ; Determine the second loss value of the probability matrix by the preset recognition loss function; train the second initial model according to the second loss value, until the parameters in the second initial model converge, and obtain the text recognition model.

In some embodiments, the above-mentioned second feature extraction network includes a plurality of groups of second convolution networks connected in sequence; each group of second convolution networks includes a convolution layer, a pooling layer, and an activation function layer connected in sequence.

In some embodiments, the above text recognition model training module is further configured to: split the feature map into at least one sub-feature map along the column direction of the feature map; the column direction of the feature map is the vertical direction of the text row direction.

In some embodiments, the above-mentioned second output network includes multiple fully-connected layers; the number of fully-connected layers corresponds to the number of sub-feature maps; the recognition model training module is further configured to: input each sub-feature map to the corresponding full-feature map respectively. In the connection layer, so that each fully connected layer outputs the output matrix corresponding to the sub-feature map.

In some embodiments, the above-mentioned classification function includes a Softmax function; the Softmax function is Wherein, e represents a natural constant; t represents the t-th probability matrix; K represents the number of different characters contained in the target training text image of the training set; m represents from 1 to K+1; ∑ represents a summation operation; is the i-th element in the output matrix; the is the ith element in the probability matrix p ^t .

In some embodiments, the above-mentioned recognition loss function includes L=-log p(y|{p ^t } _t=1...T ); wherein, y is the pre-labeled probability matrix of the target training text image; t represents the first t probability matrices; p ^t is the probability matrix corresponding to each of the sub-feature maps output by the classification function; T is the total number of the probability matrices; p represents the calculation probability; log represents the logarithmic operation.

In some embodiments, the above-mentioned recognition model training module is further used for: updating the parameters in the second initial model according to the second loss value; judging whether all the updated parameters converge; The updated second initial model is determined to be a text recognition model; if the updated parameters do not converge, continue to perform the step of determining the target training text image based on the preset training set until the updated parameters converge.

In some embodiments, the above-mentioned recognition model training module is further configured to: determine the parameter to be updated from the second initial model according to a preset rule; calculate the derivative of the parameter to be updated from the second loss value Among them, L' is the loss value of the probability matrix; W' is the parameter to be updated; update the parameter to be updated to obtain the updated parameter to be updated Among them, α' is a preset coefficient.

In some embodiments, the recognition result of the text region includes a plurality of probability matrices corresponding to the text region; the text content determination module is further configured to: determine the position of the maximum probability value in each probability matrix; In the correspondence between each position and the character, the character corresponding to the position with the maximum probability value is obtained; the obtained characters are arranged according to the arrangement order of the plurality of probability matrices; the text content in the text area is determined according to the arranged characters.

In some embodiments, the above-mentioned text content determination module is further configured to: delete repeated characters and null characters in the arranged characters according to a preset rule to obtain the text content in the text area.

In some embodiments, the above-mentioned apparatus further includes: an information acquisition module, configured to acquire the text content in each text area if the image contains multiple text areas; a sensitive information determination module, configured to obtain the text content in each text area through a pre-established sensitive word The library determines whether the textual content corresponding to the image contains sensitive information.

In some embodiments, the above-mentioned sensitive information determination module is further configured to: perform a word segmentation operation on the acquired text content; match the word segmentation obtained after the word segmentation operation with a pre-established sensitive word database one by one; if at least one word segmentation is successfully matched, Determine that the text content corresponding to the image contains sensitive information.

In some embodiments, the above-mentioned apparatus further includes: a region identification module, configured to acquire the text region to which the successfully matched word segmentation belongs, and identify the acquired text region in the image.

The implementation principle and technical effects of the device provided by the embodiment of the present invention are the same as those of the foregoing method embodiment. For brief description, for the parts not mentioned in the device embodiment, reference may be made to the corresponding content in the foregoing method embodiment.

An embodiment of the present invention further provides an electronic device, as shown in FIG. 14 , the electronic device includes a memory 100 and a processor 101, wherein the memory 100 is used to store one or more computer instructions, and the one or more computer instructions are The processor 101 executes to implement the steps of the above-mentioned text detection model training method, text area determination method, or text content determination method.

Further, the electronic device shown in FIG. 14 further includes a bus 102 and a communication interface 103 , and the processor 101 , the communication interface 103 and the memory 100 are connected through the bus 102 .

The memory 100 may include a high-speed random access memory (RAM, Random Access Memory), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The communication connection between the network element of the system and at least one other network element is implemented through at least one communication interface 103 (which may be wired or wireless), which may use the Internet, a wide area network, a local network, a metropolitan area network, and the like. The bus 102 may be an ISA bus, a PCI bus, an EISA bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one bidirectional arrow is shown in FIG. 14, but it does not mean that there is only one bus or one type of bus.

The processor 101 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above-mentioned method may be completed by an integrated logic circuit of hardware in the processor 101 or an instruction in the form of software. The above-mentioned processor 101 may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (NetworkProcessor, NP for short), etc. Circuit (Application Specific Integrated Circuit, ASIC for short), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components. Various methods, steps, and logical block diagrams disclosed in the embodiments of the present invention can be implemented or executed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the embodiments of the present invention may be directly embodied as executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory 100, and the processor 101 reads the information in the memory 100, and completes the steps of the methods in the foregoing embodiments in combination with its hardware.

Embodiments of the present invention further provide a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are invoked and executed by a processor, the machine-executable instructions cause the processor to implement The steps of the above-mentioned text detection model training method, text area determination method, or text content determination method can be found in the method embodiments, which will not be repeated here.

The text detection model training method, text area, content determination method, apparatus, and computer program product of an electronic device provided by the embodiments of the present invention include a computer-readable storage medium storing program codes, and the instructions included in the program codes can be used for The methods described in the foregoing method embodiments are executed. For specific implementation, reference may be made to the method embodiments, which will not be repeated here.

The functions, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

Finally, it should be noted that the above-mentioned embodiments are only specific implementations of the present invention, and are used to illustrate the technical solutions of the present invention, but not to limit them. The protection scope of the present invention is not limited thereto, although referring to the foregoing The embodiment has been described in detail the present invention, those of ordinary skill in the art should understand: any person skilled in the art who is familiar with the technical field within the technical scope disclosed by the present invention can still modify the technical solutions described in the foregoing embodiments. Or can easily think of changes, or equivalently replace some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be covered in the present invention. within the scope of protection. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (52)

1. a kind of text detection model training method, which is characterized in that the described method includes:

Gathered based on preset training and determines target training image；

The target training image is input to the first initial model；First initial model includes that fisrt feature extracts net Network, Fusion Features network and the first output network；

Multiple initial characteristics figures that network extracts the target training image are extracted by the fisrt feature；It is multiple described initial Scale between characteristic pattern is different；

Fusion treatment is carried out to multiple initial characteristics figures by the Fusion Features network, obtains fusion feature figure；

The fusion feature figure is input to the first output network, exports time text filed in the target training image The probability value of favored area and each candidate region；

The first of the probability value of the candidate region and each candidate region is determined by preset Detectability loss function Penalty values；First initial model is trained according to the first-loss value, until in first initial model Parameter convergence, obtains text detection model.

2. the method according to claim 1, wherein it includes sequentially connected more that the fisrt feature, which extracts network, The first convolutional network of group；First convolutional network described in every group includes sequentially connected convolutional layer, batch normalization layer and activation primitive Layer.

3. the method according to claim 1, wherein by the Fusion Features network to multiple initial spies The step of sign figure carries out fusion treatment, obtains fusion feature figure, comprising:

According to the scale of the initial characteristics figure, multiple initial characteristics figures are arranged successively；Wherein, top grade is initial The scale of characteristic pattern is minimum；The scale of the initial characteristics figure of bottom grade is maximum；

The initial characteristics figure of top grade is determined as to the fusion feature figure of the top grade；

It is in addition to the top grade, the fusion of the initial characteristics figure of current level and a upper level for the current level is special Sign figure is merged, and the fusion feature figure of current level is obtained；

The fusion feature figure of lowest hierarchical level is determined as to final fusion feature figure.

4. the method according to claim 1, wherein the first output network includes the first convolutional layer and second Convolutional layer；

The fusion feature figure is input to the first output network, exports time text filed in the target training image The step of probability value of favored area and each candidate region, comprising:

The fusion feature figure is separately input into first convolutional layer and second convolutional layer；

The first convolution algorithm, output coordinate matrix are carried out to the fusion feature figure by first convolutional layer；The coordinate Matrix includes the apex coordinate of candidate region text filed in the target training image；

The second convolution algorithm, output probability matrix are carried out to the fusion feature figure by second convolutional layer；The probability Matrix includes the probability value of each candidate region.

5. the method according to claim 1, wherein the Detectability loss function includes first function and the second letter Number；

The first function is L₁=| G^*-G|；Wherein, the G^*It is text filed in the target training image that marks in advance Coordinates matrix；G is the seat of candidate region text filed in the target training image of the first output network output Mark matrix；

The second function is L₂=-Y^*logY-(1-Y^*)log(1-Y)；Wherein, Y^*For the target training figure marked in advance The text filed probability matrix as in；Y is text filed in the target training image of the first output network output The probability matrix of candidate region；Log indicates logarithm operation；

The first-loss value L=L of the probability value of the candidate region and each candidate region₁+L₂。

6. the method according to claim 1, wherein according to the first-loss value to first initial model It is trained, up to the step of parameter in first initial model restrains, obtains text detection model, comprising:

The parameter in first initial model is updated according to the first-loss value；

Judge whether the updated parameter restrains；

If the updated parameter restrains, updated first initial model of parameter is determined as detection model；

If the updated parameter does not restrain, continues to execute and determining target training image is gathered based on preset training The step of, until the updated parameter restrains.

7. according to the method described in claim 6, it is characterized in that, updating first introductory die according to the first-loss value The step of parameter in type, comprising:

According to preset rules, parameter to be updated is determined from first initial model；

The first-loss value is calculated to the derivative of parameter to be updated described in first initial modelWherein, L is institute State first-loss value；W is the parameter to be updated；

The parameter to be updated is updated, updated parameter to be updated is obtainedWherein, α is default system Number.

8. a kind of text filed determining method, which is characterized in that the described method includes:

Obtain image to be detected；

Described image to be detected is input to the text detection model that training is completed in advance, exports text in described image to be detected The probability value of multiple candidate regions in region and each candidate region；The text detection model passes through claim The training method training of the described in any item text detection models of 1-7 obtains；

According to the overlapping degree between the probability value of the candidate region and multiple candidate regions, from multiple candidates It is determined in region text filed in described image to be detected.

9. according to the method described in claim 8, it is characterized in that, according to the probability value of the candidate region and multiple described Overlapping degree between candidate region, from the text filed step determined in multiple candidate regions in described image to be detected Suddenly, comprising:

According to the probability value of the candidate region, multiple candidate regions are arranged successively；Wherein, first candidate region Probability value is maximum, and the probability value of the last one candidate region is minimum；

Using first candidate region as current candidate region, the current candidate region is calculated one by one and except described current The overlapping degree of candidate region other than candidate region；

By in the candidate region in addition to the current candidate region, the overlapping degree is greater than the candidate of preset anti-eclipse threshold It rejects in region；

Using next candidate region in the current candidate region as new current candidate region, continues to execute and calculate institute one by one The step of stating the overlapping degree in current candidate region and the candidate region in addition to the current candidate region, until reaching last One candidate region；

Remaining candidate region after rejecting is determined as text filed in described image to be detected.

10., will be multiple described according to the method described in claim 9, it is characterized in that, according to the probability value of the candidate region Before the step of candidate region is arranged successively, the method also includes:

By in multiple candidate regions, probability value is rejected lower than the candidate region of preset probability threshold value, is obtained final more A candidate region.

11. a kind of content of text determines method, which is characterized in that the described method includes:

By the described in any item text filed determining methods of claim 8-10, obtain text filed in image；

By the text filed text identification model for being input to training completion in advance, the text filed identification knot is exported Fruit；

According to the recognition result determine it is described it is text filed in content of text.

12. according to the method for claim 11, which is characterized in that text filed be input to is trained completion in advance Before the step of identification model, the method also includes: according to pre-set dimension, text filed it is normalized to described.

13. according to the method for claim 11, which is characterized in that the text identification model has been trained by following manner At:

Gathered based on preset training and determines target training text image；

The target training text image is input to the second initial model；Second initial model includes that second feature is extracted Network, feature split network, the second output network and classification function；

The characteristic pattern that network extracts the target training text image is extracted by the second feature；

Network is split by the feature, and the characteristic pattern is split into at least one subcharacter figure；

The subcharacter figure is separately input into the second output network, exports the corresponding output square of each subcharacter figure Battle array；

The corresponding output matrix of each subcharacter figure is separately input into the classification function, exports each subcharacter Scheme corresponding probability matrix；

The second penalty values of the probability matrix are determined by preset identification loss function；According to second penalty values to institute It states the second initial model to be trained, until the parameter convergence in second initial model, obtains text identification model.

14. according to the method for claim 13, which is characterized in that it includes sequentially connected that the second feature, which extracts network, The second convolutional network of multiple groups；Second convolutional network described in every group includes sequentially connected convolutional layer, pond layer and activation primitive layer.

15. according to the method for claim 13, which is characterized in that split network by the feature and tear the characteristic pattern open The step of being divided at least one subcharacter figure, comprising:

Along the column direction of the characteristic pattern, the characteristic pattern is split into at least one subcharacter figure；The column of the characteristic pattern Direction is the vertical direction of text line direction.

16. according to the method for claim 13, which is characterized in that the second output network includes multiple full articulamentums； The quantity of the full articulamentum is corresponding with the quantity of the subcharacter figure；

It is described that the subcharacter figure is separately input into the second output network, it is corresponding defeated to export each subcharacter figure The step of matrix out, comprising: each subcharacter figure is separately input into corresponding full articulamentum, so that each described complete Articulamentum exports the corresponding output matrix of the subcharacter figure.

17. according to the method for claim 13, which is characterized in that the classification function includes Softmax function；

The Softmax function isWherein, e indicates natural constant；T indicates t-th of probability matrix；K table Show the number for the kinds of characters that the target training text image of the training set is included；M is indicated from 1 to K+1；∑ expression is asked And operation；For i-th of element in the output matrix；It is describedFor the probability matrix p^tIn i-th of element.

18. according to the method for claim 13, which is characterized in that the identification loss function include L=-log p (y | {p^t}_{T=1 ... T})；Wherein, y is the probability matrix of the target training text image marked in advance；T indicates t-th of probability square Battle array；p^tFor each of the classification function output corresponding probability matrix of the subcharacter figure；T is the sum of the probability matrix Amount；P indicates to calculate probability；Log indicates logarithm operation.

19. according to the method for claim 13, which is characterized in that according to second penalty values to second introductory die Type is trained, up to the step of parameter in second initial model restrains, obtains text identification model, comprising:

The parameter in second initial model is updated according to second penalty values；

Judge whether the updated parameter restrains；

If the updated parameter restrains, updated second initial model of parameter is determined as text identification mould Type；

If the updated parameter does not restrain, continues to execute and determining target training text is gathered based on preset training The step of image, until updated each parameter restrains.

20. according to the method for claim 19, which is characterized in that it is initial to update described second according to second penalty values In model the step of parameters, comprising:

According to preset rules, parameter to be updated is determined from second initial model；

Second penalty values are calculated to the derivative of the parameter to be updatedWherein, L ' is the loss of the probability matrix Value；W ' is the parameter to be updated；

The parameter to be updated is updated, updated parameter to be updated is obtainedWherein, α ' is default Coefficient.

21. according to the method for claim 11, which is characterized in that the text filed recognition result includes the text The corresponding multiple probability matrixs in region；

According to the recognition result determine it is described it is text filed in content of text the step of, comprising:

Determine the position of the most probable value in each probability matrix；

From in the corresponding relationship of position each in pre-set probability matrix and character, the position of the most probable value is obtained Corresponding character；

According to putting in order for multiple probability matrixs, the character got is arranged；

According to the character after arrangement determine it is described it is text filed in content of text.

22. according to the method for claim 21, which is characterized in that determine the text area according to the character after arrangement The step of content of text in domain, comprising:

According to preset rules, the repeat character (RPT) and null character in the character after deleting arrangement, obtain it is described it is text filed in Content of text.

23. according to the method for claim 11, which is characterized in that according to the recognition result determine it is described it is text filed in Content of text the step of after, the method also includes:

If include in described image it is multiple text filed, obtain it is each it is described it is text filed in content of text；

Determine in the corresponding content of text of described image whether include sensitive information by the sensitive dictionary pre-established.

24. according to the method for claim 23, which is characterized in that determine described image by the sensitive dictionary pre-established The step of whether including sensitive information in corresponding content of text, comprising:

Participle operation is carried out to the content of text got；

The participle dictionary sensitive with what is pre-established obtained after participle operation is matched one by one；

If at least one participle successful match, determines that in the corresponding content of text of described image include sensitive information.

25. according to the method for claim 24, which is characterized in that determine in the corresponding content of text of described image and include After sensitive information, the method also includes:

Obtain successful match participle belonging to it is text filed, identified in described image get it is described text filed, Or the participle of successful match.

26. a kind of text detection model training apparatus, which is characterized in that described device includes:

Training image determining module determines target training image for gathering based on preset training；

Training image input module, for the target training image to be input to the first initial model；First introductory die Type includes that fisrt feature extracts network, Fusion Features network and the first output network；

Characteristic extracting module, for extracting multiple initial spies that network extracts the target training image by the fisrt feature Sign figure；Scale between multiple initial characteristics figures is different；

Fusion Features module is obtained for carrying out fusion treatment to multiple initial characteristics figures by the Fusion Features network To fusion feature figure；

Output module exports the target training image for the fusion feature figure to be input to the first output network In text filed candidate region and each candidate region probability value；

Penalty values determination and training module, for determining the candidate region and each institute by preset Detectability loss function State the first-loss value of the probability value of candidate region；First initial model is trained according to the first-loss value, Until the parameter convergence in first initial model, obtains text detection model.

27. device according to claim 26, which is characterized in that it includes sequentially connected that the fisrt feature, which extracts network, The first convolutional network of multiple groups；First convolutional network described in every group includes sequentially connected convolutional layer, batch normalization layer and activation letter Several layers.

28. device according to claim 26, which is characterized in that the Fusion Features module is also used to:

According to the scale of the initial characteristics figure, multiple initial characteristics figures are arranged successively；Wherein, top grade is initial The scale of characteristic pattern is minimum；The scale of the initial characteristics figure of bottom grade is maximum；

The initial characteristics figure of top grade is determined as to the fusion feature figure of the top grade；

It is in addition to the top grade, the fusion of the initial characteristics figure of current level and a upper level for the current level is special Sign figure is merged, and the fusion feature figure of current level is obtained；

The fusion feature figure of lowest hierarchical level is determined as to final fusion feature figure.

29. device according to claim 26, which is characterized in that the first output network includes the first convolutional layer and the Two convolutional layers；

The output module is also used to:

The fusion feature figure is separately input into first convolutional layer and second convolutional layer；

The first convolution algorithm, output coordinate matrix are carried out to the fusion feature figure by first convolutional layer；The coordinate Matrix includes the apex coordinate of candidate region text filed in the target training image；

The second convolution algorithm, output probability matrix are carried out to the fusion feature figure by second convolutional layer；The probability Matrix includes the probability value of each candidate region.

30. device according to claim 26, which is characterized in that the Detectability loss function includes first function and second Function；

The first function is L₁=| G^*-G|；Wherein, the G^*It is text filed in the target training image that marks in advance Coordinates matrix；G is the seat of candidate region text filed in the target training image of the first output network output Mark matrix；

The second function is L₂=-Y^*logY-(1-Y^*)log(1-Y)；Wherein, Y^*For the target training figure marked in advance The text filed probability matrix as in；Y is text filed in the target training image of the first output network output The probability matrix of candidate region；Log indicates logarithm operation；

The first-loss value L=L of the probability value of the candidate region and each candidate region₁+L₂。

31. device according to claim 26, which is characterized in that the penalty values are determining and training module is also used to:

The parameter in first initial model is updated according to the first-loss value；

Judge whether the updated parameter restrains；

If the updated parameter restrains, updated first initial model of parameter is determined as detection model；

If the updated parameter does not restrain, continues to execute and determining target training image is gathered based on preset training The step of, until the updated parameter restrains.

32. device according to claim 31, which is characterized in that the penalty values are determining and training module is also used to:

According to preset rules, parameter to be updated is determined from first initial model；

The first-loss value is calculated to the derivative of parameter to be updated described in first initial modelWherein, L is institute State first-loss value；W is the parameter to be updated；

The parameter to be updated is updated, updated parameter to be updated is obtainedWherein, α is default system Number.

33. a kind of text filed determining device, which is characterized in that described device includes:

Image collection module, for obtaining image to be detected；

Detection module, for described image to be detected to be input to the text detection model that training is completed in advance, output it is described to The probability value of text filed multiple candidate regions and each candidate region in detection image；The text detection mould Type is obtained by the training method training of the described in any item text detection models of claim 1-7；

Text filed determining module, for the weight between the probability value and multiple candidate regions according to the candidate region Folded degree, it is text filed in described image to be detected from being determined in multiple candidate regions.

34. device according to claim 33, which is characterized in that the text filed determining module is also used to:

According to the probability value of the candidate region, multiple candidate regions are arranged successively；Wherein, first candidate region Probability value is maximum, and the probability value of the last one candidate region is minimum；

Using first candidate region as current candidate region, the current candidate region is calculated one by one and except described current The overlapping degree of candidate region other than candidate region；

By in the candidate region in addition to the current candidate region, the overlapping degree is greater than the candidate of preset anti-eclipse threshold It rejects in region；

Using next candidate region in the current candidate region as new current candidate region, continues to execute and calculate institute one by one The step of stating the overlapping degree in current candidate region and the candidate region in addition to the current candidate region, until reaching last One candidate region；

Remaining candidate region after rejecting is determined as text filed in described image to be detected.

35. device according to claim 34, which is characterized in that described device further include: module is rejected in region, and being used for will In multiple candidate regions, probability value is rejected lower than the candidate region of preset probability threshold value, is obtained final multiple described Candidate region.

36. a kind of content of text determining device, which is characterized in that described device includes:

Region obtains module, for obtaining in image by the described in any item text filed determining methods of claim 8-10 It is text filed；

Identification module, for exporting the text for the text filed text identification model for being input to training completion in advance The recognition result in region；

Content of text determining module, for according to the recognition result determine it is described it is text filed in content of text.

37. device according to claim 36, which is characterized in that described device further include: normalization module, for according to Pre-set dimension text filed is normalized to described.

38. device according to claim 36, which is characterized in that described device further includes text identification model training mould Block, for completing the text identification model by following manner training:

Gathered based on preset training and determines target training text image；

The target training text image is input to the second initial model；Second initial model includes that second feature is extracted Network, the second output network and classification function；

The characteristic pattern that network extracts the target training text image is extracted by the second feature；

The characteristic pattern is split into at least one subcharacter figure by second initial model；

The subcharacter figure is separately input into the second output network, exports the corresponding output square of each subcharacter figure Battle array；

The corresponding output matrix of each subcharacter figure is separately input into the classification function, exports each subcharacter Scheme corresponding probability matrix；

The second penalty values of the probability matrix are determined by preset identification loss function；According to second penalty values to institute It states the second initial model to be trained, until the parameter convergence in second initial model, obtains text identification model.

39. the device according to claim 38, which is characterized in that it includes sequentially connected that the second feature, which extracts network, The second convolutional network of multiple groups；Second convolutional network described in every group includes sequentially connected convolutional layer, pond layer and activation primitive layer.

40. the device according to claim 38, which is characterized in that the identification model training module is also used to:

Along the column direction of the characteristic pattern, the characteristic pattern is split into at least one subcharacter figure；The column of the characteristic pattern Direction is the vertical direction of text line direction.

41. the device according to claim 38, which is characterized in that the second output network includes multiple full articulamentums； The quantity of the full articulamentum is corresponding with the quantity of the subcharacter figure；

The identification model training module is also used to: each subcharacter figure is separately input into corresponding full articulamentum, So that each full articulamentum exports the corresponding output matrix of the subcharacter figure.

42. the device according to claim 38, which is characterized in that the classification function includes Softmax function；

The Softmax function isWherein, e indicates natural constant；T indicates t-th of probability matrix；K table Show the number for the kinds of characters that the target training text image of the training set is included；M is indicated from 1 to K+1；∑ expression is asked And operation；For i-th of element in the output matrix；It is describedFor the probability matrix p^tIn i-th of element.

43. the device according to claim 38, which is characterized in that the identification loss function include L=-log p (y | {p^t}_{T=1 ... T})；Wherein, y is the probability matrix of the target training text image marked in advance；T indicates t-th of probability square Battle array；p^tFor each of the classification function output corresponding probability matrix of the subcharacter figure；T is the sum of the probability matrix Amount；P indicates to calculate probability；Log indicates logarithm operation.

44. the device according to claim 38, which is characterized in that the identification model training module is also used to:

The parameter in second initial model is updated according to second penalty values；

Judge whether updated each parameter restrains；

If updated each parameter restrains, updated second initial model of parameter is determined as text and is known Other model；

If updated each parameter does not restrain, continues to execute and determining target training is gathered based on preset training The step of text image, until updated each parameter restrains.

45. device according to claim 44, which is characterized in that the identification model training module is also used to:

According to preset rules, parameter to be updated is determined from second initial model；

Second penalty values are calculated to the derivative of the parameter to be updatedWherein, L ' is the loss of the probability matrix Value；W ' is the parameter to be updated；

The parameter to be updated is updated, updated parameter to be updated is obtainedWherein, α ' is default Coefficient.

46. device according to claim 36, which is characterized in that the text filed recognition result includes the text The corresponding multiple probability matrixs in region；

The content of text determining module is also used to:

Determine the position of the most probable value in each probability matrix；

From in the corresponding relationship of position each in pre-set probability matrix and character, the position of the most probable value is obtained Corresponding character；

According to putting in order for multiple probability matrixs, the character got is arranged；

According to the character after arrangement determine it is described it is text filed in content of text.

47. device according to claim 46, which is characterized in that the content of text determining module is also used to:

According to preset rules, the repeat character (RPT) and null character in the character after deleting arrangement, obtain it is described it is text filed in Content of text.

48. device according to claim 36, which is characterized in that described device further include:

Data obtaining module, if for include in described image it is multiple text filed, obtain it is each it is described it is text filed in Content of text；

Sensitive information determining module, determining in the corresponding content of text of described image for the sensitive dictionary by pre-establishing is No includes sensitive information.

49. device according to claim 48, which is characterized in that the sensitive information determining module is also used to:

Participle operation is carried out to the content of text got；

The participle dictionary sensitive with what is pre-established obtained after participle operation is matched one by one；

If at least one participle successful match, determines that in the corresponding content of text of described image include sensitive information.

50. device according to claim 49, which is characterized in that described device further include:

Area identification module, it is text filed belonging to the participle of successful match for obtaining, acquisition is identified in described image That arrives is described text filed.

51. a kind of electronic equipment, which is characterized in that including processor and memory, the memory is stored with can be described The machine-executable instruction that processor executes, the processor execute the machine-executable instruction to realize claim 1 to 7 Described in any item text detection model training methods, the described in any item text filed determining methods of claim 8 to 10, or The step of described in any item content of text of person's claim 11 to 25 determine method.

52. a kind of machine readable storage medium, which is characterized in that the machine readable storage medium is stored with the executable finger of machine It enables, for the machine-executable instruction when being called and being executed by processor, machine-executable instruction promotes processor to realize that right is wanted Ask 1 to 7 described in any item text detection model training methods, the described in any item text filed determinations of claim 8 to 10 The step of method or the described in any item content of text of claim 11 to 25 determine method.