CN114842460A - Scene character detection method and device - Google Patents

Abstract

The invention discloses a method and a device for scene text detection, and relates to the technical field of image word processing. A specific embodiment of the method includes: acquiring an image to be detected, the image to be detected contains a text area, determining a text centerline diagram and a directional distance map of the text area, and determining the text in the text area according to the text centerline map and the directional distance map contour. The method for scene text detection according to the embodiment of the present invention can effectively separate the adjacent text through the text centerline map, and the direction distance map can detect the scene text with any shape and direction, which solves the detection problem of irregular scene text, and further can Effectively improve detection performance.

Description

A method and device for scene text detection

technical field

The invention relates to the field of image word processing, in particular to a method and device for scene word detection.

Background technique

Scene text detection technology has broad application prospects, for example, it can be used to assist driverless systems to obtain real-time road conditions and geographic information.

The current scene text detection methods mainly include the following two: one is based on regression, by predicting the offset between the candidate text frame and the actual text frame to obtain a horizontal frame, a directional rectangular frame or a quadrilateral frame, but This method is not suitable for detecting irregular-shaped text; the other is a segmentation-based method, which uses a fully convolutional network to determine whether the pixels in the image belong to a text area or a non-text area one by one, but this method is difficult to The text area is separated, which reduces the detection accuracy.

SUMMARY OF THE INVENTION

In view of this, the embodiments of the present invention provide a method and device for scene text detection, which can effectively separate the immediately adjacent scene text in the text area, and can solve the detection problem of scene text with irregular shapes and directions, and improve the detection performance. .

To achieve the above purpose, according to an aspect of the embodiments of the present invention, a method for scene text detection is provided, including:

acquiring an image to be detected, where the image to be detected includes a text area;

determining the text centerline diagram and the direction distance diagram of the text area;

Determine the text outline of the text area according to the text centerline map and the direction distance map;

Wherein, the text center line map is formed according to the text center line of the text area, and the direction distance map uses polar coordinates to return to the point on the text center line along a plurality of preset directions to reach the text center line. The distance from the edge of the text outline is obtained.

Optionally, before determining the text centerline map and the direction distance map of the text area, including:

Obtaining an image data training set, wherein the image data training samples in the image data training set include text regions;

constructing a network structure and constructing a multi-task loss function for the network structure;

The network structure is trained by using the image data training set until the value of the multi-task loss function reaches a preset condition, and a scene text detection model is obtained.

Optionally, the multi-task loss function includes a segmentation loss function and a regression loss function, the segmentation loss function is constructed based on predicted and real text centerline diagrams, and the regression loss function is constructed based on predicted and real text centerlines. The direction distance map is constructed;

The determining of the text centerline diagram and the direction distance diagram of the text area includes:

The to-be-detected image is input into the trained scene text detection model, and the text centerline map and the direction distance map corresponding to the to-be-detected image are predicted to be obtained.

Optionally, inputting the image to be detected into the trained scene text detection model, and predicting a text centerline map and a direction distance map corresponding to the image to be detected, including:

According to the scene text detection model, extract the features of the image to be detected and perform feature fusion to obtain a fusion feature map;

According to the fusion feature map and the scene text detection model, a text centerline map and a direction distance map corresponding to the image to be detected are predicted to be obtained.

Optionally, determining the text outline of the text area according to the text centerline map and the direction distance map, including:

A connected area is formed according to the aggregation of adjacent points on the text center line in the text center line diagram;

determining each sampling point on the connected region;

determining a direction point corresponding to each of the sampling points according to the sampling points and the direction distance map;

According to the direction points of each of the sampling points, the character outline of the character area is determined.

Optionally, determining each sampling point on the connected region, including:

determining the minimum rotated circumscribed rectangle of the connected region;

Divide the rectangle into n equal parts to form n-1 vertical lines; wherein, n is an integer greater than 1;

The sampling point is the midpoint of the intersection of each vertical line and the connected area.

Optionally, extract the features of the image to be detected and perform feature fusion to obtain a fusion feature map, including:

Feature extraction is performed on the to-be-detected image through the backbone network to obtain multiple feature maps of different scales;

A plurality of feature maps of different scales are fused to obtain the fused feature map.

Optionally, feature extraction is performed on the image to be detected through a backbone network to obtain multiple feature maps of different scales, including: extracting features of the image to be detected through multiple convolution modules of the backbone network to obtain multiple feature maps of different scales. Deep feature maps and shallow feature maps;

Fusion of multiple feature maps of different scales to obtain the fused feature map includes: using standard convolution and multiple atrous convolutional networks with different dilation rates to perform context feature extraction on the multiple deep feature maps, and extract the context The multiple deep feature maps after the feature are combined with the shallow feature maps in a cascaded manner to obtain the fusion feature map.

Another aspect of the embodiments of the present invention provides an apparatus for scene text detection, including:

an acquisition module, which acquires an image to be detected, where the image to be detected includes a text area;

The model prediction module determines the text centerline diagram and the direction distance diagram of the text area;

a determining module, for determining the text outline of the text area according to the text centerline diagram and the direction distance map,

Wherein, the text center line map is formed according to the text center line of the text area, and the direction distance map uses polar coordinates to return to the point on the text center line along a plurality of preset directions to reach the text center line. The distance from the edge of the text outline is obtained.

According to another aspect of the embodiments of the present invention, an electronic device is provided, including:

one or more processors;

storage means for storing one or more programs,

When the one or more programs are executed by the one or more processors, the one or more processors implement the method for scene text detection provided by the present invention.

According to another aspect of the embodiments of the present invention, there is provided a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the method for scene character detection provided by the present invention is implemented.

An embodiment of the above invention has the following advantages or beneficial effects: by acquiring an image to be detected, the image to be detected contains a text area, a text centerline diagram and a direction distance map of the text area are determined according to the to-be-detected image, and a text centerline diagram is determined according to the text centerline diagram. And the direction distance map, get the text outline of the text area. The method for scene text detection according to the embodiment of the present invention can effectively separate adjacent texts through the text centerline map, and the directional distance map obtained based on the text centerline map can detect scene texts with arbitrary shapes and directions, which solves the problem of irregular scenes. The problem of text detection can effectively improve the detection performance of the scene text detector.

Further effects of the above non-conventional alternatives will be described below in conjunction with specific embodiments.

Description of drawings

The accompanying drawings are used for better understanding of the present invention and do not constitute an improper limitation of the present invention. in:

1 is a schematic diagram of the main flow of a method for scene text detection according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of expressions of different characters according to an embodiment of the present invention;

3 is a schematic diagram of a process for obtaining a fusion feature map according to an embodiment of the present invention;

4 is a schematic diagram of a process for determining a text outline of a text area according to an embodiment of the present invention;

5 is a schematic flowchart of a method for scene text detection according to an embodiment of the present invention;

6 is a schematic diagram of main modules of an apparatus for scene text detection according to an embodiment of the present invention;

FIG. 7 is an exemplary system architecture diagram to which an embodiment of the present invention may be applied;

FIG. 8 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

In recent years, scene text detection technology based on deep learning has been widely used in various fields of computer vision, such as the field of automatic driving, image and video retrieval, text translation, etc., all need to recognize scene text. At present, a large number of scene text detection methods mostly rely on manually involved features to distinguish text and text regions, which requires a lot of feature engineering and cannot guarantee the robustness of text detection. The scene text detection method based on the deep learning algorithm includes regression and segmentation methods, but cannot effectively separate the adjacent text areas, and when detecting irregularly shaped text such as curved text, it predicts redundant background information for subsequent text recognition. It brings greater interference and low detection accuracy. In view of the above problems, embodiments of the present invention provide a method for scene text detection, which can separate adjacent text and is suitable for detecting scene texts with irregular shapes, thereby improving detection performance.

FIG. 1 is a schematic diagram of the main flow of a method for scene text detection according to an embodiment of the present invention. As shown in FIG. 1 , the method includes the following steps:

Step S101: acquiring an image to be detected, and the image to be detected includes a text area;

Step S102: Determine the text centerline map and the direction distance map of the text area;

Step S103: Determine the text outline of the text area according to the text centerline map and the direction distance map.

In the embodiment of the present invention, the image to be detected may be an image obtained in scenarios such as automatic driving, image and video retrieval, text translation, etc., the image to be detected is an image containing a text area, and the text area is an area containing scene text (text instances) , the scene text can be, for example, one or more words or text lines. The image to be detected may contain one or more text regions. Optionally, the scene text may be a regular shape or an irregular shape, for example, the scene text may be a curved shape.

In the embodiment of the present invention, the text centerline diagram is formed according to the text centerline of the text area, and the direction distance diagram uses polar coordinates to regress the points on the text centerline to reach the edge of the text outline along multiple preset directions. distance obtained.

As shown in Figure 2 is a schematic diagram of the expression of different shapes of text, in Figure 2, (a) is the text outline of the regular shape text obtained by the method of Euclidean coordinates; (b) only uses a polar coordinate It can be seen from (c) that when the text is irregular in shape, only one point with polar coordinates will not be able to cover the entire text area, so it is difficult to obtain the text more accurately. The text outline of the irregular-shaped text; it can be seen from (d) that for irregular-shaped text, the embodiment of the present invention first obtains the text centerline diagram of the text area, and then returns to the point on the text centerline along the predetermined direction to the text The distance of the outline, such as the predetermined 8 directions, the included angle between the adjacent directions is 45 degrees, so as to obtain the outline of the text with irregular shape. The method of the embodiment of the present invention can be used to precisely locate characters of any shape in a scene image.

In the embodiment of the present invention, before determining the text centerline map and the direction distance map of the text area, the steps include:

Obtain a training set of image data, the image data contains text areas;

Build the network structure and build a multi-task loss function for the network structure,

The image data training set is used to train the network structure until the value of the multi-task loss function reaches the preset condition, and the scene text detection model is obtained.

In the embodiment of the present invention, before determining the text centerline map and the directional distance map, a scene text detection model needs to be obtained, and an image data training set is obtained first. The image data training set includes a plurality of image data training samples, each image data training set The samples all contain text areas. The acquisition method of the image data is not particularly limited. Then a network structure is constructed, and the network structure is composed of CNN (Convolutional Neural Network, Convolutional Neural Network) + FPN (FeaturePyramid Networks, Feature Pyramid Network).

Model training is performed using the image training samples of the multiple images to obtain a scene text detection model, optionally, the model may be a model based on an FCN network (Fully Convolutional Networks, fully convolutional network) to implement image semantic segmentation.

In the embodiment of the present invention, the multi-task loss function includes a segmentation loss function and a regression loss function, the segmentation loss function is constructed based on the predicted and actual text centerline diagrams, and the regression loss function is based on the predicted and actual direction distance diagrams constructed.

In the embodiment of the present invention, the calculation formula of the value of the constructed multi-task loss function is shown in formula (1),

L= _{λL tcd} +L _dd formula (1),

Among them, L _tcd represents the two-class segmentation loss of TCL (text centerline map), L _dd represents the regression loss of DD (direction distance map), and λ is the weight coefficient that balances the two loss terms, such as 0.1.

For the prediction of TCL, it can be regarded as a binary classification problem of judging the image data pixel by pixel to belong to the text or background area. Since the scales of text instances in natural scenes are significantly different, if the weights of all text pixels are the same, the small text instance It may be missed because of its small contribution to the total binary classification loss. Therefore, the embodiment of the present invention adopts the Dice loss function based on instance equalization, and the constructed TCL two-class segmentation loss function is shown in formula (2),

In formula (2), G and P respectively represent the actual area and prediction area of TCL in the image; W represents the weight map of TCL, and w _c (p) of the weight p of any arbitrary pixel in W is calculated as in formula (3) ) shown in the formula to get,

In formula (3), w _c (p) represents the weight of pixel p in TCL, Area (C) is the total number of pixels in area C, C represents the set of pixels in TCL, N is the number of text instances in the image, C _p represents the centerline region in the image that contains pixel p.

In an optional implementation of the embodiment of the present invention, the predicted direction distance is the distance from the predicted point on the TCL to the text outline in multiple directions, and the DD loss function is constructed based on the Smooth L1 loss, as shown in formula (4). ,

In formula (4), d _{x, y} and d ^* _{x, y} represent the actual distance and predicted distance from the point (x, y) to the text edge, respectively, Norm _{x, y} are obtained by formula (5),

In formula (5), Box_H _{x, y} and Box_W _{x, y} represent the height and width of the edge box of the text instance where the point (x, y) is located, respectively.

In the embodiment of the present invention, a neural network (such as an FCN network) is trained in an end-to-end manner to minimize the value of the multi-task loss function, and after multiple iterations, when the value of the multi-task loss function reaches a preset convergence condition , if the increment of the value of the multi-task loss function is within the preset range, stop the iteration and obtain the scene text detection model.

In the embodiment of the present invention, determining the text centerline map and the direction distance map of the text area includes: inputting the image to be detected into the trained scene text detection model, and predicting the text centerline map and direction corresponding to the image to be detected. The distance map, specifically, includes: extracting the features of the image to be detected according to the scene text detection model and performing feature fusion to obtain a fusion feature map; according to the fusion feature map and the scene text detection model, predicting and obtaining the text centerline corresponding to the image to be detected Graph and direction distance graph.

In the embodiment of the present invention, extracting the features of the image to be detected and performing feature fusion to obtain a fusion feature map includes: performing feature extraction on the image to be detected through a backbone network to obtain multiple feature maps of different scales; The feature map is fused to obtain a fused feature map. Optionally, the ResNet-50 residual convolutional network after removing the fully connected layer is used as the backbone network (main network) to extract the features of the image to be detected.

In the embodiment of the present invention, feature extraction is performed on the image to be detected through the backbone network to obtain multiple feature maps of different scales, including: extracting features of the image to be detected through multiple convolution modules of the backbone network, and obtaining multiple feature maps of different scales Deep feature maps and shallow feature maps, using standard convolution and multiple atrous convolutional networks with different dilation rates to extract contextual features from multiple deep feature maps.

In the embodiment of the present invention, fusing multiple feature maps of different scales to obtain a fused feature map includes: merging multiple deep feature maps and shallow feature maps after extracting contextual features in a cascaded manner to obtain a fusion feature feature map.

Due to the limitation of the receptive field of standard convolution, standard convolution is not suitable for processing scene characters with large changes in shape and aspect ratio. Therefore, in this embodiment of the present invention, atrous convolution can be introduced so that the network can maintain parameters. In the case of the same amount, it has a larger receptive field, thereby improving the detection performance of long text.

The standard convolution and multiple atrous convolutions with different dilation rates are used to form a context feature extraction module (CFE).

Figure 3 is a schematic diagram of a process for obtaining a fusion feature map according to an embodiment of the present invention. Input the image to be detected (Image), and then perform feature extraction on the image to be detected through the backbone network. After the 64 convolution kernels of stage1 After the sampling (/2) operation, it enters the four stages stage2, stage3, stage4 and stage5 for downsampling (/2) operation. The number of convolution kernels are 256, 512, 1024 and 2048 respectively. After the 1*1 volume of stage2 The shallow layer feature map is obtained after the convolution layer (Conv1*1) processing. After the 1*1 convolution kernel with different channel numbers of stage3, stage4 and stage5, three deep feature maps of different scales are obtained. The CFE module is used to analyze the deep features. For context feature extraction, the CFE module consists of three atrous convolutions with different atrous rates (the atrous rate r is set to 3, 5 and 7, respectively) and a 1*1 standard convolution, which will extract three different contextual features. The scaled deep feature map and a shallow feature map are connected (Concat) along the channel axis to fuse the feature information of different receptive field ranges, that is, the shallow feature map and the deep feature map are combined in a cascade manner. , to obtain a fusion feature map of the same size as the image to be detected. Element-wise Sum is the fusion of 3 hole convolutions and standard convolutions.

In the embodiment of the present invention, determining the text outline of the text area according to the text centerline map and the direction distance map includes: forming a connected area according to the text center line; determining each sampling point on the connected area; according to the sampling point and the direction distance map , determine the direction point corresponding to each sampling point; according to the direction point, determine the text outline of the text area.

Optionally, forming a connected area according to the aggregation of adjacent points on the text centerline in the text centerline diagram includes: using an image processing connection method to aggregate adjacent points in the text centerline to form a connected area, wherein the image processing is connected. Methods may include treatment methods such as corrosion, expansion, and the like.

In the embodiment of the present invention, determining each sampling point on the connected area includes: determining the minimum rotated circumscribed rectangle of the connected area; dividing the rectangle into n equal parts to form n-1 vertical lines; wherein, n is an integer greater than 1 ; Take the midpoint of the intersection of each vertical line and the connected region as the sampling point.

Optionally, the determination of each sampling point on the connected region may also be obtained in other manners, for example, each sampling point may be determined at a preset distance on the connected region.

Optionally, a polygon bounding box can be generated according to the direction point, that is, the text outline of the text area. For example, an Alpha-shape algorithm (an algorithm that uses some feature points to describe the intuitive outline of a point set) can be used according to the direction. Points produce polygon bounding boxes.

In the embodiment of the present invention, after the text centerline map and the direction distance map are determined, post-processing is performed to obtain the text outline. Figure 4 is a schematic diagram of the process of determining the text outline of the text area. In Figure 4, (a) is the connected area formed by the two obtained text center lines, and (b) is a connected area for one of the obtained Obtain the minimum rotated circumscribed rectangle of the connected area, and then divide it into n equal parts along the long side of the rectangle, where n is 11, to obtain 10 vertical lines, with the midpoint of the intersection of each vertical line and the connected area as the sampling In (c), based on the sampling points and the direction distance map, for each sampling point, the corresponding direction points are calculated to obtain the direction points of the text edge area; in (d), based on the obtained The direction point of , uses the Alpha-shape algorithm to generate a polygon bounding box, the polygon bounding box is the text outline of the text area. Compared with segmentation-based methods such as PixelLink, TextSnake, and TextField, the post-processing method of the embodiment of the present invention is simpler and more efficient, and can improve the efficiency of scene text detection.

FIG. 5 is a schematic process diagram of a method for detecting text in a scene according to an embodiment of the present invention. In FIG. 5, (a) is an acquired image to be detected, and the image to be detected includes a text area with multiple words , a plurality of words are in a curved shape, and the adjacent distances of a plurality of words are relatively close; input the image to be detected into the scene text detection model, and obtain the single-channel text centerline diagram shown in (b) and (c) The eight-channel directional distance map shown, wherein the directional distance map is a polar coordinate return to the point on the center line of the text along eight directions (the eight directions include up, down, left, right, upper left, lower left, upper right, (lower right) to the text outline, and then post-processing based on the obtained direction distance and the text centerline diagram to obtain the text outline shown in (d), and realize the reconstruction of text instances.

The method for scene text detection provided by the embodiment of the present invention provides a way of using polar coordinates to express text, constructs a network structure to realize an end-to-end trainable deep learning model, that is, a scene text detection model; determines the image to be detected through semantic segmentation The text centerline map of the text area can effectively separate the text in the scene adjacent to the text area, and then obtain the direction distance map based on the text centerline map, and use polar coordinates to parameterize the text area. Scene text of any shape does not include redundant background information, which is beneficial to subsequent text recognition, solves the problem of detecting irregular scene text, and can more accurately reconstruct text instances of any shape. In addition, in order to solve the problem of large differences in scene text scales, atrous convolution is introduced to extract rich contextual feature information, which effectively improves the detection performance of long texts. Moreover, the post-processing method is simple and efficient, which can improve the efficiency of scene text detection.

As shown in FIG. 6 , another aspect of an embodiment of the present invention provides an apparatus 600 for scene text detection, including:

The obtaining module 601 obtains an image to be detected, and the image to be detected includes a text area;

The model prediction module 602 determines the text centerline map and the direction distance map of the text area;

The determining module 603 determines the text outline of the text area according to the text centerline diagram and the direction distance map,

The text centerline map is formed according to the text centerline of the text area, and the direction distance map is obtained by using polar coordinates to return the distances from points on the text centerline to the edge of the text outline along multiple preset directions.

In the embodiment of the present invention, the model prediction model 602 is further used to: before determining the text centerline map and the direction distance map of the text area, obtain a training set of image data, the image data includes the text area; construct a network structure and form a network structure A multi-task loss function is constructed, and the image data training set is used to train the network structure until the value of the loss function reaches the preset condition, and the scene text detection model is obtained.

In the embodiment of the present invention, the model prediction module 602 is further configured to: extract the features of the image to be detected and perform feature fusion to obtain a fused feature map; input the fused feature map into the scene text detection model to obtain the text centerline map and Direction distance map: The direction distance map is obtained by using polar coordinates to return the distances from the point on the center line of the returned text to the edge of the text outline along multiple preset directions.

In the embodiment of the present invention, the determining module 603 is further configured to: form a connected area according to the text center line; determine each sampling point on the connected area; and determine a group corresponding to each sampling point according to the sampling point and the directional distance map Direction point: Determine the text outline of the text area according to each group of direction points of each sampling point.

In this embodiment of the present invention, the determining module 603 is further configured to: determine the minimum rotated circumscribed rectangle of the connected region; divide the rectangle into n equal parts to form n-1 vertical lines; wherein n is an integer greater than 1; The midpoint of the intersection of the vertical line and the connected region is taken as the sampling point.

In this embodiment of the present invention, the model prediction module 602 is further configured to: perform feature extraction on the image to be detected through the backbone network to obtain multiple feature maps of different scales; and fuse multiple feature maps of different scales to obtain a fused feature map.

In the embodiment of the present invention, the model prediction module 602 is further configured to: extract features of the image to be detected through multiple convolution modules of the backbone network, and obtain multiple deep feature maps and shallow feature maps of different scales; Fusion of feature maps of different scales to obtain a fused feature map, including: using standard convolution and multiple atrous convolutional networks with different dilation rates to extract contextual features from multiple deep feature maps, and extract multiple deep features after contextual features. The image and the shallow feature map are combined in a cascaded manner to obtain a fusion feature map.

Another aspect of the embodiments of the present invention provides an electronic device, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, One or more processors are caused to implement the method for scene text detection provided by the embodiment of the present invention.

Another aspect of the embodiments of the present invention provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the method for scene text detection according to the embodiment of the present invention is implemented.

FIG. 7 shows an exemplary system architecture 700 of a scene text detection method or a scene text detection apparatus to which an embodiment of the present invention can be applied.

As shown in FIG. 7 , the system architecture 700 may include terminal devices 701 , 702 , and 703 , a network 704 and a server 705 . The network 704 is the medium used to provide the communication link between the terminal devices 701 , 702 , 703 and the server 705 . Network 704 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user can use the terminal devices 701, 702, 703 to interact with the server 705 through the network 704 to receive or send messages and the like. Various communication client applications may be installed on the terminal devices 701 , 702 and 703 , such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

The terminal devices 701, 702, 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

The server 705 may be a server that provides various services, such as a background management server that provides support for shopping websites browsed by the terminal devices 701 , 702 , and 703 (just an example). The background management server can analyze and process data such as images to be detected, and feed back the processing results (such as text outlines—just an example) to the terminal device.

It should be noted that the method for scene text detection provided by the embodiment of the present invention is generally executed by the server 705 , and accordingly, the apparatus for scene text detection is generally set in the server 705 .

It should be understood that the numbers of terminal devices, networks and servers in FIG. 7 are only illustrative. There can be any number of terminal devices, networks and servers according to implementation needs.

Referring next to FIG. 8 , it shows a schematic structural diagram of a computer system 800 suitable for implementing a terminal device according to an embodiment of the present invention. The terminal device shown in FIG. 8 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present invention.

As shown in FIG. 8, a computer system 800 includes a central processing unit (CPU) 801, which can be loaded into a random access memory (RAM) 803 according to a program stored in a read only memory (ROM) 802 or a program from a storage section 808 Instead, various appropriate actions and processes are performed. In the RAM 803, various programs and data necessary for the operation of the system 800 are also stored. The CPU 801 , the ROM 802 , and the RAM 803 are connected to each other through a bus 804 . An input/output (I/O) interface 805 is also connected to bus 804 .

The following components are connected to the I/O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc. ; and a communication section 809 including a network interface card such as a LAN card, a modem, and the like. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I/O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 810 as needed so that a computer program read therefrom is installed into the storage section 808 as needed.

In particular, the processes described above with reference to the flowcharts may be implemented as computer software programs in accordance with the disclosed embodiments of the present invention. For example, embodiments disclosed herein include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network via the communication portion 809, and/or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-described functions defined in the system of the present invention are performed.

It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented in special purpose hardware-based systems that perform the specified functions or operations, or can be implemented using A combination of dedicated hardware and computer instructions is implemented.

The modules involved in the embodiments of the present invention may be implemented in a software manner, and may also be implemented in a hardware manner. The described modules can also be provided in the processor, for example, it can be described as: a processor includes an acquisition module, a model prediction module and a determination module. Wherein, the names of these modules do not constitute a limitation of the module itself in some cases, for example, the acquisition module can also be described as "a module for acquiring images to be detected".

As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or may exist alone without being assembled into the device. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by a device, the device includes: acquiring an image to be detected, and the image to be detected contains a text area; determining the text of the text area Centerline diagram and direction distance diagram; determine the text outline of the text area according to the text centerline diagram and direction distance diagram.

According to the technical solution of the embodiment of the present invention, a method of using polar coordinates to express text is provided, and a network structure is constructed to realize an end-to-end trainable deep learning model, namely a scene text detection model; by determining the text center of the text area of the image to be detected Line map, which can effectively separate the text of the scene adjacent to the text area, and then obtain the direction distance map based on the text center line map, and use polar coordinates to parameterize the text outline of the text area, so that scene text of any shape can be detected. It does not include redundant background information, which is beneficial to subsequent text recognition, and solves the problem of text detection in irregular scenes, so that text instances of any shape can be more accurately reconstructed. In addition, in order to solve the problem of large differences in scene text scales, atrous convolution is introduced to extract rich contextual feature information, which effectively improves the detection performance of long texts. Moreover, the post-processing method is simple and efficient, which can improve the efficiency of scene text detection.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (11)

1. A method for detecting scene characters is characterized by comprising the following steps:

acquiring an image to be detected, wherein the image to be detected comprises a character area;

determining a character central line graph and a direction distance graph of the character area;

determining the character outline of the character area according to the character central line graph and the direction distance graph;

the character central line graph is formed according to a character central line of the character area, and the direction distance graph is obtained by regressing distances from points on the character central line to the edge of the character outline along a plurality of preset directions in a polar coordinate mode.

2. The method of claim 1, wherein determining the text center line map and the direction distance map for the text region is preceded by:

acquiring an image data training set, wherein an image data training sample in the image data training set comprises a character area;

constructing a network structure and constructing a multitask loss function for the network structure;

and training the network structure by using the image data training set until the value of the multitask loss function reaches a preset condition, and obtaining a scene character detection model.

3. The method of claim 2, wherein the multitask loss function comprises a segmentation loss function and a regression loss function, the segmentation loss function being constructed based on predicted and actual textual centerline maps, the regression loss function being constructed based on predicted and actual directional distance maps;

the determining of the character central line graph and the direction distance graph of the character area comprises the following steps:

and inputting the image to be detected into the trained scene character detection model, and predicting to obtain a character central line graph and a direction distance graph corresponding to the image to be detected.

4. The method as claimed in claim 3, wherein the step of inputting the image to be detected into the trained scene character detection model to predict a character centerline map and a direction distance map corresponding to the image to be detected comprises:

extracting the characteristics of the image to be detected and carrying out characteristic fusion according to the scene character detection model to obtain a fusion characteristic diagram;

and predicting to obtain a character central line graph and a direction distance graph corresponding to the image to be detected according to the fusion characteristic graph and the scene character detection model.

5. The method of claim 1, wherein determining the text outline of the text region from the text centerline map and the direction distance map comprises:

aggregating adjacent points on the character central line in the character central line graph to form a communication area;

determining each sampling point on the connected region;

determining a direction point corresponding to each sampling point according to the sampling points and the direction distance graph;

and determining the character outline of the character area according to the direction point of each sampling point.

6. The method of claim 5, wherein determining the respective sample points on the connected component comprises:

determining a minimum rotation circumscribed rectangle of the connected region;

dividing the rectangle into n equal parts to form n-1 vertical lines; wherein n is an integer greater than 1;

and taking the middle point of the intersection line of each vertical line and the communication area as the sampling point.

7. The method according to claim 4, wherein the extracting the features of the image to be detected and performing feature fusion to obtain a fused feature map comprises:

extracting the features of the image to be detected through a backbone network to obtain a plurality of feature maps with different scales;

and fusing a plurality of feature maps with different scales to obtain the fused feature map.

8. The method according to claim 7, wherein the extracting features of the image to be detected through a backbone network to obtain a plurality of feature maps with different scales comprises:

extracting the characteristics of an image to be detected through a plurality of convolution modules of a backbone network to obtain a plurality of deep characteristic diagrams and shallow characteristic diagrams with different scales;

fusing a plurality of feature maps with different scales to obtain the fused feature map, wherein the method comprises the following steps: and performing context feature extraction on the deep feature maps by adopting a standard convolution and a plurality of hole convolution networks with different hole rates, and merging the deep feature maps after the context features are extracted and the shallow feature maps in a cascading manner to obtain the fusion feature map.

9. An apparatus for scene text detection, comprising:

the acquisition module acquires an image to be detected, wherein the image to be detected comprises a character area;

the model prediction module is used for determining a character central line graph and a direction distance graph of the character area;

a determining module for determining the character outline of the character area according to the character central line graph and the direction distance graph,

the character central line graph is formed according to a character central line of the character area, and the direction distance graph is obtained by regressing distances from points on the character central line to the edge of the character outline along a plurality of preset directions in a polar coordinate mode.

10. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-8.

11. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-8.

Images