JP2021131784A - Image processing system, program, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To extract and save items and values from an atypical form including handwritten characters. An image processing system of the present invention uses a trained model for extracting handwritten character pixels from an image to extract handwritten character pixels from a processed image obtained by scanning a manuscript. , The handwritten area is estimated from the processed image using the trained model for estimating the handwritten area in which the handwritten characters are written from the image, and the printed area including the printed characters is extracted from the processed image. , The print area having a predetermined positional relationship with the handwritten area is specified, and the character recognition result of the handwritten area and the character recognition result of the specified print area are stored in association with each other. [Selection diagram] Fig. 9

Description

The present invention relates to an image processing system that recognizes handwritten characters described in a scanned image.

In recent years, due to changes in the working environment accompanying the spread of computers, the digitization of documents handled in business is progressing. The target of such digitization extends to forms, and technologies for reading, structuring, and storing form information (items and values) are being studied. In particular, for forms (atypical forms) that have different positions where items and values are written and do not have a unified format, the positions where items and values are written cannot be defined in advance. It is necessary to analyze the contents and structure of and search for items and values one by one. In Patent Document 1, after the contents of the form are converted into text by OCR, a character string having a possibility of a value is searched, and a character string of an item corresponding to the value is specified around the character string to correspond to both. It discloses the technology to attach and store.

JP-A-2018-55255

However, in the technique of Patent Document 1, the character string having a possibility of the value and the character string of the item for the value are determined by referring to the pattern of the character string registered in advance. Therefore, it is not possible to handle forms that include unregistered items and values.

In order to solve the above problem, the image processing system of the present invention uses a trained model for extracting handwritten character pixels from an image to obtain handwritten character pixels from a processed image obtained by scanning a document. Using a handwriting extraction means for extracting and a learned model for estimating a handwriting area in which handwritten characters are written from an image, a handwriting area estimation means for estimating a handwriting area from the processing target image, and a handwriting area estimation means for estimating the handwriting area from the processing target image, and the processing target image The type area extraction means for extracting the type area including the type characters and the type area having a predetermined positional relationship with the handwriting area are specified, and the character recognition result of the handwriting area and the character recognition result of the specified type area are specified. It is characterized in that it is provided with a storage means for storing the above in association with each other.

According to the present invention, it is possible to store items and values included in a form image including handwritten characters in association with each other.

It is a figure which showed the structure of an image processing system. FIG. 2A is a diagram showing a configuration of an image processing device. FIG. 2B is a diagram showing the configuration of the learning device. FIG. 2C is a diagram showing a configuration of an image processing server. FIG. 2D is a diagram showing the configuration of the OCR server. FIG. 3A is a diagram showing a learning sequence of the image processing system. FIG. 3B is a diagram showing a usage sequence of the image processing system. It is a figure which shows the example of a form. FIG. 5A is a diagram showing a learning document scanning screen. FIG. 5B is a diagram showing a handwritten extraction correct answer data creation screen. FIG. 5C is a diagram showing a handwriting area estimation correct answer data creation screen. FIG. 5D is a diagram showing a form processing screen. FIG. 6A is a diagram showing a flow of a document sample image generation process. FIG. 6B is a diagram showing a flow of document sample image reception processing. FIG. 6C is a diagram showing a flow of correct answer data generation processing. FIG. 7A is a diagram showing a flow of learning data generation processing. FIG. 7B is a diagram showing a flow of learning processing. FIG. 8A is a diagram showing a configuration example of learning data extracted by handwriting. FIG. 8B is a diagram showing a configuration example of learning data for handwriting area estimation. FIG. 9A is a diagram showing a flow of form text conversion request processing. FIG. 9B is a diagram showing a flow of form text conversion processing. FIG. 10 is a diagram showing an outline of a data generation process in the form text conversion process. It is a figure which shows the flow of the form text conversion processing in Example 2. It is a figure which shows the example of a form.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings with reference to specific configurations in Examples. The configuration for realizing the present invention is not limited to the configuration described in the examples. A part of the configuration described in the examples may be omitted or replaced with an equivalent to the extent that the same effect can be obtained.

(Example 1)
FIG. 1 is a diagram showing a configuration example of the image processing system of this embodiment. The image processing system 100 includes an image processing device 101, a learning device 102, an image processing server 103, and an OCR server 104. The image processing device 101, the learning device 102, the image processing server 103, and the OCR server 104 are connected via the network 105.

The image processing device 101 is a digital multifunction device called an MFP or the like, and has a printing function and a scanning function (function as an image acquisition unit 111). The image processing device 101 scans a document such as a form to generate image data. In order to create a trained model (also called a learning model) that can identify the pixels of handwritten characters and the area where handwritten characters are written from the scanned image data, a sample manuscript is used. Scan to generate image data (hereinafter this image data is referred to as "manuscript sample image"). Scan multiple documents to obtain multiple document sample images. These manuscripts include manuscripts in which characters, symbols, and the like are handwritten by the user. Then, the image processing device 101 transmits a manuscript sample image to the learning device 102 via the network 105 to create a trained model.

Further, after the trained model is created, the image processing device 101 scans the form to be processed (a manuscript including characters (handwritten symbols, handwritten figures) handwritten by the user) and sets it as the processing target. Image data to be processed is obtained (hereinafter, this image data is referred to as a "processed image"). Then, the image processing device 101 transmits the image to be processed to the image processing server 103 via the network 105.

The learning device 102 functions as an image storage unit 115 that stores a document sample image generated by the image processing device 101. Further, it functions as a learning data generation unit 112 that generates learning data from the images accumulated in this way. The training data is data used for learning a neural network that performs handwriting extraction and handwriting area estimation. The learning device 102 functions as a learning unit 113 that learns a neural network using the generated learning data. The learning process executed by the learning unit 113 generates a learning result (a learned model including neural network parameters and the like). The learning device 102 transmits the learning result (learning model) to the image processing server 103 via the network 105. In the present embodiment, deep learning using a multi-layer neural network is used as one of the machine learning methods, but the learning method is not limited to deep learning, and other learning methods may be used.

The image processing server 103 functions as an image conversion unit 114 that converts an image to be processed. The image conversion unit 114 generates an image to be handwritten OCR based on the image to be processed. That is, the image conversion unit 114 extracts the handwritten pixels and the handwritten area by applying the trained model (learned neural network) to the image to be processed generated by the image processing device 101. At this time, the image processing server 103 infers using a neural network (learned model (also referred to as a learning model)) of the learning result generated by the learning device 102, and handwritten pixels (pixel positions) in the image to be processed. Is extracted (specified). Then, a handwritten extracted image in which the specified handwritten pixel is black and the other pixels are white is obtained. Further, the image processing server 103 estimates the handwritten area of the image to be processed generated by the image processing device 101 by using a neural network. The image processing server 103 estimates (identifies) the area (handwritten area) in which the handwritten characters are written in the image to be processed by inferring from the neural network by using the learning result generated by the learning device 102. For example, it is assumed that an area (entry field) including pixels of handwritten characters and surrounded by a ruled line is estimated as a handwritten area, but the area is not limited to the area surrounded by the ruled line. Here, the actual state of this handwritten area is information indicating a partial area including handwritten characters in the image to be processed, for example, the position (coordinates) of a specific pixel on the image to be processed and the width from the pixel position. It is expressed as information consisting of and height. In addition, a plurality of handwritten areas may be obtained depending on the number of items entered in the form.

That is, the handwritten extracted image is an image obtained by extracting only the pixels estimated to be handwritten in the image to be processed. On the other hand, the handwritten area is information indicating an area of a handwritten entry portion for each item in the image to be processed. Therefore, by determining the partial area on the handwritten extracted image based on the handwritten area, the handwritten characters in the handwritten extracted image can be divided and handled as the parts written for each item. Then, the image conversion unit 114 transmits the handwritten extracted image and the handwritten area to the OCR server 104. As a result, the OCR server 104 is instructed to set the area on the handwritten extracted image corresponding to each estimated handwritten area as the target area of the handwritten OCR.

Further, since the image conversion unit 114 retains an image in which the handwritten pixels included in the handwritten extracted image are removed from the image to be processed (pixels other than the handwritten characters (parts corresponding to the printed characters) remain, the image conversion unit 114 is hereinafter referred to as a "printed image". ) Is generated. Then, the image conversion unit 114 generates information on a region on the print image that includes the print that is the target of the print OCR (hereinafter, this region is referred to as a "print region"). The generation of the print area will be described later. Then, the image conversion unit 114 transmits the print image and the print area to the OCR server 104. As a result, the OCR server 104 is instructed to perform typographic OCR on each typographic area on the typographic image.

Further, the image conversion unit 114 receives the handwritten OCR result and the printed OCR result from the OCR server 104. Then, this is integrated and transmitted as text data to the image processing device 101. Hereinafter, this text data will be referred to as "form text data".

The OCR server 104 has functions as a handwritten OCR unit 116 and a printed OCR unit 117. When the OCR server 104 receives the handwritten extracted image and the handwritten area, the OCR server 104 performs OCR processing for the handwritten character, which is suitable for recognizing the handwritten character, on the handwritten area in the handwritten extracted image, and performs text data (text data ( OCR result) is acquired. The handwritten OCR unit 116 transmits the text data of the handwritten OCR result to the image processing server 103. Further, when the OCR server 104 receives the print image and the print area, the OCR server 104 performs OCR processing for the print suitable for character recognition on the print area in the print image to acquire text data. The print OCR unit 117 transmits the print OCR text data to the image processing server 103.

<Learning sequence>
The learning sequence in this system will be described. FIG. 3A is a diagram showing a learning sequence of the image processing system.

In step 301 (referred to as S301 or the like in the following description), when the user prepares a plurality of sample originals and gives an instruction to read the originals, the image acquisition unit 111 reads the plurality of sample originals and a plurality of sheets. Generates a manuscript sample image of (S302).

The manuscript sample image generated as described above is transmitted to the learning data generation unit 112 (S303). At this time, it is advisable to add ID information to the manuscript sample image. This ID information is, for example, information for identifying an image processing device 101 that functions as an image acquisition unit 111. The ID information may be user identification information for identifying a user who operates the image processing device 101, or group identification information for identifying a group to which the user belongs.

When the image is transmitted, the learning data generation unit 112 stores the manuscript sample image in the image storage unit 115 (S304).

When the user gives an instruction to give the correct answer data regarding the manuscript sample image to the learning device 102 (S305), the learning data generation unit 112 acquires the correct answer data (information on the handwritten pixel and information on the handwritten area). Then, it is linked to the original sample image and stored in the image storage unit 115 (S306). The associated manuscript sample image and correct answer data are data used for learning the neural network. The method of assigning correct answer data will be described later. Then, the learning data generation unit 112 generates learning data for use in the learning process based on the data accumulated in this way (S307). At this time, the learning data may be generated using only the manuscript sample image based on the specific ID information. After that, the learning data generation unit 112 transmits the generated learning data to the learning unit 113 (S308). When the learning data is generated only from the image to which the specific ID information is attached, the ID information is also transmitted. The learning unit 113 performs learning processing based on the received learning data and updates the learning model (S309). The learning unit 113 may hold a learning model for each ID information and perform learning only with the corresponding learning data. By associating the ID information with the learning model in this way, it is possible to construct a learning model specialized for a specific usage environment (specific device, user group, etc.).

<Usage sequence>
The usage sequence in this system will be described. FIG. 3B is a diagram showing a usage sequence of the image processing system.

In S351, when the user gives an instruction to read the document (form), the image acquisition unit 111 reads the document and generates an image to be processed (S352). The image read here is, for example, a form 400 or a form 450 as shown in FIG. These forms include name entry fields 403 and 451, address entry fields 401 and 452, and telephone number entry fields 402 and 453, and the name, address, and telephone number are entered by hand, respectively. However, since the arrangement of these entry fields (form layout) is determined by the user who created the form, it differs for each form. A form whose layout is not defined in advance is called an atypical form.

The image to be processed read as described above is transmitted to the image conversion unit 114 (S353). At this time, it is advisable to add ID information to the transmission data.

When the image conversion unit 114 receives the data of the image to be processed, the image conversion unit 114 receives an instruction to convert the image to be processed into text (S354). At this time, the image conversion unit 114 stores the image acquisition unit 111 as a return destination of the data of the OCR processing result. The image conversion unit 114 that has received the handwritten character processing instruction requests the learning unit 113 for the latest learning model (S355). In response to this request, the learning unit 113 transmits the latest learning model to the image conversion unit 114 (S356). If the ID information is specified at the time of the request from the image conversion unit 114, the latest learning model among the learning models corresponding to the ID information is transmitted. The image conversion unit 114 performs handwriting extraction and handwriting area estimation for the image to be processed based on the acquired learning model (S357). Subsequently, the image conversion unit 114 generates a print image and a print region from the image to be processed (S358). After that, for each handwriting area, the print area closest to the handwriting area is searched, and the handwriting area is associated with the value area and the print area is associated with the item (S359).

Then, from the handwritten extracted images, a partial image (handwritten extracted image) of the portion corresponding to the area related to the value is transmitted to the handwritten OCR unit 116 (S360). The handwritten OCR unit 116 performs handwritten OCR processing on the received handwritten extracted image, and acquires text data (handwritten) which is a character recognition result of the handwritten character image (S361). The handwriting OCR unit 116 transmits the acquired text data (handwriting) to the image conversion unit 114 (S362).

Further, from the print image, a partial image (print image) of the portion corresponding to the area related to the item is transmitted to the print OCR unit 117 (S363). The type OCR unit 117 performs the type OCR process on the type image and acquires the text data (type) of the type character image (S364). Then, the type OCR unit 117 transmits the acquired text data (type) to the image conversion unit 114 (S365). The image conversion unit 114 transmits the text data (printed type) relating to the item thus obtained and the text data (handwriting) relating to the value to the image acquisition unit 111 as form text data (S366). The image acquisition unit 111 that has acquired the form text data presents the form text data usage screen to the user (S367). After that, the image acquisition 111 outputs the form text data according to the usage of the form text data. For example, it may be sent to a separate external business system (not shown), or printed and output.

<Device configuration>
In order to realize the above-mentioned system, each device has the following configuration. FIG. 2A is a diagram showing a configuration of an image processing device. FIG. 2B is a diagram showing the configuration of the learning device. FIG. 2C is a diagram showing a configuration of an image processing server. FIG. 2D is a diagram showing a configuration of an OCR server.

As shown in FIG. 2A, the image processing apparatus 101 includes the following. It includes a CPU 201, a ROM 202, a RAM 204, a printer device 205, a scanner device 206, a document transport device 207, a storage 208, an input device 209, a display device 210, and an external interface 211. The devices are connected to each other by the data bus 203 so as to be able to communicate with each other.

The CPU 201 is a controller for comprehensively controlling the image processing device 101. The CPU 201 boots the OS (operating system) by the boot program stored in the ROM 202. The controller program stored in the storage 208 is executed on this OS. The controller program is a program for controlling the image processing device 101. The CPU 201 comprehensively controls each device connected by the data bus 203. The RAM 204 operates as a temporary storage area such as a main memory or a work area of the CPU 201.

The printer device 205 prints image data on paper (recording material, sheet). These include an electrophotographic printing method using a photoconductor drum and a photoconductor belt, and an inkjet method in which ink is ejected from a minute nozzle array to print an image directly on paper, but any method may be used. The scanner device 206 scans a document such as paper using an optical reader such as a CCD, obtains electrical signal data, converts the data, and generates image data. Further, the document transport device 207 such as an ADF (auto document feeder) transports the documents placed on the platen on the document transport device 207 to the scanner device 206 one by one.

The storage 208 is a non-volatile memory such as an HDD or SSD that can be read and written, and various data such as the controller program described above are recorded in the storage 208. The input device 209 is an input device composed of a touch panel, hard keys, and the like. The input device 209 receives the user's operation instruction. Then, the instruction information including the instruction position is transmitted to the CPU 201. The display device 210 is a display device such as an LCD or a CRT. The display device 210 displays the display data generated by the CPU 201. The CPU 201 determines which operation has been performed from the instruction information received from the input device 209 and the display data displayed on the display device 210. Then, the image processing device 101 is controlled according to the determination result, and new display data is generated and displayed on the display device 210.

The external interface 211 transmits and receives various data such as image data to and from an external device via a network such as a LAN, a telephone line, or a proximity radio such as infrared rays. The external interface 211 receives PDL data from an external device such as a learning device 102 or a PC (not shown). The CPU 201 interprets the PDL data received by the external interface 211 and generates an image. The generated image is printed by the printer device 205 or stored in the storage 108. Further, the external interface 211 receives image data from an external device such as the image processing server 103. The received image data is printed by the printer device 205, stored in the storage 108, and transmitted to another external device by the external interface 211.

The learning device 102 of FIG. 2B includes a CPU 231, a ROM 232, a RAM 234, a storage 235, an input device 236, a display device 237, an external interface 238, and a GPU 239. Each unit can send and receive data to and from each other via the data bus 233.

The CPU 231 is a controller for controlling the entire learning device 102. The CPU 231 boots the OS by a boot program stored in the ROM 232 which is a non-volatile memory. The learning data generation program and the learning program stored in the storage 235 are executed on this OS. The learning data is generated by the CPU 231 executing the learning data generation program. In addition, the CPU 231 learns a neural network that performs handwriting extraction by executing a learning program. The CPU 231 controls each unit via a bus such as the data bus 233.

The RAM 234 operates as a temporary storage area such as a main memory or a work area of the CPU 231. The storage 235 is a non-volatile memory that can be read and written, and records the above-mentioned learning data generation program and learning program.

The input device 236 is an input device composed of a mouse, a keyboard, and the like. The display device 237 is the same as the display device 210 described with reference to FIG. 2 (a).

The external interface 238 is the same as the external interface 211 described with reference to FIG. 2 (a).

The GPU 239 is an image processing processor that generates image data and learns a neural network in cooperation with the CPU 231.

The image processing server 103 of FIG. 2C includes a CPU 261 and a ROM 262, a RAM 264, a storage 265, an input device 266, a display device 267, and an external interface 268. Each unit can send and receive data to and from each other via the data bus 263.

The CPU 261 is a controller for controlling the entire image processing server 103. The CPU 261 boots the OS by a boot program stored in the ROM 262, which is a non-volatile memory. The image processing server program stored in the storage 265 is executed on this OS. By executing this image processing server program, the CPU 261 performs handwriting extraction and handwriting area estimation for the image to be processed. The CPU 261 controls each unit via a bus such as the data bus 263.

The RAM 264 operates as a temporary storage area such as a main memory or a work area of the CPU 261. The storage 265 is a non-volatile memory that can be read and written, and records the above-mentioned image processing program.

The input device 266 is the same as the input device 236 described with reference to FIG. 2 (b). The display device 267 is the same as the display device 210 described with reference to FIG. 2 (a).

The external interface 268 is the same as the external interface 211 described with reference to FIG. 2 (a).

The OCR server 104 of FIG. 2D includes a CPU 291 and a ROM 292, a RAM 294, a storage 295, an input device 296, a display device 297, and an external interface 298. Each unit can send and receive data to and from each other via the data bus 293.

The CPU 291 is a controller for controlling the entire OCR server 104. The CPU 291 boots the OS by a boot program stored in the ROM 292, which is a non-volatile memory. The OCR server program stored in the storage 295 is executed on this OS. By executing this OCR server program, the CPU 291 recognizes the handwritten characters and the printed characters of the handwritten extracted image and the printed image and converts them into text. The CPU 291 controls each unit via a bus such as a data bus 293.

The RAM 294 operates as a temporary storage area such as a main memory or a work area of the CPU 291. The storage 295 is a non-volatile memory that can be read and written, and records the above-mentioned image processing program.

The input device 296 is the same as the input device 236 described with reference to FIG. 2 (b). The display device 297 is the same as the display device 210 described with reference to FIG. 2 (a).

The external interface 298 is the same as the external interface 211 described with reference to FIG. 2 (a).

<Operation screen>
The user's instruction shown in S301 is given in the image processing device 101 via the following operation screen. FIG. 5A is a diagram showing an operation screen (hereinafter, referred to as a learning document scanning screen) when scanning a sample document and instructing to create a learning model.

The learning document scanning screen 500 is an example of a screen displayed on the display device 210. As shown in FIG. 5A, the learning document scan screen 500 includes a preview area 501, a scan button 502, and a transmission start button 503.

The scan button 502 is a button for starting scanning of the document set in the scanner device 206. When scanning is complete, a document sample image is generated and displayed in the preview area 501. By setting another document in the scanner device 206 and pressing the scan button 502 again, a plurality of document sample images can be held together.

When the document is read, the transmission start button 503 can be pressed. When the transmission start button 503 is instructed to be selected by the user, the original sample image is transmitted to the learning device 102.

The instruction to give the correct answer data by the user shown in S305 is performed on the following operation screens (FIGS. 5B to 5C). FIG. 5B is a diagram showing an operation screen (hereinafter, referred to as a handwritten extraction correct answer data creation screen) when designating the pixels of the handwritten character portion in the manuscript sample image. Further, FIG. 5C is a diagram showing an operation screen (hereinafter, referred to as a handwritten area estimation correct answer data creation screen) when designating a handwritten area including handwritten characters in the manuscript sample image. The user operates based on the display contents of the handwriting extraction correct answer data creation screen and the handwriting area estimation correct answer data creation screen to create correct answer data.

The handwritten extraction correct answer data creation screen 520 is an example of a screen displayed on the display device 237. As shown in FIG. 5B, the handwritten extraction correct answer data creation screen 520 includes an image display area 521, an image selection button 522, an enlargement button 523, a reduction button 524, an extraction button 525, an estimation button 526, and a save button 527. ..

The image selection button 522 is a button for selecting a document sample image received from the image processing device 101 and stored in the image storage unit 115. When the image selection button 522 is instructed, a selection screen (not shown) is displayed, and a document sample image can be selected. When the original sample image is selected, the selected original sample image is displayed in the image display area 521. The user operates on the original sample image displayed in the image display area 521 to create correct answer data.

The enlargement button 523 and the reduction button 524 are buttons for enlarging or reducing the display of the image display area 521. By instructing the enlargement button 523 or the reduction button 524, the original sample image displayed in the image display area 521 can be enlarged or reduced and displayed so that the correct answer data can be easily created.

The extraction button 525 and the estimation button 526 are buttons for selecting whether to create correct answer data of handwriting extraction or handwriting area estimation. If you select one, the selected button will be highlighted. When the extraction button 525 is selected, the correct answer data for handwriting extraction is created. When this button is selected, the user creates the correct answer data for handwriting extraction by operating as follows. As shown in FIG. 5B, the user operates the mouse cursor via the input device 236 and specifies by tracing the handwritten characters in the manuscript sample image displayed in the image display area 521. , Select the pixel corresponding to the handwritten character. Upon receiving this operation, the learning data generation unit 112 records the pixel positions on the document sample image selected by the above operation. That is, the correct answer data of the handwriting extraction is the position of the pixel corresponding to the handwriting on the manuscript sample image.

On the other hand, when the estimation button 526 is selected, the correct answer data for the handwriting area estimation is created. When this button is selected, the user creates the correct answer data for the handwriting area estimation by operating as follows. As shown by the dotted line frame in FIG. 5C, the user operates the mouse cursor via the input device 236 to create a ruled line on which handwritten characters in the original sample image displayed in the image display area 521 are written. Select the position of the enclosed area (in the entry field, not including the ruled line). This is an operation of selecting an area for each entry field of the form. Upon receiving this operation, the learning data generation unit 112 records the area selected by the above operation. That is, the correct answer data for estimating the handwritten area is an area (area in which handwritten characters can be entered) in the entry field on the manuscript sample image (hereinafter, the area in which the handwritten characters are entered is referred to as a "handwritten area". ).

The save button 527 is a button for saving the created correct answer data.

The correct answer data extracted by handwriting is stored in the image storage unit 115 as the following image. The size of the image showing the correct answer data of the handwritten extraction has the same size (width and height) as the original sample image. The value of the pixel at the handwritten character position selected by the user is a value indicating handwriting (for example, 0, and so on). The values of the other pixels are values indicating that they are not handwritten (for example, 255, and so on). Hereinafter, the image which is the correct answer data of such handwriting extraction will be referred to as "handwriting extraction correct answer image". An example of a handwritten extracted correct image is shown in FIG. 4 (c).

Further, the correct answer data for estimating the handwriting area is stored in the image storage unit 115 as the following image. The size of the image showing the correct answer data of the handwriting area estimation has the same size (width and height) as the original sample image. The value of the pixel corresponding to the handwriting area selected by the user is a value indicating that it is a handwriting area (for example, 0, and so on). Further, the values of the other pixels are values indicating that the area is not a handwriting area (for example, 255, and the same applies thereafter). Hereinafter, the image which is the correct answer data of such handwriting area estimation will be referred to as "handwriting area estimation correct answer image". An example of the handwritten area estimation correct image is shown in FIG. 4 (d).

The user's instruction shown in S351 is given in the image processing device 101 via the following operation screen. FIG. 5D is a diagram showing an operation screen (hereinafter, referred to as a form processing screen) displayed when scanning and processing a form to be processed. As shown in FIG. 5D, the form processing screen 520 includes a preview area 541, a scan button 542, and a transmission start button 543.

The scan button 542 is a button for starting scanning of the document set in the scanner device 206. When the scan is completed, the image to be processed is generated and displayed in the preview area 541.

When the document is read, the transmission start button 543 can be pressed. When the transmission start button 543 is pressed by the user, the image to be processed is transmitted to the image processing server 103.

<Original sample image generation process>
Next, the original sample image generation process by the image processing device 101 will be described. FIG. 6A is a diagram showing a flow of a document sample image generation process. This process is realized by the CPU 201 reading the controller program recorded in the storage 208, expanding it in the RAM 204, and executing it. This is started by the user operating the input device 209 of the image processing device 101.

The CPU 201 determines in S601 whether or not the document scanning instruction has been given. When the user performs a predetermined operation (instruction of the scan button 502) for scanning the document via the input device 209, it is determined as YES and the process proceeds to S602. If not, it is determined as NO, and the process proceeds to S604.

In S602, the CPU 201 controls the scanner device 206 and the document transport device 207 to scan the sample document and generate a document sample image. The manuscript sample image is generated as grayscale image data, but is not limited to this, and may be a color multi-valued image.

In S603, the CPU 201 transmits the document sample image generated in S602 to the learning device 102 via the external interface 211.

The CPU 201 determines in S604 whether or not to end the process. When the user performs a predetermined operation to end the original sample image generation process, it is determined as YES and the process is terminated. If not, it is determined as NO, and the process proceeds to S601.

Through the above processing, the image processing device 101 generates a document sample image and transmits it to the learning device 102. A plurality of document sample images are acquired according to the user's operation and the number of documents placed on the document transport device 207.

<Original sample image reception processing>
Next, the document sample image reception process by the learning device 102 will be described. FIG. 6B is a diagram showing a flow of document sample image reception processing. This process is realized by the CPU 231 reading the learning data generation program recorded in the storage 235, expanding it in the RAM 234, and executing it. This is started when the user turns on the power of the learning device 102.

The CPU 231 determines in S621 whether or not the original sample image has been received. If the CPU 231 has received the image data via the external interface 238, the CPU 231 determines YES and transitions to S622. If not, it is determined as NO, and the process proceeds to S623.

In S622, the CPU 231 records the received document sample image in a predetermined area of the storage 235.

The CPU 231 determines in S623 whether or not to end the process. When the user performs a predetermined operation for ending the document sample image reception process such as turning off the power of the learning device 102, it is determined as YES and the process is terminated. If not, it is determined as NO, and the process proceeds to S621.

<Correct answer data generation process>
Next, the correct answer data generation process by the learning device 102 will be described. FIG. 6C is a diagram showing a flow of correct answer data generation processing.

This process is realized by the learning data generation unit 112 of the learning device 102. This is started when the user performs a predetermined operation via the input device 236 of the learning device 102.

The CPU 231 determines in S641 whether or not a selection instruction for a document sample image has been made. When the user performs a predetermined operation (instruction of the image selection button 522) for selecting the original sample image via the input device 236, it is determined as YES and the process proceeds to S642. If not, it is determined as NO, and the process proceeds to S643.

In S642, the CPU 231 reads the original sample image selected by the user in S641 from the storage 235 and displays it on the operation screen (image display area 521).

In S643, the CPU 231 determines whether or not the user has given an input instruction for correct answer data. For the user to trace the handwritten characters on the manuscript sample image or to specify the handwritten area in which the handwritten characters are written, as described above with reference to FIGS. 5 (b) and 5 (c), via the input device 236. If the operation of is performed, it is determined as YES, and the transition to S644 occurs. If not, it is determined as NO, and the process proceeds to S647.

In S644, the CPU 231 determines whether the correct answer data input by the user is an operation for creating correct answer data for handwriting extraction or an operation for creating correct answer data for the handwritten area. If the user has performed an operation instructing the creation of correct answer data for handwritten extraction (selection of the extraction button 525), the CPU 231 determines YES and transitions to S645. On the other hand, in the case of the operation of instructing the correct answer data of the handwriting area estimation (the estimation button 526 is selected), the transition to S646 occurs.

In S645, the CPU 231 temporarily stores the correct answer data of the handwritten extraction input by the user in the RAM 234. As described above, the correct answer data of the handwriting extraction is the position information of the pixel corresponding to the handwriting in the manuscript sample image.

In S646, the CPU 231 temporarily stores the correct answer data of the handwriting area estimation input by the user in the RAM 234. As described above, the correct answer data for the handwriting area estimation is the area information corresponding to the handwriting area on the manuscript sample image.

The CPU 231 determines in S647 whether or not the instruction to save the correct answer data has been given. When the user performs a predetermined operation (instruction of the save button 527) for saving the correct answer data via the input device 236, it is determined as YES and the process proceeds to S648. If not, it is determined as NO, and the process proceeds to S650.

In S648, the CPU 231 generates a handwritten extraction correct answer image and saves it as handwritten extraction correct answer data. The CPU 231 generates a handwritten extraction correct answer image as follows. The CPU 231 generates an image having the same size as the original sample image read in S642 as the handwritten extraction correct image. All pixels of the image are set to values indicating that they are not handwritten. Next, in S645, the position information temporarily stored in the RAM 234 is referred to, and the value of the pixel at the corresponding position on the handwritten extracted correct image is changed to a value indicating that the handwriting is performed. The handwritten extraction correct answer image generated in this way is associated with the original sample image read out in S642 and stored in a predetermined area of the storage 235.

In S649, the CPU 231 generates a handwriting area estimation correct answer image and saves it as correct answer data for the handwriting area estimation. The CPU 231 generates a handwritten area estimation correct answer image as follows. The CPU 231 generates an image having the same size as the original sample image read in S642 as the handwritten area estimation correct image. All the pixels of the image are set to values indicating that they are not in the handwriting area. Next, in S646, the area information temporarily stored in the RAM 234 is referred to, and the value of the pixel in the corresponding area on the handwritten area estimated correct answer image is changed to a value indicating that it is a handwritten area. The handwritten area estimated correct answer image generated in this way is associated with the original sample image read out in S642 and stored in a predetermined area of the storage 235.

The CPU 231 determines in S650 whether or not to end the process. When the user performs a predetermined operation to end the correct answer data generation process, it is determined as YES and the process is terminated. If not, it is determined as NO and the transition to S641 is performed.

<Learning data generation process>
Next, the learning data generation process by the learning device 102 will be described. FIG. 7A is a diagram showing a flow of learning data generation processing. This process is realized by the learning data generation unit 112 of the learning device 102. This is started when the user performs a predetermined operation via the input device 209 of the image processing device 101.

First, in S701, the CPU 231 selects and reads out the original sample image stored in the storage 235. Since a plurality of document sample images are recorded in the storage 235 by the processing step of S622 in the flowchart of FIG. 6B, one of them is randomly selected.

In S702, the CPU 231 reads out the handwritten extraction correct answer image stored in the storage 235. Since the handwritten extraction correct answer image associated with the original sample image read out in S701 is stored in the storage 235 by the process of S648, it is read out.

In S703, the CPU 231 reads out the handwritten area estimated correct answer image stored in the storage 235. Since the handwritten area estimated correct answer image associated with the original sample image read out in S701 is stored in the storage 235 by the process of S649, it is read out.

In S704, the CPU 231 cuts out a part (for example, a size of vertical x horizontal = 256 x 256) in the original sample image read in S701 to generate an input image to be used for training data. The cutting position is randomly determined.

In S705, the CPU 231 cuts out a part of the hand-drawn extraction correct-answer image read in S702 to generate a correct-answer label image (teacher data, correct-answer image data) used for learning data of hand-drawn extraction. Hereinafter, this correct label image will be referred to as a "handwritten extracted correct label image". The position and size to be cut out are the same as the position and size to cut out the input image from the original sample image in S704.

In S706, the CPU 231 cuts out a part of the handwritten area estimation correct answer image read in S703 to generate a correct answer label image to be used for the learning data of the handwritten area estimation (hereinafter, this correct answer label image is referred to as "handwritten area estimated correct answer label". Called "image"). The position and size to be cut out are the same as the position and size to cut out the input image from the original sample image in S704.

In S707, the CPU 231 associates the input image generated in S704 with the handwritten extraction correct answer label image generated in S706, and stores it in a predetermined area of the storage 235 as learning data for handwritten extraction. In this embodiment, the learning data as shown in FIG. 8A is stored.

In S708, the CPU 231 associates the input image generated in S704 with the handwritten area estimation correct answer label image generated in S707, and stores it in a predetermined area of the storage 235 as learning data for handwriting area estimation. In this embodiment, the learning data as shown in FIG. 8B is stored.

The CPU 231 determines in S709 whether or not to end the learning data generation process. If the CPU 231 has generated training data for the number of predetermined learning data (determined by the user via the input device 236 of the learning device 102 at the start of this flowchart), YES. Judge and end the process. If not, it is determined as NO, and the process proceeds to S701.

As described above, the learning data used for the learning process of the neural network for extracting the handwritten pixels and the learning data used for the learning process of the neural network for estimating the handwritten area are generated. In order to increase the versatility of the neural network, the training data may be processed. For example, the input image is randomly selected from a predetermined range (for example, between 50% and 150%) and scaled at a variable magnification determined. The correct label image for handwriting extraction and handwriting area estimation is similarly scaled. Alternatively, the input image is rotated at a rotation angle determined by randomly selecting from a predetermined range (for example, between -10 degrees and 10 degrees). The correct label image for handwriting extraction and handwriting area estimation also rotates in the same manner. Considering scaling and rotation, when cutting out the input image and the correct label image for handwriting extraction and handwriting area estimation in S704, S705, and S706, use a slightly larger size (for example, length x width = 512 x 512). break the ice. Then, after scaling and rotation, the image is cut out from the central portion so as to have the size of the final input image and the correct label image for handwriting extraction and handwriting area estimation (for example, length x width = 256 x 256). Alternatively, the brightness of each pixel of the input image may be changed for processing. That is, the brightness of the input image is changed by using gamma correction. The gamma value is randomly selected and determined from a predetermined range (for example, between 0.1 and 10.0).

<Learning process>
Next, the learning process by the learning device 102 will be described. FIG. 7B is a diagram showing a flow of learning processing. This process is realized by the learning unit 113 of the learning device 102. This is started when the user performs a predetermined operation via the input device 236 of the learning device 102. In this embodiment, the mini-batch method is used for learning the neural network.

First, the CPU 231 initializes the handwriting extraction and the handwriting area estimation neural networks in S731, respectively. That is, the CPU 231 constructs two neural networks, and randomly determines and initializes the values of each parameter included in these neural networks. Various structures of these neural networks can be used, and for example, they may take the form of FCN (Full Convolutional Networks), which is a known technique. In addition, the neural network for estimating the handwritten area may take the form of YOLO (You Only Look None), which is a known technique.

The CPU 231 acquires the learning data in S732. The CPU 231 executes the learning data generation process shown in the flowchart of FIG. 7A to acquire a predetermined number of learning data (for example, 10 mini-batch size learning data).

The CPU 231 calculates the error of the handwritten extraction neural network in S733. That is, the input image included in each learning data of handwriting extraction is input to the neural network of handwriting extraction to obtain an output. The output has the same image size as the input image, and as a prediction result, the pixel value of the pixel determined to be handwritten is the value indicating handwriting, and the pixel value determined to be not is the pixel value. It is an image which is a value indicating that it is not handwritten. Then, the difference between the output and the handwritten extraction correct label image included in the learning data is evaluated to obtain an error. Cross entropy can be used as an index for the evaluation.

The CPU 231 adjusts the parameters of the handwritten extraction neural network in S734. That is, the parameter value of the handwritten extraction neural network is changed by the backpropagation method based on the error calculated in S733.

The CPU 231 calculates the error of the neural network for estimating the handwriting area in S735. That is, the input image included in each learning data of the handwriting area estimation is input to the neural network of the handwriting area estimation to obtain an output. The output has the same image size as the input image, and as a prediction result, the pixel value determined to be the handwritten area is the value indicating the handwritten area, and the pixel determined not to be the handwritten area is the pixel. It is an image which is a value indicating that the value is not a handwriting area. Then, the difference between the output and the handwritten area estimation correct label image included in the learning data is evaluated to obtain an error. As the index of the evaluation, cross entropy can be used as in the case of handwritten extraction.

In S736, the CPU 231 adjusts the parameters of the neural network for estimating the handwriting area. That is, the parameter value of the neural network for estimating the handwriting area is changed by the backpropagation method based on the error calculated in S735.

The CPU 231 determines in S737 whether or not to end the learning. This is done as follows. The CPU 231 determines whether or not the processes of S732 to S736 have been performed a predetermined number of times (for example, 60,000 times). The predetermined number of times can be determined by the user inputting an operation at the start of this flowchart. If it has been performed a predetermined number of times, it is determined as YES, and the process transitions to S738. If not, the process proceeds to S732 and the learning of the neural network is continued.

In S738, the CPU 231 transmits, as a learning result, the parameters of the neural network for handwriting extraction and handwriting area estimation adjusted in S734 and S736 (that is, the learned learning model) to the image processing server 103, respectively.

<Form text request processing>
Next, the form text conversion request processing by the image processing device 101 will be described. The image processing device 101 scans a form in which handwritten characters are written to generate an image to be processed. Then, the image data to be processed is transmitted to the image processing server 103 to request the form text conversion. FIG. 9A is a diagram showing a flow of form text conversion request processing. This process is realized by the CPU 201 of the image processing device 101 reading the controller program recorded in the storage 208, expanding it into the RAM 204, and executing it. This is started when the user performs a predetermined operation via the input device 209 of the image processing device 101.

First, in S901, the CPU 201 controls the scanner device 206 and the document transport device 207 to scan the document and generate an image to be processed. The image to be processed is generated as grayscale image data.

In S902, the CPU 201 transmits the image to be processed generated in S901 to the image processing server 103 via the external interface 211.

The CPU 201 determines in S903 whether or not the processing result has been received from the image processing server 103. When the processing result is received from the image processing server 103 via the external interface 211, it is determined as YES, and the process proceeds to S904. If not, it is determined as NO, and the processing step of S903 is repeated.

The CPU 201 outputs the processing result received from the image processing server 103 in S904, that is, the form text data generated by recognizing the handwritten characters and printed characters included in the processing target image generated in S901. For example, the user can transmit the form text data to the transmission destination set by operating the input device 209 via the external interface 211.

<Form text processing>
Next, the form text conversion process by the image processing server 103 will be described. FIG. 9B is a diagram showing a flow of form text conversion processing. FIG. 10 is a diagram showing an outline of a data generation process in the form text conversion process. The image processing server 103 that functions as the image conversion unit 114 receives the image to be processed from the image processing device 101, and obtains text data by OCRing the printed characters and handwritten characters included in the scanned image data. The OCR for the type is executed by the type OCR unit 117. The handwritten OCR unit 116 executes the OCR for the handwritten characters. The form text conversion process is realized by the CPU 261 reading the image processing server program stored in the storage 265, deploying it in the RAM 264, and executing it. This is started when the user turns on the power of the image processing server 103.

First, in S951, the CPU 261 loads a neural network (learning model) that performs handwriting extraction and a neural network (learning model) that estimates the handwriting area. The CPU 261 constructs the same neural network as in the case of S731 in the flowchart of FIG. 7 (b). Then, the CPU 261 reflects the learning results (parameters of the neural network for handwriting extraction and parameters of the neural network for estimating the handwriting area) transmitted from the learning device 102 in the process of S738 to the constructed neural network, respectively. , Load the trained neural network (learning model).

The CPU 261 determines in S952 whether the image to be processed has been received from the image processing device 101. If the image to be processed has been received via the external interface 268, it is determined as YES, and the process proceeds to S953. If not, it is determined as NO, and the process proceeds to S965. As an example, here, it is assumed that the form 400 of FIG. 10 (form 400 shown in FIG. 4) is received as the image to be processed.

In S953, the CPU 261 extracts handwritten pixels from the processing target image received from the image processing device 101. The CPU 261 inputs the image to be processed into the neural network for handwriting extraction constructed in S951 to estimate the handwritten pixels. The following image data can be obtained as the output of the neural network. Pixels that have the same image size as the image to be processed and are determined to be handwritten as a prediction result have a value indicating that they are handwritten, and pixels that are determined not to be handwritten have a value indicating that they are not handwritten. , The recorded binary image data can be obtained. Then, the processing target image extracts the pixel at the same position as the pixel of the value indicating that it is handwritten in the image data to generate the handwritten extracted image. As a result, an image including only the pixels determined to be handwritten, such as the handwritten extracted image 1001 of FIG. 10, can be obtained.

In S954, the CPU 261 estimates the handwriting area from the processing target image received from the image processing device 101. The CPU 261 inputs the image to be processed into the neural network for estimating the handwriting area constructed in S951 to estimate the handwriting area. The following image data can be obtained as the output of the neural network. Pixels that have the same image size as the image to be processed and are determined to be in the handwriting area as a prediction result have a value indicating that they are in the handwriting area, and pixels that are determined not to be in the handwriting area are not in the handwriting area. Binary image data in which the values indicating the above are recorded can be obtained. In S305, the user created the correct answer data for the handwriting area estimation for each entry item of the form in consideration of the ruled line frame (entry column). Since the neural network that estimates the handwriting area learns this, it outputs the pixels estimated to be the handwriting area for each entry field (entry item). Since the output of the neural network for estimating the handwriting area is the prediction result for each pixel, the estimated pixels in the handwriting area do not necessarily form a rectangle. In the present embodiment, in order to make the handwriting area easy to handle, an circumscribed rectangle circumscribing the estimated handwriting area is set. A known technique can be applied to the setting of the circumscribing rectangle. Each circumscribed rectangle can be expressed as information consisting of the upper left end point and the width and height on the image to be processed. The rectangular information group obtained in this way is used as a handwriting area. In 1002 of FIG. 10, the position of the handwritten area estimated with respect to the image to be processed (form 400) is shown by a dotted line frame and illustrated (1011 to 1013).

In S955, the CPU 261 generates a print image by removing pixels determined to be handwritten from the image to be processed based on the output of the neural network of handwriting extraction obtained in S953. The CPU 261 sets the pixels of the image to be processed and the pixels at the same positions as the pixels whose pixel values are the values indicating handwriting in the image data output by the neural network to white (RGB = (255, 255, 255)). change. As a result, the print image 1003 of FIG. 10 is obtained.

In S956, the CPU 261 extracts a print area from the print image generated in S955. The CPU 261 extracts a partial area on the type image including the type as the type area. Here, the partial area is a group (object) of print contents, for example, a character line consisting of a plurality of characters, a sentence consisting of a plurality of character lines, or an object such as a figure, a photograph, a table, or a graph. .. As a method for extracting this partial region, for example, the following method can be adopted. A binary image is generated by binarizing a printed image into black and white. In this binary image, a portion where black pixels are connected (connected black pixels) is extracted, and a rectangle circumscribing this is created. By evaluating the shape and size of the rectangle, a character or a group of rectangles that are a part of the character can be obtained. By evaluating the distance between the rectangles and integrating the rectangles having a distance equal to or less than a predetermined threshold value, the rectangle group which is a character can be obtained. When rectangles of characters of similar size are lined up close together, they can be integrated to obtain a group of rectangles of character lines. When rectangles of character lines with the same short side length are arranged at equal intervals, they can be integrated to obtain a group of rectangles in a sentence. It is also possible to obtain rectangles containing objects other than characters, lines, and sentences, such as figures, photographs, tables, and graphs. From the rectangles extracted above, a single character or a rectangle that is a part of a character is excluded. The remaining rectangle is used as a partial area. In 1004 of FIG. 10, the type region extracted with respect to the type image is illustrated by a dotted line frame (1014-1018).

In S957, the CPU 261 selects one of the handwriting areas estimated in S954 and acquires the corresponding print area. Specifically, the closest printing area located to the left or upward with respect to the handwriting area is searched for, and the corresponding area is set as the printing area corresponding to the handwriting area. That is, the type area having a predetermined positional relationship with the handwriting area (the type area closest to the left direction or the upward direction) is acquired as the type area corresponding to the handwriting area. For example, in FIG. 10, since the printing area closest to the handwriting area 1011 is the printing area 1016, the area 1011 and the area 1016 are associated with each other. FIG. 101005 shows how the handwriting area and the print area are associated with each other.

In S958, the CPU 261 cuts out only the part of the handwritten area selected in S957 from the handwritten extracted image generated in S953, and transmits it to the handwritten OCR unit 116 via the external interface 268. Then, the handwritten OCR is executed on the cut out handwritten extracted image. That is, the character recognition process is executed based on the handwritten pixels included in the handwritten area. A known technique can be applied to the handwritten OCR to realize it. By handwriting OCR of the handwritten extracted image, for example, as shown in the address entry field 401 and the telephone number entry field 453 in FIG. 4, the printed contents such as the type and the mark in the handwriting entry field are subject to the handwriting OCR. It reduces the possibility of becoming.

The CPU 261 determines in S959 whether or not the handwritten OCR result is received from the handwritten OCR unit 116. The handwritten OCR result is text data acquired by the handwritten OCR unit 116 recognizing handwritten characters. If the handwritten OCR result has been received from the handwritten OCR unit 116 via the external interface 268, it is determined as YES, and the process proceeds to S955. If not, the process of S959 is repeated.

In S960, the CPU 261 cuts out only the part of the type area corresponding to the handwriting area acquired in S957 from the type image generated in S955, transmits it to the type OCR unit 117 via the external interface 268, and transmits the type OCR. Let it run. A known technique can be applied to the type OCR to realize it.

The CPU 261 determines in S961 whether or not the print OCR result has been received from the print OCR unit 117. The type OCR result is text data acquired by the type OCR unit 117 recognizing the type included in the type area. If the print OCR result has been received from the print OCR unit 117 via the external interface 268, it is determined as YES, and the process proceeds to S961. If not, the process of S961 is repeated.

In S962, the CPU 261 generates and stores the form text data 1006 shown in FIG. 10 using the print OCR result obtained in S961 as an item and the handwritten OCR result obtained in S959 as a value.

In S963, the CPU 261 determines whether or not the processing after S957 has been performed on all the handwriting areas estimated in S954. When processing is performed for all the handwriting areas, it is determined as YES, and the process proceeds to S964. If not, it is determined as NO, and the process proceeds to S957.

In S964, the CPU 261 transmits and outputs the form text data generated in S962 to the image processing device 101. In the present embodiment, the form text data is transmitted to the image processing device 101, but the present invention is not limited to this, and if the output destination external server is specified in advance, the form text data is directly sent to the external server. It may be configured to transmit.

In S965, the CPU 261 determines whether or not to end the process. When the user performs a predetermined operation such as turning off the power of the image processing server 103, it is determined as YES and the process is terminated. If not, it is determined as NO, and the process proceeds to S952.

As described above, as shown in this embodiment, it is possible to read the described information (items and values) from the scanned image of the form including the handwritten entry without registering the items and values in advance. As a result, even atypical forms for which it is not known in advance what kind of items are included can be digitized without relying on human labor.

In this embodiment, the handwritten extracted image is cut out for each handwritten area to generate and transmit a plurality of partial images, but this is not the case. The handwritten extracted image generated from the image to be processed and the handwritten area may be transmitted to the handwritten OCR unit, and the processing after S957 may be performed for each handwritten area from the OCR result.

Further, in the present embodiment, the handwritten extraction is performed on the image to be processed to generate the handwritten extracted image, but this is not the case. The handwriting extraction may be performed only within the area estimated by the handwriting area estimation. Even in this way, the image that is the target of the handwritten OCR is a handwritten area indicating an entry item and includes only handwritten pixels.

(Example 2)
In the form that the user fills out by hand, the entry fields are often crowded in order to make it easier to notice the omission. Of these, forms with vertical entry fields often have an item name on the left and an entry field on the right, so there is a high possibility that there will be an item on the left of the handwritten entry. On the other hand, a form in which entry fields are lined up side by side often has an item name at the top and an entry field at the bottom, so there is a high possibility that there are items above the written handwriting.

In this embodiment, a case will be described in which the search direction of the print area is limited according to the positional relationship between the handwritten areas when the print area corresponding to the handwritten area is acquired in the form text conversion process by utilizing this tendency. The configuration of the image processing system of this embodiment is the same as that of the first embodiment except for the feature portion. Therefore, similar reference numerals will be given to the same configurations, and detailed description thereof will be omitted.

<Form text processing>
The process in the second embodiment will be described with reference to FIG. The processes from S951 to S956 are the same as the processes of the same reference numerals in Example 1.

The CPU 261 selects one of the handwriting areas estimated in S954 in S1101.

In S1102, the CPU 261 determines whether the handwriting area selected in S1101 is densely packed in the left-right direction or densely packed in the vertical direction. Specifically, when the other handwriting area closest to the one selected in S1101 is on the left or right, it is concentrated in the left-right direction, and when the other handwriting area closest to the one selected in S1101 is on the top or bottom, it is dense in the vertical direction. Is determined. The form 1200 shown in FIG. 12 will be described as an example. It is assumed that the form 1200 is received in S952, the areas 1201 to 1204 are estimated as the handwritten area (indicated by the long dotted line) in S954, and the areas 1205 to 1211 are extracted as the printed area (indicated by the short dotted line) in S956. Of these, since the handwriting area 1201 is the area 1202 in which the closest handwriting area is below, it is determined that the handwriting area 1201 is dense in the vertical direction. On the other hand, since the handwriting area 1203 is the area 1204 with the closest handwriting area on the right, it is determined that the handwriting area 1203 is dense in the left-right direction. If it is dense in the left-right direction, it is determined as YES, and the process transitions to S1103. If not, it is determined as NO, and the process proceeds to S1104.

In S1103, the CPU 261 acquires the print area closest to the upward direction from the handwriting area selected in S1101. In the form 1200 of FIG. 12, the handwriting area 1203 is densely packed in the left-right direction, and the print area closest to the handwriting area 1203 in the upward direction is the area 1210. Therefore, the area 1210 is acquired as the print area corresponding to the handwriting area 1203.

In S1104, the CPU 261 acquires the print area closest to the left from the handwriting area selected in S1101. In the form 1200 of FIG. 12, the handwriting area 1201 is densely packed in the vertical direction, and the print area closest to the left direction from the handwriting area 1201 is the area 1207. Therefore, the area 1207 is acquired as the print area corresponding to the handwriting area 1201.

The processes from S958 to S967 are the same as the processes of the same reference numerals in Example 1.

According to the second embodiment, the configuration of the handwritten area (= entry field of the form) is analyzed, and the items are searched for in the direction in which there is a high possibility that the item corresponding to each handwritten value exists. This makes it easier to associate items that are more suitable for handwritten values.

Although the description is omitted in this embodiment, if the print area is not found above or in the left direction of the handwriting area in S1103 or S1104, the print area may be searched again in the left or upward direction, respectively. Further, a threshold value may be set for the distance between the handwriting area and the print area, and the print area farther than the threshold value from the handwriting area may be controlled so as not to be associated with each other.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The present invention may be applied to a system composed of a plurality of devices or a device composed of one device. For example, in the embodiment, the learning data generation unit 112 and the learning unit 113 are realized by the learning device 102, but they may be realized by separate devices. In this case, the device that realizes the learning data generation unit 112 transmits the learning data generated by the learning data generation unit 112 to the device that realizes the learning unit 113. Then, the learning unit 113 learns the neural network based on the received learning data. Although the image processing device 101 and the image processing server 103 have been described as separate devices, the image processing device 101 may have the function of the image processing server 103. Although the image processing server 103 and the OCR server 104 have been described as separate devices, the image processing server 103 may have a function as the OCR server 104.

The present invention is not limited to the above examples, and various modifications (including organic combinations of each example) are possible based on the gist of the present invention, and these are excluded from the scope of the present invention. is not it. That is, all the configurations in which each of the above-described examples and modifications thereof are combined are also included in the present invention.

In the embodiment, as shown in S956, the extraction of the print region is determined based on the connectivity of the pixels, but it may be estimated using a neural network in the same manner as the handwriting region estimation. Handwritten area estimation In the same way as when creating a correct image, the user selects the type area, creates correct answer data based on it, newly configures a neural network that estimates the type OCR area, and refers to the correct answer data. To learn.

In the embodiment, the learning data was generated from the learning data generation process at the time of the learning process. However, a large amount of learning data may be generated in advance by the learning data generation process, and a mini-batch size may be sampled from the learning data at any time during the learning process.

In the embodiment, the input image is generated as a grayscale image, but it may be generated as another format such as a full-color image.

The definitions of abbreviations appearing in each embodiment are as follows. The MFP is a Multi Function Peripheral. The ASIC is an Application Special Integrated Circuit. The CPU is a Central Processing Unit. RAM is Random-Access Memory. ROM is a Read Only Memory. HDD stands for Hard Disk Drive. SSD stands for Solid State Drive. LAN is a Local Area Network. PDL is Page Description Language. The OS is an Operating System. A PC is a Personal Computer. OCR stands for Optical Character Recognition / Reader. CCD is a Charge-Coupled Device. The LCD is a Liquid Crystal Display. ADF stands for Auto Document Feeder. CRT stands for Cathode Ray Tube. GPU is a Graphics Processing Unit. The ID is an identity.

Claims (11)

A handwriting extraction means for extracting the pixels of the handwritten characters from the image to be processed obtained by scanning the original using a trained model for extracting the pixels of the handwritten characters from the image.
A handwriting area estimation means for estimating a handwriting area from the image to be processed by using a learned model for estimating a handwriting area in which handwritten characters are written from an image, and a handwriting area estimation means.
A type area extraction means for extracting a type area including a type character from the image to be processed, and a type area extraction means.
A storage means for identifying the type area having a predetermined positional relationship with the handwriting area and storing the character recognition result of the handwriting area and the character recognition result of the specified type area in association with each other.
An image processing system characterized by being equipped with. The image processing system according to claim 1, wherein the character recognition result of the handwritten area is obtained by performing handwritten character recognition processing based on the pixels of the handwritten character included in the estimated handwritten area. .. The printing area extraction means is characterized in that the printing area is extracted by extracting the printing area from the image generated by removing the pixels of the handwritten character from the processing target image. Item 2. The image processing system according to Item 1 or 2. Any one of claims 1 to 3, wherein the character recognition result of the type area is obtained by performing the character recognition process of the type on the type characters included in the extracted type area. The image processing system described in. Any of claims 1 to 4, wherein the type area having a predetermined positional relationship with the handwriting area is a type area closest to the handwriting area in the left direction or the upward direction. The image processing system according to item 1. The print area having a predetermined positional relationship with the handwriting area is a print area closest to the handwriting area in the left direction when the handwriting area is densely arranged in the vertical direction with another handwriting area. The first to fourth aspects of claim 1 to 4, wherein when the handwriting area is densely arranged in the left-right direction with another handwriting area, the handwriting area is the print area closest to the handwriting area in the upward direction. The image processing system according to any one of the items. The trained model for extracting the pixels of the handwritten character from the image is created by learning based on the correct answer data created based on the pixels of the handwritten character portion specified by the user on the sample image. The image processing system according to any one of claims 1 to 6, wherein the image processing system is characterized by the above. The trained model for estimating the handwritten area in which handwritten characters are written from the image is learned based on correct answer data created based on the position of the entry field specified by the user on the sample image. The image processing system according to any one of claims 1 to 7, wherein the image processing system is created. The trained model for extracting the pixels of the handwritten character from the image and the trained model for extracting the pixels of the handwritten character from the image are trained models constructed by deep learning. The image processing system according to any one of claims 1 to 8, wherein the image processing system is characterized. A program for causing a computer to function as each means of the image processing system according to any one of claims 1 to 8. A handwriting extraction step that extracts the pixels of the handwritten characters from the image to be processed obtained by scanning the manuscript using a trained model for extracting the pixels of the handwritten characters from the image.
Using a trained model for estimating the handwritten area in which handwritten characters are written from the image, a handwritten area estimation step for estimating the handwritten area from the image to be processed, and a handwriting area estimation step.
A type area extraction step for extracting a type area including a type character from the image to be processed, and a type area extraction step.
A save step of identifying the type area having a predetermined positional relationship with the handwriting area, and saving the character recognition result of the handwriting area and the character recognition result of the specified type area in association with each other.
An image processing method characterized by having.

Images