JP6540597B2 - INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING METHOD, AND PROGRAM - Google Patents

Description

  The present disclosure relates to an information processing apparatus, an information processing method, and a program capable of correcting and outputting an image obtained by photographing an original.

  There is known a technology capable of correcting and outputting an image obtained by photographing a document (for example, Patent Document 1 to Patent Document 5).

Unexamined-Japanese-Patent No. 2004-062350 JP, 2015-210749, A Unexamined-Japanese-Patent No. 11-243481 gazette JP, 2012-22413, A JP 2012-14430 A

  It is desirable that the corrected image be more easily visually recognizable by the user.

  In view of the circumstances as described above, the object of the present disclosure is to make it easier for the user to visually recognize an image after correction in a technology for correcting and outputting an image obtained by photographing an original. It is.

An information processing apparatus according to an aspect of the present disclosure is
An original area extracting unit for extracting an original area image which is an image of an area of the original from an original photographed image obtained by photographing the original;
A correction unit that corrects distortion of the extracted document area image to generate a corrected document image;
A determination unit that determines a corrected character image included in the corrected original image and identifies a character indicated by the corrected character image;
A mask image generation unit that generates a mask image including a character mask area that masks the corrected character images determined by the determination unit one-on-one;
An editable area calculation unit for calculating an editable area including all the character mask areas in the mask image;
The character mask area is detected from the editable area of the mask image, and text data indicating a character indicated by the corrected character image to be masked by the character mask area is arranged in the character mask area of the mask image, And an arranging unit for generating a composite image.

  According to this embodiment, since the corrected character image is determined after correcting the distortion of the document area image, the accuracy of character identification is improved.

  The determination unit determines a corrected character image included in the corrected document image by optical character recognition.

  The mask image generation unit changes the size of the generated mask image so as to match the size of the composite image.

  The determination unit further determines a corrected drawing image included in the corrected original image.

  The determination unit does not determine characters included in the corrected image as the corrected character image.

  According to the present embodiment, the text data is not arranged for the characters included in the corrected drawing image.

The mask image generation unit generates the mask image further including a drawing mask area that masks the corrected drawing image determined by the determination unit one-on-one,
The editable area calculation unit calculates the editable area further including all the drawing mask areas in the mask image;
The arrangement unit further detects the figure mask area from the editable area of the mask image, and the corrected figure image or the corrected figure image to be masked by the figure mask area in the figure mask area of the mask image Different images are further arranged to generate the composite image.

  According to the present embodiment, the arranging unit may typically arrange the corrected drawing image itself determined by the judging unit, or may arrange an image different from the corrected drawing image based on the user's selection. The image which processed the correction | amendment figure image as needed may be arrange | positioned.

  The information processing apparatus further includes an image forming unit that forms the composite image on a sheet.

  According to the present embodiment, the information processing apparatus is, for example, an image forming apparatus such as a printer.

  The information processing apparatus further includes an image reading unit that reads the document photographed image.

  According to the present embodiment, the information processing apparatus is, for example, an image forming apparatus having a scanner function.

An information processing method according to an aspect of the present disclosure is
An original area image which is an image of an area of the original is extracted from an original photographed image obtained by photographing the original,
Correcting distortion of the extracted original area image to generate a corrected original image;
Determining a corrected character image included in the corrected original image, and identifying a character indicated by the corrected character image;
Generating a mask image including a character mask area for masking the determined corrected character image one by one;
Calculating an editable area in the mask image including all of the character mask area;
The character mask area is detected from the editable area of the mask image, and text data indicating a character indicated by the corrected character image to be masked by the character mask area is arranged in the character mask area of the mask image, Generate a composite image.

A program according to an embodiment of the present disclosure is
An information processor,
A document area extraction unit for extracting a document area image which is an image of a region of the document from a document photographed image obtained by photographing a document;
A correction unit that corrects distortion of the extracted document area image to generate a corrected document image;
A determination unit that determines a corrected character image included in the corrected original image and identifies a character indicated by the corrected character image;
A mask image generation unit that generates a mask image including a character mask area that masks the corrected character image determined by the determination unit on a one-on-one basis;
An editable area calculation unit for calculating an editable area including all of the character mask area in the mask image, the character mask area is detected from the editable area of the mask image, and the character mask of the mask image is detected Text data indicating characters indicated by the corrected character image to be masked by the character mask area is arranged in the area, and functions as an arrangement unit for generating a composite image.

  According to the present disclosure, in the technology of correcting and outputting an image obtained by photographing an original, it is possible to make the image after correction more easily visually recognizable by the user.

FIG. 1 is a block diagram showing a hardware configuration of an image forming apparatus according to an embodiment of the present disclosure. FIG. 2 is a block diagram showing a functional configuration of the image forming apparatus. 5 is a flowchart showing an operation of the image forming apparatus. FIG. 6 is a diagram for explaining the operation of the image forming apparatus.

  Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

(1. Hardware configuration of image forming apparatus)
FIG. 1 is a block diagram showing a hardware configuration of an image forming apparatus according to an embodiment of the present disclosure.

  An information processing apparatus according to each embodiment of the present invention is an image forming apparatus (for example, an MFP or multifunction peripheral), and is hereinafter referred to as an MFP.

  The MFP 1 includes a control unit 11. The control unit 11 includes a central processing unit (CPU), a random access memory (RAM), a read only memory (ROM), a dedicated hardware circuit, and the like, and controls the overall operation of the MFP 1. A computer program that causes the MFP 1 to function as each functional unit (described later) is stored in a non-transitory storage medium such as a ROM.

  The control unit 11 is connected to the image reading unit 12, the image processing unit 14, the image memory 15, the image forming unit 16, the operation unit 17, the storage unit 18, the network communication unit 13 and the like. The control unit 11 controls the operation of each of the connected units, and transmits and receives signals or data to and from each unit.

  The control unit 11 operates the respective functions such as the scanner function, the print function and the copy function according to a job execution instruction input from the user through the operation unit 17 or a personal computer (not shown) connected to the network or the like. Control the drive and processing of the mechanism necessary to carry out the control.

  The image reading unit 12 reads an image from a document.

  The image processing unit 14 processes the image data of the image read by the image reading unit 12 as necessary. For example, the image processing unit 14 performs image processing such as shading correction in order to improve the quality after the image read by the image reading unit 12 is formed.

  The image memory 15 has an area for temporarily storing data of a document image obtained by reading by the image reading unit 12 and temporarily storing data to be printed by the image forming unit 16.

  The image forming unit 16 forms an image of image data or the like read by the image reading unit 12.

  The operation unit 17 includes a touch panel unit and an operation key unit that receive instructions from the user regarding various operations and processes that can be executed by the MFP 1. The touch panel unit includes a display unit 17 a such as an LCD (Liquid Crystal Display) provided with a touch panel.

  The network communication unit 13 is an interface for connecting to a network.

  The storage unit 18 is a large-capacity storage device such as a hard disk drive (HDD) that stores an original image and the like read by the image reading unit 12.

(2. Functional configuration of image forming apparatus)
FIG. 2 is a block diagram showing a functional configuration of the image forming apparatus.

  The MFP 1 loads an information processing program stored in the ROM, which is an example of a non-transitory storage medium that can be read by a computer, into the RAM and executes the information processing program. These functions as the mask image generation unit 104, the editable area calculation unit 105, and the arrangement unit 106.

  The document area extraction unit 101 reads a document photographed image from the image memory 15. The original area extraction unit 101 extracts an original area image I-2 from the read original photographed image.

  The correction unit 102 corrects distortion of the document area image extracted by the document area extraction unit 101 and generates a corrected document image.

  The determination unit 103 determines a corrected figure image and a corrected character image included in the corrected original image. The determination unit 103 refers to the character recognition database 111 to identify the character indicated by each of the plurality of extracted corrected character images.

  The mask image generation unit 104 generates a mask image based on the corrected original image.

  The editable area calculation unit 105 calculates an editable area in the mask image.

  The arranging unit 106 generates a composite image by arranging the corrected figure image and the text data indicating the character indicated by the corrected character image in the editable area in the mask image.

(3. Operation of image forming apparatus)
FIG. 3 is a flowchart showing the operation of the image forming apparatus. FIG. 4 is a diagram for explaining the operation of the image forming apparatus.

  The image reading unit 12 as the image input unit 110 scans and reads a document photographed image I-1 formed (printed) on a sheet. Alternatively, the network communication unit 13 as the image input unit 110 receives a document photographed image I-1 from an information processing apparatus (not shown) connected to the network N. The image memory 15 set in the image memory 15 stores the document photographed image I-1 acquired by the image reading unit 12 or the network communication unit 13 as the image input unit 110 (step S1).

  Specifically, “original photographed image I-1” is image data obtained by photographing an original with a digital camera or the like. The "document" is a document or the like in which characters and figures are visibly recorded. The "character" is typically a print of text data, but may be a handwritten character. In the present disclosure, “figure” means a recording target other than “character” on a sheet or the like. Specific examples of the “figure” include, but are not limited to, photographs, handwritten drawings such as figures and tables drawn electronically, and the like.

  The document area extraction unit 101 reads the document photographed image I-1 from the image memory 15. The document area extraction unit 101 extracts a document area image I-2 from the read document photographed image I-1 (step S2). The "original area image" is an image of the area of the original. More specifically, the "document area image" is an area bordered by the outline of the document (paper) in the entire document photographed image I-1, and includes characters, figures and a background. Specifically, the document area extraction unit 101 searches for an outline of a document (paper) included in the document photographed image I-1. When the document area extraction unit 101 detects one point P on the contour line, the document area extraction unit 101 searches the contour line by tracing the contour line starting from the point P. The document area extraction unit 101 determines, as a document area, a closed area surrounded by an outline whose start point and end point of search are the same and whose outer periphery is a predetermined distance or more. The document area extraction unit 101 extracts an image of the determined document area as a document area image I-2.

  The correction unit 102 corrects the distortion of the document area image I-2 extracted by the document area extraction unit 101 (step S3). When a document such as a rectangular sheet of paper is photographed by a digital camera or the like, the document area image I-2 is likely to be distorted by a trapezoid or the like. Therefore, the correction unit 102 corrects the trapezoidal original area image I-2 into a rectangular shape (see the arrow and dotted line in FIG. 4) by a known trapezoidal distortion correction technique or the like to generate a corrected original image I-3. .

  Discrimination unit 103 discriminates a corrected figure image included in corrected original image I-3 by a known method (edge detection or the like). For example, if the determination unit 103 determines that a graphic formed by consecutive edges (if the graphic is two-dimensional such as a rectangle, the occluded region surrounded by the outline of the graphic (the graphic has a specific size or less and is a single color) For example, a ruled line or a symbol such as a black circle or a hyphen may be discriminated as a corrected figure image, and the “corrected figure image” is included in the corrected original image I-3. The discrimination unit 103 discriminates the corrected character image included in the area (character area) excluding the corrected figure image in the corrected original image I-3 by optical character recognition (OCR, Optical Character Recognition). Specifically, the determination unit 103 extracts a plurality of corrected character images from the character area of the corrected document image I-3 excluding the correction image, that is, the determination unit 103 performs the correction. Included in figure image The “corrected character image” is an image of one character and one character (may be one word and one word depending on the language) included in the character area of the corrected original image I-3. The determination unit 103 refers to the character recognition database 111 to identify the character indicated by each of the plurality of extracted corrected character images Specifically, the character recognition database 111 includes the character image pattern and the character code The discrimination unit 103 searches the character recognition database 111 for an image pattern of the extracted corrected character image, and the character associated with the image pattern obtained by the search Get code

  The mask image generation unit 104 generates a mask image I-4 based on the corrected original image I-3 (step S5). The “mask image I-4” includes a figure mask area for masking the corrected figure image included in the corrected original image I-3 one to one and a character mask area for masking the corrected character image one to one. Typically, in the mask image I-4, pixels in the figure mask area and the character mask area are black (all bits are 1), and pixels in the other background areas are white (all bits are 0). Furthermore, the mask image generation unit 104 compares the size of the mask image I-4 with the size of the image to be output (“composite image I-6” described later), and based on the ratio, the mask image I-4 is Change the size to match the size of the image to be output. Furthermore, the mask image generation unit 104 labels each of the figure mask area and the character mask area included in the mask image I-4. “Labeling” means that the figure mask area and the character mask area included in the mask image I-4 correspond to the corrected figure image and the corrected character image included in the corrected original image I-3, respectively. It is to attach an identification number.

  The pixels in the background area may be colors other than white, such as the ground color of the document. The pixels in the figure mask area and the character mask area may be colors other than black, such as white, as long as they are different in color from the background area. Furthermore, the pixels of the figure mask area and the character mask area may have different colors.

  The editable area calculation unit 105 calculates an editable area I-5 in the mask image I-4 (step S6). The “editable area I-5” is an area within one frame including all the character mask area and the figure mask area in the mask image I-4. Specifically, the “editable area I-5” is an area within the smallest frame that can include all character mask areas and figure mask areas in the mask image I-4.

  The arranging unit 106 detects a figure mask area from the editable area I-5 in the mask image I-4. The arranging unit 106 arranges the corrected figure image to be masked by the figure mask area (in other words, the corrected figure image labeled in the figure mask area) in the figure mask area in the mask image I-4 (step S7) . At this time, the arrangement unit 106 deforms the corrected figure image such that the size of the corrected figure image matches the size of the figure mask area. The arranging unit 106 typically arranges the corrected figure image itself determined by the determining unit 103 (step S4). Alternatively, the arrangement unit 106 may arrange an image different from the corrected drawing image based on the user's selection, or may arrange an image obtained by processing the corrected drawing image as necessary.

  The arranging unit 106 detects a character mask area from the editable area I-5 in the mask image I-4. The arranging unit 106 is a text data indicating a character indicated by a corrected character image (in other words, a corrected character image labeled in the character mask area) masked by the character mask area in the character mask area in the mask image I-4. Are placed (step S8). Specifically, the arranging unit 106 arranges the text data of the character code determined by the determining unit 103 by the OCR in the character mask area in the mask image I-4. The font of the text data may be a font estimated from the corrected character image extracted by the determination unit 103. The size of each character may be the maximum size that the outline of each character mask area can encompass (in this case, the font size of all characters is not necessarily the same). Alternatively, the font sizes of all the characters may be the same.

  As described above, the arranging unit 106 arranges the corrected drawing image or another image in the drawing mask area of the mask image I-4, and generates the composite image I-6 by arranging the text data in the character mask area. Do.

  The display unit 17a, for example, allows the user to easily visually confirm the composite image I-6 generated by the arrangement unit 106 and the document photographed image I-1 (step S1) read by the image reading unit 12 Display side by side. An output method (printing, mail transmission, BOX storage, etc.) of the composite image I-6 is selected by the user via the operation unit 17. The image output unit 112 outputs the composite image I-6 according to the selected output method (step S9). For example, when printing is selected as the output method, the image forming unit 16 as the image output unit 112 forms (prints) the composite image I-6 on a sheet.

(4. Modification)
An image forming apparatus (MFP) has been described as an example of the information processing apparatus. Instead of this, as an embodiment of the information processing apparatus, a terminal device such as a personal computer, a tablet computer, a smart phone, a digital camera or the like may be used. In that case, the terminal device may acquire the document captured image scanned and read by the MFP through the network in step S1 (input of the document captured image). Alternatively, the terminal device may acquire an image captured by another terminal device (for example, a digital camera or a terminal device incorporating a digital camera) as a document captured image through the network. Alternatively, the terminal device may acquire an image captured by a camera incorporated in the terminal device itself as a document captured image. In addition, the terminal device may output the print data to the MFP through the network or output (display) to the display unit of the terminal device in step S9 (output a composite image).

  Alternatively, as an embodiment of the information processing apparatus, a computer as a server apparatus connected to a network (cloud) may be used. In that case, the server device may acquire the document photographed image scanned and read by the MFP through the network as Step S1 (input of the document photographed image). Alternatively, the server device may acquire an image captured by a terminal device (for example, a digital camera or a terminal device incorporating a digital camera) through a network as a document captured image. In step S9 (output a composite image), the server device may output the print data to the MFP through the network or may output (display) the display data to the display unit of the terminal device through the network.

(5. Summary)
2. Description of the Related Art In recent years, the spread of terminal devices such as smartphones and tablet computers incorporating digital cameras and the improvement of image quality of digital cameras have been remarkable. For this reason, instead of storing or printing document data created by word processor software, a document may be photographed by a terminal device, and an image obtained by photographing the document may be stored or printed. However, an image obtained by photographing a document may be distorted due to various conditions such as the angle of the camera. It may be less accurate to process the distorted image.

  Therefore, according to the present embodiment, the document area image I-2 is extracted from the document photographed image I-1 (step S2), the distortion of the document area image I-2 is corrected (step S3), and the corrected character is obtained by the OCR. An image is determined (step S4), and text data indicating a corrected character image is placed (step S8). According to the present embodiment, since the distortion of the document area image I-2 is corrected (step S3) and the corrected character image is determined by the OCR (step S4), the accuracy of the OCR is improved. Further, the document area image I-2 is extracted (step S2), and distortion of the document area image I-2 is corrected (step S3). Therefore, the area other than the document (background, desk on which the document is placed, etc.) As a result, it is possible to obtain an output result close to the document data created by the word processor software. In addition, the determination unit 103 does not determine the characters included in the corrected image as a corrected character image (step S4). As a result, since the text data is not arranged for the characters included in the corrected drawing image, the corrected drawing image to be arranged (step S7) becomes natural for the user without discomfort. Further, a mask image I-4 including a mask area for masking the correction drawing image and the correction character image included in the correction original image I-3 one to one is generated (step S5), and the text data is arranged (step S8) Therefore, it does not take time for the user to specify the location of the placement destination. As described above, the user can automatically convert the photographed image I-1 into an electronic document.

  On the other hand, in Patent Document 1, a document to be imaged (an existing document) is read as a photo and rearranged as an electronic document, but the user designates coordinates and selects a recognition range, a pasting position, etc. It takes time and effort.

1 ... MFP
101 Document area extraction unit 102 Correction unit 103 Discrimination unit 104 Mask image generation unit 105 Editable region calculation unit 106 Arrangement unit 110 Image input unit 111 Character recognition database 112 Image output unit

Claims (8)

An original area extracting unit for extracting an original area image which is an image of an area of the original from an original photographed image obtained by photographing the original;
A correction unit that corrects distortion of the extracted document area image to generate a corrected document image;
A corrected drawing image included in the corrected original image is determined, a corrected character image for each character is determined from a character area which is an area excluding the corrected drawing image in the corrected original image, and included in the corrected drawing image . A discrimination unit for identifying characters indicated by the corrected character image without judging characters as the corrected character image;
A mask image generation unit that generates a mask image including a character mask area that masks the corrected character image for each character determined one by one by the determination unit;
An editable area calculation unit for calculating an editable area including all the character mask areas in the mask image;
The character mask area is detected from the editable area of the mask image, and text data indicating a character indicated by the corrected character image to be masked by the character mask area is arranged in the character mask area of the mask image, An information processing apparatus comprising: an arranging unit that generates a composite image. The information processing apparatus according to claim 1, wherein
An information processing apparatus, wherein the determination unit determines a corrected character image included in the corrected document image by optical character recognition. The information processing apparatus according to claim 1 or 2, wherein
The mask image generation unit changes the size of the generated mask image so as to match the size of the composite image .
The placement unit sets the font size of the character corresponding to the text data to the maximum size that can be included in the outline of the character mask area without making the font size of all the characters the same. apparatus. An information processing apparatus according to any one of claims 1 to 3 , wherein
The mask image generation unit generates the mask image further includes an illustration mask area to mask one-to-one to the correction diagram image the determination unit has determined,
The editable area calculation unit calculates the editable area further including all the drawing mask areas in the mask image;
The arrangement unit further detects the figure mask area from the editable area of the mask image, and the corrected figure image or the corrected figure image to be masked by the figure mask area in the figure mask area of the mask image An information processing apparatus that further arranges different images to generate the composite image. An information processing apparatus according to any one of claims 1 to 4 , wherein
An information processing apparatus, further comprising: an image forming unit that forms the composite image on a sheet. An information processing apparatus according to any one of claims 1 to 5 , wherein
An information processing apparatus, further comprising: an image reading unit that reads the document photographed image. An original area image which is an image of an area of the original is extracted from an original photographed image obtained by photographing the original,
Correcting distortion of the extracted original area image to generate a corrected original image;
A corrected drawing image included in the corrected original image is determined, and a corrected character image which is an image for each character is determined from a character area which is an area excluding the corrected drawing image in the corrected original image, and the corrected drawing image is determined . Identifying the character indicated by the corrected character image without discriminating the character contained in the image as the corrected character image;
Generating a mask image including a character mask area in which the determined corrected character image for each character is masked one-to-one,
Calculating an editable area in the mask image including all of the character mask area;
The character mask area is detected from the editable area of the mask image, and text data indicating a character indicated by the corrected character image to be masked by the character mask area is arranged in the character mask area of the mask image, An information processing method that generates a composite image. An information processor,
A document area extraction unit for extracting a document area image which is an image of a region of the document from a document photographed image obtained by photographing a document;
A correction unit that corrects distortion of the extracted document area image to generate a corrected document image;
A corrected drawing image included in the corrected original image is determined, and a corrected character image which is an image for each character is determined from a character area which is an area excluding the corrected drawing image in the corrected original image, and the corrected drawing image is determined . A discrimination unit for discriminating the character indicated by the corrected character image without discriminating the character contained in the image as the corrected character image;
A mask image generation unit that generates a mask image including a character mask area that masks the corrected character image for each character determined one by one by the determination unit;
An editable area calculation unit for calculating an editable area including all of the character mask area in the mask image, the character mask area is detected from the editable area of the mask image, and the character mask of the mask image is detected A program which causes text data indicating characters indicated by the corrected character image to be masked by the character mask region to be arranged in an area and to function as an arranging unit for generating a composite image.

Images