JP2013037419A - Handwritten information reflection system, handwritten information reflection method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a handwritten information reflection system and the like to reflect handwritten information extracted from a paper medium in reflow type electronic book data.SOLUTION: In S1, a page space of a paper document 4 having marking lines is read and read image data is generated. In S2, the read image data is scanned and areas having different background colors are extracted. In S3, edges in each extracted area are calculated and binarized. In S4 to S6, row areas each representing a row except for spacing between rows are extracted from each binarized area, character areas each representing a character are extracted from each row area, and character recognition is performed on each character area to identify its character code. In S7, a group of the character codes in each extracted area is stored as handwritten information data. Then extracted character strings of the handwritten information data are checked against electronic book data, and the handwritten information data is reflected in matching positions.

Description

  The present invention relates to a handwritten information reflecting system, a handwritten information reflecting method, and a program, and more particularly to a handwritten information reflecting system that reflects handwritten information extracted from a paper medium in electronic book data.

  2. Description of the Related Art In recent years, electronic books using digital data displayed on a display for browsing have been provided in order to save paper and reduce bulky paper transportation and storage costs.

  Various utilization methods are possible by digitizing a book. In Patent Document 1, a vertical stripe pattern having the same thickness as that of an actual book corresponding to the reading progress information is displayed, so that the user can determine which position the currently displayed page corresponds to as a whole book. A technique for intuitively and roughly recognizing is proposed.

  In Patent Document 2, keywords as indexes are extracted for each page, and by creating a table of contents from the keywords, it is not necessary to input a character string required for the search, and these digitized documents can be easily searched.・ Display technology has been proposed.

JP 2003-150618 A JP 2005-275756 A

  However, with the techniques of Patent Literature 1 and Patent Literature 2, there is a problem that a marker or memo written on a paper medium is extracted, and the extracted marker or memo cannot be reflected as rewritten electronic book data as handwritten information. there were.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide a handwritten information reflecting system that reflects handwritten information extracted from a paper medium in reflow electronic book data. .

In order to achieve the above-described object, the first aspect of the present invention is a storage means for storing reflow type electronic book data and a read image read from a paper book, and based on a predetermined extraction condition from the read image. Extraction means for extracting handwritten information and a reflection position, and reflection means for reflecting the handwritten information in the reflection position extracted by the extraction means in the reflow type electronic book data. This is a handwritten information reflection system.
According to the first invention, handwritten information extracted from a paper medium can be easily reflected in electronic book data.

The extraction unit extracts a memo image including a memo as the handwritten information, and extracts a character string at a position close to the memo image as the reflection position.
Thereby, the memo image extracted as a memo and the character string at a position close to the memo image can be associated with each other and stored as handwritten information data.

The extraction means calculates a histogram of the memo image and distinguishes the memo image from the book image according to the distribution of the histogram.
Thereby, the book image extracted as a memo image by mistake can be excluded from the target of the memo image.

The reflection means displays a selection screen for the memo image to be reflected, and reflects the memo image selected on the selection screen in the reflow electronic book data.
Thereby, since the memo image to be reflected can be confirmed, the user can reliably reflect only the memo image in the electronic book data.

The extraction unit extracts a marker area as the handwritten information, and extracts a character string in the marker area as the reflection position.
Thereby, the character string in a marker area | region can be preserve | saved as handwritten information data for every marker area | region.

2nd invention memorize | stores the read image read from reflow type | mold electronic book data and a paper book, The extraction step which extracts handwritten information and a reflection position from the said read image based on predetermined | prescribed extraction conditions And a reflecting step of reflecting the handwritten information in the reflecting position extracted by the extracting means in the reflow type electronic book data.
According to the second invention, handwritten information extracted from a paper medium can be easily reflected in electronic book data.

  3rd invention memorize | stores the reflow type | mold electronic book data and the reading image read from the paper book, The extraction step which extracts handwritten information and a reflection position from the said reading image based on predetermined | prescribed extraction conditions And a reflecting step of reflecting the handwritten information in the reflecting position extracted by the extracting means in the reflow type electronic book data.

  By installing the program according to the third invention in the computer, the handwritten information reflecting system according to the first invention can be obtained.

  According to the present invention, it is possible to provide a handwritten information reflection system that reflects handwritten information extracted from a paper medium in reflow electronic book data.

It is a figure which shows the structural example of the handwritten information reflection system which concerns on the 1st Embodiment of this invention. It is a figure which shows the hardware structural example of the terminal in 1st Embodiment. It is an example of the information memorize | stored in the memory | storage part of the terminal in 1st Embodiment. It is a flowchart explaining the handwritten information extraction process in 1st Embodiment. It is a figure which shows the example of the read image data in 1st Embodiment. It is a figure which shows the example which extracts the line area | region for every line except the line space of the binarization area | region. It is a figure which shows the example which extracts the character area for every character for every line area. It is a figure which shows the example of the handwritten information data in 1st Embodiment. It is a flowchart explaining the handwritten information reflection process in 1st Embodiment. It is a figure which shows an example of the display file based on the XML file of the electronic book data to which the marker tag was added, and the file. It is a flowchart explaining the handwritten information extraction process in 2nd Embodiment. It is a figure which shows the example of the read image data in 2nd Embodiment. It is a figure which shows the example of the histogram calculated from the extraction area | region image. It is a figure which shows the example of a display of a reflection memo selection screen. It is a figure which shows the example of the handwritten information data in 2nd Embodiment. It is a flowchart explaining the handwritten information reflection process in 2nd Embodiment. It is a figure which shows the example of the display screen based on the XML file of the electronic book data to which the memo tag was added, and the file. It is a figure which shows the example of the read image data in a modification. It is a figure which shows the example of the handwritten information data in a modification.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[First embodiment of the present invention]
FIG. 1 is a diagram illustrating a configuration example of a handwritten information reflecting system 1 according to an embodiment of the present invention.

  The handwritten information reflecting system 1 shown in FIG. 1 is configured by connecting a terminal 2 and a reading device 3 to each other. The terminal 2 is a computer system including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), an HDD (Hard Disc Drive), etc. For example, a personal computer, a tablet terminal, a mobile phone, It is a smartphone.

  The terminal 2 stores reflow electronic book data. Electronic book data includes bibliographic data (book titles, authors, publishers, publication dates, etc.), table of contents data (headings indicating the content of the book text, etc.), text data (electronic data related to the text of the book), Document list data (book author, title, publisher, date of publication, etc.) are included.

  Reflow-type e-book data means that the text will reflow to fit the screen even if the size of the text is changed, such as increasing the text to increase the number of pages and reducing the text to decrease the number of pages. This is electronic book data that can be read only by page feed.

  The terminal 2 receives and stores read image data read from the paper book 4 by the reading device 3. The terminal 2 extracts handwritten information from the read image data based on a predetermined extraction condition, extracts a position matching the handwritten information from the electronic book data as a reflection position, and reflects the handwritten information at the reflection position. The handwritten information referred to in the first embodiment is, for example, a portion provided with a marker line by a highlighter pen.

  The reading device 3 is a scanner or a camera, for example, and reads the paper surface of the paper book 4 to generate read image data. In the first embodiment, the reading device 3 may be anything as long as it can image one page of the paper book 4. When a camera is mounted on the terminal 2, the terminal 2 and the reading device 3 are integrated.

  FIG. 2 is a diagram illustrating a hardware configuration example of the terminal 2. Note that the hardware configuration in FIG. 2 is an example, and various configurations can be adopted depending on the application and purpose.

  The terminal 2 is connected to a control unit 11, a storage unit 12, a media input / output unit 13, a communication control unit 14, an input unit 15, a display unit 16, a peripheral device I / F unit 17, and the like via a bus 18.

  The control unit 11 includes a CPU, a ROM, a RAM, and the like. The CPU calls a program stored in the storage unit 12, ROM, recording medium, etc. to a work memory area on the RAM, executes it, drives and controls each device connected via the bus 18, and is described later by the terminal 2. To achieve the process. The ROM is a non-volatile memory and permanently stores a boot program for the terminal 2, programs such as BIOS (Basic Input / Output System), data, and the like. The RAM is a volatile memory, and temporarily stores a program, data, and the like loaded from the storage unit 12, ROM, recording medium, and the like, and includes a work area used by the control unit 11 to perform various processes.

  The storage unit 12 is an HDD or the like, and stores a program executed by the control unit 11, data necessary for program execution, an OS (operating system), and the like. As for the program, a control program corresponding to the OS and an application program for causing the terminal 2 to execute processing to be described later are stored. Each of these program codes is read by the control unit 11 as necessary, transferred to the RAM, read by the CPU, and executed as various means.

  The media input / output unit 13 (drive device) inputs / outputs data, for example, media such as a CD drive (-ROM, -R, -RW, etc.), DVD drive (-ROM, -R, -RW, etc.) Has input / output devices.

  The communication control unit 14 includes a communication control device, a communication port, and the like, is a communication interface that mediates communication between the terminal 2 and the network, and performs communication control between other devices via the network. The network may be wired or wireless.

  The input unit 15 inputs data and includes, for example, a keyboard, a pointing device such as a mouse, and an input device such as a numeric keypad. An operation instruction, an operation instruction, data input, and the like can be performed on the terminal 2 via the input unit 15.

  The display unit 16 includes a display device such as a liquid crystal panel, and a logic circuit or the like (video adapter or the like) for realizing the video function of the terminal 2 in cooperation with the display device.

The peripheral device I / F (interface) unit 17 is a port for connecting a peripheral device to the terminal 2, and the terminal 2 transmits and receives data to and from the peripheral device via the peripheral device I / F unit 17. The peripheral device I / F unit 17 is a USB (Universal Serial Bus) or IEEE (The Institute of Electrical and
Electronics Engineers) 1394, RS (Recommended Standard) -232C, etc., and usually has a plurality of peripheral devices I / F. The connection form with the peripheral device may be wired or wireless. The bus 18 is a path that mediates transmission / reception of control signals, data signals, and the like between the devices.

  FIG. 3 is an example of information stored in the storage unit 12 of the terminal 2.

  As illustrated in FIG. 3, the storage unit 12 of the terminal 2 stores read image data 21, electronic book data 22, handwritten information data 23, handwritten information reflecting AP (application) 24, and an electronic book viewer 25.

  The read image data 21 is data obtained by imaging the paper book 4 read by the reading device 3. The electronic book data 22 is data for one book, including bibliographic data, table of contents data, text data, document list data, and the like, and is data produced in a reflow type.

  The handwritten information data 23 is handwritten information data extracted from the read image data 21 by the handwritten information reflecting application 24 based on a predetermined extraction condition.

  The handwritten information reflecting application 24 extracts handwritten information from the read image data 21 based on a predetermined extraction condition, and generates handwritten information data 23. The handwritten information reflecting application 24 extracts a position matching the handwritten information from the electronic book data 22 as a reflected position, and reflects the extracted handwritten information at the reflected position.

  The electronic book viewer 25 acquires the electronic book data 22 in which the handwritten information is reflected, and displays the contents.

  FIG. 4 is a flowchart for explaining the handwritten information extraction process in the first embodiment, which is executed by the handwritten information reflection system 1. 4 will be described with reference to FIGS. 5 to 9.

  In step S <b> 1, the reading device 3 reads the paper surface of the paper book 4 provided with a marker line by a highlighter pen or the like, and generates read image data.

  FIG. 5 is a diagram illustrating an example of the read image data 21a and 21b read and generated by the reading device 3.

  As shown in FIG. 5, in the read image data 21 a, a marker 30 a is added to the characters “My cat is a cat” and a marker 30 b is added to the characters “Yes. No name yet”. Has been granted. In addition, the read image data 21b is provided with a marker 30c on the characters "But at that time, I didn't think anything, so I didn't think it was awful." In the example of FIG. 5, the marker 30a and the marker 30b included in the read image data 21a are not in contact with each other marker line, but the marker 30c included in the read image data 21b is in contact with a plurality of marker lines. .

  Returning to the description of FIG. In step S2, the handwritten information reflecting application 24 of the terminal 2 scans the read image data generated by the reading device 3 in the process of step S1, and extracts regions (for example, fluorescent colors) having different background colors. For example, in FIG. 5, markers 30a, 30b, and 30c (regions surrounded by diagonal lines) are determined to be different regions, and are extracted as extraction regions 31a, 31b, and 31c, respectively, as indicated by the tip of arrow S2.

  Returning to the description of FIG. In step S3, the handwritten information reflecting application 24 of the terminal 2 calculates and binarizes the edge in the extraction area for each extraction area extracted in the process of step S2. For example, in FIG. 5, the edges in the extraction areas 31a, 31b, and 31c are calculated and binarized, and binarized areas 32a, 32b, and 32c are obtained as indicated by the tip of the arrow S3.

  Returning to the description of FIG. In step S4, the handwritten information reflecting application 24 of the terminal 2 extracts a line area for each line excluding the line spacing for each binarized area binarized in the process of step S3.

  FIG. 6 is a diagram illustrating an example of extracting the row area for each row excluding the line spacing of the binarized region 32c.

  As shown in FIG. 6, the binarized area 32 c includes a line that includes the character string “But”, a line that includes the character string “Nothing to think at that time”, and “ ", A line including the character string" I didn't expect it ", and a line space (blank). In the binarized area 32c, as indicated by the tip of the arrow S4, line areas 33a, 33b, 33c, and 33d for each line excluding the line spacing are extracted. Since the binarized areas 32a and 32b in FIG. 5 are character strings consisting of one line that does not include a line space, the binarized areas 32a and 32b are directly used as line areas (not shown).

  Returning to the description of FIG. In step S5, the handwritten information reflecting application 24 of the terminal 2 extracts a character area for each character for each line area extracted in the process of step S4.

  FIG. 7 is a diagram illustrating an example of extracting a character area for each character for each row area 33a.

  As shown in FIG. 7, the line area 33a includes the character string “But”, however, as indicated by the tip of the arrow S5, the character area 34a of “shi” and the character area 34b of “ka”. , “Shi” character area 34c, “so” character area 34d, and “no” character area 34e, respectively.

  Returning to the description of FIG. In step S6, the handwritten information reflecting application 24 of the terminal 2 performs character recognition for each character area extracted in the process of step S5, and specifies a character code. In step S7, the handwritten information reflecting application 24 of the terminal 2 stores the character code group in the extracted area in the storage unit 12 as handwritten information data (marker) for each extracted area extracted in the process of step S2.

  FIG. 8 is a diagram illustrating an example of the handwritten information data 23a.

  As shown in FIG. In the data storage area 1, handwritten information data (character code group in the extraction area 31 a) with the type “marker” and the extracted character string “my gaai is a cat” is stored. In the data storage area 2, handwritten information data (character code group in the extraction area 31 b) with the type “marker” and the extracted character string “. No name yet” is stored. In the data storage area 3, handwritten information data (character code group in the extraction area 31c) whose type is “marker” and whose extracted character string is “but at that time, I did not think it was afraid of anything.” ) Is saved.

  As described above, the handwritten information reflecting system 1 can read the paper surface of the paper book 4 provided with the marker line by the highlighter pen, extract the marker area as handwritten information, and save it.

  FIG. 9 is a flowchart for explaining handwritten information reflecting processing in the first embodiment executed by the handwritten information reflecting system 1. In the description of FIG. 9, the specific processing contents will also be described with reference to FIG.

  In step S11, the handwritten information reflecting application 24 of the terminal 2 matches the character string of the electronic book data 22 that matches the extracted character string of the handwritten information data 23a (FIG. 8) stored in the storage unit 12 at a rate equal to or higher than the threshold. Search for.

  Specifically, the handwritten information reflection application 24 of the terminal 2 performs matching with the electronic book data 22 character by character from the beginning of the extracted character string of the handwritten information data 23a to the end of the character string. If not, scoring is performed as 0, and it is determined whether or not to match based on a preset threshold value. For example, the score must be 19 in order to completely match the extracted character string of 19 characters, but a fuzzy search is possible by setting a threshold value (ratio of matching).

  In step S12, the handwritten information reflecting application 24 of the terminal 2 adds a marker tag surrounding the character string of the electronic book data 22 searched in the process of step S11.

  Specifically, in the read image data 21 a shown in FIG. 5, one sentence of “My friend is a cat. No name yet” forms one paragraph. That is, in the electronic book data 22 described in the XML (Extensible Markup Language) format, “<P>” is described in the former stage of “My friend is a cat. There is no name yet.” By describing “</ P>”, it is indicated that the portion surrounded by <P> to </ P> is one paragraph.

  As shown in FIG. 8, the extracted character strings “My cat is a cat” and “No name yet.” In the handwritten information data 23 a are extracted separately. In the file, “<span class =“ mark ”>” is described at the front of “My cat is my cat”, and “</ span>” is described at the back. In addition, “<span class =“ mark ”>” is described in the previous stage of “Yes. No name yet”, and “</ span>” is described in the subsequent stage. Thereby, a marker tag (marker function) is added to a portion surrounded by <span class = “mark”> to </ span>.

  Returning to the description of FIG. In step S13, the handwritten information reflection application 24 of the terminal 2 integrates continuous marker tags among the marker tags added in the process of step S12. As a result, marker tags have been added to “My friend is a cat” and “No name yet” in the XML file of the electronic book data 22, but they are integrated into one.

  As shown in FIG. 8, the extracted character string “But I didn't think it was scary because I had no idea at that time” in the handwritten information data 23a was extracted in a row, so one marker tag Is added and there is no need for marker tag integration.

  FIG. 10 is a diagram illustrating an example of an XML file 41a of electronic book data 22 to which a marker tag is added and a display screen 42a based on the file.

  As shown in FIG. 10, in the XML file 41a of the electronic book data 22, marker tags are described before and after “My friend is a cat. There is no name yet.” I didn't think anything, so I didn't think it was awful. "Marker tags are written before and after. The electronic book viewer 25 can display the display screen 42a on the display unit 16 based on the XML file 41a. In the example of FIG. 10, the handwritten marker given on the paper surface of the paper book 4 is reflected and displayed in the electronic book data 22 (in the figure, the area surrounded by the diagonal lines).

[Effect in the first embodiment of the present invention]
1. As described above, a marker attached to a paper book can be easily reflected in a reflow electronic book. For example, in the case of school textbooks and reference books, the contents marked on a plurality of books during the class can be reflected on the electronic book, so that they can be confirmed together on a train.

  2. By reflecting a marker provided by a celebrity in a reflow-type electronic book, an electronic book having a new added value can be provided. For example, a celebrity fan may be interested in purchasing an e-book that is marked with a sentence recommended by a celebrity, even if he is not interested in the original e-book. .

[Second embodiment of the present invention]
Next, a second embodiment of the present invention will be described.

  In the second embodiment, the basic system configuration (FIG. 1), the hardware configuration of the terminal 2 (FIG. 2), and the information stored in the storage unit 12 (FIG. 3) are the same as those in the first embodiment. It is the same. In the second embodiment, note writing is extracted as handwritten information and reflected in the electronic book data 22.

  FIG. 11 is a flowchart for describing handwritten information extraction processing in the second embodiment, which is executed by the handwritten information reflecting system 1. In the description of FIG. 11, specific processing contents will also be described with reference to FIGS. 12 to 15.

  In step S <b> 21, the reading device 3 reads the paper surface of the paper book 4 in which handwritten characters are written, and generates read image data.

  FIG. 12 is a diagram illustrating an example of the read image data 21c and 21d read and generated by the reading device 3.

  As shown in FIG. 12, in the read image data 21c, a handwritten character 51a of "Memo A." is written above the basic printing plate 52 (near the line spacing of "Sai" and "I"). Yes. In the read image data 21 d, a handwritten character 51 b of “Memo B.” is written below the basic plate surface 52. The read image data 21d includes a book image 53.

  Returning to the description of FIG. In step S <b> 22, the handwritten information reflecting application 24 of the terminal 2 acquires the basic version of the paper surface of the paper book 4. For example, the handwritten information reflection application 24 calculates the horizontal size from the edge of the leftmost character string and the edge of the rightmost character string in the text on the page, and the edge of the uppermost character string and the character string of the lowermost character string. By calculating the vertical size from the edges, the basic plate surface can be obtained.

  In step S23, the handwritten information reflecting application 24 of the terminal 2 calculates an edge outside the basic printing surface acquired in the process of step S22, and extracts a region with many edges. That is, since there is no text outside the basic version, an area with many edges is extracted as a memo writing candidate. For example, in FIG. 12, the handwritten character 51b included in the read image data 21d is extracted as the extraction region image.

  In step S24, the handwritten information reflecting application 24 of the terminal 2 calculates an edge in the basic printing plate acquired in the process of step S22, and extracts an area where the appearance position of the edge is irregular.

  Specifically, the widths of the line area and the inter-line area are calculated in advance based on a sheet without a note, and the edge is searched by scanning the line area with respect to the sheet with a note. . Although the text has regular edges, the memos and images are irregular, so the areas where the edges appear irregular are extracted as memo candidates. According to this method, an image or the like is also extracted. For example, in FIG. 12, a handwritten character 51a included in the read image data 21c and a handwritten character 51b included in the read image data 21d are extracted as extraction region images.

  In step S25, the handwritten information reflecting application 24 of the terminal 2 calculates a histogram for each extracted region image extracted in the processes in steps S23 and S24, and determines that the extracted region image in which the histogram is widely distributed is a book image.

  FIG. 13 is a diagram illustrating an example of a histogram calculated from the extracted region image.

  FIG. 13A shows a histogram 62a calculated from the extraction region image 61a corresponding to the handwritten character 51a shown in FIG. FIG. 13B shows a histogram 62b calculated from the extraction area image 61b corresponding to the book image 53 shown in FIG. FIG. 13C shows a histogram 62c calculated from the extraction region image 61c corresponding to the handwritten character 51b shown in FIG.

  As can be seen from these figures, only the histogram 62b is widely distributed, and the distributions of the histograms 62a and 62c are biased at both ends. Therefore, the extraction area image 61b that is the original image in calculating the histogram 62b can be determined as a book image, not a memo.

  Returning to the description of FIG. In step S <b> 26, the handwritten information reflection application 24 of the terminal 2 generates a reflection memo selection screen based on the extracted area image extracted in steps S <b> 23 and S <b> 24 and displays it on the display unit 16.

  FIG. 14 is a diagram illustrating a display example of the reflected memo selection screen 71.

  As shown in FIG. 14, on the reflected memo selection screen 71, the extraction area images 61a, 61b, and 61c are displayed together with the file names, and a check box is arranged on the left side of each image, so that the handwritten memo is determined. The extraction area image is checked. In the case of the example in FIG. 14, the extraction area image 61 b is determined to be a book image in the process of step S <b> 25, so the check is removed.

  In addition, the reflected memo selection screen 71 cancels the reflection to the electronic book data 22, the decision button 71 a selected when the handwritten memo (extracted area image) whose check box is checked is reflected in the electronic book data 22. A cancel button 71b that is selected when the operation is performed is also displayed.

  Returning to the description of FIG. In step S27, the handwritten information reflecting application 24 of the terminal 2 sets the character string at a position close to the extraction area in the read image data 21 for each of the extraction area images 61a to 61c displayed on the reflection memo selection screen 71 in the process of step S26. Identify each. In step S28, the handwritten information reflecting application 24 of the terminal 2 associates the extracted area images 61a to 61c with the extracted character string specified in the process of step S27 and saves it in the storage unit 12 as handwritten information data (memo). To do.

  FIG. 15 is a diagram illustrating an example of the handwritten information data 23b.

  As shown in FIG. In the data storage area 1, handwritten information data having a type “memo”, an extraction area image “Memo001.png”, and a character string “xx (waigai)” near the extraction area is stored. In the data storage area 2, handwritten information data whose type is “memo”, the extracted area image is “Memo003.png”, and the character string at the position close to the extracted area is “beginning”. Saved. In addition, No. Among the character strings stored in one data storage area, “xx” indicates a character that could not be recognized.

  As described above, the handwritten information reflecting system 1 can read the paper surface of the paper book 4 on which the memo is written, extract an area with many edges or irregular edge appearance positions as handwritten information, and save it. it can.

  FIG. 16 is a flowchart for explaining handwritten information reflecting processing in the second embodiment, which is executed by the handwritten information reflecting system 1. In the description of FIG. 16, the specific processing contents will also be described with reference to FIG.

  In step S31, the handwritten information reflecting application 24 of the terminal 2 matches the character string at a position close to the extraction region of the handwritten information data 23b (FIG. 15) stored in the storage unit 12 with a ratio equal to or higher than the threshold. The character string of the data 22 is searched.

  Specifically, the handwritten information reflecting application 24 of the terminal 2 matches the electronic book data 22 character by character from the beginning of the character string of the handwritten information data 23b to the end of the character string, as in the first embodiment. Then, 1 is scored if there is a match, 0 is scored if there is no match, and it is determined whether or not to match based on a preset threshold value.

  In step S32, the handwritten information reflecting application 24 of the terminal 2 adds a memo tag surrounding the character string of the electronic book data 22 searched in the process of step S31.

Specifically, as shown in FIG. 15, by the above-described processing, “xx (my own)” and “beginning” are close to the handwritten memo extraction area in the handwritten information data 23 b. Since it is extracted as a character string of the position, the “<memo”
src = “Memo001.png”> ”is described, and“ </ memo> ”is described in the subsequent stage. In addition, “<memo src =“ Memo001.png ”>” is described in the front stage of “It will be the beginning”, and “</ memo>” is described in the subsequent stage. As a result, a memo tag (memo function) is added to the area surrounded by <memo src = “Memo001.png”> to </ memo>.

  FIG. 17 is a diagram illustrating an example of an XML file 41b of electronic book data 22 to which a memo tag is added and a display screen 42b based on the file.

  As shown in FIG. 17, in the XML file 41b of the electronic book data 22, memo tags are described before and after “myself”, and before and after “beginning”. Has a memo tag. The electronic book viewer 25 can display the display screen 42b on the display unit 16 based on the XML file 41b. In the example of FIG. 17, the memo writing on the paper book 4 is reflected in the electronic book data 22 and displayed as balloons of “Memo A.” and “Memo B.”. In the example of FIG. 17, a balloon is displayed, but actually, it is displayed only when the character string surrounded by the memo tag is tapped (or the mouse is passed).

[Effect in the second embodiment of the present invention]
1. As described above, a memo written on a paper book can be easily reflected in a reflow electronic book. For example, in the case of school textbooks and reference books, the contents recorded in a plurality of books during a class can be confirmed together on a train by reflecting them in an electronic book.

  2. By reflecting a memo written by a celebrity in a reflow-type electronic book, an electronic book having a new added value can be provided. For example, a celebrity fan may be interested in purchasing an electronic book with a note directly written by a celebrity, even if he / she is not interested in the original electronic book.

[Modification]
In the above, the marker or memo written on the paper book is reflected in the reflow-type electronic book data. However, the present invention is not limited to this. For example, a place where a corner of the page is folded (bookmark) is detected. The top character string of the page may be extracted and a bookmark function may be added to the character string matching the character string of the electronic book data.

  For example, as shown in FIG. 18, it is assumed that the read image data 21e and 21f are generated by the reading device 3. In the read image data 21f, a bookmark is bent at the upper right corner.

  When the handwritten information reflecting application 24 of the terminal 2 searches for four corners outside the basic version of the paper surface of the paper book 4 and detects a bend (a triangular edge in the example of FIG. 18), it extracts it as a bookmark. Then, the character string at the top of the page from which the bending is extracted is extracted. Then, as shown in FIG. 19, the handwritten information reflecting application 24 stores, in the storage unit 12, handwritten information data 23 c of the type “bookmark” and the extracted character string “is a story about eating and eating”. Can be saved.

  As described above, the handwritten information reflecting system 1 can read the paper surface of the paper book 4 in which the bookmark is folded, extract the folding as the handwritten information, and save it.

  The preferred embodiments of the handwritten information reflecting system and the like according to the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to such examples. It will be apparent to those skilled in the art that various changes or modifications can be conceived within the scope of the technical idea disclosed in the present application, and these naturally belong to the technical scope of the present invention. Understood.

DESCRIPTION OF SYMBOLS 1 ......... Handwritten information reflecting system 2 ......... Terminal 3 ......... Reading device 4 ......... Paper book 11 ......... Control unit 12 ......... Storage unit 16 ......... Display unit

Claims (7)

Storage means for storing reflow type electronic book data and a read image read from a paper book;
Extraction means for extracting handwritten information and a reflection position from the read image based on a predetermined extraction condition;
Reflecting means for reflecting the handwritten information in the reflecting position extracted by the extracting means among the reflow type electronic book data;
A handwritten information reflection system comprising: 2. The handwritten information reflecting system according to claim 1, wherein the extracting unit extracts a memo image including a memo as the handwritten information, and extracts a character string at a position close to the memo image as the reflecting position. . The handwriting information reflecting system according to claim 2, wherein the extraction unit calculates a histogram of the memo image and distinguishes the memo image from the book image according to the distribution of the histogram. The reflection unit displays a selection screen of the memo image to be reflected, and reflects the memo image selected on the selection screen in the reflow type electronic book data. The handwritten information reflection system described. 5. The handwritten information reflection according to claim 1, wherein the extraction unit extracts a marker region as the handwritten information, and extracts a character string in the marker region as the reflection position. system. A storage step of storing reflow type electronic book data and a read image read from a paper book;
An extraction step for extracting handwritten information and a reflection position from the read image based on a predetermined extraction condition;
A reflecting step of reflecting the handwritten information in the reflecting position extracted by the extracting means in the reflow type electronic book data;
A method for reflecting handwritten information, comprising: On the computer,
A storage step of storing reflow type electronic book data and a read image read from a paper book;
An extraction step for extracting handwritten information and a reflection position from the read image based on a predetermined extraction condition;
A reflecting step of reflecting the handwritten information in the reflecting position extracted by the extracting means in the reflow type electronic book data;
A program for running

Images