JP7184411B1 - Character Recognition Program Evaluation Method, Character Recognition Program Evaluation System, and Method of Manufacturing Teacher Data for Character Recognition Program - Google Patents

Abstract

A character recognition program evaluation method capable of objectively evaluating a character recognition program.
A step of obtaining image data of a handwritten character, a step of obtaining a plurality of machine character recognition results by a character recognition calculation unit using a plurality of different character recognition programs, and a step of obtaining a plurality of machine character recognition results from the image data. a step of obtaining matching rate data regarding a matching rate of a result; a step of obtaining a human character recognition result that is a result of character recognition by human visual recognition of the image data; determining whether the plurality of machine character recognition results are correct or not; a step of obtaining an accuracy rate of the plurality of machine character recognition results; the image data; at least one of the plurality of machine character recognition results and correct/wrong thereof; the human character recognition result; and a step of storing character recognition program evaluation results associated with the accuracy rate.
[Selection drawing] Fig. 1

Description

The present invention relates to a character recognition program evaluation method used in OCR, a character recognition program evaluation system, and a method of producing teacher data for a character recognition program.

In recent years, a technique called OCR (Optical Character Recognition) has been used in various fields. In addition, in OCR, techniques such as improving the program (algorithm) for recognizing characters by machine learning have been proposed. For example, image data (including characters) contained in training data can be recognized by the program to be learned, and the correct character recognition result (correct answer) of the image data can be conveyed to the learning function part of the target program. It is expected to improve accuracy rate in program character recognition.

Here, conventional character recognition programs have a fairly high character recognition accuracy for printed characters, but the character recognition accuracy for handwritten characters is considerably lower than that for printed characters, and an improvement is expected. However, handwritten characters are extremely diverse depending on the personality of the writer, and abbreviations such as abbreviated characters are frequently used depending on the situation. Against the background of such diversity of handwritten characters, it is difficult to objectively evaluate the character recognition accuracy of a specific program for handwritten characters. The inability to do so prevents character recognition programs from improving the character recognition accuracy of handwritten characters.

Patent No. 5464474

The present invention relates to a character recognition program evaluation method and evaluation system capable of objectively evaluating a character recognition program, and to a method of producing teaching data effective for improving character recognition accuracy for handwritten characters.

A character recognition program evaluation method according to the present invention includes:
obtaining image data of handwritten characters;
a step of performing character recognition on the image data by a character recognition calculation unit using a plurality of different character recognition programs to obtain a plurality of machine character recognition results, which are character recognition results of the image data by the plurality of character recognition programs; ,
obtaining match rate data relating to the match rate of the plurality of machine character recognition results obtained from the image data;
a step of obtaining a human character recognition result, which is a character recognition result by human visual recognition of the image data;
comparing each of the plurality of machine character recognition results with the human character recognition result to determine whether the plurality of machine character recognition results are correct or not, and obtaining an accuracy rate of the plurality of machine character recognition results; ,
a character recognition program evaluation result in which the image data, at least one of the plurality of machine character recognition results and correct/incorrect thereof, the human character recognition result, the matching rate data, and the accuracy rate are associated with each other; and storing.

According to the character recognition program evaluation method according to the present invention, image data of actual handwritten characters are subjected to character recognition by a plurality of character recognition programs. Moreover, an objective evaluation result can be obtained. In addition, evaluation efficiency can be improved by recognizing common image data using a plurality of character recognition programs. In addition, since multifaceted indices such as the accuracy rate and the matching rate can be obtained for each image data, multifaceted and effective knowledge can be obtained as to whether the machine character recognition result for the image data is correct or incorrect.

Further, for example, in the step of obtaining the human character recognition result, selecting at least two humans as evaluators;
determining tentative recognition results that are character recognition results obtained from the at least two humans;
In the step of determining the provisional recognition result, if the provisional recognition results match each other, the provisional recognition result is recognized as the human character recognition result, and if the provisional recognition results do not match, at least two people may return to the step of selecting as the evaluator.

In such a character recognition program evaluation method, it is possible to effectively suppress errors in evaluation results caused by variations in the character recognition results of human evaluators.

Further, for example, the step of obtaining the human character recognition result includes a step of storing an evaluator confidence level relating to the probability that the temporary recognition result of each evaluator matches the human character recognition result,
In the step of selecting at least two persons as evaluators, at least two persons may be selected as evaluators using the matching rate data and the evaluator reliability.

In the evaluation method of such a character recognition program, for example, for image data with a high matching rate, a person with a high evaluation reliability is given priority and selected as an evaluator, and for image data with a low matching rate, the evaluation reliability is selected. By preferentially selecting a person with a low degree of accuracy as an evaluator, it is possible to improve the efficiency of obtaining human character recognition results. This is because, in the step of obtaining human character recognition results, differences in the abilities of evaluators are more likely to appear in evaluation efficiency and evaluation accuracy in situations where image data with a high matching rate is evaluated.

Further, for example, in the step of obtaining the human character recognition result, any one of the image data, the plurality of machine character recognition results, and the plurality of machine character recognition results can be selected as the human character recognition result or a candidate thereof. A graphical interface may be displayed on the evaluator's terminal.

In such a character recognition program evaluation method, it is possible to reduce the amount of input work by the evaluator and improve the efficiency of the step of obtaining human character recognition results by humans.

Further, in the character recognition program evaluation system according to the present invention, an image data acquisition unit for acquiring image data of handwritten characters;
a machine character recognition result acquisition unit for obtaining a plurality of machine character recognition results, which are character recognition results of the image data by a plurality of different character recognition programs performing character recognition on the image data by the plurality of character recognition programs; ,
a matching rate data obtaining unit for obtaining matching rate data relating to the matching rates of the plurality of machine character recognition results obtained from the image data;
a human character recognition result obtaining unit for obtaining a human character recognition result, which is a character recognition result of the image data visually recognized by a human;
Each of the plurality of machine character recognition results and the human character recognition result are compared to obtain a correctness determination unit for determining whether the plurality of machine character recognition results are correct and an accuracy rate of the plurality of machine character recognition results. an accuracy rate calculator;
a character recognition program evaluation result in which the image data, at least one of the plurality of machine character recognition results and correct/incorrect thereof, the human character recognition result, the matching rate data, and the accuracy rate are associated with each other; and an evaluation result storage unit for storing.

Further, in a method for producing teacher data for a character recognition program according to the present invention, a step of obtaining image data of handwritten characters;
a step of performing character recognition on the image data by a plurality of different character recognition programs to obtain a plurality of machine character recognition results, which are character recognition results of the image data by the plurality of character recognition programs;
obtaining match rate data relating to the match rate of the plurality of machine character recognition results obtained from the image data;
a step of obtaining a human character recognition result, which is a character recognition result by human visual recognition of the image data;
each of the plurality of machine character recognition results and the human character recognition result are compared to determine whether the plurality of machine character recognition results are correct or not; a step of obtaining an accuracy rate representing the percentage of character recognition programs that performed correct character recognition;
and storing the image data, the human character recognition result, the match rate data, and teacher data associated with the correct rate.

According to the method for producing teacher data for a character recognition program according to the present invention, image data of actual handwritten characters are produced through character recognition by a plurality of character recognition programs, so that actual handwritten characters are used for character recognition. It is possible to produce teacher data that is consistent with the situation and has a high learning effect. Further, by recognizing common image data using a plurality of character recognition programs, the production efficiency of teacher data can be improved. In addition, since the image data included in the training data has multifaceted indices such as the accuracy rate and the matching rate, it is possible to obtain multifaceted and effective knowledge about the expected learning effect of using the training data. can.

The disclosed technology includes content related to a system that simultaneously evaluates the results of multiple artificial intelligences. Also, Multiple Artificial Intelligence, Automatic processing results, Reasonable modifications, Intelligence data generation, Automatic processing of corrections ), including findings related to

FIG. 1 is a conceptual diagram of a character recognition program evaluation system that performs a character recognition program evaluation method according to an embodiment of the present invention. FIG. 2 is a conceptual diagram showing an example of information obtained by the character recognition program evaluation method. FIG. 3 is a conceptual diagram showing an example of evaluator selection criteria in the step of obtaining human character recognition results. FIG. 4 is a conceptual diagram illustrating an evaluator's input method in the step of obtaining human character recognition results. FIG. 5 is a conceptual diagram showing an example of character recognition properties of image data included in training data. FIG. 6 is a conceptual diagram showing an example of analytical knowledge obtained from information included in evaluation results. FIG. 7 is a conceptual diagram showing a reliability information storage unit that stores reliability information of evaluators. FIG. 8 is a conceptual diagram showing information included in evaluation results and teacher data. FIG. 9 is a conceptual diagram showing a specific example of information included in evaluation results and teacher data. FIG. 10 is a flow chart showing an example of processing in the step of obtaining a human character recognition result.

FIG. 1 is a conceptual diagram showing a character recognition program evaluation system 10 that performs a character recognition program evaluation method according to an embodiment of the present invention. As shown in FIG. 1, the character recognition program evaluation system 10 includes an image data acquisition unit 20, a machine character recognition result acquisition unit 30, a machine character recognition tendency calculation unit 40, a human character recognition result acquisition unit 50, a scoring unit 60, an information It has an integration unit 70 and the like.

The character recognition program evaluation system 10 has an arithmetic unit that performs arithmetic processing, a storage unit that stores information, a communication unit that communicates with other servers and terminals, and the like, and is based on physical servers and cloud computing. It is implemented by a virtual server or the like. Moreover, various calculation units, storage units, and the like included in the character recognition program evaluation system 10 may be centrally arranged in one place, or may be arranged dispersedly via a communication network.

Hereinafter, a method for evaluating a character recognition program and a method for producing teacher data for a character recognition program by the character recognition program evaluation system 10 will be described with specific examples. However, the character recognition program evaluation method and the character recognition program teacher data production method by the character recognition program evaluation system 10 are not limited to these specific examples.

In the evaluation method and the like by the character recognition program evaluation system 10, first, in the image data acquisition unit 20, a step of acquiring image data 20a of handwritten characters is performed. The image data acquisition unit 20 has a reading unit 24 such as a scanner or a camera, and reads characters written on the form 22 or the like to acquire image data 20a. The acquired image data 20a is stored in the storage unit of the image data 20 or the like. The number of image data 20a to be acquired and the number of characters included in the image data 20a are not particularly limited.

The handwritten characters acquired as the image data 20a are not limited to the forms 22, and may be handwritten characters included in other documents such as various application forms, letters, and greeting cards. However, the handwritten characters acquired as the image data 20a are preferably handwritten for some practical purpose. By using handwritten characters written for practical purposes, the evaluation results obtained by the evaluation system 10 are closer to the practical character recognition accuracy of the character recognition program.

Next, in the evaluation system 10, the machine character recognition result obtaining unit 30 performs character recognition on the image data 20a, and obtains the machine character recognition result, which is the character recognition result by the character recognition program. As shown in FIG. 1, the machine character recognition result acquisition unit 30 includes a first character recognition calculation unit 31, a second character recognition calculation unit 32, a third character recognition calculation unit, which are character recognition calculation units using a plurality of different character recognition programs. It has a part 33 .

The first character recognition calculation unit 31, the second character recognition calculation unit 32, and the third character recognition calculation unit 33 are controlled by mutually different character recognition programs. character recognition. As a result, the machine character recognition result acquisition unit 30 obtains a first machine character recognition result 31a, a second machine character recognition result 32a, and a third machine character recognition result, which are character recognition results obtained by a plurality of character recognition programs for the image data 20a. 33a is obtained.

As shown in FIG. 1, the machine character recognition result acquisition unit 30 has independent databases corresponding to the respective first character recognition calculation unit 31, second character recognition calculation unit 32, and third character recognition calculation unit 33. is preferable from the viewpoint of maintaining the computation speed. In each database, the first to third machine character recognition results 31a, 32a, 33a, which are the character recognition results by the character recognition calculation units 31, 32, 33, are stored. It is copied and stored in the database. As a result, reading and writing from the database by the first character recognition calculation unit 31, the second character recognition calculation unit 32, and the third character recognition calculation unit 33 are performed independently of each other. This contributes to improving the processing speed of the unit 30 .

The number of character recognition programs that perform character recognition on the image data 20a in the machine character recognition result acquisition unit 30 is not particularly limited as long as it is plural, but is preferably about 3 to 8192, more preferably about 10 to 256. . As the number of character recognition programs increases, the efficiency of evaluation and training data production increases, and the objectivity of the matching rate and accuracy rate (described later) obtained in evaluation and the like increases. On the other hand, by limiting the number of character recognition programs to a predetermined number or less, it is possible to prevent the resources required for the evaluation system 10 from increasing excessively.

Next, in the evaluation system 10, the machine character recognition tendency calculation unit 40 performs a step of obtaining match rate data regarding the match rates of the plurality of machine character recognition results 31a, 32a, and 33a obtained from the image data 20a. As shown in FIG. 1, the machine character recognition tendency calculation unit 40 has a match rate data acquisition unit 42 for acquiring match rate data relating to the match rates of the machine character recognition results 31a, 32a, and 33a.

FIG. 2 is a conceptual diagram showing a correspondence relationship of information obtained by processing by the evaluation system 10. As shown in FIG. The concordance rate data acquisition unit 42 shown in FIG. 1 reads out the first to third machine character recognition results 31a to 33a from each database included in the machine character recognition result acquisition unit 30, and obtains concordance rate data 42a.

In FIG. 2A, five different character recognition calculation units 31, 32, and 33 (fourth and fifth character recognition calculation units are shown in FIG. are omitted) to obtain five machine character recognition results 31a to 33a. As can be understood from the five machine character recognition results 31a to 33a shown in FIG. 2(a), the image data 20a of "Nippon Taro" was subjected to character recognition by five different character recognition calculation units. Three different machine character recognition results are obtained, of which at most three machine character recognition results agree.

In the case shown in FIG. 2(a), the match rate data acquisition unit 42 matches up to three machine character recognition results as a result of character recognition performed by five different character recognition calculation units. Therefore, the match rate data acquisition unit 42 determines that the match rate for the image data 20a of "Nippon Taro" is 60% (3/5).

On the other hand, in the case shown in FIG. 2B, three different machine character recognition results are obtained as a result of character recognition performed by three different character recognition calculation units on the image data 20a of handwritten characters "same as above". are obtained, of which at most two machine character recognition results match. In the case shown in FIG. 2B, as a result of character recognition performed by five different character recognition calculation units, a maximum of two machine character recognition results match. The matching rate for the image data 20a of "same as above" is determined to be 40% (2/5).

As shown in FIG. 2, matching rate data 42a is calculated for each image data 20a that is the object of character recognition. The matching rate data 42a is calculated by comparing a plurality of machine character recognition results 31a to 33a with respect to the common image data 20a, regardless of whether the matching machine character recognition result is correct or incorrect. In other words, the matching rate data 42a is calculated regardless of the human character recognition result 56a, which will be described later, or before the human character recognition result 56a is obtained.

Note that the matching rate data 42a becomes a meaningful value (value to be shifted) only when character recognition results are obtained by a plurality of mutually different character recognition programs. This is because the match rate is always 100% when one character recognition result is obtained by one character recognition program.

Next, in the evaluation system 10, the human character recognition result obtaining unit 50 performs a step of obtaining a human character recognition result 56a, which is a character recognition result of the image data 20a visually recognized by a human. As shown in FIG. 1, the human character recognition result acquisition unit 50 includes an evaluator selection unit 52, a temporary recognition result acquisition unit 54, a determination unit 56, an evaluator database 52a, a storage unit for storing human character recognition results 56a, and the like. have.

As shown in FIG. 1, the human character recognition result acquisition unit 50 communicates with a plurality of evaluator terminals 94 operated by the evaluator via a communication network or the like, and remotely obtains the character recognition results visually recognized by humans. Obtaining a certain human character recognition result 56a is preferable from the viewpoint of efficient and inexpensive evaluation. 1, the evaluator selection unit 52 selects at least two persons (linked to the evaluator terminal 94) as evaluators; and determining tentative recognition results, which are character recognition results obtained from two humans (evaluators), to obtain human character recognition results.

FIG. 10 is a flow chart showing an example of processing in a step in which the human character recognition result acquisition unit 50 acquires the human character recognition result 56a. In step S0002, the human character recognition result acquisition unit 50, which has started the process of acquiring the human character recognition result 56a for the predetermined image data 20a in step S001, acquires the matching rate data 42a (see FIG. 2) regarding the matching rate of the corresponding image data 20a. ).

Next, in step S003, the evaluator selection unit 52 of the human character recognition result acquisition unit 50 shown in FIG. . In step S003, the evaluator selection unit 52 selects two evaluators by referring to the matching rate data 42a read out in step S002 and the evaluator reliability indicating the reliability of the candidate as an evaluator. do.

Here, the evaluator reliability used by the evaluator selection unit 52 for selecting evaluators is obtained by the processing of the human character recognition result acquisition unit 50 shown in FIG. It relates to the probability (evaluator correct answer rate) of matching with the human character recognition result 56a finally determined by . However, the evaluator reliability is not limited to being composed only of the evaluator's rate of correct answers for each evaluator. can be anything.

FIG. 3 is a correspondence table showing criteria for selecting evaluators by the evaluator selection unit 52 in step S003. When the match rate of the image data 20a to be processed is high (for example, 80% or more=A rank), the evaluator selection unit 52 preferentially selects candidates with higher evaluator reliability as evaluators. In addition, when the matching rate of the image data 20a to be processed is low (for example, 40% or less = C rank), the evaluator selection unit 52 preferentially selects candidates with relatively low evaluator reliability as evaluators. Select. In addition, when the match rate of the image data 20a to be processed is medium (for example, more than 40% and less than 80% = B rank), the evaluator selection unit 52 selects a candidate with a medium degree of evaluator reliability as an evaluator. to be selected preferentially.

As described above, for the image data 20a with a high matching rate, compared to the case for the image data 20a with a low matching rate, a person with a high evaluation reliability is selected as an evaluator, giving priority to human character recognition. The efficiency of obtaining the result 56a can be improved. As will be described later with reference to FIG. 4, in a situation where image data 20a with a high matching rate is evaluated, there is a high possibility that correct character recognition results are included in the machine character recognition results 31a to 33a. In addition, the ability to search out the machine character recognition results 31a to 33a that match the image data 20a is important, and the ability difference of the evaluators tends to appear. On the other hand, in a situation where image data with a low matching rate is evaluated, there is a high possibility that there is no correct character recognition result among the machine character recognition results 31a to 33a, and the frequency of typing temporary recognition results increases. However, there are many cases where it takes a long time, and it is relatively difficult to show differences in the ability of evaluators.

Next, in step S004, the temporary recognition result acquiring unit 54 of the human character recognition result acquiring unit 50 sends the temporary recognition result (or An input screen for the human character recognition result 56a) is displayed. As shown in FIG. 4, the human character recognition result acquiring unit 50 obtains one of the image data 20a, the plurality of machine character recognition results 31a to 33a, and the plurality of machine character recognition results 31a to 33a as a human character recognition result 56a. Alternatively, the first graphical interface 58 that can be selected as the candidate provisional recognition result is displayed on the display unit of the evaluator terminal 94 .

As shown in FIG. 4, the first graphical interface 58 is a check box for checking the correct character recognition result of the image data 20a from among the plurality of machine character recognition results 31a to 33a. . As indicated by the dotted lines in FIG. 4, the evaluator selects the check boxes corresponding to the machine character recognition results 31a to 33a that are correct character recognition results, and the character recognition results for the image data 20a by the evaluator are displayed. is determined.

In step S005, the provisional recognition result input by the evaluator to the evaluator terminal 94 is sent to the provisional recognition result acquisition unit 54 of the human character recognition result acquisition unit 50 through the input of the confirmation button 58a and the like. The acquisition unit 54 acquires the temporary recognition result. At this time, the temporary recognition result acquisition unit 54 acquires temporary recognition results from the two evaluators selected in step S003. As shown in FIG. 4, the display unit of the evaluator terminal 94 may display a second graphical interface 59 such as a text box into which the temporary recognition result can be typed. The evaluator can also input the provisional recognition result by directly inputting to the second graphical interface 59 .

Also, as shown in FIG. 4 , when any of the machine character recognition results is selected by the first graphical interface 58 , the selected machine character recognition result is copied to the second graphical interface 59 . Further, the display section of the evaluator terminal 94 is provided with an automatic judgment display column 57 for displaying whether each of the machine character recognition results 31a to 33a is a correct answer or an incorrect answer when the provisional recognition result is input. may have been

In step S006, the determination unit 56 shown in FIG. 1 determines whether or not the temporary recognition results, which are the character recognition results obtained from the two evaluators acquired in step S005, match each other. In step S006, when the temporary recognition results obtained by the two selectors match each other, the process proceeds to step S007, and the determination unit 56 shown in FIG. It is authorized and stored in the storage section of the human character recognition result acquisition section 50 .

On the other hand, if the provisional recognition results of the two selectors do not match each other, it is determined in steps S008 and S009 that the number of repetitions does not exceed a predetermined number (5 times), and then the two persons are identified. The process returns to the step of selecting the evaluator (step S003). If it is determined in steps S008 and S009 that the number of repetitions exceeds the predetermined number (5 times), the process proceeds to step S010. In step S010, the human character recognition result acquisition unit 50 suspends the acquisition of the human character recognition result 56a for the target image data 20a to prevent endless repetition of the process. For the image data 20a for which acquisition of the human character recognition result 56a has been suspended in step S010, the human character recognition result 56a is determined by, for example, the supervisor of the evaluation system 10 or the like.

After step S007 or step S010, the process proceeds to step S011, and the process of obtaining the human character recognition result 56a by the human character recognition result obtaining unit 50 ends. Note that when the human character recognition result 56a for the image data 20a is determined in step S007 or the like shown in FIG. Store or update rater confidence in terms of probability of matching with recognition result 56a. FIG. 7 is a conceptual diagram showing the evaluator reliability levels 52a1 to 52a6 stored in the evaluator database 52a of the human character recognition result acquisition unit 50 shown in FIG. Evaluator reliability levels 52a1 to 52a6 of each evaluator are stored in the evaluator database 52a. The evaluator database 52a stores other information about the evaluators, such as the login time of each evaluator and the number of processes.

Also, in FIG. 10 and the like, the step of acquiring the human character recognition result is explained by exemplifying the process of selecting two evaluators and acquiring the human character recognition result 56a, but only two evaluators are selected. is not limited, and may be one or three or more. For example, when the human character recognition result 56a is acquired by one evaluator, the human character recognition result 56a is certified based on the results input by the evaluator on the screen shown in FIG.

After the human character recognition result 56a is obtained by the human character recognition result obtaining unit 50 shown in FIG. The grading unit 60 has a correct/wrong judging unit 62, a correct answer rate calculation unit 64, and a storage unit that stores the correct/wrong answer 62a and the correct answer rate 64a of the machine character recognition result 31a.

The correct/wrong judging section 62 of the scoring section 60 shown in FIG. 1, as shown in FIG. The result 56a is compared to determine whether each of the machine character recognition results 31a to 33a is correct or incorrect 62a. The correctness determination unit 62 determines that the machine character recognition results 31a to 33a match the human character recognition result 56a as a correct answer, and the machine character recognition results 31a to 33a do not match the human character recognition result 56a. Judge things as wrong answers. The correct/wrong judging section 62 stores the correct/wrong 62a of the machine character recognition results 31a to 33a in the storage section of the scoring section 60. FIG. The same applies to FIG. 2(b).

Further, the accuracy rate calculation unit 64 of the scoring unit 60 shown in FIG. 1 can perform correct character recognition on the predetermined image data 20a among the plurality of machine character recognition results 31a to 33a for the predetermined image data 20a. A correct answer rate 64a indicating the ratio of the character recognition program is calculated and stored in the storage section of the scoring section 60. FIG. As shown in FIG. 2(a), for image data 20a of handwritten characters "Nippon Taro", three of machine character recognition results 31a to 33a corresponding to five different character recognition programs are correct answers, Two are wrong answers. That is, with respect to the image data 20a of "Nippon Taro", three character recognition programs, which are 60%, out of the five character recognition programs to be evaluated give correct answers, so the calculated correct answer rate 64a is 60%. be.

On the other hand, as shown in FIG. 2B, with respect to the image data 20a of handwritten characters "same as above", one of the machine character recognition results 31a to 33a corresponding to five different character recognition programs is correct. , are incorrect answers, the calculated accuracy rate 64a is 20%. As can be understood from FIGS. 2A and 2B, when the matching rate and the correct answer rate 64a for the predetermined image data 20a are compared, the matching rate is higher than the correct answer rate 64a, and the correct answer rate 64a is higher than the correct answer rate 64a. and the rate of coincidence.

Next, in the evaluation system shown in FIG. 1, the information integrating section 70 stores the evaluation result 72b of the character recognition program and the teacher data 72a for the character recognition program with machine learning function in the storage section. The information integration unit 70 has an information processing unit 72, and the information processing unit 72 integrates information generated by each unit of the evaluation system 10 to generate a character recognition program evaluation result 72b and teacher data 72a.

FIG. 8 is a conceptual diagram showing information included in the character recognition program evaluation result 72b. As shown in FIG. 8, the character recognition program evaluation result 72b includes the image data 20a obtained by the image data obtaining unit 20, the machine character recognition results 31a to 33a obtained by the machine character recognition result obtaining unit 30, and the score. The correct/wrong 62a obtained by the unit 60, the human character recognition result 56a obtained by the human character recognition result obtaining unit 50, the coincidence rate data 42a obtained by the machine character recognition tendency calculation unit 40, and the scoring unit 60 and the obtained accuracy rate 64a.

FIG. 9 is a conceptual diagram showing a specific example of information included in the character recognition program evaluation result 72b. As shown in FIG. 9, in the character recognition program evaluation result 72b, a first machine character recognition result 31a by a predetermined character recognition program, its correctness/incorrectness 62a, a human character recognition result 56a, matching rate data 42a, and an accuracy rate 64a are stored in association with the corresponding image data 20a.

Since the character recognition program evaluation result 72b includes multifaceted indices such as the accuracy rate 64a and the match rate for each image data 20a, it provides multifaceted and effective knowledge about the correctness 62a of the machine character recognition result for the image data 20a. Obtainable. FIG. 6 is a conceptual diagram showing an example of analytical knowledge obtained from the information included in the character recognition program evaluation result 72b.

For example, as shown in the second line of FIG. 6, when a predetermined character recognition program gives an incorrect answer to image data with a high matching rate and a high accuracy rate, the character recognition ability for that image data will be evaluated by another character recognition program. It can be analyzed that it is relatively inferior to In addition, as shown in the third line of FIG. 6, when the predetermined character recognition program gives an incorrect answer to image data with a high match rate and a low accuracy rate, the character recognition program will be notified of the character recognition ability for that image data. It can be analyzed as having a common underlying problem. Also, as shown in the fourth row of the table shown in FIG. 6, when the predetermined character recognition program gives a correct answer to image data with a high match rate and a low accuracy rate, the character recognition ability for that image data is evaluated by another program. It can be analyzed that it has an innovative superiority that character recognition programs do not have.

Further, as shown in the fifth line of FIG. 6, when a predetermined character recognition program gives an incorrect answer to image data with a low matching rate and a low accuracy rate, the character recognition ability for that image data is determined by each character recognition program. It can be analyzed that there is a large variation and that there is a somewhat deep-seated problem. In addition, as shown in the sixth line of FIG. 6, when a predetermined character recognition program gives a correct answer to image data with a low matching rate and a low accuracy rate, the character recognition ability for that image data is determined by another character recognition program. It can be analyzed that it has a relative advantage over

As with the character recognition program evaluation results 72b shown in FIGS. 8 and 9, the teacher data 72a shown in FIG. are stored in association with However, unlike the character recognition program evaluation result 72b, the teacher data 72a may omit the first machine character recognition result 31a by a predetermined character recognition program and its correctness/incorrectness 62a.

As with the character recognition program evaluation result 72b, the teacher data 72a also includes multifaceted indices such as the accuracy rate 64a and the match rate for each image data 20a. And effective knowledge can be obtained. FIG. 5 is a conceptual diagram showing an example of analytical knowledge obtained from information included in the teacher data 72a.

As shown in the second line of FIG. 5, image data with a high matching rate and a high accuracy rate can be expected to have a learning effect of confirming the current attainment level at which many character recognition programs can give correct answers. In addition, as shown in the third line of FIG. 5, image data with a high match rate and a low accuracy rate can be expected to have a learning effect that improves problems common to character recognition programs. Further, as shown in the fourth line of FIG. 5, image data with a low match rate and a low accuracy rate is expected to have a learning effect of improving character recognition, which has large individual differences between character recognition programs.

As described above, according to the evaluation system 10 shown in FIG. 1, the image data 20a of the actual handwritten characters are subjected to character recognition by a plurality of character recognition programs. Consistent and objective evaluation results are obtained. Further, character recognition of common image data 20a by a plurality of character recognition programs can increase the number of program evaluations per result of human character recognition, thereby improving efficiency of evaluation.

10 Evaluation system for character recognition program (evaluation system)
20 Image data acquisition unit 22 Handwritten characters 20a Image data 30 Machine character recognition result acquisition unit 31 First character recognition calculation unit 31a First machine character recognition result 32 Second character recognition calculation unit 32a Second Machine character recognition result 33 Third character recognition calculation unit 33a Third machine character recognition result 40 Machine character recognition tendency calculation unit 42 Concordance rate data acquisition unit 42a Concordance rate data 50 Human character recognition result acquisition unit 52 Evaluator selection unit 54 Temporary recognition result acquisition unit 56 Judgment unit 52a Evaluator database 52a1 to 52a6 Evaluator reliability 56a Human character recognition result 58 First graphical interface 59 Second graphical interface 57 Automatic judgment Display column 60 Scoring unit 62 Correctness determination unit 64 Accuracy rate calculation unit 64a Accuracy rate 70 Information integration unit 72b Character recognition program evaluation result 72a Teacher data 94 Evaluator terminal

Claims (6)

A character recognition program evaluation method by a character recognition program evaluation system,
an image data acquisition unit acquiring image data of handwritten characters;
A character recognition calculation unit using a plurality of different character recognition programs performs character recognition on the image data, and a plurality of machine character recognition results, which are the character recognition results of the plurality of character recognition programs for the image data, are converted into machine characters. a step obtained by the recognition result obtaining unit ;
a matching rate data obtaining unit obtaining matching rate data relating to a matching rate of the plurality of machine character recognition results obtained from the image data;
a step in which a human character recognition result obtaining unit obtains a human character recognition result, which is a character recognition result of the image data visually recognized by a human;
An accuracy rate calculation unit compares each of the plurality of machine character recognition results with the human character recognition result to determine whether the plurality of machine character recognition results are correct or not, and determines whether the plurality of machine character recognition results are correct or not. a step of obtaining an accuracy rate;
The information integration unit generates a character associated with the image data, at least one of the plurality of machine character recognition results and correct/wrong thereof, the human character recognition result, the matching rate data, and the accuracy rate. and a step of storing recognition program evaluation results. In the step of obtaining the human character recognition result, the human character recognition result acquisition unit selects at least two humans as evaluators;
determining tentative recognition results that are character recognition results obtained from the at least two humans;
In the step of judging the provisional recognition result, the human-character recognition result acquisition unit recognizes the provisional recognition result as the human-character recognition result when the provisional recognition results match each other; 2. The method of evaluating a character recognition program according to claim 1, wherein if there is no match, the process returns to the step of selecting at least two persons as evaluators. In the step of obtaining the human character recognition result, the human character recognition result acquisition unit stores an evaluator confidence level regarding the probability that the temporary recognition result of each evaluator matches the human character recognition result,
In the step of selecting at least two people as evaluators, the human character recognition result acquisition unit selects at least two people as evaluators using the matching rate data and the evaluator reliability. 3. The character recognition program evaluation method according to claim 2. In the step of obtaining the human character recognition result, the human character recognition result obtaining unit obtains one of the image data, the plurality of machine character recognition results, and the plurality of machine character recognition results as the human character recognition result or 2. The character recognition program evaluation method according to claim 1, further comprising the step of displaying a graphical interface selectable as the candidate on the evaluator's terminal. an image data acquisition unit that acquires image data of handwritten characters;
a machine character recognition result acquisition unit for obtaining a plurality of machine character recognition results, which are character recognition results of the image data by a plurality of different character recognition programs performing character recognition on the image data by the plurality of character recognition programs; ,
a matching rate data obtaining unit for obtaining matching rate data relating to the matching rates of the plurality of machine character recognition results obtained from the image data;
a human character recognition result obtaining unit for obtaining a human character recognition result, which is a character recognition result of the image data visually recognized by a human;
Each of the plurality of machine character recognition results and the human character recognition result are compared to obtain a correctness determination unit for determining whether the plurality of machine character recognition results are correct and an accuracy rate of the plurality of machine character recognition results. an accuracy rate calculator;
a character recognition program evaluation result in which the image data, at least one of the plurality of machine character recognition results and correct/incorrect thereof, the human character recognition result, the matching rate data, and the accuracy rate are associated with each other; An evaluation system for a character recognition program, comprising: an evaluation result storage unit for storing. A method for producing teacher data for a character recognition program using a character recognition program evaluation system,
an image data acquisition unit acquiring image data of handwritten characters;
A plurality of different character recognition programs perform character recognition on the image data, and a machine character recognition result acquisition unit obtains a plurality of machine character recognition results, which are character recognition results of the image data by the plurality of character recognition programs. a step;
a matching rate data obtaining unit obtaining matching rate data relating to a matching rate of the plurality of machine character recognition results obtained from the image data;
a step in which a human character recognition result obtaining unit obtains a human character recognition result, which is a character recognition result of the image data visually recognized by a human;
A correctness rate calculation unit compares each of the plurality of machine character recognition results with the human character recognition result to determine whether the plurality of machine character recognition results are correct or not, and determines whether the plurality of character recognition programs are correct. obtaining a rate;
A method for producing teaching data for a character recognition program , wherein an information integration unit stores teaching data associated with the image data, the human character recognition result, the matching rate data, and the accuracy rate.

Images