JP4192729B2 - Method for highlighting free-form annotation, annotation highlighting device, and program for highlighting free-form annotation - Google Patents

Description

  The present invention generally relates to systems and methods for highlighting high-value annotations in a free-form environment. That is, the present invention relates to a method for highlighting free-form annotations, a free-form annotation enhancement system that emphasizes high-value free-form annotations, and a program for highlighting free-form annotations.

  Traditional user interfaces in digital libraries tend to focus on search. That is, the user can search for documents online or on a display screen. However, the user must print the document and manually mark and annotate important phrases / phrases in the document. Annotations can help the reader understand the document content at a glance, provide a more recognizable structure for the reader's own thoughts, and / or make it the most appropriate and useful part of a long document Concentrate readers.

  When a reader marks a document, the reader's annotations are not equally important. That is, the mark attached by the reader has a different function, and the degree of the value is different. Differences in value allow the user to personalize the document so that key phrases or ideas or important citations stand out in the overall text summary. An automatic electronic summary of the document can then be created based on the annotation.

  A number of methods have been attempted to display and present annotations in documents. For example, a thumbnail display or other content display is used. However, these methods generally depend on the type of annotation rather than the importance and / or value of the phrase as indicated by the type and / or amount of annotation.

  Conventionally, a method for detecting and extracting free-form annotations for use in organizing reader annotations into meaningful structures has been disclosed (see, for example, Patent Document 1). However, as in the case of the technology in other collaborative applications, the method and system shown in US Pat. Is not distinguished from other low-value clauses or annotations. These methods and systems can only focus on and extract the annotations themselves, or the clauses to which the annotations are associated.

As a result, in these methods and systems, the format of the annotation itself is converted. For example, as shown in FIG. 1, multi-thumbnail display is commonly used to display long document clauses. However, based on this multi-thumbnail display, the user cannot identify which pages or sections of the document are particularly important. More specifically, the highlighted portion is not conspicuous. In other examples, the function of the annotation, i.e., the commentator's thoughts about the importance or relevance of the annotation and / or the corresponding clause, is lost because the annotation symbol cannot be identified. That is, the margin (margin) can no longer be read, and all annotations appear to have the same weight and value.
Conventionally, a method of detecting and “peeling” an annotation using an “annotation lifting” technique has been disclosed. This method is a method of comparing a marked reference document (first image) with an unmarked reference document (second image) and extracting a pixel displaying the mark (for example, (See Patent Document 2). A method for detecting an annotation using a shape recognition algorithm is also disclosed. This method is a method of searching and matching gesture-based data such as handwritten input without performing a conventional recognition technique (see, for example, Patent Document 3). Furthermore, a system is disclosed that can search a database in an electronic notebook based on characteristics (note mark type), keywords (schematic contents of note marks on a digital page), and speech enhancement (for example, patent documents). 4). However, the above-mentioned problems are not solved by these disclosures.
US Pat. No. 6,279,014 US Pat. No. 5,692,073 US Pat. No. 5,687,254 US Pat. No. 5,970,455 Poon et al., “Scribbler: A Tool for Searching Digital Ink”, CHI'95 Mosaic of Creativity, 1995 May 7-11, pp. 252-253 Shipman et al., “Identifying Useful Passages in Documents based on Annotation Patterns”, FX Palo Alto Laboratory, pp. 197 1-8 Bederson et al., “Pad ++: A Zooming Graphical Interface for Exploring Alternate Interface Physics (Pad ++: A Zooming Graphical Interface for Exploring Alternate Interface Physics)”, UIST, November 2, 1994— 4th, pp. 17-26

  Thus, the inventors have sought to improve the usefulness of annotations using a system that can rank a series of detected free-form annotations according to importance and display the importance of the annotation.

  The present invention provides a system and method for detecting and enhancing high-value free-form annotations.

  The present invention provides a system and method for identifying important or high value annotations.

  The present invention provides a system and method for marking and / or displaying a determined high value annotation in a recognizable format.

  In various exemplary embodiments of the systems and methods according to this invention, a particular annotation style used by a particular user to create an annotation on a document is determined. This annotation style may be individual for the particular individual who created the annotation, or it may be a standardized style. Once the annotation style is determined such that the high value annotation mark can be distinguished from the low value annotation mark, the annotation in the document is analyzed to locate the high value annotation mark.

  In various exemplary embodiments, statistical analysis, time display information, handwriting detection, and / or other to classify annotations that can separate high-value annotations from the remaining annotations Based on the approach, high value annotation marks can be distinguished from low value annotation marks.

According to a first aspect of the present invention, there is provided a method for emphasizing a free-form annotation, wherein document data indicating a document annotated to alert a reader is received by a receiving means, and the received data The document indicated by the document data is analyzed by the analysis unit to identify the type of annotation attached to the document, and the determination unit determines the importance of each annotation based on a predetermined criterion according to the identified annotation type. The display of the annotation in a state where at least a part of the document indicated by the document data is reduced and displayed on the display means is reduced to the display of the annotation in a state where it is not reduced and displayed. According to this method, the control is performed so that the higher the importance of the annotation is, the more visually emphasized it is displayed, compared with the display when it is reduced .

According to a second aspect of the present invention, in the first aspect, the predetermined criterion is such that the lower the appearance frequency for each type of annotation is, the higher the importance is, and indicated by the received document data. In this method, the document is analyzed by an analysis means to determine the frequency of appearance for each type of annotation, and the smaller the frequency of appearance, the higher the importance and the importance of each annotation is determined .

According to a third aspect of the present invention, in the first and second aspects described above, the appearance of the annotation is changed so that the higher the determined importance level is visually emphasized and displayed on the display means. And display it .

A fourth aspect of the present invention is the method according to the third aspect, wherein the changing the appearance includes changing at least one of font size, font color, underline, highlighting, and iconification. .

According to a fifth aspect of the present invention, in the first to fourth aspects, the determination of the importance is performed by grouping annotated parts or phrases, and ranking the relative importance of a plurality of groups, A method that includes typing .

A sixth aspect of the present invention is a method according to the fifth aspect, wherein the part or phrase annotated with a plurality of annotations is classified as a multimark .

  According to a seventh aspect of the present invention, there is provided an annotation emphasis display device, which is indicated by a receiving unit that receives document data indicating a document annotated to alert a reader, and the received document data. Determining means for analyzing the document by the analyzing means to identify the type of annotation attached to the document and determining the importance of each annotation based on a predetermined criterion according to the identified type of the annotation; The display of the annotation in a state where at least a part of the document indicated by the document data is displayed in a reduced size is a display when the display of the annotation in a state in which the document is not reduced is reduced according to the reduction. In comparison, the annotation emphasis display device includes control means for controlling the annotation so that the higher the importance of the annotation is, the more visually emphasized it is displayed.

  According to an eighth aspect of the present invention, in the seventh aspect, the predetermined criterion is such that the lower the appearance frequency for each type of annotation is, the higher the importance is. An annotation highlighting device that analyzes the document indicated by the document data by using an analysis means to determine the appearance frequency for each type of annotation, and determines the importance of each annotation by increasing the importance as the appearance frequency decreases. is there.

  According to a ninth aspect of the present invention, in the seventh or eighth aspect, the control means changes the appearance of the annotation and displays it visually enhanced as the determined importance is higher. An annotation emphasis display device that controls display means.

  In a tenth aspect of the present invention, in the ninth aspect, an annotation highlighting display wherein the changing the appearance includes changing at least one of font size, font color, underline, highlighting, and iconification. Device.

  According to an eleventh aspect of the present invention, in the seventh to tenth aspects, the determination of the importance by the determination means is performed by grouping annotated parts or clauses, and calculating a relative importance of a plurality of groups. An annotation highlighting device that includes ranking and typing.

  A twelfth aspect of the present invention is the annotation highlighting apparatus according to the eleventh aspect, wherein a part or a phrase with a plurality of annotations is classified as a multimark.

  According to a thirteenth aspect of the present invention, there is provided a program for emphasizing free-form annotations, wherein the computer receives document data indicating a document annotated to alert the reader, and receiving The document indicated by the document data is analyzed by an analysis means to identify the type of annotation attached to the document, and the importance of each annotation is determined based on a predetermined criterion corresponding to the type of the identified annotation. A determination means for determining the degree, and the display of the annotation in a state where at least a part of the document indicated by the document data is displayed in a reduced size is reduced to the display of the annotation in a state in which the document is not displayed in a reduced size And a control means for controlling the display so that the higher the importance of the annotation is, the higher the importance of the annotation is. It is.

  These and other features and advantages of the present invention will be set forth in or apparent from the following detailed description of the apparatus / system and method according to the present invention.

  For example, reduced displays such as thumbnails, schematic outlines, outlines, and summaries are used in many interfaces. Using these interfaces, a user can view and interact with many documents at once, or many pages of a document on a standard size display. When a reader places a mark on a document, he or she indicates a phrase that is important to the reader or others. Some of these annotations are particularly valuable and others are of little importance except for specific reading ranges. Emphasizing high value annotations according to the system and method of the present invention can improve reduced document display after automatically detecting such high value annotations according to the system and method of the present invention. Emphasizing high-value annotations eliminates noise from the reduced display, and also allows the user to focus attention on relevant parts of a long document or an extensive collection of documents.

  FIG. 1 shows a collection of thumbnails displaying individual pages of a long document. This thumbnail display is adopted by the XLibris (trademark) document reader available from Fuji Xerox (Tokyo, Japan), but rendered like Adobe Acrobat (http://www.adobe.com/prodindex/acrobat/) Programs, scanning front-end programs such as Pagis (http://www.pagis.com/home.html), The integrator specified in R. Trigg et al., “Moving document collections online: The evolution of a shared repository” (ECSCW'99 newsletter) Displaying a document used in such a document management system is a common method. The page of a particular document, indicated by the thumbnail shown in FIG. 1, is read by the user and annotated or marked. Figure 1 shows that it is difficult to distinguish one page from another, but this is because many of the pages are marked with highlight markers and all highlighted parts are Because it looks almost the same. For example, if the user tries to find an important annotated phrase at the beginning of the document, it may be difficult to select between page 17 and page 25 of the document shown in FIG. . It is a high overhead operation for the user to consider each page by enlarging the page to full size. However, the user has already indicated which of these pages are of interest when the document is first read and highlighted. On pages 17 and 25, there are phrases marked with stars. However, to show all the pages of a document as thumbnails, a compact display as shown in FIG. 1 is required, and these star annotations are not visible.

  It is important to note that none of the methods described below depend on the document that was originally annotated or the marking method that was annotated with digital ink. Indeed, it is easy to assume a first step in which a marked document is scanned. An annotation is detected and “peeled” using an “annotation lifting (lifting)” technique (see, for example, Patent Document 2).

A wide variety of user interfaces for information systems use reduced display of documents. For example, a program that acts as a front end to a scanner, such as "Pagis" or Visioneer's "PaperPort" document management system, uses thumbnails as one of the principle modes for simultaneously presenting multiple pages or multiple documents. There are many cases. For example, as shown by HYPERLINK "http://www.amico.org/home.html" http://www.amico.org/home.html , the database of images as provided by "AMICO" is Thus, the search results are presented in the same manner.

  For example, other tools for manipulating documents, such as MS Word and MS PowerPoint, use the document structure to build a reduced display. To generate a table of content or an outline that allows the user to view and interact with one or more pages of a document at a time, such as tagged documents such as section headings, individual PowerPoint slides, etc. Discrete boundaries may be used by these programs. In many cases, search tools, such as Altavista, present search results as a calculated reduced document display such as a summary. This reduced display corresponds to the gist of the document.

  It should be understood that these examples are not intended to limit the adaptable types of reduced display to which the systems and methods according to the present invention are applied, but rather to illustrate their scope.

  While users annotate documents extensively, not all these annotations are worth the same to the reader over time. Some annotations are temporary traces of the reader's attention as they struggle to read difficult explanations. Other annotations select phrases that will cause the reader to react. Yet another annotation is commentary, useful for the next reading or future activity. For example, if the user expects a word to be quoted verbatim in a summary document that the lawyer is preparing, the lawyer can place a star on the phrase.

  FIG. 2 illustrates an example annotation style that may be utilized by a user. FIG. 2 illustrates various methods for annotating documents that can be analyzed and emphasized using the free form annotation enhancement system and method according to the present invention. The annotation style in the upper left phrase of FIG. 2 indicates that this particular part or phrase is important by using (a) an asterisk in the margin of the document adjacent to this part or phrase of the document. . In various exemplary embodiments according to the present invention, this star is used as an objective indicator that the highlighted portion or phrase of the document is of high value.

  The clause on the upper right side of FIG. 2 shows another annotation style. For example, (b) the importance of a part of a document or section of a sentence highlighted in advance by (c) a circle or (d) underlining in a part or section of the document highlighted. Or the emphasis is further strengthened. In the lower left clause shown in FIG. 2, a part or clause of the document is more important by the user attaching (f) brackets to the portion or clause that was previously underlined (e) in the document. Has been. Similarly, (g) a double underline can be placed on this part or clause to indicate that the clause is more important. In the lower right clause shown in FIG. 2, (h) free-form annotations such as handwriting in margins indicate that adjacent parts or clauses of the document are more important, ie, more valuable.

  FIG. 3 shows a specific example of an electronic document. The upper half of the page shown in FIG. 3 includes digital annotations such as (a) underline and (b) margin bars. These annotations are considered to be less important than (d) annotations (c) written on the page and adjacent text clauses, such as circled or indicated by arrows. Each part of the page is enumerated in the text and annotated with out-of-line writing to recall such part of the text. The next (e) commentary on the page is not emphasized as much as the previous marginal writing, but is considered to be of high value as well. This example illustrates the changes and relevance of reader annotations to operations including reread and write.

  The first aspect to note in the page shown in FIG. 3 is only how much weight the page is annotated with. Few pages have no reader mark. This example also illustrates the various stages of processing that must be performed to detect high value annotations. First, the reader's marking style must be characterized. Thereby, the value of the annotation is normalized. In order to normalize the annotation values, the background of the document and the frequency of the reader's mark inside the page must be determined, and the type of mark that the reader tends to mark must be identified. In particular, the reader usually has to decide on whether to underline, highlight, use margin bars, write symbols in the margins, and / or to write out of the margins extensively. If the reader usually makes few marks on the document, the highlighted phrase is likely a high-value annotation. Alternatively, if the reader uses highlight markers on most pages, the single highlight example is less likely to be as important. Style characterization assigns weights to individual annotations to evaluate the correlation values of these annotations.

  After the reader's style has been characterized, the characteristics of the annotation itself must be analyzed to detect which types of annotation among the various annotations attached by the reader are of high value to the reader. I must. In order to determine which types of annotations are high-value annotations for the reader, the annotation type heuristic assessment is now used, for example, textual marks such as highlighting and underlining, commentary and Emphasis was placed on the distinction between annotation means, such as white space marks such as symbols. Further distinctions between margin writing are: range (ie, the space occupied by the annotation), shape (eg, length and width), placement (ie, where the annotation is associated with the source document on the page), well known Depending on the features that will be developed or will be developed at a later date. Further, if it is desired, the annotation itself may be analyzed to determine the standard shape. For example, in FIG. 3, “1” and “2” surrounded by a pair of (f) stars or (g) circles may be detected using a shape recognition algorithm (see, for example, Patent Document 3). reference). Even if not completely accurate, the relative length of specific parts of the commentary can be easily determined by handwriting recognition. In general, readers find that long commentary is more valuable than short commentary.

  Finally, the marking pattern can be used to help evaluate relative values. For example, clauses associated with one or more types of marks can be identified. In FIG. 3, it is possible to detect the fact that there are circles, arrows, and extra-marginal notes adjacent to these symbols. The confluence of these factors, i.e. the reader encircles the text rather than underlining it, the reader marks both sides with a symbol, or the reader writes out of the box, A decision can be made that the clauses in the text are considered particularly important.

  The emphasis of high value annotations is based on a set of techniques that can be used to visually reveal the reader's high value marks in a reduced document display. Color or size character emphasis (by saturation) of the reader's high-value mark annotation is a simple way to make such high-value annotation better visible in the schematic display. This enhancement may be performed at the same time as other annotations are not highlighted, for example, by reducing the color saturation of low-value annotations. As an alternative or in addition, such high value annotation marks may be stylized or iconified. If the reader uses regular stars to call up important phrases and display them in the next writing task, these stars will be visible in, for example, “*” in a reduced display. Can be made more visible by substituting a target substitute. Third, and more generally, the entire set of predictable mappings may be applied to convert such high value annotations to something better visible. Since the yellow mark is not visible in the reduced display, it may be converted into a red mark. Similarly, important marks in phrases can be converted to font features in the content table. Fourth, low-value marks may be omitted or disappear. Finally, important markings can also be used as a basis for further processing, such as automatically building a summary.

  The inventors have found that users use two main annotation type categories. The first category includes selection marks such as highlights, underlines, circles, and margin bars that identify passages in the document. The second category includes descriptive marks such as comments and symbols that respond to the content. Needless to say, the roles of these two classes of marks are not completely distinguished, but because the action of selecting text from the body is descriptive action, writing out of the margin is inevitably necessary to pay attention to nearby text. This is because the focus is selected.

  Such observation has led to the design of a mark parser (syntactic analysis system) that identifies the structure and importance in the pen stroke information of a reader such as XLibris. The mark parser classifies groups of marks into annotation types and builds a phrase-based grouping of annotations that can be used to identify high importance sections of the document.

  The text and annotations shown in FIG. 4 illustrate how the mark parser classifies mark groups, builds annotation groups, and identifies high-value marks. In this example, the red ink mark must be identified as a comment attached to the same phrase as the adjacent highlight mark. This marks the two clauses as multi-mark clauses, for example perceived importance, which is an importance value for comparing such marked clauses with other marked clauses. Relative weighting may be applied.

  It should be understood that a digital ink annotation includes data representing a temporal sequence of marks, which can be advantageously used by a mark parser in parsing.

  The mark parser performs a three-pass process that groups, types, and ranks the relative importance of the reader's annotation marks. In the first pass, the mark parser clusters time rank marks into low level groups based on time, distance, and / or pen type. At this stage, individual pen strokes can be combined into clusters that can be determined to be a single annotation.

  In the first pass, the mark parser uses temporal and spatial distance functions to determine breaks in mark groupings. Pen type differences are utilized, at least in part, in determining temporal and spatial gaps within a stroke sequence, because highlight markers and pens are used differently by annotation. In general, the temporal gap between related marks, such as the strokes that make up a single annotation, is larger when using a pen than with highlight markers. This may reflect the fact that readers tend to ponder more when writing descriptive comments and symbols than simply highlighting phrases. During the first stage, the parser further groups pen strokes that display temporally continuous but spatially distinguishable marks, such as multiline comments. This process results in grouping one-level or multi-level hierarchical clustering of marks, as shown in FIG.

  It should be understood that the top set of highlight marker strokes in FIG. 4 is divided by a large temporal gap between the strokes, as in the bottom comment. When setting the boundaries for clustering between errors (if) strokes that should be grouped eventually become individual clusters and errors (if) unrelated strokes are placed in the same cluster There are exchange conditions (trade off). Since the path after the mark parser joins the clusters into a high-level structure, mistakes on the side with more clusters can be grouped further later in any case So it seems suitable.

  In the second pass, the mark parser classifies mark clusters into annotation types using the pen type, the number of component marks, and / or the size and / or shape of the cluster range. In the example shown in FIG. 4, the six high-level clusters shown as rectangles include three highlight annotations and three comment annotations. In various exemplary embodiments, the stroke type may be based on the pen type, the number of overlapping words, and / or the shape of the stroke. For example, a cluster composed of five or more pen marks is considered a comment as opposed to a highlight marker. As clusters are classified, importance values are further assigned. One exemplary embodiment of the category types and a list of basic importance values assigned to these category types is shown in Table 1 below.

  These are a range of three basic importance values for the importance values assigned to the mark cluster. For example, an importance value within a low value range is assigned to a mark that roughly identifies a target region, such as region highlighting or surrounding a region with a circle. Selection marks with more emphasis, such as underlines, margin bars, highlights, and circles, are assigned values within the intermediate value range. Descriptive marks such as comments, symbols, calls (callouts), etc. are assigned values in the high price range. The initial value of the importance value is selected such that clauses with mark cluster combinations are higher in rank or equivalent than clauses marked with only one type of cluster. In the example shown in FIG. 4, three comment clusters have an importance value of 20, while three highlighted clusters have an importance value of 15.

  During the third pass, the mark parser groups each mark cluster with a spatially relevant clause or clause from the source text. Due to the duplication of clauses in several mark clusters, these mark clusters are grouped into a single multi-mark cluster, creating the highest entity in the resulting parse tree. A phrase is associated with each stroke by selecting the sentence adjacent to the mark and identifying the sentence closest to the stroke. In this way, a larger horizontal gap is allowed compared to the vertical gap. In the example shown in FIG. 4, highlight markers and comments are attached to the same sentence in the text. As a result, the two phrases are correctly recognized as multimark annotations or clusters, as indicated by the blue rectangle in FIG.

  The importance value of such a multimark cluster is the sum of the importance values of the constituent clusters of the multimark cluster. Thus, for the two marked phrases in the example shown in FIG. 4, the upper and lower phrases have importance values 50 and 55, respectively.

FIG. 4 further illustrates one exemplary embodiment of the clustering method used in various embodiments of the present invention. In FIG. 4, the lowest cluster is displayed by a small inner box, and the higher cluster is displayed by a box that is larger and contains more. The larger box has a type and importance value attached. Due to the overlap between clusters, the overlapping clusters are grouped into multi-mark clusters. Multimark clusters are assigned the highest group value. For example, because the highlight marker and handwritten comment are attached to the same sentence in the text, these two marks are correctly recognized as multimarks, as indicated by the large rectangle that clusters the marks. As an example shown in FIG. 4, there is a sufficient opportunity to recognize incorrectly in the parsing process. Stroke clustering, attached information, and grouping clauses with clusters are heuristic. Nevertheless, the overall result is consistent with the desired outcome. Since low-level structural analysis involves inaccurate subgrouping, it is important to design the use of recognized structures that do not depend on exact clustering results. Recognizing a group of marks, classifying the type of mark, and attaching importance values to the group provides the possibility of various interface level functions. Four examples of such functions are shown below.
(1) A presentation that draws reader interest to the location of high-value annotations in the thumbnail display of a document (2) A presentation that combines annotations to help stakeholders share their annotations more effectively (3) Annotations Or a mechanism to select meaningful groupings of associated text (4) a mechanism to help interested parties search via annotations according to a specific pattern

  FIG. 5 shows a first example of a multi-thumbnail display of a document 500 generated according to the system and method of the present invention. In FIG. 5, high value phrases are highlighted by icons. The thumbnail image shows an asterisk icon 510 to indicate a high value annotation associated with the first portion of the document 500. An asterisk icon 520 indicates a second high value annotation associated with the second portion of the document 500. An asterisk icon 530 indicates a third high value annotation associated with the third portion of the document 500. It will be apparent that each icon can be supplemented with known or later developed icons or display attributes for displaying high value without departing from the spirit or scope of the present invention.

  FIG. 6 shows a second example of a multi-thumbnail display of a document 500 generated according to the method and system of the present invention. In FIG. 6, the high-value clause or part of the document 500 is highlighted again by an icon. For example, the asterisk icon 510 displays a first high value annotation associated with a first portion or phrase of the document 500. The asterisk icon 520 displays a second high value annotation associated with the second portion or phrase of the document 500. The asterisk 530 displays a third high-value annotation that is associated with a third portion or phrase of the document 500. The call (call up) 540 displays the third high value annotation in a magnified view while maintaining the orientation of the document 500 by the reader.

  FIG. 7 illustrates one exemplary embodiment of a system 100 that includes a free-form annotation enhancement system 200. As shown in FIG. 7, one or more user input devices 300 are connected to the free form annotation enhancement system 200 via one or more links 310. One or more user input devices 300 are known or later developed that can be used to enter data and / or commands, such as, for example, a keyboard, mouse, voice recognition system, digital ink pen, trackball, trackpad, etc. Device. The free-form annotation enhancement system 200 is connected to the data source 120 of the system 100 either directly via the link 320 or via the links 130, 150 and the communication network 140 of the system 100. The links 130, 150, 310, 320 and the communication network 140 may be any known or later developed device that can be used to connect the user input device 300 and the data source 120 to the free form annotation enhancement system 200. , Direct line connections, connections on wide area networks or local networks, connections on intranets or connections on extranets, connections on the Internet, or connections on other distributed processing networks or systems. In general, the links 130, 150, 310, 320 and the communication network 140 may be any suitable well-known or later-developed connection system or structure.

  Data source 120 includes electronic document 110. Media data such as electronic document 110 can be retrieved from data source 120 by free-form annotation enhancement system 200 and shared by components of free-form annotation enhancement system 200. Data source 120 can be a digital camera, scanner, locally or remotely located computer, or any other well-known or later developed system that can be used to generate electronic data or document information, such as annotation information. It should be understood. Similarly, data source 120 may be any suitable device that stores and / or transmits electronic image data, such as a network client or server. The data source 120 can be integrated into the free form annotation enhancement system 200 or can be remotely connected to the free form annotation enhancement system 200 over a communication network 140, for example.

  The electronic document 110 provided by the data source 120 is an actual document or a scanned image of a photograph, a video taken, media data generated electronically using software such as word processing software, or well-known or later Media data generated using a developed programming language and / or computer software program, application window content on the user's desktop (eg, toolbar, Windows® decoration, spreadsheet shown in spreadsheet program) Etc.), it should also be understood that it may be a live broadcast of a video image, or any other well-known or later-developed electronic document. Further, the electronic document may be an analog image and / or an analog object, such as a photograph, a drawing, and an object using digital ink.

  FIG. 8 shows a block diagram of an exemplary embodiment of a free form enhancement system 200 according to the present invention. As shown in FIG. 8, one or more user input devices 300 are connected to the free-form annotation enhancement system 200 through the input / output interface 210 via one or more communication links 310. Data source 120 is also connected to free-form annotation display system 200 through link 320 through input / output interface 210.

  The freeform annotation enhancement system 200 includes an input / output interface 210, a controller 220, a memory 230, an importance determination circuit or routine 240, an annotation modification circuit or routine 250, and a freeform annotation detection circuit or routine 260, all of which are data. And / or interconnected on a control bus 280. The display device 330 is also connected to the input / output interface 210 via the link 340.

  The controller 220 controls the operation of other components of the freeform enhancement system 200. The controller 220 controls the flow of data between the other components of the free form annotation enhancement system 200 as needed. The memory 230 can store information that enters or exits the free form annotation enhancement system 200, can also store necessary programs and / or data that implement the functions of the free form annotation enhancement system 200, and Data and / or annotation information can also be stored at various stages of processing.

  The memory 230 can be implemented using an appropriate combination of changeable, volatile or non-volatile memory, non-changeable fixed memory, and the like. Changeable memory (either volatile or non-volatile) can be static or dynamic RAM, floppy (R) disk and disk drive, writable or rewritable optical disk and disk drive, hard drive, flash memory, etc. It can be implemented using any one or more of these. Similarly, the non-changeable memory or the fixed memory can be mounted using any one or more of ROM such as CD-ROM and DVD-ROM disk, PROM, EPROM, EEPROM, optical ROM disk, and disk drive.

  A free form annotation detection circuit or routine 260 is invoked by the controller 220 to detect and identify annotations included in the electronic document 110 provided by the data source 120, as outlined above. The free-form annotation detection circuit or routine 260 may be developed as a prior art system and method (see, for example, US Pat. No. 6,057,049) or other well-known or later developed to detect annotations contained within a document. Can be executed using the system. It should be understood that the free-form annotation detection circuit or routine 260 can optionally be omitted from the free-form annotation enhancement system 200 if the data source 120 provides the electronic document 110 for annotation information. It should also be understood that the free form annotation detection circuit or routine 260 operates equally well with all types of electronic information. For example, if the data source 120 is a scanner, the free-form annotation detection circuit or routine 260 may detect annotations from within the scanned document, as outlined above.

  The free form annotation detection circuit or routine 260 is used, for example, to create a digital annotation time, the physical size of the annotation, the color or other visual attributes of the annotation, and / or the digital ink device or annotation. Annotation information, such as the pen type of a physical device, can be collected and / or determined.

  The importance determination circuit or routine 240 receives the electronic document 110 from either the memory 230 or the input / output interface 210 under the control of the controller 220. The importance determination circuit or routine 240 retrieves the annotation information determined by the free form annotation detection circuit or routine 260 directly from the free form annotation detection circuit or routine 260 or from the memory 230. The importance determination circuit or routine 240 groups the annotations into a first group of marks, which is when the annotation is made, the physical size of the annotation, the color or other visual attributes, and / or when annotating There is a pen type etc. used for. Using such information, the importance determination circuit or routine 240 groups the marks into one or more high level groups as outlined above, and ranks and types the relative importance of the marks. To do. The importance determination circuit or routine 240 assigns weights to high level groups. One or more high-value annotations are determined based on the assigned weight values.

  The annotation modification circuit or routine 250 modifies these high value annotations and adds special icons, as outlined above. An annotation modification circuit or routine 250 is activated by the controller 220 to change the appearance of the annotation and / or to indicate the importance of the annotation as determined and provided by the importance determination circuit or routine 240. Add special icons to be displayed. An annotation modification circuit or routine 250 modifies one or more features of the system and method shown in US patent application Ser. No. 09 / 942,666, which is hereby incorporated by reference in its entirety. And / or other well-known or later-developed methods for adding special icons etc. to the corresponding reduced display.

  FIG. 9 details an exemplary embodiment of the importance determination circuit or routine 240. As shown in FIG. 9, the importance determining circuit or routine 240 includes a mark parser 241, a mark clustering circuit or routine 242, and a frequency counter 243.

  In the exemplary embodiment shown in FIG. 9, frequency counter 243 determines the frequency of occurrence of each particular type of one or more annotations included within electronic document 110 provided by data source 120. In order to do this, the annotation information provided by the free-form annotation detection circuit or routine 260 is analyzed. Mark parser 241 uses this frequency information to characterize the particular type of annotation by the reader. For example, the frequency counter 243 can determine the frequency of annotations in the electronic document 110, with 87% of the annotations highlighted, 10% of the annotations circled, 2% numbers, and the remaining 1% Is a handwritten note. Once the frequency counter 243 determines this relationship, two methods of determining high value annotations can be used by the mark parser 241 as outlined above.

  In the first operation mode, among the electronic documents, the handwriting type digital annotation has the lowest frequency of appearance, so the mark parser 241 can determine that the handwritten memo has the highest value. . The mark parser 241 thus instructs the controller 220 that the handwritten memo is a high-value annotation. An annotation modification circuit or routine 250 modifies these identified high value annotations. For example, handwritten notes are shaded in red.

  However, in the second more complex mode of operation, the importance determination circuit or routine 240 detects which is valuable after the frequency counter 243 determines the frequency information that displays the reader's annotation style. Analyze annotation characteristics. For this purpose, the mark parser 241 inputs the frequency information from the frequency counter 243. Mark parser 241 identifies the structure of the group of annotations and the importance as indicated, for example, by the user's usage frequency. The mark parser 241 characterizes a group of annotations by annotation type and builds a document-based annotation grouping to identify high value sections of the document.

  The mark parser 241 uses the three-pass process outlined above. Cluster circuit or routine 242 highlights mark parser 241 in the first pass of mark parser 241 to generate and group one-level or multi-level hierarchical clustering of annotations, as described above. Operate.

  In the second pass, the mark parser 241 classifies the mark into an annotation type using, for example, but not limited to, pen type, number of marks, and / or cluster size and / or shape.

  In the third pass, mark parser 241 groups mark clusters together with clauses from the target document. Due to the overlap between mark cluster clauses, these mark clusters will be grouped into multi-mark clusters. Multimark clusters are assigned the highest value of the group.

  In this exemplary embodiment, when the mark parser 241 assigns importance values to the annotations in the target document, the importance determination circuit or routine 240 may issue a command to display high-value annotations to the user. Transmit to the controller 220. Table 1 shows examples of importance values attached to annotation types.

  FIG. 10 shows a flowchart of one exemplary embodiment of a method for highlighting free-form annotations according to the present invention. Operation starts at step S100 and moves to step S200, where an annotated document is entered or selected for analysis and enhancement of high value annotation. In step S300, annotations in the document are identified as outlined above. Next, in step S400, the reader's annotation style is determined. In various exemplary embodiments, annotation information is parsed to determine, for example, the frequency of occurrence of a particular mark. Operation further continues to step S500.

  In step S500, the determined annotations are classified as outlined above. Next, in step S600, the classified annotations are analyzed to determine overlapping or adjacent annotations. In step S700, the analyzed annotations are clustered together based on the overlapping state of the annotations. Operation further continues to S800.

  In step S800, the annotation is assigned a value based on various criteria such as cluster overlap, cluster spatial size, and time between pen strokes. In step S900, a high-value annotation based on the assigned value is displayed to the user. Operation of the method ends at step S1000.

  It should be understood that these steps are shown for illustrative purposes only, and that various other exemplary embodiments can be used to determine high-value annotations for display using many other methods.

  In the various exemplary embodiments outlined above, the freeform annotation enhancement system 200 can be implemented using a programmed general purpose computer. However, the free form annotation enhancement system 200 is also capable of hardware electronics such as special purpose computers, programmed microprocessors or microcontrollers and peripheral integrated circuit elements, and ASICs or other integrated circuits, digital signal processors, discrete element circuits, or the like. It can be executed using a logic circuit, a programmable logic element such as PLD, PLA, FPGA, PAL or the like. In general, any device capable of executing a finite state machine capable of implementing the flowchart shown in FIG. 10 can be used to implement the free form annotation enhancement system 200.

  The circuits and elements of the various exemplary embodiments of the free form annotation enhancement system 200 outlined above can each be implemented as part of a suitable programmed general purpose computer. Alternatively, the circuits and elements of the various embodiments of the freeform annotation enhancement system 200 outlined above are each as a physically identifiable hardware circuit in an ASIC, or as an FPGA, PDL, PLA Alternatively, it can be executed using PAL or the like, or using individual logic elements or individual circuit elements. The particular shape employed by each of the circuits and elements of the various embodiments of the freeform annotation enhancement system 200 outlined above has design choices that will be clear and foreseeable to those skilled in the art. is there.

  Further, the various exemplary embodiments of the freeform annotation enhancement system 200 outlined above and / or the various circuits and elements described above each execute on a software routine, programmed general purpose computer, respectively. Can be executed as manager or object, special purpose computer, microprocessor, etc. In this case, the various exemplary embodiments of the freeform annotation enhancement system 200 and / or the various circuits and elements described above each include one or more routines incorporated into the communication network, resources residing on the server, and the like. Can be run as Each of the various exemplary embodiments of the free form annotation enhancement system 200 and the various circuits and elements described above can be converted into a free form annotation enhancement system 200, software, such as a web server or client device hardware and software system, respectively. It can also be implemented by being physically incorporated into the system and / or hardware system.

  One of ordinary skill in the art will recognize many applications for the system and method according to the present invention, including but not limited to display devices such as file browser devices, systems for displaying personal computer applications, handheld devices, and the like. In summary, the present invention is compatible with known or later developed systems and devices that can use digital ink or high value annotations.

  Although the present invention has been described in connection with the exemplary embodiments outlined above, it will be apparent to those skilled in the art that many alternatives, modifications, and variations will be apparent to those skilled in the art. . Accordingly, as indicated above, the exemplary embodiments of the present invention are for purposes of illustration and are not limited. Various changes can be made without departing from the spirit and scope of the invention.

It is a figure which shows the specific example of the typical multi thumbnail display of the long document with the annotation. FIG. 6 is a diagram illustrating various examples of annotation styles used by different readers. It is a figure which shows a part of document as a specific example containing an annotation. FIG. 6 illustrates an example of clustered annotations according to various example embodiments of the system and method of the present invention. FIG. 2 illustrates a first example embodiment of a multi-thumbnail display with high value annotations that has been determined and highlighted, according to various example embodiments of the system and method of the present invention. FIG. 6 illustrates a second example embodiment that displays in detail an annotated portion corresponding to a selected high-value annotation, according to various example embodiments of the systems and methods of the present invention. It is. FIG. 2 illustrates an exemplary embodiment of a system that can be used to highlight free-form annotations according to the present invention. 1 is a block diagram illustrating an exemplary embodiment of a free form enhancement system according to the present invention. FIG. FIG. 9 is a block diagram illustrating an example embodiment of an importance determination system using the example free form annotation enhancement system shown in FIG. 8. FIG. 5 is a flowchart schematically illustrating an exemplary embodiment of a method for highlighting free-form annotations.

Claims (13)

Document data indicating the annotated document is received by the receiving means in order to alert the reader.
The document indicated by the received document data is analyzed by the analysis unit to identify the type of annotation attached to the document, and the determination unit is configured based on a predetermined criterion corresponding to the identified type of the annotation. Determine the importance of each annotation,
When the display of the annotation in a state where at least a part of the document indicated by the document data is displayed on the display means is reduced, the display of the annotation in a state where the document is not reduced is reduced according to the reduction. A method for emphasizing a free-form annotation that is controlled so that the higher the importance of the annotation is, the higher the importance of the annotation is displayed .   The predetermined criterion is such that the lower the appearance frequency for each annotation type, the higher the importance,
  The document indicated by the received document data is analyzed by an analysis means to determine the appearance frequency for each type of annotation, and the smaller the appearance frequency, the higher the importance and the importance of each annotation is determined.
  The method of claim 1.   For the display means, the appearance of the annotation is changed and displayed so that the higher the determined importance is, the more visually emphasized it is displayed.
  The method according to claim 1 or claim 2. Changing the appearance includes changing at least one of font size, font color, underline, highlighting, and iconification.
The method of claim 3.   Determining the importance includes grouping annotated parts or clauses, ranking and typing the relative importance of a plurality of groups;
  The method according to any one of claims 1 to 4.   Parts or clauses with multiple annotations are classified as multimarks
  The method of claim 5. Receiving means for receiving document data indicating a document annotated to alert the reader;
  The document indicated by the received document data is analyzed by an analysis means to identify the type of annotation attached to the document, and each annotation is determined based on a predetermined criterion corresponding to the identified type of annotation. A decision means to determine the importance,
  The display of the annotation in a state where at least a part of the document indicated by the document data is displayed in a reduced size is a display when the display of the annotation in a state in which the document is not reduced is reduced according to the reduction. In comparison, control means for controlling the annotation so that the higher the importance of the annotation, the more visually emphasized it is displayed.
  An annotation highlighting device comprising:   The predetermined criterion is such that the lower the appearance frequency for each annotation type, the higher the importance,
  The determination means analyzes the document indicated by the received document data by the analysis means to obtain an appearance frequency for each annotation type, and the smaller the appearance frequency, the higher the importance and the importance of each annotation Determine the degree
  The annotation emphasis display device according to claim 7. The control means controls the display means so that the higher the determined importance, the more visually changed the appearance of the annotation is displayed.
  The annotation emphasis display device according to claim 7 or 8. Changing the appearance includes changing at least one of font size, font color, underline, highlighting, and iconification.
  The annotation emphasis display device according to claim 9.   Determining the importance by the determining means includes grouping annotated parts or clauses, ranking and typing the relative importance of a plurality of groups;
  The annotation emphasis display device according to any one of claims 7 to 10.   Parts or clauses with multiple annotations are classified as multimarks
  The annotation emphasis display device according to claim 11. Computer
Receiving means for receiving document data indicating a document annotated to alert the reader;
The document indicated by the received document data is analyzed by an analysis means to identify the type of annotation attached to the document, and each annotation is determined based on a predetermined criterion corresponding to the identified type of annotation. A decision means to determine the importance,
The display of the annotation in a state where at least a part of the document indicated by the document data is displayed in a reduced size is a display when the display of the annotation in a state in which the document is not reduced is reduced according to the reduction. In comparison, control means for controlling the annotation so that the higher the importance of the annotation, the more visually emphasized it is displayed.
A program for highlighting free-form annotations to function as

Images