[go: up one dir, main page]

EP1944756A3 - Method and system for determining similarity between two documents - Google Patents

Method and system for determining similarity between two documents Download PDF

Info

Publication number
EP1944756A3
EP1944756A3 EP08103135A EP08103135A EP1944756A3 EP 1944756 A3 EP1944756 A3 EP 1944756A3 EP 08103135 A EP08103135 A EP 08103135A EP 08103135 A EP08103135 A EP 08103135A EP 1944756 A3 EP1944756 A3 EP 1944756A3
Authority
EP
European Patent Office
Prior art keywords
document
input
documents
input documents
determining similarity
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP08103135A
Other languages
German (de)
French (fr)
Other versions
EP1944756A2 (en
Inventor
Takahiko Kawatani
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
HP Inc
Original Assignee
Hewlett Packard Co
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hewlett Packard Co filed Critical Hewlett Packard Co
Publication of EP1944756A2 publication Critical patent/EP1944756A2/en
Publication of EP1944756A3 publication Critical patent/EP1944756A3/en
Withdrawn legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/3331—Query processing
    • G06F16/334—Query execution
    • G06F16/3347—Query execution using vector based model
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00—Data processing: database and file management or data structures
    • Y10S707/99931—Database or file accessing
    • Y10S707/99932—Access augmentation or optimizing
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00—Data processing: database and file management or data structures
    • Y10S707/99931—Database or file accessing
    • Y10S707/99933—Query processing, i.e. searching

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Image Analysis (AREA)

Abstract

A method and system for determining similarity between given two input documents is described. Terms that occur in each of the input documents are detected. Each of input document is segmented into document segments, each segment being an appropriately sized chunk. Document segment vectors are generated, each vector including as its element values according to occurrence frequencies of said terms occurring in said respective document segments. For each of the two input documents, a squared inner product for all combinations of said document segment vectors contained in each input document is calculated. Similarity is then determined between the two input documents based on a sum of said squared inner products.
EP08103135A 2000-11-20 2001-11-15 Method and system for determining similarity between two documents Withdrawn EP1944756A3 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2000353475A JP2002169834A (en) 2000-11-20 2000-11-20 Computer and method for making vector analysis of document
EP01996067A EP1336174A4 (en) 2000-11-20 2001-11-15 Method of vector analysis for a document

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
EP01996067A Division EP1336174A4 (en) 2000-11-20 2001-11-15 Method of vector analysis for a document

Publications (2)

Publication Number Publication Date
EP1944756A2 EP1944756A2 (en) 2008-07-16
EP1944756A3 true EP1944756A3 (en) 2008-07-23

Family

ID=18826235

Family Applications (3)

Application Number Title Priority Date Filing Date
EP08103139A Withdrawn EP1944757A3 (en) 2000-11-20 2001-11-15 Method and system for identifying document segment relationships
EP01996067A Withdrawn EP1336174A4 (en) 2000-11-20 2001-11-15 Method of vector analysis for a document
EP08103135A Withdrawn EP1944756A3 (en) 2000-11-20 2001-11-15 Method and system for determining similarity between two documents

Family Applications Before (2)

Application Number Title Priority Date Filing Date
EP08103139A Withdrawn EP1944757A3 (en) 2000-11-20 2001-11-15 Method and system for identifying document segment relationships
EP01996067A Withdrawn EP1336174A4 (en) 2000-11-20 2001-11-15 Method of vector analysis for a document

Country Status (5)

Country Link
US (2) US7562066B2 (en)
EP (3) EP1944757A3 (en)
JP (1) JP2002169834A (en)
CN (1) CN1227614C (en)
WO (1) WO2002041557A2 (en)

Families Citing this family (57)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8241274B2 (en) 2000-01-19 2012-08-14 Medtronic, Inc. Method for guiding a medical device
US7308138B2 (en) * 2000-12-12 2007-12-11 Hewlett-Packard Development Company, L.P. Document segmentation method
US7194461B2 (en) * 2001-03-02 2007-03-20 Hewlett-Packard Development Company, L.P. Document and information retrieval method and apparatus
AU2002326118A1 (en) 2001-08-14 2003-03-03 Quigo Technologies, Inc. System and method for extracting content for submission to a search engine
JP4452012B2 (en) * 2002-07-04 2010-04-21 ヒューレット・パッカード・カンパニー Document uniqueness evaluation method
DE60335472D1 (en) 2002-07-23 2011-02-03 Quigo Technologies Inc SYSTEM AND METHOD FOR AUTOMATED IMAGING OF KEYWORDS AND KEYPHRASES ON DOCUMENTS
JP2004280661A (en) * 2003-03-18 2004-10-07 Fujitsu Ltd Search method and program
JP2005158010A (en) * 2003-10-31 2005-06-16 Hewlett-Packard Development Co Lp Apparatus, method and program for classification evaluation
US7827025B2 (en) * 2004-04-06 2010-11-02 Microsoft Corporation Efficient capitalization through user modeling
US7567959B2 (en) 2004-07-26 2009-07-28 Google Inc. Multiple index based information retrieval system
US7580921B2 (en) * 2004-07-26 2009-08-25 Google Inc. Phrase identification in an information retrieval system
US7426507B1 (en) 2004-07-26 2008-09-16 Google, Inc. Automatic taxonomy generation in search results using phrases
US7580929B2 (en) * 2004-07-26 2009-08-25 Google Inc. Phrase-based personalization of searches in an information retrieval system
US7584175B2 (en) * 2004-07-26 2009-09-01 Google Inc. Phrase-based generation of document descriptions
US7711679B2 (en) 2004-07-26 2010-05-04 Google Inc. Phrase-based detection of duplicate documents in an information retrieval system
US7702618B1 (en) 2004-07-26 2010-04-20 Google Inc. Information retrieval system for archiving multiple document versions
US7536408B2 (en) * 2004-07-26 2009-05-19 Google Inc. Phrase-based indexing in an information retrieval system
US7599914B2 (en) * 2004-07-26 2009-10-06 Google Inc. Phrase-based searching in an information retrieval system
JP2006091994A (en) * 2004-09-21 2006-04-06 Toshiba Corp Device, method and program for processing document information
US7533094B2 (en) * 2004-11-23 2009-05-12 Microsoft Corporation Method and system for determining similarity of items based on similarity objects and their features
US8290962B1 (en) * 2005-09-28 2012-10-16 Google Inc. Determining the relationship between source code bases
DE102005051617B4 (en) * 2005-10-27 2009-10-15 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Automatic, computer-based similarity calculation system for quantifying the similarity of textual expressions
US8166021B1 (en) 2007-03-30 2012-04-24 Google Inc. Query phrasification
US7925655B1 (en) 2007-03-30 2011-04-12 Google Inc. Query scheduling using hierarchical tiers of index servers
US8166045B1 (en) 2007-03-30 2012-04-24 Google Inc. Phrase extraction using subphrase scoring
US7702614B1 (en) 2007-03-30 2010-04-20 Google Inc. Index updating using segment swapping
US7693813B1 (en) 2007-03-30 2010-04-06 Google Inc. Index server architecture using tiered and sharded phrase posting lists
US8086594B1 (en) 2007-03-30 2011-12-27 Google Inc. Bifurcated document relevance scoring
US8117223B2 (en) * 2007-09-07 2012-02-14 Google Inc. Integrating external related phrase information into a phrase-based indexing information retrieval system
US8577884B2 (en) * 2008-05-13 2013-11-05 The Boeing Company Automated analysis and summarization of comments in survey response data
EP2128774A1 (en) * 2008-05-29 2009-12-02 Accenture Global Services GmbH Techniques for computing similarity measurements between segments representative of documents
JP2010122823A (en) * 2008-11-18 2010-06-03 Nec Corp Text processing system, information processing apparatus, method for processing text and information, and processing program
EP2306339A1 (en) * 2009-09-23 2011-04-06 Adobe Systems Incorporated Algorith and implementation for fast computation of content recommendation
WO2011115024A1 (en) * 2010-03-15 2011-09-22 日本電気株式会社 Information processing device, information processing method, and information processing program
WO2012000185A1 (en) * 2010-06-30 2012-01-05 Hewlett-Packard Development Company,L.P. Method and system of determining similarity between elements of electronic document
CN102385574B (en) * 2010-09-01 2014-08-20 株式会社理光 Method and device for extracting sentences from document
EP2715474A4 (en) * 2011-05-24 2015-11-18 Namesforlife Llc Semiotic indexing of digital resources
US8566156B2 (en) * 2011-07-05 2013-10-22 Yahoo! Inc. Combining segments of users into vertically indexed super-segments
CN102360372B (en) * 2011-10-09 2013-01-30 北京航空航天大学 A Cross-lingual Document Similarity Detection Method
EP2870543A4 (en) * 2012-10-12 2016-04-06 Hewlett Packard Development Co A combinatorial summarizer
US9501506B1 (en) 2013-03-15 2016-11-22 Google Inc. Indexing system
CN104216894B (en) 2013-05-31 2017-07-14 国际商业机器公司 Method and system for data query
US9483568B1 (en) 2013-06-05 2016-11-01 Google Inc. Indexing system
US12147459B2 (en) * 2014-08-07 2024-11-19 Cortical.Io Ag Methods and systems for mapping data items to sparse distributed representations
CN105630767B (en) * 2015-12-22 2018-06-15 北京奇虎科技有限公司 The comparative approach and device of a kind of text similarity
KR102703923B1 (en) * 2016-12-29 2024-09-06 주식회사 엔씨소프트 Apparatus and method for learning narrative of document, apparatus and method for generating narrative of document
JP6842167B2 (en) * 2017-05-08 2021-03-17 国立研究開発法人情報通信研究機構 Summary generator, summary generation method and computer program
CN107239552A (en) * 2017-06-08 2017-10-10 上海德衡数据科技有限公司 A kind of network documentation semantic retrieving method for keyword
CN107330052A (en) * 2017-06-29 2017-11-07 上海德衡数据科技有限公司 A kind of network documentation intelligent retrieval prototype based on semanteme
CN107330044A (en) * 2017-06-29 2017-11-07 上海德衡数据科技有限公司 A kind of network documentation intelligent retrieval framework based on semanteme
US10891943B2 (en) * 2018-01-18 2021-01-12 Citrix Systems, Inc. Intelligent short text information retrieve based on deep learning
CN110147533B (en) * 2019-01-24 2023-08-29 腾讯科技（深圳）有限公司 Encoding method, apparatus, device and storage medium
CN110147532B (en) * 2019-01-24 2023-08-25 腾讯科技（深圳）有限公司 Encoding method, apparatus, device and storage medium
US11829386B2 (en) 2020-01-30 2023-11-28 HG Insights, Inc. Identifying anonymized resume corpus data pertaining to the same individual
US11734332B2 (en) 2020-11-19 2023-08-22 Cortical.Io Ag Methods and systems for reuse of data item fingerprints in generation of semantic maps
US20240370407A1 (en) * 2023-05-02 2024-11-07 Crowdstrike, Inc. Techniques for detecting file similarity
CN117112609B (en) * 2023-06-29 2024-05-10 南京国电南自轨道交通工程有限公司 Method for improving retrieval efficiency of monitoring historical data by using key element matrix

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5930784A (en) * 1997-08-21 1999-07-27 Sandia Corporation Method of locating related items in a geometric space for data mining

Family Cites Families (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3668702A (en) * 1970-10-30 1972-06-06 Itt Adaptive matched filter for radar signal detector in the presence of colored noise
US5301109A (en) * 1990-06-11 1994-04-05 Bell Communications Research, Inc. Computerized cross-language document retrieval using latent semantic indexing
US5634021A (en) 1991-08-15 1997-05-27 Borland International, Inc. System and methods for generation of design images based on user design inputs
US5983251A (en) * 1993-09-08 1999-11-09 Idt, Inc. Method and apparatus for data analysis
US5619709A (en) * 1993-09-20 1997-04-08 Hnc, Inc. System and method of context vector generation and retrieval
US6038533A (en) * 1995-07-07 2000-03-14 Lucent Technologies Inc. System and method for selecting training text
US5901244A (en) 1996-06-18 1999-05-04 Matsushita Electric Industrial Co., Ltd. Feature extraction system and face image recognition system
US5857179A (en) * 1996-09-09 1999-01-05 Digital Equipment Corporation Computer method and apparatus for clustering documents and automatic generation of cluster keywords
US5819258A (en) 1997-03-07 1998-10-06 Digital Equipment Corporation Method and apparatus for automatically generating hierarchical categories from large document collections
US6070133A (en) * 1997-07-21 2000-05-30 Battelle Memorial Institute Information retrieval system utilizing wavelet transform
US6356864B1 (en) * 1997-07-25 2002-03-12 University Technology Corporation Methods for analysis and evaluation of the semantic content of a writing based on vector length
JP4074687B2 (en) * 1997-07-28 2008-04-09 株式会社ジャストシステム Summary sentence creation support system and computer-readable recording medium recording a program for causing a computer to function as the system
US6996572B1 (en) * 1997-10-08 2006-02-07 International Business Machines Corporation Method and system for filtering of information entities
US6134541A (en) * 1997-10-31 2000-10-17 International Business Machines Corporation Searching multidimensional indexes using associated clustering and dimension reduction information
US6347315B1 (en) * 1997-12-12 2002-02-12 Canon Kabushiki Kaisha Method and apparatus for selecting and utilizing one of computers or databases

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5930784A (en) * 1997-08-21 1999-07-27 Sandia Corporation Method of locating related items in a geometric space for data mining

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
CAID W R ET AL: "Learned Vector-Space Models for Document Retrieval", INFORMATION PROCESSING & MANAGEMENT, ELSEVIER, BARKING, GB, vol. 31, no. 3, 1 May 1995 (1995-05-01), pages 419 - 429, XP004011612, ISSN: 0306-4573 *
JEROME R BELLEGARDA: "A Multispan Language Modeling Framework for Large Vocabulary Speech Recognition", IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, IEEE SERVICE CENTER, NEW YORK, NY, US, vol. 6, no. 5, 1 September 1998 (1998-09-01), XP011054322, ISSN: 1063-6676 *
NEWBY G B: "Context-based statistical sub-spaces", SIXTH TEXT RETRIEVAL CONFERENCE, 19 November 1997 (1997-11-19) - 21 November 1997 (1997-11-21), Gaithersburg, USA, pages 735 - 745, XP002482620 *
SALTON G ET AL: "TERM-WEIGHTING APPROACHES IN AUTOMATIC TEXT RETRIEVAL", INFORMATION PROCESSING & MANAGEMENT, ELSEVIER, BARKING, GB, vol. 24, no. 5, 1988, pages 513 - 523, XP000716208, ISSN: 0306-4573 *

Also Published As

Publication number Publication date
EP1944757A2 (en) 2008-07-16
CN1470047A (en) 2004-01-21
EP1336174A4 (en) 2007-05-30
CN1227614C (en) 2005-11-16
US8171026B2 (en) 2012-05-01
EP1944756A2 (en) 2008-07-16
EP1944757A3 (en) 2008-07-23
EP1336174A2 (en) 2003-08-20
JP2002169834A (en) 2002-06-14
WO2002041557A2 (en) 2002-05-23
US20040068396A1 (en) 2004-04-08
WO2002041557A3 (en) 2003-01-30
US20090216759A1 (en) 2009-08-27
US7562066B2 (en) 2009-07-14

Similar Documents

Publication Publication Date Title
EP1944756A3 (en) Method and system for determining similarity between two documents
EP1335607A3 (en) Image smoothing apparatus and method
US6198850B1 (en) System and method for segmentation dependent lossy and lossless compression for higher quality
EP1494143A3 (en) Method and system for representing information items in a pseudo-image
EP0933727A3 (en) Image information processing apparatus and its method
EP0982949A3 (en) Image processing apparatus and method
EP0822710A3 (en) Improved method of combining two digital images
CA2345218A1 (en) A method for compressing digital documents with control of image quality and compression rate
EP1455284A3 (en) Image processing method and image processing system
MXPA01007847A (en) Texture description method and texturebased image retrieval method using gabor filter in frequency domain.
EP1233349A3 (en) Data display method and apparatus for use in text mining
EP0932108A3 (en) Maintaining document state history
HUP9801780A2 (en) Image preprocessing process and device for motion estimation
EP0851659A3 (en) Information processing system and method therefor
EP0961488A3 (en) Gamut mapping method and apparatus
EP1528486A3 (en) Classification evaluation system, method, and program
EP0833515A3 (en) Device for and method of coding/decoding image information
EP1130547A3 (en) Image creation device
EP1072982A3 (en) Method and system for similar word extraction and document retrieval
EP0849940A3 (en) An apparatus for converting gray levels of an image, a method thereof, a program storage device thereof, and an infrared camera
EP1306798A3 (en) Banded compositor for variable data
EP1081648A3 (en) Method for processing a digital image
EP1045317A3 (en) Data processing method and apparatus
WO2002060053A3 (en) Methods, devices and computer program products for compressing an audio signal
EP2267715A3 (en) Signal processing device and signal processing method

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

PUAL Search report despatched

Free format text: ORIGINAL CODE: 0009013

AC Divisional application: reference to earlier application

Ref document number: 1336174

Country of ref document: EP

Kind code of ref document: P

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): DE FR GB

AK Designated contracting states

Kind code of ref document: A3

Designated state(s): DE FR GB

17P Request for examination filed

Effective date: 20090123

17Q First examination report despatched

Effective date: 20090227

AKX Designation fees paid

Designated state(s): DE GB

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20110601