KR100933270B1 - Method, system and computer-readable recording medium for performing web search based on image information - Google Patents

Abstract

The present invention relates to a method, a system for performing a web search using an image created and input by a user, and a computer readable recording medium for recording a computer program for executing the method. According to an aspect of the present invention, a method for providing a web search service by converting an image input by a user into a markup language, the method comprising: recognizing an image, at least one equation, a symbol corresponding to the recognized image, Extracting a character, converting the at least one extracted expression, symbol and / or character into a markup language, and performing a web search for the expression, symbol and / or character converted to the markup language. A method is provided that includes performing a step.

Image input interface, formula, chemical formula, sign, character

Description

TECHNICAL, SYSTEM AND COMPUTER-READABLE RECORDING MEDIUM FOR PERFORMING WEB SEARCH BASED ON IMAGE INFORMATION}

The present invention relates to a method, a system for performing a web search using an image created and input by a user, and a computer readable recording medium for recording a computer program for executing the method. More specifically, the present invention provides an image of a formula, chemical formula, symbol, and / or character input by an image input device such as a mouse, pen mouse, electronic pen, digitizer, touch pad, touch panel, tablet, or the like. Recognize and extract them as formulas, formulas, symbols and / or characters, and then convert the extracted formulas, formulas, symbols and / or characters into the form of markup language so that a web search based on them can be performed. And a computer readable recording medium for recording a computer program for executing the method.

In recent years, as the use of the Internet is becoming more common, various types of information can be obtained through the Internet. Companies providing web services through web sites provide various kinds of services to meet the needs of more and more diverse internet users, and the types of such services are also increasing day by day.

Users are exposed to the services provided by these companies in various forms, and they want to obtain various web contents such as news information, dictionary information, professional information, local information, and shopping information through web sites. In addition, users use a web search service to obtain a desired content among an infinite amount of content that is diverse.

Recently, the web search using the portal site has been actively performed, and users enter the key keywords related to the content they are looking for in the search box of the portal site to use the web search service. The keyword input to the search box is mainly made by the keyboard, and the keyboard is not inconvenient for inputting commonly used text such as Korean or English.

However, as the needs of users are diversified, the necessity for searching for contents including professional knowledge is increasing.However, due to the limitation of the text input by the keyboard, the user can change the formulas, chemical formulas, symbols (special symbols) and characters. It was almost impossible to perform a web search using keywords including (special characters) and the like. Moreover, the problem becomes more serious when the user uses the user terminal device which does not include a conventional keyboard as the aspect which utilizes a web service becomes various day by day.

Accordingly, there is a need for the development of a technology that enables a user to easily input keywords including formulas, formulas, symbols, characters, and the like, and enables the web search by such keywords to be efficiently performed.

An object of the present invention is to solve all the problems of the prior art as described above.

In addition, another object of the present invention is to enable a web search of content including formulas, formulas, symbols, characters, and the like.

In addition, another object of the present invention is to enable users to easily input keywords including formulas, chemical formulas, symbols, characters, etc. using an input device capable of inputting an image.

In addition, another object of the present invention is to simplify the user's operation for inputting equations, formulas, symbols, characters, etc., to increase the accuracy of keyword input, and also to save time spent searching for desired content. It is.

The characteristic constitution of the present invention for achieving the object of the present invention as described above and for achieving the specific effects of the present invention described below is as follows.

According to an aspect of the present invention, a method for providing a web search service by converting an image input by a user into a markup language, the method comprising: recognizing an image, at least one equation, a symbol corresponding to the recognized image, Extracting a character, converting the at least one extracted expression, symbol and / or character into a markup language, and performing a web search for the expression, symbol and / or character converted to the markup language. A method is provided that includes performing a step.

According to another aspect of the present invention, a method for providing a web search service by converting an image input by a user into a markup language, the method comprising: recognizing an image, at least one recommendation expression corresponding to the recognized image, recommendation Extracting a symbol and / or recommendation character, converting at least one expression, symbol and / or character selected by the user from the at least one extracted recommendation expression, recommendation symbol and / or recommendation character into a markup language , And performing a web search for expressions, symbols, and / or characters translated into the markup language.

According to yet another aspect of the present invention, a system for providing a web search service by converting an image input by a user into a markup language, wherein the system recognizes an image and at least one expression, symbol and / or corresponding to the recognized image. Or an image processing unit for extracting characters, a conversion unit for converting the at least one extracted expression, symbol and / or character into a markup language, and a web search for the expression, symbol and / or character converted into the markup language. A system is provided that includes a web search unit that provides a service.

According to another aspect of the present invention, a system for providing a web search service by converting an image input by a user into a markup language, the system recognizes an image and at least one recommendation expression and recommendation symbol corresponding to the recognized image. And / or an image processing unit for extracting a recommendation character, and converting at least one expression, symbol and / or character selected by the user among the at least one extracted recommendation expression, recommendation symbol, and / or recommendation character into a markup language. And a web search unit for providing a web search service for expressions, symbols, and / or characters converted into the markup language.

In addition, according to another aspect of the present invention, there are further provided other methods, a system for executing the method, and the like, and a computer readable recording medium.

The effects of the present invention achieved due to the characteristic constitution of the present invention are as follows.

1. Web search for contents including formulas, chemical formulas, symbols, characters, etc. can be efficiently performed.

2. The user can easily construct keywords including formulas, chemical formulas, symbols, characters, etc. using the image input device.

3. Keywords including formulas, formulas, symbols, characters, etc. can be entered more accurately.

4. Saves time spent by users searching the web.

DETAILED DESCRIPTION The following detailed description of the invention refers to the accompanying drawings that show, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein may be embodied in other embodiments without departing from the spirit and scope of the invention with respect to one embodiment. In addition, it is to be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present invention, if properly described, is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled. Like reference numerals in the drawings refer to the same or similar functions throughout the several aspects.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the present invention.

Configuration of the entire system

1 is a schematic diagram illustrating an overall configuration of a system for performing a web search based on an image input by a user according to an embodiment of the present invention.

As shown in FIG. 1, the system of the present invention may include a user terminal device 100, a web server 200, and an image processing server 300, and the user terminal device 100 and the web server 200. The image processing server 300 may communicate through various network communication networks 500.

Here, the network communication network 500 may be configured regardless of the communication mode such as wired and wireless, but may be a known World Wide Web (WWW).

The user terminal device 100 is a terminal device used by a user to use a web service provided by the web server 200, and is a terminal device that can access the web server 200 through the network communication network 500. A digital device that has a computing power by mounting a microprocessor such as a personal computer (for example, a desktop computer, a notebook computer, a tablet computer, a palmtop computer, a workstation), a workstation, a PDA, a web pad, a mobile phone, etc. Any number may be adopted as the user terminal device 100 according to the present invention.

The web server 200 is an operation server of a web site for providing news information, dictionary information, professional information, local information, shopping information, and the like to the user terminal device 100. Can be. The web server 200 may have an enhanced web search function so that a user can quickly obtain desired content.

The image processing server 300 recognizes an image input from the user terminal device 100 through a known image recognition algorithm, and extracts equations, chemical formulas, symbols, characters, etc. having a profile similar to the recognized image. In this regard, those skilled in the art to which the present invention belongs may refer to the character recognition technology disclosed in the Korean Information Science Society, Vol. 34, No. 1 (C), which is a well-known technical document. have.

Preferably, the image processing server 300 may recommend formulas, formulas, symbols, characters, etc. in order of high similarity with the image input by the user and provide them to the user terminal device 100. When such a recommendation is made, the user can construct a keyword for web search by selecting at least one of the recommended formula, formula, symbol, and text. The image processing server 300 may be implemented separately from the web server 200, but may be implemented as one or more program modules that perform the same function as necessary and included in the web server 200. In addition, if necessary, one or more program modules that perform at least some of the functions of the image processing server 300 may be included in the user terminal device 100 in the form of an application program.

The database 400 is connected to the image processing server 300 and, at the request of the image processing server 300, data such as profiles for a plurality of formulas, chemical formulas, symbols, characters, and the like, and web searchable markup languages corresponding thereto. To provide.

Take an example of the formula stored in the database 400. The root formula, one of the most famous formulas in human history, can be expressed in the following image.

However, this image is also a fact that it is quite difficult to search by entering text such as this, this formula not only difficult to target the main official web searches and b ² -4ac part of.

Therefore, the database 400 according to an embodiment of the present invention may include a description of the formula using MathML (Mathematical Markup Language), which is a kind of a known markup language. Specifically, this is as follows.

That is, the database 400 according to an embodiment of the present invention includes the image of the root formula as described above, the expression of the MathML form of the root formula, and preferably, the profile of the image of the root formula (for example, , a set of information such that the characters of x contains a formula such as the information, the image segments displayed on the top right image segment in the left most, the -4ac b ²⁾ may be included.

Configuration of User Terminal Device

Hereinafter, a configuration of the user terminal device 100 according to an exemplary embodiment of the present invention will be described with reference to FIG. 2.

As shown in FIG. 2, the user terminal device 100 includes an input unit 110, a display unit 130, and an image component 150.

According to a preferred embodiment of the present invention, the input unit 110 is a hardware device for receiving an image by a user, a mouse, a pen mouse, an electronic pen, a digitizer, a touch pad, a touch panel, a tablet, and the like. It may include an image input interface (an embodiment will be described later) that stores an image input using a hardware device in a buffer (not shown) and provides for processing of the user terminal device 100.

According to an embodiment of the present invention, the display unit 130 is a known display means such as a liquid crystal display (LCD), a cathode-ray tube (CRT), a light emitting diode (LED) monitor, an image input interface and the present invention. The apparatus for displaying a graphical user interface required for a web search according to. In particular, the graphical user interface for searching the web may be a web browser such as Internet Explorer ^TM, Netscape ^TM, Mozilla ^TM, Firefox ^TM. Meanwhile, the display unit 130 may include a touch screen. In this case, the display unit 130 may also perform a function of the input unit 110.

The image component 150 according to an embodiment of the present invention transmits the image input by the user through the input unit 110 to the web server 200. According to an embodiment of the present invention, the image component 150 receives information about a formula, a chemical formula, a text, and / or a symbol, which is recommended in response to an image input by a user, transmitted from the web server 200. This may be displayed through a graphic user interface displayed on the display unit 130 or an image input interface to be described later.

The image component 150 may additionally perform recognition / recommendation such as recognition of an image and similar equations described above to be performed by the image processing server 300. The image component 150 may be an application program module downloaded and installed from the web server 200 to the user terminal device 100.

The process of web search based on image information

<First Embodiment>

Hereinafter, FIG. 3 illustrates an image input interface according to an embodiment of the present invention and a web browser linked thereto, and FIG. 4 illustrates a web search process based on image information according to the first embodiment of the present invention. By reference, the present invention will be described further. For convenience of description, in FIG. 4, exemplary processes on the user terminal device 100 side and the server side are shown separately.

First, when a user accesses the web server 200 through the user terminal device 100 (S410), the web server 200 displays the image component 150 together with the webpage information to be displayed on the user terminal device 100. The user terminal device 100 transmits the data to the user terminal device 100 (S415).

The user terminal device 100 displays the image input interface 630 while displaying the web page 600 on the web browser 610 on the user terminal device 100 using the received webpage information (S420). (S425). The image input interface 630 may be activated as a user manipulates a predetermined button located on a menu bar or a toolbar of the web browser 610. Meanwhile, as illustrated, the image input interface 630 may include an image input window 631, a recommendation window 633, an Undo button, a Clear button, and a Close button. Can be.

The user may freely draw equations, symbols, characters, etc. in the image input window 631 using an input device such as a mouse, an electronic pen, a tablet, and the like (S430). Meanwhile, according to an embodiment of the present invention, instead of directly inputting an image, the user may perform image input by loading an image file including an equation, a symbol, a character, and the like.

When the user inputs an image (S430), the image component 150 generates image data with information including the coordinate information of the input image, an input order, click and release (for mouse use), and the like. When the input device is a tablet, additional information such as the strength of the pressure applied when the image is input may also be included in the image data.

Meanwhile, in the process of inputting an image, the user may use a cancel button, a delete button, and a close button. Among these, the cancel button is for a function for erasing a recently input stroke among the images input by the user in the image input window 631, and the delete button is for a function for erasing the entire image input by the user. Meanwhile, the close button inactivates and disappears from the image input interface 630.

와 같은 수식을 입력한다고 가정하면,

,

및

와 같이 기호나 문자가 입력되는 순서대로 그 개별 이미지가 웹 서버(200)에 전송될 수도 있고,

가 모두 입력된 후에 전체 이미지가 한 번에 전송될 수도 있다. 또한, 일부 이미지를 전송할 때에는, 입력 장치에 의한 입력에 있어 불연속적인 부분이 있는 경우, 예를 들면, 전자 펜을 이용한 이미지 작성 시에 전자 펜이 입력 패널에서 떨어질 때마다, 그때까지 작성된 이미지를 웹 서버(200)에 전송하는 방식을 취할 수도 있다(자명하게도, 이렇게 전송된 개개의 이미지에 대하여 후술하는 S440 내지 S455의 단계가 수행될 수도 있다). 한편, 입력된 이미지가 전체적으로 전송되는 경우에는, 사용자가 이미지 입력 인터페이스(630)를 통해 이미지 작성이 모두 끝났다는 신호를 입력함으로써, 전체 이미지가 웹 서버(200)로 전송될 수 있다(마찬가지로, 전송된 전체 이미지에 대하여 후술하는 S440 내지 S455의 단계가 수행될 수도 있다). 이를 위해, 이미지 입력 인터페이스(630)에는 입력 완료 버튼이 추가적으로 포함될 수 있다.Next, the image component 150 transmits the input image to the web server 200 (S435) (which may soon be sent back to the image processing server 300). Sending the image to the web server 200 may be performed for the entire image, but may not be. For example, if a user uses an input device

Suppose you enter a formula such as

,

And

The individual images may be transmitted to the web server 200 in the order in which the symbols or characters are input, as shown in FIG.

After all are input, the entire image may be transmitted at once. In addition, when transmitting some images, if there is a discontinuity in the input by the input device, for example, every time the electronic pen falls from the input panel when creating an image using the electronic pen, the image created up to that time is displayed on the web. A method of transmitting to the server 200 may be employed (obviously, the steps S440 to S455 described below may be performed with respect to the individual images thus transmitted). On the other hand, when the input image is transmitted in its entirety, the entire image may be transmitted to the web server 200 by the user inputting a signal indicating that the image creation is completed through the image input interface 630 (similarly, transmission Steps S440 to S455 described below may be performed on the entire image). To this end, the image input interface 630 may additionally include an input completion button.

In this way, when an image is transmitted to the web server 200 (S435), the web server 200 transmits it to the image processing server 300, and the image processing server 300 transmits the image transmitted through the image recognition algorithm. Recognize (S440).

Thereafter, the image processing server 300 extracts a formula, a symbol, a character, etc. corresponding to the recognized image as a recommendation formula, a recommendation symbol, a recommendation character, and the like, and through the web server 200, the user terminal device 100. In step S450). At this time, the recommended equation, symbol, character, etc. may be determined for an individual part of the input image, or may be determined for the entire image.

Meanwhile, the image processing server 300 may refer to the database 400 as described above when extracting the recommendation formula, recommendation symbol, recommendation character, or the like. As described above, the order of the recommended formulas, symbols, characters, etc. may be adjusted according to the similarity between the input image and the formulas, symbols, characters, etc. stored in the database 400.

Equations, symbols, characters, and the like recommended from the image processing server 300 to the user terminal device 100 are displayed on the recommendation window 633 of the image input interface 630 by the image component 150, and the user may At least one is selected (S455). Preferably, the recommendation window 633 may be divided into a plurality of regions, and the above-described formulas, symbols, characters, etc. may be displayed in the order of high similarity to low similarity.

In S455, the user may sequentially select recommended equations, symbols, characters, etc. for each individual part of the image input by the user, or may select recommended equations, etc. for the entire image.

When the user selects one or more equations, symbols, characters, etc. displayed on the recommendation window 633 and constructs a keyword by using the same, the keyword is transmitted to the web server 200 (S460). On the other hand, even in the case where the user selection is made to the individual image in S455 it is preferable that the configuration of the keyword is made in the user terminal device 100, but less preferably using a formula, symbols, characters, etc. selected in the web server 200 Keywords may be constructed.

Then, the received keyword is converted into a markup language such as MathML or XML (eXtensible Markup Language) and used for web search (S470). In order for the web search according to the present invention to be performed quickly and accurately, the web server 200 or another content server (not shown) connected thereto may be written in a markup language or easily searched in a markup language. Various contents that can be may be database.

Second Embodiment

Hereinafter, the present invention will be further described with reference to FIG. 3 and FIG. 5, which illustrates a web search process based on image information according to a second embodiment of the present invention. In FIG. 5, for convenience of description, the exemplary processes on the user terminal device 100 side and the server side are shown separately.

In the present embodiment, when the user connects to the web server 200 through the user terminal device 100 (S510), the web server 200 may display an image component along with web page information to be displayed on the user terminal device 100. 150 is transmitted to the user terminal device 100 (S515). The user terminal device 100 displays the web page 600 on the web browser 610 on the user terminal device 100 based on the received webpage information (S520), and displays the image input interface 630 together. In operation S525, the user may input equations, symbols, characters, and the like into the image input window 631 through the input unit 110 (S530). Since the above processes are the same as in the first embodiment described above, a detailed description thereof will be omitted.

In the present embodiment, when the user inputs the image to the image input window 631 through the input unit 110, the image component 150 recognizes the image through an image recognition algorithm (S540). Thereafter, the image component 150 extracts equations, symbols, and letters corresponding to the recognized images as recommendation equations, recommendation symbols, and recommendation characters (S550).

In this case, the image component 150 may refer to a database (not shown) on the user terminal device 100 in which information on equations, symbols, and characters are stored, and the database may be referred to as a web server 200 in S515. May be transmitted together with the image component 150 to the user terminal device 100. Meanwhile, the database may correspond to the writing of a specific user of the user terminal device 100 and the equations, symbols, and characters specified according to each other, and may store information on the information in the database and update it from time to time.

Next, the image component 150 displays the extracted recommendation formula, recommendation symbol, and recommendation character in the recommendation window 633 of the image input interface 630, awaiting the user's selection, and the user displays the recommendation window 633. After searching the formulas, symbols, and characters to be selected, at least one of them is selected (S555).

If the user selects one or more equations, symbols, characters, etc. displayed on the recommendation window 633, and constructs a keyword by using the same, the keyword is transmitted to the web server 200 (S560).

Then, the received keyword is converted into a markup language such as MathML or XML and used for web search (S570).

Embodiments according to the present invention described above may be implemented in the form of program instructions that may be executed by various computer components, and may be recorded in a computer-readable recording medium. The computer readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs, DVDs, and magneto-optical media such as floptical disks. And hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine code, such as produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter. The hardware device may be configured to operate as one or more software modules to perform the process according to the invention, and vice versa.

Although the present invention has been described by specific embodiments such as specific components and the like, but the embodiments and the drawings are provided to assist in a more general understanding of the present invention, the present invention is not limited to the above embodiments. For those skilled in the art, various modifications and variations can be made from such descriptions.

Therefore, the spirit of the present invention should not be limited to the embodiments described above, and all of the equivalents or equivalents of the claims, as well as the claims below, are included in the scope of the spirit of the present invention. I will say.

1 is a schematic diagram illustrating an overall configuration of a system for performing a web search based on an image input by a user according to an embodiment of the present invention.

2 is a diagram illustrating a detailed configuration of a user terminal device according to an exemplary embodiment.

3 is a diagram illustrating an image input interface and a web browser linked thereto according to an embodiment of the present invention.

4 is a flowchart illustrating a web search process based on image information according to the first embodiment of the present invention.

5 is a flowchart illustrating a web search process based on image information according to a second embodiment of the present invention.

<Description of the symbols for the main parts of the drawings>

100: user terminal device 200: Web server

300: image processing server 110: input unit

130: display unit 150: image component

Claims (16)

A method of providing a web search service by converting an image input by a user into a markup language, Recognizing the image, Extracting at least one expression, symbol, or character corresponding to the profile of the recognized image, Converting the at least one extracted expression, symbol or character into a markup language, and Performing a web search for an expression, symbol or character converted to the markup language Including, Extracting the at least one expression, symbol or character is performed with reference to a database storing information relating to a profile of the plurality of expressions, symbols or characters. The method of claim 1, The image is an image input by at least one of a mouse, a pen mouse, an electronic pen, a digitizer, a touch pad, a touch panel, and a tablet. delete The method of claim 1, The extracting the at least one expression, symbol or character comprises calculating a similarity between the profile of the recognized image and a plurality of expressions, symbols or characters stored in the database. The method of claim 1, The markup language is Mathmatic Markup Language (MathML) or eXtensible Markup Language (XML). The method of claim 1, And converting the at least one expression, symbol or character into a markup language, with reference to a database storing information about a plurality of expressions, symbols or characters and the markup language corresponding thereto. A method of providing a web search service by converting an image input by a user into a markup language, Recognizing the image, Extracting at least one recommendation expression, recommendation symbol, or recommendation character corresponding to the profile of the recognized image, Converting at least one expression, symbol or character selected by the user from the at least one extracted recommendation expression, recommendation symbol or recommendation character into a markup language, and Performing a web search for an expression, symbol or character converted to the markup language Including, Extracting the at least one recommendation expression, recommendation symbol, or recommendation character is performed by referring to a database storing information on a profile of a plurality of expressions, symbols or characters. The method of claim 7, wherein The image is an image input by at least one of a mouse, a pen mouse, an electronic pen, a digitizer, a touch pad, a touch panel, and a tablet. delete The method of claim 7, wherein Extracting the at least one recommendation expression, recommendation symbol or recommendation character comprises calculating a similarity between the profile of the recognized image and a plurality of expressions, symbols or characters stored in the database. The method of claim 10, And displaying the at least one extracted recommendation expression, recommendation symbol, or recommendation character according to the calculated similarity order. The method of claim 7, wherein The markup language is MathML or XML. The method of claim 7, wherein And converting the at least one expression, symbol or character into a markup language, with reference to a database storing information about a plurality of expressions, symbols or characters and the markup language corresponding thereto. A system for providing a web search service by converting an image input by a user into a markup language, An image processor for recognizing an image and extracting at least one expression, symbol, or character corresponding to the profile of the recognized image; A conversion unit for converting the at least one extracted expression, symbol or character into a markup language, and Web search unit that provides a web search service for the expression, symbol or character converted to the markup language Including, And the image processor references a database that stores information about a profile of a plurality of expressions, symbols, or characters. A system for providing a web search service by converting an image input by a user into a markup language, An image processing unit for recognizing an image and extracting at least one recommendation expression, recommendation symbol, or recommendation character corresponding to a profile of the recognized image; A conversion unit for converting at least one expression, symbol, or character selected by the user from the at least one extracted recommendation expression, recommendation symbol, or recommendation character into a markup language, and Web search unit that provides a web search service for the expression, symbol or character converted to the markup language Including, And the image processor references a database that stores information about a profile of a plurality of expressions, symbols, or characters. 14. A computer readable recording medium having recorded thereon a computer program for executing the method according to any one of claims 1, 2, 4 to 8 and 10 to 13.

Images