[go: up one dir, main page]

CN105631051A - Character recognition based mobile augmented reality reading method and reading system thereof - Google Patents

Character recognition based mobile augmented reality reading method and reading system thereof Download PDF

Info

Publication number
CN105631051A
CN105631051A CN201610111436.7A CN201610111436A CN105631051A CN 105631051 A CN105631051 A CN 105631051A CN 201610111436 A CN201610111436 A CN 201610111436A CN 105631051 A CN105631051 A CN 105631051A
Authority
CN
China
Prior art keywords
image
keyword
text
server
mobile
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201610111436.7A
Other languages
Chinese (zh)
Inventor
吕建明
石嘉琪
代涵宣
刘宇阳
徐辰沁
马芮
黄洁晶
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
South China University of Technology SCUT
Original Assignee
South China University of Technology SCUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by South China University of Technology SCUT filed Critical South China University of Technology SCUT
Priority to CN201610111436.7A priority Critical patent/CN105631051A/en
Publication of CN105631051A publication Critical patent/CN105631051A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/951—Indexing; Web crawling techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Processing Or Creating Images (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

本发明公开了一种基于文字识别的移动增强现实阅读方法，包括以下步骤：1、移动设备获取所拍摄到的包含文字的原始图像；2、移动设备对所获得到的原始图像进行预处理，并上传到服务器；3、服务器获得文字集合以及各个文字在图像中的位置信息；4、服务器获得关键字集合以及各个关键字的位置信息；5、对于每个关键字，服务器在知识库中检索与其相关的多媒体资源，并将相应的结果回传给移动设备；6、移动终端针对每组接受到的结果，将多媒体资源精确叠加在原始图像上。本发明还公开了一种实现基于文字识别的移动增强现实阅读方法的阅读系统，包括：手机端和服务器端；手机端和服务器端通过互联网进行通信。具有服务器的存储代价较小等优点。

The invention discloses a mobile augmented reality reading method based on character recognition, which comprises the following steps: 1. The mobile device acquires the captured original image containing characters; 2. The mobile device preprocesses the obtained original image, and upload to the server; 3. The server obtains the text collection and the location information of each text in the image; 4. The server obtains the keyword collection and the location information of each keyword; 5. For each keyword, the server searches the knowledge base 6. The mobile terminal accurately superimposes the multimedia resources on the original image for each group of received results. The invention also discloses a reading system for realizing the mobile augmented reality reading method based on character recognition, comprising: a mobile phone terminal and a server terminal; the mobile phone terminal and the server terminal communicate through the Internet. It has advantages such as less storage cost of the server.

Description

Based on mobile augmented reality reading method and the reading system thereof of Text region
Technical field
The present invention relates to a kind of enhancing reality system technology, in particular to a kind of mobile augmented reality reading method based on Text region and reading system thereof.
Background technology
In conventional books, newspaper reading pattern, the information that people obtain only comes from the books and newspapers read, the quantity of information obtained is less and has limitation, if wanting to understand more multi information for interested content, it usually needs input keyword in the search engine of PC end or mobile terminal and search for. This kind of mode of operation is loaded down with trivial details, and reader and the interactivity read between report are poor.
In view of above-mentioned reading model Problems existing, the present invention proposes a kind of enhancing reality (AugmentedReality in conjunction with Text region, knowledge base coupling technology, it is called for short AR) technology, relevant word, image, video are accurately added on the word content of readers ' reading, help reader conveniently to obtain more information in the process read, and make the type diversification more of reading information.
So-called augmented reality is a kind of by the technology of true world information and virtual world information Seamless integration-. traditional augmented reality based on mobile equipment, it is take image by mobile phone camera, and by the image of this real world and be kept in background data base in advance store image compare, if the image mated mutually can be found, then by the word relevant with this image, the virtual information superposition of video or image is displayed in the preview window of mobile phone camera, user is allowed to see image and seamless being superimposed of virtual information in the true world, so that user can obtain more information and have the sensory experience of exceeding reality, reality is had more understanding. augmented reality has good application in e-magazine, placard publicity, virtual furnishings displaying etc.
But existing augmented reality, it is generally required to background data base preserves in advance and processes for generation of the image strengthening real effect, only when the image photographed comprises the image that these set in advance time, just display strengthens real content accordingly. This kind as identifying using image and in the way of matching vector, needs typing in advance in service end and stores the view data of a large amount of offer couplings on the one hand, and storage cost is relatively big, and early stage, image typing preparation work was loaded down with trivial details; On the other hand, owing to default image could can only be identified and produce to strengthen real effect, mobile enhancing realizes terminal and can only play a role under very limited specific image scene, constrains more greatly the widespread use of augmented reality.
Summary of the invention
The primary and foremost purpose of the present invention is to overcome the shortcoming of prior art with not enough, it is provided that a kind of mobile augmented reality reading method based on Text region.
Another object of the present invention is in overcoming the shortcoming of prior art and deficiency, thering is provided a kind of reading system being applied to the mobile augmented reality reading system based on Text region, this system is a kind of in conjunction with the mobile augmented reality reading system of Text region, knowledge base coupling.
The primary and foremost purpose of the present invention is achieved through the following technical solutions: a kind of reading method being applied to the mobile augmented reality reading system based on Text region, comprises the following steps:
S1. mobile equipment obtain taken by the image P comprising word.
S2. the image P that step S1 is obtained by mobile equipment carries out pre-treatment and obtains image P', then uploads onto the server.
S3. the image P' received is carried out text detection and identification by server, obtains word set { WiAnd each word appear at the positional information { Loc in image P'i. Wherein WiRepresent i-th word detected, LociRepresent that this word appears at the position in image P'.
S4. server is according to predefined keywords dictionary, and the word in the image P' obtained in step S3 is carried out keyword match, obtains set of keywords { Tj, and each keyword appears at the position { Pos in image P'j. Wherein TjRepresent jth the keyword detected, PosjRepresent this keyword TjAppear at the position in image P'.
S5. each keyword T that server obtains according to step S4j, carry out retrieving and T in knowledge basejRelevant multimedia resource S setj. And by result for retrieval set { (Tj,Sj,Posj) return to mobile equipment. Wherein PosjIt is keyword TjAppear at the position in image P'.
S6. result (the T that mobile terminal receives for often groupj,Sj,Posj), by multimedia resource Sj, accurately it is superimposed upon the Pos of the image P that step S1 obtainsjOn position.
Abovementioned steps S1 is specially: the camera utilizing mobile equipment, is taken by the reading material including word, obtains image P.
Abovementioned steps S2 is specially: image P is adjusted resolving power by mobile equipment, and carries out image enhaucament and binary conversion treatment, obtains image P', then uploads onto the server.
Abovementioned steps S3 is specially: server, after obtaining image P', is detecting the character area in P', thus obtaining position in the picture, each word place. And call based on the word in the recognition engine text identification character area of optical character recognition (OCR) technology.
Abovementioned steps S4 is specially: the generation method of Keywords Dictionary is: for the ample resources (comprising article, picture, video etc.) collected in advance, utilize the Chinese lexical analysis device with functions such as Chinese word segmentation, part of speech mark, named entity recognition, new word identification to extract key noun wherein as keyword the title of all kinds of resource or title, and add in Keywords Dictionary. Keyword in Keywords Dictionary sorts according to temperature. When carrying out keyword match, the word sequence in the image P' of the acquisition in step S3 is first carried out Chinese word segmentation, then to each word obtained, search in Keywords Dictionary; Finally be retained in Keywords Dictionary occur word as keyword, constitute set of keywords { Tj. Each keyword TjPosition PosjThe position of first word being defined as this keyword in image P'.
Abovementioned steps S5 is specially: each keyword T that server obtains according to step S4j, carry out retrieving and T in knowledge basejRelevant multimedia resource S setj. In knowledge base, the multimedia resource information of record can be word, picture, video or three-dimensional model, and information source can be the inside resource of the web retrieval to World Wide Web or particular organization. Knowledge base adopts the mode of the table of falling row index that the descriptor of resource is carried out index, and supports the full-text search based on keyword.
Abovementioned steps S6 is specially: the concrete grammar of resource superposition is, the result (T that mobile terminal receives for often groupj,Sj,Posj), position Pos in image PjNear region carry out highlighted highlighting, prompting reader this be the region that can click. When reader clicks this region time, this areas adjacent will show and keyword TjThe resource information S being associatedj��
Another object of the present invention is achieved through the following technical solutions: a kind of mobile augmented reality reading system based on Text region, comprising: mobile phone terminal and server end; Described mobile phone terminal is communicated by internet with server end; Described mobile phone terminal comprises taking module, image pre-processing module and resource laminating module; Described server end comprises Text region module, keyword match module and knowledge base retrieval module; Described taking module comprises the image of word by mobile phone camera shooting; The image photographed is carried out pre-treatment by described image pre-processing module; The image received is carried out text detection and identification by described Text region module; Described keyword match module by Keywords Dictionary to the word in image carries out keyword match; Described knowledge base retrieval module retrieves the multimedia resource set relevant with keyword in knowledge base; Multimedia resource is accurately superimposed upon on the image of described mobile phone terminal shooting by described resource laminating module.
The principle of work of the present invention: the present invention is by the word in the reading material captured by character recognition technology identification mobile terminal, and in knowledge base, carry out information retrieval according to the word identified, the related text of acquisition, image or video resource are accurately added in the shooting picture of mobile terminal, it may also be useful to family obtains more relevant information based on reading on the basis of thing. The mobile augmented reality system based on character recognition technology that the present invention proposes breaks through above-mentioned limitation, when mobile terminal takes the material that arbitrary magazine, newpapers and periodicals etc. comprise word, to first carry out Text region, then knowledge base by word and backstage is compared, and then relevant Word message, picture information or video information is accurately added in the preview screen of mobile terminal. This kind, based on the mode of Text region, has following advantage, does not need to preserve in the server in advance corresponding image on the one hand, and the storage cost of server is less, also without the need to the image typing preparation work in early stage; On the other hand, the material comprising word arbitrarily can be identified and produce to strengthen real effect by mobile terminal, greatly expands the scope of application of this system.
The present invention has following advantage and effect relative to prior art:
1, to compensate for conventional books and newspapers reading method obtaining information amount few in the present invention, the shortcoming that interactivity is poor, the information that reader is obtained is not limited to reading matter, it is possible to merged mutually with the content of nature, efficiently mode with reality reading matter by related resource, it is provided that to the reading material that reader enriches more.
2, the reading material comprising word arbitrarily can be identified and produce to strengthen real effect by the present invention, and do not need the image to reading material to carry out typing in advance and process, greatly expands the scope of application of this system. Only need reading material comprises specific keyword, near keyword, superposition will can click mutual content of multimedia for user.
3, the augmented reality based on Text region proposed of the present invention, does not need the image preserving reading material in advance in the server in advance, and the storage cost of server is less.
Accompanying drawing explanation
Fig. 1 is the method flow diagram of invention.
Fig. 2 is the schematic diagram of reading matter.
Fig. 3 is the image process schematic diagram obtaining and comprising word.
Fig. 4 is schematic diagram shooting picture being carried out keyword recognition and highlighting.
Fig. 5 is the image process schematic diagram obtaining and comprising word.
Fig. 6 is schematic diagram shooting picture being carried out keyword recognition and highlighting.
Fig. 7 is the Resources list displaying figure.
Fig. 8 is resource display figure.
Fig. 9 is the reading system block diagram of the present invention.
Embodiment
Below in conjunction with embodiment and accompanying drawing, the present invention is described in further detail, but embodiments of the present invention are not limited to this.
Embodiment
As shown in Figure 1, a kind of mobile augmented reality reading method based on Text region, mainly comprises following six steps:
The image P comprising word taken by the acquisition of S1, mobile equipment.
The image P that step S1 is obtained by S2, mobile equipment carries out pre-treatment and obtains image P', then uploads onto the server.
The image P' received is carried out text detection and identification by S3, server, obtains word set { WiAnd each word appear at the positional information { Loc in image P'i. Wherein WiRepresent i-th word detected, LociRepresent that this word appears at the position in image P'.
S4, server, according to predefined keywords dictionary, carry out keyword match in the word in the image P' of acquisition in step s3, obtain set of keywords { Tj, and each keyword appears at the position { Pos in image P'j. Wherein TjRepresent jth the keyword detected, PosjRepresent this keyword TjAppear at the position in image P'.
Each keyword T that S5, server obtain according to step S4j, carry out retrieving and T in knowledge basejRelevant multimedia resource S setj. And by result for retrieval set { (Tj,Sj,Posj) return to mobile equipment. Wherein PosjIt is keyword TjAppear at the position in image P'.
Result (the T that S6, mobile terminal receive for often groupj,Sj,Posj), by multimedia resource Sj, accurately it is superimposed upon the Pos of the image P that step S1 obtainsjOn position.
Abovementioned steps S1 is specially: the camera utilizing mobile equipment, is taken by the reading material including word, obtains image P.
Abovementioned steps S2 is specially: image P is adjusted resolving power by mobile equipment, and carries out image enhaucament and binary conversion treatment, obtains image P', then uploads onto the server.
Abovementioned steps S3 is specially: server, after obtaining image P', is detecting the character area in P', thus obtaining position in the picture, each word place. And call based on the word in the recognition engine text identification character area of optical character recognition (OCR) technology.
Abovementioned steps S4 is specially: the generation method of Keywords Dictionary is: for the ample resources (comprising article, picture, video etc.) collected in advance, utilize the Chinese lexical analysis device with functions such as Chinese word segmentation, part of speech mark, named entity recognition, new word identification to extract key noun wherein as keyword the title of all kinds of resource or title, and add in Keywords Dictionary. Keyword in Keywords Dictionary sorts according to temperature. When carrying out keyword match, the word sequence in the image P' of the acquisition in step S3 is first carried out Chinese word segmentation, then to each word obtained, search in Keywords Dictionary; Finally be retained in Keywords Dictionary occur word as keyword, constitute set of keywords { Tj. Each keyword TjPosition PosjThe position of first word being defined as this keyword in image P'.
Abovementioned steps S5 is specially: each keyword T that server obtains according to step S4j, carry out retrieving and T in knowledge basejRelevant multimedia resource S setj. In knowledge base, the multimedia resource information of record can be word, picture, video or three-dimensional model, and information source can be the inside resource of the web retrieval to World Wide Web or particular organization. Knowledge base adopts the mode of the table of falling row index that the descriptor of resource is carried out index, and supports the full-text search based on keyword.
Abovementioned steps S6 is specially: the concrete grammar of resource superposition is, the result (T that mobile terminal receives for often groupj,Sj,Posj), position Pos in image PjNear region carry out highlighted highlighting, prompting reader this be the region that can click. When reader clicks this region time, this areas adjacent will show and keyword TjThe resource information S being associatedj��
In order to show the mobile augmented reality reading method based on Text region and the reading system thereof of the present invention visually, it is described in detail below in conjunction with accompanying drawing and embodiment:
As shown in Figure 2, being the schematic diagram of reading matter, reading matter 201 can be any article comprising word such as magazine, placard or teaching material. Reading matter 201 is opened to page 202 and page 203, and page 202 contains a picture 204 built and the descriptive text 205 about picture 204, and page 203 contains one section of text description 206. In conventional reading model, when we read reading matter 201, the information acquired is only the content that page 202 and page 203 show, and the type of information is also only word and picture. If wanting to do further understanding for some things mentioned in the page, user needs to input corresponding keyword in a browser, obtains resource, and this process is comparatively loaded down with trivial details. By helping, user obtains the information wanting to understand when reading with a kind of form easily more in the present invention.
As shown in Figure 3, being the schematic diagram of the process using mobile equipment 301 to obtain the image comprising word, mobile equipment 301 can be any mobile equipment comprising network savvy and camera, such as mobile phone, palm panel computer etc. Based on the preview function of the built-in camera of mobile equipment 301, on the screen 302 of mobile equipment 301, the image 303 of display is the content of the page 202 of reading matter 201, not only comprises word, also comprise picture simultaneously in image 303. User determines to obtain after image 303, and image is by upload server after pretreatment, and detection is published picture the character area in picture by server, and the word in character area is identified, keyword match and obtain associated multimedia resource. The result of process is as shown in Figure 4, image 303 image 401 after treatment will be shown on mobile equipment 301, keyword Guangzhou tower is gone out by frame 402 frame and does highlighted highlighting, to remind user to click herein, it can be seen that relevant more multimedia resource.
As shown in Figure 5, it is the schematic diagram that another use mobile equipment 301 obtains the process of the image comprising word, the camera preview image 304 of mobile equipment 301 contains the word paragraph on reading matter 201 page 203. After user determines to obtain image 304, result is as shown in Figure 6 after treatment, and by the image 403 after display process on mobile equipment 301, keyword Guangzhou tower is gone out by frame 404 frame and does highlighted highlighting, to remind user to click herein, it can be seen that relevant more multimedia resource.
When user is on mobile equipment 301 screen when click on area 402 or region 404, display on mobile equipment 301 screen is as shown in Figure 7, by multimedia resource list 501 relevant for display Guangzhou tower on mobile equipment 301, wherein comprise multiple resource type, comprise video resource, article resource, 3D model etc. Click the resource items in the Resources list 501, it is possible to check detailed resource information. Such as, click on area 502, will obtain Guangzhou tower 3D model as shown in Figure 8. In Fig. 8, the Guangzhou tower 3D model 601 of display on mobile equipment 301, according to the change of the distance between mobile equipment 301 and reading matter 201, the size of Guangzhou tower 3D model 601 also will change accordingly; When changing the angle between mobile equipment 301 and reading matter as user, by the Guangzhou tower 3D model 601 of display different angles on the screen 302 of mobile equipment 301. Thus it can be seen that the different behaviors according to user are made corresponding change by the enhancing real-life asset shown, contribute to user comprehensive go understanding information, by force interactive.
As shown in Figure 9, a kind of reading system realizing the described mobile augmented reality reading method based on Text region, comprising: mobile phone terminal and server end; Described mobile phone terminal is communicated by internet with server end; Described mobile phone terminal comprises taking module, image pre-processing module and resource laminating module; Described server end comprises Text region module, keyword match module and knowledge base retrieval module; Described taking module comprises the image of word by mobile phone camera shooting; The image photographed is carried out pre-treatment by described image pre-processing module; The image received is carried out text detection and identification by described Text region module; Described keyword match module by Keywords Dictionary to the word in image carries out keyword match; Described knowledge base retrieval module retrieves the multimedia resource set relevant with keyword in knowledge base; Multimedia resource is accurately superimposed upon on the image of described mobile phone terminal shooting by described resource laminating module.
Above-described embodiment is that the present invention preferably implements mode; but embodiments of the present invention are not restricted to the described embodiments; the change done under the spirit of other any the present invention of not deviating from and principle, modification, replacement, combination, simplification; all should be the substitute mode of equivalence, it is included within protection scope of the present invention.

Claims (7)

1.一种基于文字识别的移动增强现实阅读方法，其特征是，包括以下步骤：1. A mobile augmented reality reading method based on text recognition, is characterized in that, comprises the following steps: 步骤S1、移动设备获取所拍摄到的包含文字的图像P；Step S1, the mobile device acquires the captured image P containing text; 步骤S2、移动设备对步骤S1获得的图像P进行预处理得到图像P'，然后上传到服务器；Step S2, the mobile device preprocesses the image P obtained in step S1 to obtain an image P', and then uploads it to the server; 步骤S3、服务器对接收到的图像P'进行文字检测和识别，获得文字集合{Wi}以及各个文字出现在图像P'中的位置信息{Loci}；其中Wi表示检测到的第i个文字,Loci表示该文字出现在图像P'中的位置；Step S3, the server detects and recognizes characters on the received image P', and obtains the character set {W i } and the location information {Loc i } where each character appears in the image P'; where W i represents the i-th detected A text, Loc i represents the position where the text appears in the image P'; 步骤S4、服务器根据预定义关键字词典，在步骤S3中的获取的图像P'中的文字中进行关键字匹配，获得关键字集合{Tj}，以及各个关键字出现在图像P'中的位置{Posj}；其中Tj表示检测到的第j个关键字，Posj表示该关键字Tj出现在图像P'中的位置；Step S4. According to the predefined keyword dictionary, the server performs keyword matching in the text in the image P' obtained in step S3, and obtains the keyword set {T j }, and each keyword that appears in the image P' Position {Pos j }; where T j represents the jth keyword detected, and Pos j represents the position where the keyword T j appears in the image P'; 步骤S5、服务器根据步骤S4所获得的每个关键字Tj，在知识库中进行检索和Tj相关的多媒体资源集合Sj，并将检索结果集合{(Tj,Sj,Posj)}回传给移动设备；其中Posj是关键字Tj出现在图像P'中的位置；Step S5, the server retrieves the multimedia resource set S j related to T j in the knowledge base according to each keyword T j obtained in step S4, and collects the search result set {(T j , S j , Pos j ) } back to the mobile device; where Pos j is the position where the keyword T j appears in the image P'; 步骤S6、移动终端针对每组接收到的结果(Tj,Sj,Posj)，将多媒体资源Sj，精确叠加在步骤S1所获得的图像P的Posj位置上。Step S6, for each group of received results (T j , S j , Pos j ), the mobile terminal accurately superimposes the multimedia resource S j on the position of Pos j of the image P obtained in step S1. 2.如权利要求1所述的结合文字识别、知识库匹配的移动增强现实阅读系统，其特征是，在前述步骤S1中，利用移动设备的摄像头，对包含有文字的阅读材料进行拍摄，获得图像P。2. The mobile augmented reality reading system combined with text recognition and knowledge base matching as claimed in claim 1, characterized in that, in the aforementioned step S1, the camera of the mobile device is used to photograph the reading material containing the text to obtain Image P. 3.如权利要求1所述的结合文字识别、知识库匹配的移动增强现实阅读系统，其特征是，在前述步骤S2中，移动设备对图像P调整分辨率，并进行图像增强及二值化处理，得到图像P'，然后上传到服务器。3. The mobile augmented reality reading system combined with character recognition and knowledge base matching as claimed in claim 1, characterized in that, in the aforementioned step S2, the mobile device adjusts the resolution of the image P, and performs image enhancement and binarization processing to obtain the image P', and then upload it to the server. 4.如权利要求1所述的结合文字识别、知识库匹配的移动增强现实阅读系统，其特征是，在前述步骤S3中，服务器在获得图像P'后，检测P'中的文字区域，从而获得每个文字处在图像中的位置，并调用基于光学字符识别技术的文字识别引擎识别文字区域中的文字。4. The mobile augmented reality reading system combined with text recognition and knowledge base matching as claimed in claim 1, characterized in that, in the aforementioned step S3, after the server obtains the image P', it detects the text area in P', thereby Obtain the position of each character in the image, and call the character recognition engine based on optical character recognition technology to recognize the characters in the character area. 5.如权利要求1所述的结合文字识别、知识库匹配的移动增强现实阅读系统，其特征是，在前述步骤S5中，知识库中记录的多媒体资源信息可以是文字、图片、视频或者三维模型，信息来源可以是对万维网的网页采集或者是特定机构的内部资源。5. The mobile augmented reality reading system combined with text recognition and knowledge base matching as claimed in claim 1, characterized in that, in the aforementioned step S5, the multimedia resource information recorded in the knowledge base can be text, pictures, videos or three-dimensional Model, the source of information can be a collection of web pages on the World Wide Web or internal resources of a particular organization. 6.如权利要求1所述的结合文字识别、知识库匹配的移动增强现实阅读系统，其特征是，在前述步骤S6中，资源叠加的具体方法是，移动终端针对每组接收到的结果(Tj,Sj,Posj)，将关键字出现在图像P中的区域Posj进行突出高亮显示，提示读者这是可以点击的区域，当读者点击关键字的时候，会在关键字区域附近，显示资源信息Sj。6. the mobile augmented reality reading system that combines text recognition, knowledge base matching as claimed in claim 1, is characterized in that, in aforementioned step S6, the specific method of resource superimposition is, mobile terminal receives the result ( T j , S j , Pos j ), highlight the area Pos j where the keyword appears in the image P, and remind the reader that this is an area that can be clicked. When the reader clicks on the keyword, it will be in the keyword area Nearby, resource information S j is displayed. 7.一种实现权利要求1所述的基于文字识别的移动增强现实阅读方法的阅读系统，其特征是，包括：手机端和服务器端；所述手机端和服务器端通过互联网进行通信；所述手机端包括拍摄模块、图像预处理模块和资源叠加模块；所述服务器端包括文字识别模块，关键字匹配模块和知识库检索模块；所述拍摄模块通过手机摄像头拍摄包含文字的图像；所述图像预处理模块对拍摄到的图像进行预处理；所述文字识别模块对接收到的图像进行文字检测和识别；所述关键字匹配模块通过关键字词典对图像中的文字中进行关键字匹配；所述知识库检索模块在知识库中检索和关键字相关的多媒体资源集合；所述资源叠加模块将多媒体资源精确叠加在所述手机端拍摄的图像上。7. A reading system that realizes the mobile augmented reality reading method based on character recognition claimed in claim 1, is characterized in that, comprising: a mobile phone terminal and a server terminal; the mobile phone terminal and the server terminal communicate through the Internet; the The mobile phone terminal includes a shooting module, an image preprocessing module and a resource superimposition module; the server side includes a text recognition module, a keyword matching module and a knowledge base retrieval module; the shooting module captures an image containing text through a mobile phone camera; the image The preprocessing module preprocesses the captured image; the text recognition module performs text detection and recognition on the received image; the keyword matching module performs keyword matching in the text in the image through a keyword dictionary; The knowledge base retrieval module searches the knowledge base for a collection of multimedia resources related to keywords; the resource overlay module accurately overlays the multimedia resources on the image captured by the mobile phone.
CN201610111436.7A 2016-02-29 2016-02-29 Character recognition based mobile augmented reality reading method and reading system thereof Pending CN105631051A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610111436.7A CN105631051A (en) 2016-02-29 2016-02-29 Character recognition based mobile augmented reality reading method and reading system thereof

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610111436.7A CN105631051A (en) 2016-02-29 2016-02-29 Character recognition based mobile augmented reality reading method and reading system thereof

Publications (1)

Publication Number Publication Date
CN105631051A true CN105631051A (en) 2016-06-01

Family

ID=56045983

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610111436.7A Pending CN105631051A (en) 2016-02-29 2016-02-29 Character recognition based mobile augmented reality reading method and reading system thereof

Country Status (1)

Country Link
CN (1) CN105631051A (en)

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106484843A (en) * 2016-09-30 2017-03-08 维沃移动通信有限公司 A kind of lyrics poster generation method and server
CN106650727A (en) * 2016-12-08 2017-05-10 宇龙计算机通信科技(深圳)有限公司 Information display method and AR (augmented reality) device
CN107612992A (en) * 2017-09-14 2018-01-19 葛立洲 A kind of experiential method based on AR technologies
CN108197621A (en) * 2017-12-28 2018-06-22 北京金堤科技有限公司 Company information acquisition methods and system and information processing method and system
CN108230428A (en) * 2017-12-29 2018-06-29 掌阅科技股份有限公司 E-book rendering method, electronic equipment and storage medium based on augmented reality
CN109087439A (en) * 2018-07-03 2018-12-25 百度在线网络技术（北京）有限公司 Bill method of calibration, terminal device, storage medium and electronic equipment
CN109145141A (en) * 2018-09-06 2019-01-04 百度在线网络技术（北京）有限公司 Information displaying method and device
CN109215416A (en) * 2018-10-24 2019-01-15 天津工业大学 A kind of Chinese character assistant learning system and method based on augmented reality
CN109871753A (en) * 2019-01-08 2019-06-11 上海玄彩美科网络科技有限公司 A kind of text handling method and equipment based on augmented reality
CN110019906A (en) * 2017-11-22 2019-07-16 百度在线网络技术（北京）有限公司 Method and apparatus for showing information
CN110781703A (en) * 2018-07-30 2020-02-11 罗伯特·博世有限公司 Method, mobile device and analytical processing computer for generating shipping information
CN112506398A (en) * 2020-11-25 2021-03-16 飞毯信息技术有限公司 Image-text display method, device and computer readable medium for same
CN112861504A (en) * 2021-02-08 2021-05-28 北京百度网讯科技有限公司 Text interaction method, device, equipment, storage medium and program product
CN113761257A (en) * 2020-09-08 2021-12-07 北京沃东天骏信息技术有限公司 Picture analysis method and device
CN115100666A (en) * 2022-05-18 2022-09-23 东北大学 AR conference system based on significance detection and super-resolution reconstruction and construction method

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120323562A1 (en) * 2006-10-13 2012-12-20 Syscom Inc. Method and system for converting image text documents in bit-mapped formats to searchable text and for searching the searchable text
CN102855480A (en) * 2012-08-07 2013-01-02 北京百度网讯科技有限公司 Method and device for recognizing characters in image
CN103050025A (en) * 2012-12-20 2013-04-17 广东欧珀移动通信有限公司 Mobile terminal learning method and learning system thereof
CN103718174A (en) * 2011-08-05 2014-04-09 黑莓有限公司 System and method for searching for text and displaying found text in augmented reality
CN104199834A (en) * 2014-08-04 2014-12-10 徐�明 Method and system for interactively obtaining and outputting remote resources on surface of information carrier

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120323562A1 (en) * 2006-10-13 2012-12-20 Syscom Inc. Method and system for converting image text documents in bit-mapped formats to searchable text and for searching the searchable text
CN103718174A (en) * 2011-08-05 2014-04-09 黑莓有限公司 System and method for searching for text and displaying found text in augmented reality
CN102855480A (en) * 2012-08-07 2013-01-02 北京百度网讯科技有限公司 Method and device for recognizing characters in image
CN103050025A (en) * 2012-12-20 2013-04-17 广东欧珀移动通信有限公司 Mobile terminal learning method and learning system thereof
CN104199834A (en) * 2014-08-04 2014-12-10 徐�明 Method and system for interactively obtaining and outputting remote resources on surface of information carrier

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
钟阳等: "基于图像识别的智能文字阅读系统", 《数字技术与应用》 *

Cited By (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106484843A (en) * 2016-09-30 2017-03-08 维沃移动通信有限公司 A kind of lyrics poster generation method and server
CN106484843B (en) * 2016-09-30 2019-07-26 维沃移动通信有限公司 A method and server for generating lyrics poster
CN106650727A (en) * 2016-12-08 2017-05-10 宇龙计算机通信科技(深圳)有限公司 Information display method and AR (augmented reality) device
CN106650727B (en) * 2016-12-08 2020-12-18 宇龙计算机通信科技(深圳)有限公司 Information display method and AR equipment
CN107612992A (en) * 2017-09-14 2018-01-19 葛立洲 A kind of experiential method based on AR technologies
CN110019906A (en) * 2017-11-22 2019-07-16 百度在线网络技术（北京）有限公司 Method and apparatus for showing information
CN110019906B (en) * 2017-11-22 2022-07-08 百度在线网络技术（北京）有限公司 Method and apparatus for displaying information
CN108197621A (en) * 2017-12-28 2018-06-22 北京金堤科技有限公司 Company information acquisition methods and system and information processing method and system
CN108230428A (en) * 2017-12-29 2018-06-29 掌阅科技股份有限公司 E-book rendering method, electronic equipment and storage medium based on augmented reality
CN108230428B (en) * 2017-12-29 2019-02-01 掌阅科技股份有限公司 E-book rendering method, electronic equipment and storage medium based on augmented reality
CN109087439A (en) * 2018-07-03 2018-12-25 百度在线网络技术（北京）有限公司 Bill method of calibration, terminal device, storage medium and electronic equipment
CN109087439B (en) * 2018-07-03 2021-02-09 百度在线网络技术（北京）有限公司 Bill checking method, terminal device, storage medium and electronic device
CN110781703A (en) * 2018-07-30 2020-02-11 罗伯特·博世有限公司 Method, mobile device and analytical processing computer for generating shipping information
CN109145141A (en) * 2018-09-06 2019-01-04 百度在线网络技术（北京）有限公司 Information displaying method and device
CN109215416A (en) * 2018-10-24 2019-01-15 天津工业大学 A kind of Chinese character assistant learning system and method based on augmented reality
CN109871753A (en) * 2019-01-08 2019-06-11 上海玄彩美科网络科技有限公司 A kind of text handling method and equipment based on augmented reality
CN113761257A (en) * 2020-09-08 2021-12-07 北京沃东天骏信息技术有限公司 Picture analysis method and device
CN112506398A (en) * 2020-11-25 2021-03-16 飞毯信息技术有限公司 Image-text display method, device and computer readable medium for same
CN112506398B (en) * 2020-11-25 2023-06-09 飞毯信息技术有限公司 Image-text display method and device and computer readable medium for the same
CN112861504A (en) * 2021-02-08 2021-05-28 北京百度网讯科技有限公司 Text interaction method, device, equipment, storage medium and program product
CN115100666A (en) * 2022-05-18 2022-09-23 东北大学 AR conference system based on significance detection and super-resolution reconstruction and construction method
CN115100666B (en) * 2022-05-18 2024-06-18 东北大学 AR conference system and construction method based on saliency detection and super-resolution reconstruction

Similar Documents

Publication Publication Date Title
CN105631051A (en) Character recognition based mobile augmented reality reading method and reading system thereof
US8788529B2 (en) Information sharing between images
US9372920B2 (en) Identifying textual terms in response to a visual query
US10331729B2 (en) System and method for accessing electronic data via an image search engine
CN102625937B (en) Architecture for responding to visual query
CN104021150B (en) Face recognition with social networks auxiliary
Erol et al. HOTPAPER: multimedia interaction with paper using mobile phones
JP7009769B2 (en) Recommended generation methods, programs, and server equipment
US8244037B2 (en) Image-based data management method and system
US10192279B1 (en) Indexed document modification sharing with mixed media reality
BRPI0614864A2 (en) use of image derived information as search criteria for internet and other search agents
CN102855298A (en) Image retrieval method and system
US20140344238A1 (en) System And Method For Accessing Electronic Data Via An Image Search Engine
CN103237165A (en) Method and electronic equipment for checking extended name card information in real time
CN110471886B (en) System for searching for documents and persons based on detecting documents and persons around a table
WO2024193538A1 (en) Video data processing method and apparatus, device, and readable storage medium
Jing et al. Flood event image recognition via social media image and text analysis
US20160004789A1 (en) Visual Search Engine
Takeda et al. Memory reduction for real-time document image retrieval with a 20 million pages database
Nguyen et al. Augmented media for traditional magazines
JP7231529B2 (en) Information terminal device, server and program
Uchiyama et al. On-line document registering and retrieving system for AR annotation overlay
Chiaro et al. NoisyArt: Exploiting the Noisy Web for Zero-shot Classification and Artwork Instance Recognition
Wengert et al. Kooaba interactive posters
Patel VISUAL SEARCH APPLICATION FOR ANDROID

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
WD01 Invention patent application deemed withdrawn after publication
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20160601