JP2020521219A - Efficiently enhance images with related content - Google Patents

Abstract

The subject matter herein relates generally to providing content related to text drawn in images. In one aspect, the system includes a data processing device configured to extract text from the image. The extracted text is divided into a plurality of blocks. The plurality of blocks are presented as respective first user selectable targets on a user interface at a first zoom level. A user selection of a first block of the plurality of blocks is detected. In response to detecting the user selection of the first block, the portion of the extracted text in the first block is displayed on the user interface at a second zoom level that is greater than the first zoom level. Is presented as a user-selectable target for. In response to detecting the user selection of the portion of the extracted text in the first block, an action is initiated based on the content of the user selected text.

Description

The present invention relates to efficient enhancement of images with related content.

The Internet provides access to various resources such as image files, audio files, video files, and web pages. The search system can identify the resource in response to the query. The query may be a text query that includes one or more search terms or phrases, an image query that includes images, or a combination of text and image queries. The search system ranks the resources and provides search results that may link to the identified resources and may provide content related to the query.

Images (eg, digital photos) have become a popular way for users to gather information and take action, as can be seen by the widespread popularity of smartphones with cameras and camera-first applications. .. For example, a user who has taken a picture with his smartphone or other device may want to get more information about the content of the picture. In some cases, the user may need to open a different application to enter a query based on the content identified in the photo, which can be cumbersome and result in inaccurate or invalid queries.

This specification describes systems, methods, devices, and techniques for presenting content related to text drawn in images.

In general, an innovative aspect of the subject matter described herein can be implemented in a method that includes extracting text from a captured image by a user device. The user device may divide the extracted text into blocks. The user device may present the blocks as respective user selectable targets on the user interface of the user device at the first zoom level. User selection of a first block of the plurality of blocks can be detected. In response to detecting the user selection of the first block, the user device selects a portion of the extracted text in the first block of the user device at a second zoom level that is greater than the first zoom level. Each may be presented on the user interface as a respective second user selectable target. In response to detecting the user selection of the portion of the extracted text in the first block, the user device may initiate an action based on the content of the text of the user selection. Other implementations of this aspect include corresponding apparatus, methods, and computer programs configured to perform the actions of the encoded methods on a computer storage device.

These and other implementations can optionally include one or more of the following features. In some aspects, initiating an action may include initiating communication to a destination address identified by the user-selected text.

Initiating the action includes generating a search query based on the user-selected textual content and sending the search query by the user device to a search engine. Initiating an action includes generating two or more candidate search queries based on user-selected textual content, displaying the candidate search queries in a user interface, and selecting one of the candidate queries. Sending the selected candidate query to the search engine by the user device in response to the selection.

Some aspects may include extracting context from an image by a user device. Generating a search query or a candidate search query may include generating a search query or a candidate search query further based on the extracted context of the image.

Some aspects include extracting a context from an image by a user device and using the extracted context of the image by the user device to determine if the image falls into any of a number of predefined categories. Deciding whether to do so. Generating a search or candidate search query may include generating a search or candidate search query based on the extracted context if the image falls into one of the predefined categories, If not, the step of generating a search query or a candidate search query is based solely on the content of the user-selected text.

In some aspects, the user device generates one or more search queries for the at least one block and based on the extracted text of the block before the user selects any of the blocks. be able to. The user device may submit one or more search queries from the user device to the search engine. The user device may generate and send one or more search queries before presenting the block as each user-selectable target on the user interface.

In some aspects a user device may receive search result content from a search engine based on one or more search queries and store the received search result content in a local memory of the user device. Generating one or more search queries based on the extracted text of the block parses the extracted text of the block into multiple first sets of text at a first level text-based granularity. And generating a search query for each first set of text.

Some aspects, in response to detecting the user selection of the block, extract the extracted text of the block from the plurality of text at a second level text-based granularity that is greater than the first level text-based granularity. Parsing into a second set of text, and generating an additional search query for each second set of text before receiving a user selection of some of the extracted text in the first block; , Sending each additional search query from the user device to the search engine; receiving additional search result content based on the additional search query by the user device and from the search engine; Storing in a local memory of the user device.

In some aspects dividing the extracted text into blocks is based at least in part on a semantic analysis of the extracted text. Some aspects may include displaying on a user interface content received at a user device from a search engine.

In general, another innovative aspect of the subject matter described herein can be implemented in a method that includes extracting text from an image by a user device. The user device may divide the extracted text into blocks. The user device may present the plurality of blocks as a respective first user selectable target on the user interface of the user device at the first zoom level. User selection of a first block of the plurality of blocks can be detected. In response to detecting the user selection of the first block, the user device selects a portion of the extracted text in the first block of the user device at a second zoom level that is greater than the first zoom level. Each may be presented on the user interface as a respective second user selectable target. In response to detecting the user selection of the portion of the extracted text in the first block, the user device may initiate an action based on the content of the text of the user selection.

In some aspects, initiating an action may include initiating communication to a destination address identified by the user-selected text. Initiating the action includes generating a search query based on the user-selected textual content and sending the search query to a search engine.

Some aspects are, by the user device, extracting context from the image, and using the extracted context of the image to determine that the image belongs to one of a number of predefined categories, Adjusting the search query based on the extracted context in response to determining that the image belongs to one of the predefined categories.

Some aspects may include receiving search result content selected by a user device and from a search engine based on a search query. The search engine can rank the search results based on one predefined category to which the image belongs. Some aspects include extracting, by the user device, a context from the image, and using the context of the extracted image to determine that the image does not belong to one of a number of predefined categories. Can be included. Generating a search query can include generating a general search query based only on user-selected text.

In some aspects, initiating the action includes generating two or more candidate search queries based on the content of the user-selected text; displaying the candidate search queries in a user interface; Responding to a user selection of one of the following by the user device sending a candidate query of the user selection to a search engine.

Some aspects relate to one or more search queries based on the extracted text of the first block by the user device before the user selects any of the blocks and for the first block. And sending one or more search queries from the user device to the search engine.

In some aspects, the user device generates and sends one or more search queries prior to presenting the blocks on the user interface as respective first user-selectable targets. Some aspects may include receiving search result content from a search engine based on one or more search queries and storing the received search result content in a local memory of the user device.

In some aspects, generating one or more search queries based on the extracted text of the first block includes extracting the extracted text of the first block from a first level text-based It may include parsing into a plurality of first sets of text at a granularity, and generating a search query for each first set of text.

Some aspects, in response to detecting the user selection of the first block, extract the extracted text of the first block from a second level text base that is larger than a first level text base granularity. Parsing into a plurality of second sets of text at a granularity of, and an additional search for each second set of text before receiving a user selection of a portion of the extracted text in the first block. Generating a query, sending each additional search query from the user device to the search engine, receiving additional search result content based on the additional search query by the user device and from the search engine, Storing additional search result content in a local memory of the user device.

In some aspects dividing the extracted text into blocks is based at least in part on a semantic analysis of the extracted text. Some aspects may include displaying on a user interface content received at a user device from a search engine. Displaying the content received at the user device from the search engine on the user interface may include displaying the content in a viewport of the user device with at least a portion of the image.

It has been proposed to use OCR (Optical Character Recognition) to extract text from physical documents (especially via scanned documents). However, the extracted text is usually treated as a block of text. In contrast, aspects of the present disclosure include all (or some) words, phrases, and/or blocks of text (eg, including numbers such as telephone numbers) within an image, each user-selectable. Target. The user can get more information about the content of the image by invoking a query based on the text item in the image by selecting the block containing/comprising the text item. The content returned by the query is displayed on the user interface of the user device. Compared to the traditional process where a user has to manually enter text from an image into a search engine to initiate a query, aspects of the present invention cause errors when the user manually enters a search term. Eliminate the possibility of having to re-enter search terms. This avoids the possibility that the search engine responds to mistyped queries and the user may have to repeat the query and retrieval process when they recognize the error and resubmit the corrected query. Allows more efficient use of search engine resources.

Particular embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. Rather than requiring the user to manually enter a text query, the network is identified by identifying content (e.g., via a query) based on the text drawn in the image and selected by the user. By reducing the number of submitted queries that go through, by reducing the number of search queries processed by search engines, and by reducing the number of search results served over the network in response to search queries. , Will be able to use network and search engine resources more efficiently. For example, a user may mistakenly enter text that appears in an image as a query, and one or more query corrections and subsequent network communications and search engines until the correct query yields search results that meet the user's information needs. We may submit queries that do not accurately reflect the content of the images that require processing. These additional queries consume network and search engine resources, reducing the network and search engine bandwidth available to handle other network traffic and other queries, and/or much of the search engine. Search engine response time is especially slow when scaled up across users. Therefore, identifying content according to a user's selection of text that accurately reflects the user's information needs can increase network bandwidth, reduce network and search engine response times, and/or reduce the query processed by the search engine. Leads to an increase.

Allowing the user to select the text rather than re-enter it is also more efficient for the user and allows the user to more quickly request content related to the text. It also makes it easier for the user to submit multiple queries by simply selecting different targets for different parts of the text.

The zooming techniques described herein allow a user to quickly and efficiently select the text for which they want to receive additional content. For example, if you create a user-selectable target for each piece of text when the user device is at a low zoom level (for example, the lowest zoom level where the text is at its minimum), the user will be adjacent to the text that the user was trying to select. You may select the wrong text, such as the text you want. This can result in unintended queries being sent over the network to search engines, all of which have the drawbacks mentioned above, but can lead to user frustration. The user can more easily select the correct text by creating a user-selectable target for a block of text at the appropriate zoom level, and then creating a user-selectable target for a smaller portion of the text at a higher zoom level. it can. The zoom technique reduces the number of unintended queries that are sent to search engines over the network, thus increasing network bandwidth, reducing network response time, increasing the number of queries processed by search engines, and increasing search engine It can bring about a reduction in response time.

The content of the image may be pre-cached at the user device based on the zoom level and/or the text in the image presented in the viewport of the user device at that zoom level. When the zoom level changes, additional (and/or alternative) content can be pre-cached based on the updated zoom level and/or the text presented in the viewport at the updated zoom level. .. In this way, the content associated with the text in the image can be presented more quickly in response to the user's selection of the text.

Various features and advantages of the above subject matter are described below with reference to the drawings. Additional features and advantages will be apparent from the subject matter and claims set forth herein.

For the avoidance of doubt, "text" as used herein may include letters, numbers, and/or symbols, such as, for example, telephone numbers or email addresses may constitute blocks of text. Intended.

FIG. 6 illustrates an exemplary environment in which a user device presents content related to text depicted in an image. FIG. 6 illustrates an example screenshot of an example user interface that allows a user to select text to request additional content associated with the user-selected text. FIG. 6 illustrates an exemplary screenshot of an exemplary user interface that allows a user to select text and view content associated with the selected text. FIG. 6 illustrates an exemplary screenshot of an exemplary user interface that allows a user to select text and view content associated with the selected text. 3 is a flow chart of an exemplary process for initiating an action based on user-selected textual content. 6 is a flow chart of an exemplary process for providing content in response to a search query.

In general, the systems and techniques described herein identify and provide additional content related to text and/or other content depicted in images. The user can use the camera of the smartphone or other user device to capture the image. The text in the image can then be used to identify additional content. For example, the image may depict a restaurant menu. The user device may present a user-selectable target in or around the text to allow the user to select the text and view additional content associated with the selected text. For example, a user may select a menu item and view content related to the menu item (eg, nutrition information), an image of the menu item, or other types of content related to the menu item. Additional content may be presented instantly (or near instantly) in response to selection using the pre-cache techniques described below.

The user device may divide the text drawn in the image into blocks and present each block as a user-selectable target using, for example, semantic analysis of the text and/or placement of the text within the image. it can. For example, the user device may display the text in the image of the menu in multiple blocks, including blocks for each section of the menu (for example, a block of appetizers, a block of main dishes, etc.) based on the layout and semantic analysis of the text in the image. It can be divided into blocks. When the user selects one of the blocks, the user device may present the selected block at a higher zoom level and present some of the text within the block as a user selectable target. For example, the user device may identify phrases, menu items, or other relevant parts of the text, and present at least some of the relevant parts of the text as user selectable targets at higher zoom levels. .. By changing the zoom level and the user selectable target based on the different zoom levels, the user can more easily select the text that they want to view for additional content.

At each zoom level, content may be pre-cached at the user device based on the text presented by the user device at the zoom level. As the user adjusts the zoom level, additional and/or different (alternative) content may be added to the user device based on the text presented by the user device at the updated zoom level (e.g., before the user selects text). Can be pre-cached at. In this way, the content associated with the text selected by the user can be provided instantly after the user has made the selection, or at least faster than sending the request over the network.

FIG. 1 is a diagram illustrating an exemplary environment 100 in which a user device 110 presents content related to text depicted in an image. User device 110 may be a smartphone, tablet computer, wearable device, or other type of mobile or non-mobile electronic device. User device 110 includes a camera 112 that captures images 113 and/or video. User device 110 also includes a display 120 that presents images 113 and/or videos and content related to images 113. As described in more detail below, the display 120 can include an input device such as a touch screen that allows a user to interact with the text depicted in the image 113 presented on the display 120.

The user device 110 includes an image application 111 that allows a user to capture an image using the camera 112 or view objects within the field of view of the camera's lens using the viewfinder. The image application 111 also allows the user to browse and/or listen to content related to the text drawn on the image. For purposes of this specification, image refers to the image captured by the camera 112 and visible content in the viewfinder that shows the field of view of the lens of the camera. The image application 111 also allows a user to view additional content related to images stored on a user device, such as the local memory 124 of the user device 110, or text drawn on images stored in different locations. become. For example, the user selects an image stored on the user device 110 (or a networked location) and uses the image application 111 to view content related to the image and/or the text drawn on the image. be able to. The local memory 124 may be internal memory of the user device 110 (for example, flash memory of the user device 110) or external memory connected to the user device 110 (for example, memory connected to a universal serial bus (USB) port of the user device 110). ) Can be included.

The image application 111 includes a text processor 114 that processes the text drawn on the image. The text processor 114 can recognize the text drawn in the image using, for example, optical character recognition (OCR). In some implementations, the text processor 114 executes on the user device 110 to increase the speed at which text is identified in images as compared to, for example, a full service OCR application in a networked location. Including a lightweight OCR engine. A lightweight OCR engine is an OCR engine that uses a small amount of memory (for example, less than a threshold amount of memory) and/or uses the minimum processing power of the user device 110 to identify text in images. is there. The lightweight OCR engine may use the additional processing power of the user device 110 as compared to sending the image to the OCR engine at the networked location, but the lightweight OCR engine allows the image to be sent over the network. And processed by the OCR engine and the text recognition can be performed faster than if the text were sent back to the user device 110. This reduces the waiting time when recognizing the text and presenting the content related to the text. The lightweight OCR engine can be implemented as a software development kit (SDK) that includes features for identifying text in images.

Text processor 114 may also process the text drawn in the image to identify relevant blocks, phrases, or other portions of text. In some implementations, text processor 114 may identify related text based on the placement of the text within the image. For example, text processor 114 may identify text that is grouped together as related text and/or has the same (or similar) visual characteristics, such as the same font type, size, and/or color. it can. A group of text can include text that is within a threshold distance of other text in the group and/or text that is within a visible edge or boundary in an image. For example, a group of text includes text within a visible block drawn on an image (e.g., a square, rectangle, or other suitable shape with a visible border) or on one side of a dividing line drawn on the image. obtain.

In some implementations, text processor 114 performs a semantic analysis of the text drawn in the image to identify relevant text. For example, text processor 114 can use semantic analysis to identify phrases, phrases, sentences, or paragraphs in the text depicted in the image as relevant text. Text processor 114 may also use semantic analysis and/or placement of text within images to identify related phrases, phrases, sentences, paragraphs, or blocks of individual words. For example, by analyzing all recognized words (or most of them) in an image, text processor 114 determines the category of the image as described below and may have multiple meanings. Image categories can be used to clarify the meaning of words. In another example, the text processor 114 determines that a particular portion of an image containing a particular heading contains a word and uses the text alignment to use the heading to determine the meaning of the word. can do. For example, if the word “cheek” is in the menu image and is under the heading “BBQ main dish”, the text processor 114 may determine that the word “cheek” refers to food rather than a body part. You can

In some cases, blocks of text include multiple lines of text. For example, a block of appetizer text in a menu can include one or more lines of text for each appetizer. In other cases, the block of text may include a single phrase or a single word. For example, a block of text may include text drawn next to a boat, or text engraved on a sign in a park.

The text processor 114 can output image data 115 that specifies the text identified in the processed image and each block (or other portion) of the text identified in the image. For each block of text, the image data 115 can specify the actual text contained in the block (eg, actual words and phrases). The image data 115 can also specify the location of each portion of text within the image. For example, the image data 115 can include, for a block of text, the coordinates within the image surrounding the block of text, or the coordinates of each corner of a rectangular block of text. As described below, the content acquisition unit 116 of the image application 111 retrieves the image data 115 in order to obtain the content associated with the text identified in the image, eg, pre-cache the content in the local memory 124. Can be used.

The user interface controller 118 of the image application 111 can receive the image 113 and generate a user interface 122 that presents the image 113 on the display 120. In some implementations, the image application 111 includes a user interface controller 118 for displaying the image 113 captured by the camera 112, for example, by the display 120 of the user device 110 before the image is processed by the text processor 114. To provide directly to. In this way, the image 113 is displayed immediately after the image is captured.

The user interface controller 118 can also receive image data 115 of the image 113 output by the text processor 114. User interface controller 118 generates image data 115 (e.g., blocks of text and data indicating their position within image 113) to generate a user-selectable target of the block of text identified by text processor 118. Can be used. The user selectable target allows the user to select a block of text to obtain content related to the text in the block, zoom in on the block to select text in the block, and/or Or it is possible to start another action. Each user-selectable target can include a selectable portion of the display (eg, a selectable touch screen area) and a visual indicator that the block is selectable. For example, a user-selectable target for a block of text can be a visual indicator, with a selectable icon, a highlighted box (or other shape) around the block, different background colors for the text within the block, and within the block. Text in different font colors, highlighted or bolded text, or other suitable visual indicator to indicate to the user that a block is selectable. The user interacts with the target, for example, by clicking on the target, placing the pointer over the target, touching the touch screen at the target's location, or using any other suitable user interaction. A user selectable target can be selected.

The user interface controller 118 can present a user selectable target in the image 113 at the user interface 122. User interface controller 118 can also generate a user-selectable target for the portion of text contained in the block. For example, the user interface 122 may include, for a block of text, a user-selectable target that allows the user to zoom in on the block to make the text in the block larger. The user interface 122 also includes, for each one or more portions of text in the block, a user-selectable target that allows a user to select a portion of text to request content associated with the portion of text. Can be included. As mentioned above, in some implementations where the image 113 is displayed immediately after the image is captured, when the image is processed by the text processor, the user interface controller 118 causes the user to select the first displayed image. Can be supplemented with possible targets.

In some implementations, the user interface controller 118 can include two user-selectable targets in the text block. For example, a first user-selectable target allows a user to zoom in on a block of text, and a second user-selectable target requests that the user request content related to the text in the block. enable. In another example, the user interface controller 118 can zoom in on a block of text in response to a user selecting a location within the block that does not contain text, such as a space between the text. When the user selects the text in the block, the user interface controller 118 can initiate a request for user-selected text or content related to the text in the block.

The user interface controller 118 can determine whether to zoom in on a block of text or present content depending on a user selection of the block of text. In some implementations, the user interface controller 118 zooms in or presents the content based on the text in the selected text block, for example, based on the semantic content of the text in the selected text block. You can decide what to do. For example, if a text block contains text associated with one particular entity or particular type of content, the user interface controller 118 may decide to present the content associated with the entity rather than zooming in on the text block. can do. In another example, if the text block includes text associated with a particular dish, the user interface controller 118 may determine to present content associated with the particular dish in response to a user selection of the text block. .. The content may be, for example, a character avatar next to a text block containing a review of the dish in an animated callout. If the text block includes text associated with multiple entities or multiple types of content, the user interface controller 118 can determine to zoom in on the text block in response to a user selection of the text block.

The user interface controller 118 may adjust the zoom level each time the user selects a user-selectable target for a portion of the text, for example, until a particular zoom level is reached. If the user selects a block of text, the user interface controller 118 can zoom in on the block of text so that the text in the block is larger on the display 120. For example, the user interface controller 118 can zoom in on an image on the display 120 such that the user selected block fills the viewport of the display 120 in at least one dimension (eg, horizontal and/or vertical).

The user interface controller 118 can also present at least a portion of the text within the user selection block as a user selectable target. In this way, the user can select a portion of text, for example a word or phrase, to request content related to the word or phrase. In some implementations, when the user selects a phrase or multiple portions of text that include multiple words, the user interface controller 118 zooms in further on the portion of the text and makes a user selection of a subportion of the selected portion. Possible targets may be presented. For example, if the user selects a phrase, the user interface controller 118 may zoom in so that the phrase fills the viewport of the display in at least one dimension (eg, horizontal and/or vertical). The user interface controller 118 can also present sub-portions of text as user-selectable targets. For example, if the user selects the phrase "red wine" in the image of the menu, the user interface controller 118 zooms in on that phrase and presents the word "red" as one user-selectable target, called "wine". The word can be presented as a second user-selectable target.

The user interface controller 118 may select a portion of text within a block of user selections for presentation as a user-selectable target based on the content of the portion of text. In some implementations, the user interface controller 118 can select text based on the content available for the text portion. For example, the user interface controller 118 can access a list of known entities (eg, entities for which a knowledge panel containing a summary of information about the entity is available) for which a particular type of content is available. If the piece of text includes a name, or other reference to an entity in the list of known entities, the user interface controller 118 can present the piece of text as a user-selectable target, thus allowing the user to select it. Can select a portion of the text to request additional content associated with the entity.

In some implementations, the user interface controller 118 ranks the portion of text within the block of user selections, presents the upper ranked portion of the text as a user selectable target, and the lower ranked portion of the text. Do not present as a possible target. For example, the user interface controller 118 can select a particular number or percentage of the portions of text with the highest rank, or a portion with a rank score that meets (eg, exceeds) a threshold score. The user interface controller 118 may determine the amount of content available for each piece of text, the association between each piece of text and the category of images, the popularity of the entity referenced by the piece of text (e.g., for each entity). Portions of the text may be ranked (and/or a rank score may be determined) based on the number of queries received) and/or other suitable criteria.

The user interface controller 118 can also zoom out of the text in response to user interaction. For example, the user interface controller 118 can zoom out to present more images in response to the user pinching on the touch screen or selecting an icon for zooming out.

For example, when a user initiates a request for content by selecting a user-selectable target for a portion of text, UI controller 118 may send request data 123 to content acquisition unit 116. The request data 119 can specify the selected text. For example, if the user selects a user-selectable phrase depicted in the image presented at user interface 122, user interface controller 118 may generate request data 119 that specifies each word in the phrase. In another example, if the user selects a block of text, the user interface controller 118 can generate request data 119 that specifies each word in the block of text.

In some implementations, the request data 119 includes context data. For example, the request data 119 can include other text that is not selected. In a particular example, request data 119 includes each word (or some of the words) identified in the image, or each word (or word of words) that is visible in user interface 122 based on the current zoom level. Some of them). As described below, the contextual data can be used to identify categories of images, which in turn can identify content to present in order to clarify selected words and in response to user selection. Can be used to select. For example, additional words in the image of the menu can be used to determine that the selected word, such as "shoulder," corresponds to food and to select the appropriate content.

The content acquisition unit 116 can receive the request data 119 from the user interface controller 118 and acquire the content from the search system 150 (or the local memory 124) based on the request data 119. In some implementations, the content acquisition unit 118 generates query data that specifies one or more search queries based on the request data 119, for example, a local area network (LAN), a wide area network (WAN). Query data 130 to a search system 150 via a data communication network 140, such as a mobile network, the Internet, or a combination thereof. The one or more queries can include text selected by the user. The query data 130 can also include contextual data.

Search system 150 includes a search engine 152 and a context classifier 154. The search engine 150 may use the index 156 to identify content based on words included in one or more search queries included in the query data 130. The index 156 may be provided in response to a query or referenced in search results provided in response to a query, such as an electronic resource (eg, web page, image, video, and/or other Content) can be included.

The context classifier 154 uses one or more machine learning models to classify the image (or the portion of the image visible in the user interface 122) based on the image or text drawn on the portion of the image. Can be used. Machine learning models can be trained using labeled training data. The labeled training data can include a set of training images, data that specifies, for each image, the words that are drawn in the image, and labels that specify the category of the image. Exemplary categories may include restaurant menus, signs (eg, road signs or store signs), magazines, movie posters, music posters, and so on. Although FIG. 1 illustrates context classifier 154 as part of search system 150, in some implementations context classifier 154 may be implemented on user device 110.

When the query data is received, the context classifier 154 attempts to classify the image into one of a set of predefined categories that the machine learning model has been trained to classify the image. Contextual data and machine learning models can be used. In some implementations, the search engine 152 adjusts the search query and/or adjusts the ranking of resources based on the classification and/or whether the image was successfully classified.

If the context classifier 154 classifies the image into one of the categories, the search engine 152 (or content acquisition unit 116 if the context classifier 154 is implemented on the user device 110) may classify the image. The search query can be tailored to include one or more terms based on the category. For example, if the image is classified as a movie poster, search engine 152 may add the word "movie" to the search query. In a particular example, the user-selected text may be the movie title “Alien”. The context classifier 154 may use additional words drawn on the movie poster, such as the actor's name or the movie description, to classify the image as a movie poster. The search engine 152 may then add the word “movie” to the query “alien” to form the query “alien movie”. In this way, the search results are more likely to be associated with the movie "alien" than to other content generally associated with aliens.

In some implementations, the search engine 152 adjusts the ranking of search results based on the image classification. For example, the search engine 152 may improve search results for resources that are related to image categories (for example, increase search result rank) and/or search results that are not related to image categories, rather than modifying the search query. Can lower the ranking. In some implementations, the search engine 152 may adjust the ranking of resource types based on image categories. For example, if the image category is a music poster, the search engine 152 can boost audio and video resources and lower web page resource rankings.

If the context classifier 154 does not classify the image into one of the categories, the search engine 152 uses the user-selected text to search for the generic text, for example, because the context data is not associated with one of the categories. Search can be performed. This may be an alternative option for images whose category cannot be determined based on the context data received by the context classifier 154. For example, an image of a store sign may include only a single word with no additional contextual text that can be used to classify the image.

The search engine 152 may provide the content 132 to the context acquisition unit 116 based on the search query and the ranking of the resources identified in response to the search query. For example, the search engine 152 may provide the content 132 with a certain number of highest ranked resources. In another example, the search engine 152 may provide the content 132 with search results that reference, eg, link to, the highest ranked resource. For example, as described below, the content may be presented in images, or a search results page may be presented that includes search results selected based on user-selected text. The content acquisition unit 116 can receive the content 132 and provide the content 132 to the user interface controller 118. The user interface controller 118 can then present at least some of the resources included in the content 132 in the user interface 122 along with the images on the search results page or search results. For example, the user interface controller 118 may include, as content, text and/or images extracted from a web page resource, a link to a video hosted by the website, and/or a video in the user interface if selected by the user. A thumbnail image of the video that starts playing can be presented.

The content acquisition unit 116 may also pre-cache the content in local memory, for example, prior to receiving a selection of text drawn on the image. In some implementations, the content acquisition unit 116 sends the query data 130 to the search system 150 based on the image data 115 output by the text processor 114. The query data 130 can include all text identified in the image by the text processor 114. The search engine 152 can identify content based on the text and/or category of the image determined by the context classifier 154. The content acquisition unit 116 can receive the identified content from the search engine 152 and store the content in the local memory 124. In this way, when the user requests content related to the entire image, the content acquisition unit 116 does not send additional query data to the search system 150 via the network 140 after the user request, but rather from the local memory. Content can be acquired. This reduces latency in presenting content to the user and allows the user to instantly, or almost instantaneously, receive additional content related to the selected text (for example, at a time below a threshold such as 1 second). Allows you to browse.

In some implementations, the content acquirer 116 generates different text-based granularity queries for pre-caching based on the zoom level of the user interface 122. The text-based granularity defines the amount of text used to generate each query. For example, in one text-based granularity, individual words may be used to generate a query. At higher text-based granularity, individual phrases or other sets of words can be used up to a threshold number of words to generate queries.

The zoom level allows the content acquisition unit to generate one or more queries for each block of text for which a user-selectable target is presented, if the entire image is viewable in the user interface. For example, text processor 114 may identify multiple distinct blocks of text in an image and request separate content for each block, for example, prior to receiving a user selection of one of the blocks. In a particular example, the image of the menu may include a first block of text for the appetizer, a second block of text for the main dish, and a third block of text for the drink. In this example, the query data 130 may include a list of appetizers in the first block, a list of main dishes in the second block, and a list of drinks in the third block. The search engine 152 can individually identify the content (eg, electronic resource) for each block and provide the content to the content acquisition unit 116. The content acquisition unit 116 can store the content in the local memory 124. In this way, if the user selects one of the blocks, the content acquisition unit 116 will retrieve the local memory 124 from the local memory 124 without sending additional query data to the search system 150 via the network 140 after the user selection. The content of the block selected by the user can be obtained.

The content acquisition unit 116 may update the content stored in the local memory 124 in response to a user interaction with the user interface 122 that changes the zoom level of the user interface 122. For example, when the user selects one of the blocks, the user interface controller 118 may zoom in on the selected block and present at least a portion of the text within the user-selected block as a user-selectable target. .. The content acquisition unit 116 can also generate queries with different text-based granularity based on the updated zoom level. For example, the content acquirer may generate a query for each phrase identified in the selected block or for each phrase in the selected block that the user interface controller 118 presents as a user selectable target. ..

Continuing with the previous menu example, if the user selects the appetizer block, the content acquisition unit 116 may generate one or more queries for each phrase representing the appetizer. The content acquisition unit 116 may, for example, search engine query data 130 that specifies a query before the user selects one of the appetizers or before the appetizer is presented in the user interface 122 as a user-selectable target. Can be sent to 152. The search engine 152 can identify the content for each appetizer and provide the identified content to the content acquisition unit 116. The content acquisition unit 116 can store the content in the local memory 124. In this way, if the user selects one of the appetizers, the content acquisition unit 116 does not send additional query data to the search system 150 via the network 140 after the user has selected the local memory 124. The content of the appetizer selected by the user can be acquired from.

In some implementations, the user interface controller 118 may perform actions other than requesting content associated with the user-selected text based on the content of the text. For example, if the image shows a phone number, the user interface controller 118 can initiate a call on the smartphone in response to the user selecting the phone number display by the smartphone. In another example, if the image shows an address, the user interface controller 118 may open a map application and, in response to a user selection of the address, request that the map application display a map of the location referenced by the address. You can In another example, if the image shows a destination address of a web resource, such as a universal resource locator (URL), the user interface controller 118 opens an application (e.g., a web browser) and the application is instructed to use the electronic address referenced by the network address. You can request to navigate to a resource. In another example, if the image shows an email address for a web resource, the user interface controller 118 may, for example, open an email application and open a blank email address for that email address in response to a user selection of the email address. By displaying the email, an email to that email address can be initiated.

In some implementations, the user interface controller 118 produces a user interface that enhances the content displayed in the viewfinder. For example, the user can point the camera 112 at a menu or other object. The user interface controller 118 generates a user-selectable target for the text block and/or other portion of the text currently displayed in the viewfinder and presents the text in the viewfinder to the user-selectable target. You can When the user moves the camera 112, the user interface controller 118 can move the user-selectable targets along with the text in which they were generated to their proper location. When the new text becomes visible in the viewfinder, the user interface controller 118 generates the user selectable target and renders the text as described above to present the content in response to the user selection of the user selectable target. Can be processed.

FIG. 2 is a diagram illustrating an exemplary screenshot of an exemplary user interface 205 that allows a user to select text to request additional content associated with the user-selected text. A first screenshot 210 of user interface 205 shows user interface 205 presenting an image 211 of a menu from an exemplary restaurant. For example, the user may have used the camera of the smartphone to capture the image 211 of the menu. Exemplary menus include an appetizer section, a sandwich section, a main dish section, and a dessert section.

The second screenshot 220 of the user interface 205 shows user selectable targets 221-224 created for different sections of the menu. For example, a user interface controller, such as user interface controller 118 in FIG. 1, may identify each section of the menu as a block of associated text and present each section as a user-selectable target. Specifically, the user selectable target 221 allows the user to select the appetizer section of the menu, and the user selectable target 222 allows the user to select the main dish section of the menu. Possible targets 223 allow the user to select the sandwich section of the menu, and user selectable targets 224 allow the user to select the dessert section of the menu.

In this example, each user selectable target 221-224 includes a thick line around that section to indicate to the user that a section of the menu is selectable. In some implementations, the lines around each section may be of a particular color and have a particular brightness, in order to draw the user's attention to the fact that the sections may be selected. Good. In other examples, the background color of the text in each section may be highlighted or changed to indicate that the section is selectable, or the text to indicate that the section is selectable. The color of may be changed.

The example user interface 205 has also been updated to include an allergy indicator 227 in the screenshot 220. In some implementations, an image application, eg, image application 111, may access user data provided by the user to present additional content related to the user and related to the text drawn on the image. it can. An example of user data is a list of foods that the user is allergic to. The image application can compare the food in the menu to the list of allergies and, if there is a match, present an allergy indicator next to the item in the menu. In this example, the user is allergic to shrimp and the allergy indicator 227 indicates to the user menu items.

The user can either touch within the user-selectable target (if the user device has a touch screen), click the user-selectable target with the pointer and mouse, or another suitable method. , A user selectable target can be selected. In this example, the user is using the pointer 226 to select the user selectable target 221 and zoom in on the appetizer section of the menu.

In response to the selection of the user-selectable target 221, the user interface 205 is updated to present a zoomed-in view of the appetizer section, as shown in the exemplary screenshot 230. For example, the menu image 211 may be displayed at a first zoom level, eg, a default zoom level. When a user-selectable target is selected, the user interface zoom level is updated to a second zoom level greater than the first zoom level to present the text in the selected section at a higher zoom level. be able to. This allows the user to more appropriately view the text in the selected section and more appropriately select the text that the user wants to zoom in further or request content.

In the exemplary screenshot 230, the user interface 205 also shows each appetizer as a user-selectable target. For example, appetizer chips and salsa are presented as user selectable target 231, appetizer staff mushrooms are presented as user selectable target 232, and appetizer buffalo wings are presented as user selectable target 233. The appetizer mozzarella stick is presented as a user selectable target 234. As described above, when a user selects a block of text, at least a portion of the text within the selected block can be presented as a user selectable target.

When the user selects a user-selectable target, the user interface 205 can be updated to zoom in further or to present content related to the content of the user-selected target. In this example, the user uses the pointer 236 to select the Chips and Salsa user-selectable target 231, and accordingly, to the phrase Chips and Salsa, as shown in the exemplary screenshot 240. The user interface 205 is updated to zoom in.

Also, the user interface 205 has also been updated to present a user-selectable target for the word "phrase and salsa". Specifically, the user interface 205 includes a user selectable target 241 for the word "chips" and a user selectable target 243 for the word "salsa". The user interface 205 may not include the user-selectable target for the word "and", or other words that are unlikely to be selected by the user. For example, as described above, the user interface controller may determine which blocks or other parts of the text are selected based on the content available for the text content (eg, the entity referenced by the text), or other criteria. You can choose whether a word or phrase should be presented as a user-selectable target.

If the user selects a user-selectable target 241, the user interface 205 can be updated to present content related to chips, such as nutrition information for chips, images of chips, recipes for chips, and the like. Similarly, if the user selects a user selectable target 243, the user interface 205 may be updated to present salsa related content, such as salsa nutrition information, salsa images, salsa recipes, and the like. it can.

FIG. 3 is a diagram illustrating example screenshots 310 and 320 of an example user interface 305 that allows a user to select text and view content associated with the selected text. Screenshot 310 is similar to screenshot 230 of FIG. In the screenshot 310, the user interface 305 presents a partial view (eg, a zoomed-in view) of the image of the menu. Specifically, the user interface 305 presents the portion of the image that includes the appetizer section of the menu. The user interface 305 also presents the user selectable targets 311-314 of the appetizer item in the menu. In this example, the user has selected a user selectable target 311 of the appetizer “Chips and Salsa” using pointer 316.

In response to a user selection, the user interface 305 is updated to present chips and salsa related content on the image of the menu, as shown in screenshot 320. In this example, the content includes a text block 322 containing information about chips and salsa and images 323 and 324 of chips and salsa. The text block 322 and images 323 and 324 may be presented over the portion of the image of the menu that was visible in screenshot 310 so that the selected user-selectable target 311 remains visible. For example, text block 322 and images 323 and 324 are presented over portions of the image that depict other appetizers while the chips and salsa text remains visible.

In some implementations, different parts of the image are visible, such that the selected text remains visible and the content associated with the selected text is presented below the selected text. , The user interface 305 can be updated. In some implementations, the visible portion of the image is not changed and the content is presented on a portion of the image that does not include the selected text, eg, above the selected text or on the side of the selected text.

FIG. 4 is a diagram illustrating exemplary screenshots 410 and 420 of an exemplary user interface 405 that allows a user to select text and view content associated with the selected text. Screenshot 410 is similar to screenshot 230 of FIG. In the screenshot 410, the user interface 405 presents a partial view (eg, a zoomed-in view) of the image of the menu. Specifically, the user interface 405 presents the portion of the image that includes the appetizer section of the menu. The user interface 405 also presents the user selectable targets 411-414 of the appetizer item in the menu. The user has selected the user selectable target 411 of the appetizer “Chips and Salsa” using the pointer 416.

In this example, the user interface 405 has been updated to present the search results page 421 as shown in screenshot 420, rather than presenting the content on the image as in the example of FIG. The search results page 421 can be used to initiate a search based on the text selected by the user. For example, the search results page 421 automatically responds to the text "Chips and Salsa" by the image application in response to the user selecting the user selectable target 411 for the text "Chips and Salsa". Contains the entered search box 422. The user then submits the modified query, such as "Chips and Salsa Nutrition Information", and modifies the query in the search box 422 to view the modified query's search results on the search results page 421. can do.

Search results page 421 also includes search results 423-425 identified based on the query "Chips and Salsa". For example, the image application may submit a query to a search engine and submit search results received from the search engine to a search results page in response to detecting the selection of the user selectable target 411. When the user modifies the query and submits the modified query, new search results for the modified query may be presented on the search results page, for example, instead of search results 423-425.

FIG. 5 is a flow chart of an exemplary process 500 for initiating an action based on user-selected textual content. The operations of process 500 may be implemented by a user device including, for example, one or more data processing devices, such as user device 110 of FIG. Process 500 may also be implemented by instructions stored on a computer storage medium that causes a data processing device to perform the operations of process 500 by execution of the instructions by a system that includes the data processing device.

The user device extracts text from the captured image (502). The image may be captured using the camera of the user device, may be captured by the camera of another device sent to the user device, or may be obtained from storage. User devices can use OCR techniques to extract text from captured images.

The user device divides the extracted text into blocks (504). The user device is configured to identify the group of related texts based on the placement of the extracted text within the captured image, based on one or more semantic analyzes of the extracted text, and/or to identify groups of related text. Appropriate techniques can be used to divide the extracted text into blocks of related text.

The user device presents (506) blocks of text as respective first user-selectable targets on a user interface of the user device. The user interface may present the plurality of blocks as a user selectable target at a first zoom level, eg, a default zoom level that displays the entire captured image.

The user device detects 508 a user selection of a first block of the plurality of blocks. The user can interact with the user-selectable target to select text. For example, the user can touch the touch screen of the display of the user device at the location of the text or use the mouse to click the text.

In response to the user selection of text, the user device presents the extracted text in the first block as respective second user selectable targets on the user interface of the user device (510). For example, the user device may update the user interface to zoom in on the first block of text. That is, the user device can display the first block of text at a second zoom level that is greater than the first zoom level. In this way, the text in the first block will be larger and it will be easier for the user to select the appropriate text to request the content associated with that text.

The user device detects a user selection of a portion of the extracted text in the first block of text (512). In response, the user device initiates an action based on the content of the selected text (514). The action may differ depending on the content of the selected text. For example, if the selected text content is a phone number, the action may be to make a call using the phone number. If the selected text content is an address, the action may be to open a map application to that address.

If the selected textual content is an entity or has no corresponding action, the action may be to request content related to the selected textual content. For example, the user device may generate one or more search queries based on the selected textual content and initiate a search using the selected textual content.

In some implementations, the action may be to display a plurality of candidate search queries on the user interface to allow the user to select a favorite search query. The candidate search query can be based on the content of the selected text. For example, a candidate search query may include the selected textual content and one or more suggested terms associated with the selected textual content. When the user selects one of the candidate queries, the user device can submit the query to a search engine and present search results or content related to the selected query to a user interface.

FIG. 6 is a flowchart of an exemplary process 600 for providing content in response to a search query. The operations of process 600 may be implemented, for example, by a system that includes one or more data processing devices, such as search system 150 of FIG. Process 600 may also be implemented by instructions stored on a computer storage medium that causes a data processing device to perform the operations of process 600 by execution of the instructions by a system that includes the data processing device.

The system receives query data (602). Query data may include captured image queries and context data. The query may be drawn in an image and may include textual content selected by the user. The contextual data can include other text drawn on the image but not selected by the user.

The system attempts to classify the image based on the contextual data (604). For example, the system can use one or more machine learning models to attempt to classify an image based on the text of the image contained in the contextual data. The output of the machine learning model may be one of a predefined set of categories (eg, menus, movies, music, billboards, etc.) or unclassified.

If the system determines that the image belongs to one of the categories, the system will use the electronic resource (e.g., web page, image, video, etc.) based on the combination of the user-selected textual content and the category. Are identified and ranked (608). For example, the system can modify the content-based query of the selected text to include one or more terms based on the identified category. In another example, the system may increase the rank of resources associated with categories and/or reduce the rank of resources not associated with categories.

If the system determines that the image does not belong to any of the categories, then the system uses a general search to identify and rank the resources (610). For example, the system cannot modify queries or adjust resource rankings. Instead, the system can identify resources based solely on the content of the selected text.

The system provides search result content for presentation at a user device where the query data was received (612). In some implementations, the search result content may include the resource identified based on the query and optionally the category of the image if the image was classified as belonging to one of the categories. For example, the user device may display content that includes an image for which the text is selected (eg, text on a web page, an image, an embedded video, or a link to a video, etc.).

In some implementations, the search result content includes search results for the resource identified based on the query and, optionally, a category of the image if the image was classified as belonging to one of the categories. For example, the user device may present the search results on a search results page, eg, instead of or over the image.

Embodiments of the subject matter and operations described herein include digital electronic circuitry, or computer software, firmware, or hardware, or the like, including the structures disclosed herein and their structural equivalents. It can be implemented in one or more combinations. Embodiments of the subject matter described herein include one or more computer programs, i.e., computer programs encoded on a computer storage medium, for execution by a data processing device or for controlling operation of the data processing device, i.e. , Can be implemented as one or more modules of computer program instructions. Alternatively or additionally, the program instructions are artificially generated propagated signals, e.g., generated to encode information for transmission to a suitable receiver device for execution by a data processing device. It can be encoded with machine-generated electrical, optical, or electromagnetic signals. The computer storage medium may be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more thereof. Moreover, while the computer storage medium is not a propagated signal, the computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagated signal. Computer storage media may also be and included in one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).

The operations described herein may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or data received from other sources. ..

The term "data processing device" includes, by way of example, programmable processors, computers, systems on a chip, or any number or combination of the foregoing, for any type of device, device, and machine for processing data. Includes. The device may include, for example, a dedicated logic circuit such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus may also include, in addition to hardware, code that creates an execution environment for the computer program in question, such as processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or one of them. Codes making up one or more combinations may also be included. The device and execution environment can implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any form of programming language, including compiled or interpreted, declarative or procedural. It can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. The computer program may, but need not, correspond to files in the file system. This program may be part of a file that holds other programs or data (for example, one or more scripts stored in a markup language document), a single file dedicated to the program in question, or multiple reconciliation files. (Eg, a file that stores one or more modules, subprograms, or portions of code). The computer program can be deployed to run on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network. ..

The processes and logic flows described herein are one or more programmable programs that execute one or more computer programs to perform actions by manipulating input data and producing outputs. It can be executed by a processor. The processes and logic flows may also be performed by special purpose logic circuits, such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits), and devices may be implemented as such.

Suitable processors for the execution of computer programs include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The key elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, the computer also includes, receives data from, or transfers data to, one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks. Operably coupled so that However, the computer does not require such a device. In addition, a computer is another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, just to name a few. (Eg Universal Serial Bus (USB) flash drive). Suitable devices for storing computer program instructions and data include, by way of example, semiconductor memory devices such as EPROMs, EEPROMs, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magnetic disks, and CDs. Includes all forms of non-volatile memory, media and memory devices, including ROM and DVD-ROM discs. The processor and memory may be supplemented by, or incorporated in, dedicated logic circuitry.

To provide user interaction, embodiments of the subject matter described herein include display devices for displaying information to a user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, a user. Can be implemented on a computer with a keyboard and pointing device capable of providing input to the computer, such as a mouse or trackball. Other types of devices may also be used to provide interaction with the user, for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback. The input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, the computer sends and receives documents to and from the device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser. , Can interact with the user.

Embodiments of the subject matter described herein include backend components, eg, as data servers, or middleware components such as application servers, or users interact with implementations of the subject matter described herein. In a computing system that includes front-end components, such as client computers having a graphical user interface or web browser, or one or more such back-end components, middleware components, or any combination of front-end components. Can be implemented in. The components of the system can be interconnected by any form or medium of digital data communication, eg, a communication network. Examples of communication networks include local area networks (“LAN”) and wide area networks (“WAN”), internetworks (eg, the Internet), and peer-to-peer networks (eg, ad hoc peer-to-peer networks).

The computing system can include clients and servers. Clients and servers are typically remote from each other and typically interact through a communications network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client and server relationship to each other. In some embodiments, the server sends data (eg, an HTML page) to a client device (eg, displays the data to a user who interacts with the client device and receives user input from a user who interacts with the client device). In purpose). Data generated at the client device (eg, the result of user interaction) can be received at the server from the client device.

While the specification includes details of many specific implementations, these are not limitations on the scope of the invention or on what is claimed, but as an illustration of features unique to particular embodiments of the particular invention. Should be interpreted. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Further, although features are described above as operating in particular combinations and may be claimed as such initially, one or more features from the claimed combination may optionally be deleted from the combination. Yes, the claimed combinations can be directed to sub-combinations or variations of sub-combinations.

Similarly, although the acts are depicted in a particular order in the drawings, this may be performed in the particular order shown, or in a sequential order, to achieve the desired result. Or should not be understood as requiring that all described acts be performed. Multitasking and parallel processing may be advantageous in certain environments. Furthermore, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the described program components and systems generally represent a single software. It should be understood that it can also be integrated together in a product and packaged in multiple software products. Thus, particular embodiments of the subject matter have been described. Other embodiments are within the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Moreover, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

100 example environments
110 User device
111 Image Application
112 camera
113 images
114 text processor
115 image data
116 Content Acquisition Department
118 User interface controller, content acquisition unit
118 UI controller
119 Request data
120 display
122 User Interface
123 Request data
124 Local memory
130 query data
132 contents
140 data communication network
150 search system
152 search engine
154 Context classifier
156 Index
205 user interface
210 first screenshot
211 images
220 second screenshot
221 User selectable target
222 User selectable target
223 User selectable target
224 User selectable targets
226 pointer
227 Allergy Indicator
230 screenshots
231 User selectable target
232 User selectable target
233 User selectable target
234 User selectable target
236 pointer
240 screenshots
241 User selectable target
243 User selectable target
305 User Interface
310 screenshot
311 to 314 User selectable targets
311 User selectable target
316 pointer
320 screenshots
322 Text Block
323 images
324 images
405 user interface
410 screenshot
411-414 User selectable target
411 User selectable target
416 pointer
420 screenshots
421 Search Results Page
422 Search Box
423 ~ 425 search results
500 processes
600 processes

Claims (60)

Extracting text from the captured image by the user device,
Dividing the extracted text into a plurality of blocks by the user device;
Presenting the block as a respective user selectable target on the user interface of the user device at a first zoom level by the user device;
Detecting a user selection of a first block of the plurality of blocks,
In response to detecting the user selection of the first block, by the user device, at a second zoom level that is greater than the first zoom level, one of the extracted text in the first block. Presenting a section as a respective second user-selectable target on the user interface of the user device,
Responsive to detecting a user selection of a portion of the extracted text in the first block, initiating an action by the user device based on the content of the user selected text. .. The method of claim 1, wherein initiating the action comprises initiating communication to a destination address identified by the user-selected text. The step of initiating the action comprises:
Generating a search query based on the content of the user-selected text;
Sending the search query to a search engine by the user device. The step of initiating the action comprises:
Generating two or more candidate search queries based on the content of the user-selected text;
Displaying the candidate search query in the user interface;
Sending the selected candidate query to a search engine by the user device in response to a user selection of one of the candidate queries. Further comprising the step of extracting context from the image by the user device,
The method of claim 3 or 4, wherein generating the search query or the candidate search query comprises generating the search query or the candidate search query further based on the extracted context of the image. Method. Extracting context from the image by the user device;
Further using the extracted context of the image by the user device to determine whether the image falls into one of a plurality of predefined categories.
The step of generating the search query or the candidate search query, if the image falls into one of the predefined categories, generates the search query or the candidate search query further based on the extracted context. 5. The method of claim 3 or 4, wherein, if not, the step of generating the search query or the candidate search query is based solely on the content of the user-selected text. Before the user selects any of the blocks,
Generating, for at least one block, one or more search queries by the user device based on the extracted text of the block;
Sending the one or more search queries from the user device to a search engine;
7. The method according to any one of claims 1 to 6, further comprising: 8. The method of claim 7, wherein the user device generates and sends the one or more search queries before presenting the block as respective user-selectable targets on the user interface. Receiving search result content from the search engine based on the one or more search queries at the user device;
Storing the received search result content in a local memory of the user device. Generating the one or more search queries based on the extracted text of the block,
Parsing the extracted text of the block into a plurality of first sets of text at a first level text-based granularity;
Generating a search query for each first set of text. In response to detecting the user selection of the block,
Parsing the extracted text of the block into a plurality of second sets of text at a second level of text-based granularity that is greater than the first level of text-based granularity;
Before receiving the user selection of the portion of the extracted text in the first block,
Generating an additional search query for each second set of text,
Sending each additional search query from the user device to the search engine;
Receiving additional search result content based on the additional search query by the user device and from the search engine;
11. The method of claim 10, further comprising storing the additional search result content in a local memory of the user device. 12. The method according to any one of claims 1 to 11, wherein the step of dividing the extracted text into the blocks is based at least in part on a semantic analysis of the extracted text. 12. A method according to any one of claims 3 to 11, or dependent on any one of claims 3 to 11, further comprising displaying on the user interface content received at the user device from a search engine. The method according to item 12. A computer storage medium encoded with instructions that, when executed by one or more computers, causes the one or more computers to perform the method defined in any one of claims 1-13. A processor configured to perform the method as defined in any one of claims 1-13. Extracting text from the image by the user device,
Dividing the extracted text into a plurality of blocks by the user device;
Presenting the plurality of blocks by the user device at a first zoom level on a user interface of the user device as respective first user selectable targets;
Detecting a user selection of a first block of the plurality of blocks,
In response to detecting the user selection of the first block, by the user device, at a second zoom level that is greater than the first zoom level, one of the extracted text in the first block. Presenting a section as a respective second user-selectable target on the user interface of the user device,
Responsive to detecting a user selection of a portion of the extracted text in the first block, initiating an action by the user device based on the content of the user selected text. .. 17. The method of claim 16, wherein initiating the action comprises initiating communication to a destination address identified by the user-selected text. The step of initiating the action comprises:
Generating a search query based on the content of the user-selected text;
Sending the search query to a search engine by the user device. Extracting context from the image by the user device;
Determining that the image belongs to one of a plurality of predefined categories using the extracted context of the image;
19. Responsive to determining that the image belongs to one of the predefined categories, further adjusting the search query based on the extracted context. .. Further comprising receiving search result content selected by the user device and from the search engine based on the search query, the search engine based on the one predefined category to which the image belongs. 20. The method of claim 19 wherein the search results are ranked. Extracting context from the image by the user device;
Further using the context of the extracted image to determine that the image does not belong to one of a plurality of predefined categories,
19. The method of claim 18, wherein generating the search query comprises generating a general search query based solely on the user-selected text. The step of initiating the action comprises:
Generating two or more candidate search queries based on the content of the user-selected text;
Displaying the candidate search query in the user interface;
Sending a user-selected candidate query to a search engine by the user device in response to a user selection of one of the candidate queries. Before the user selects any of the blocks,
Generating, by the user device, one or more search queries for the first block based on the extracted text of the first block;
Sending the one or more search queries from the user device to a search engine;
17. The method of claim 16, further comprising: 24. The method of claim 23, wherein the user device generates and sends the one or more search queries prior to presenting the plurality of blocks as respective first user-selectable targets on the user interface. The method described. Receiving search result content from the search engine based on the one or more search queries;
24. The method of claim 23, further comprising storing the received search result content in a local memory of the user device. Generating the one or more search queries based on the extracted text of the first block,
Parsing the extracted text of the first block into a plurality of first sets of text at a first level text-based granularity;
24. Generating a search query for each first set of text. In response to detecting the user selection of the first block,
Parsing the extracted text of the first block into a plurality of second sets of text at a second level text-based granularity that is greater than the first level text-based granularity;
Before receiving the user selection of the portion of the extracted text in the first block,
Generating an additional search query for each second set of text,
Sending each additional search query from the user device to the search engine;
Receiving additional search result content based on the additional search query by the user device and from the search engine;
27. The method of claim 26, further comprising storing the additional search result content in a local memory of the user device. The method of claim 16, wherein dividing the extracted text into the blocks is based at least in part on a semantic analysis of the extracted text. The method of claim 16, further comprising displaying content received at the user device from a search engine on the user interface. 30. Displaying content received at the user device from the search engine on the user interface comprises displaying the content in a viewport of the user device with at least a portion of the image. The method described. A data processing device,
A memory device in data communication with the data processing device for storing instructions executable by the data processing device, wherein the instructions, when executed in such a way, cause the data processing device to:
Extracting the text from the image,
Splitting the extracted text into a plurality of blocks,
Presenting the plurality of blocks as a respective first user selectable target on a user interface at a first zoom level;
Detecting a user selection of a first block of the plurality of blocks;
In response to detecting the user selection of the first block, the user extracts a portion of the extracted text in the first block at a second zoom level that is greater than the first zoom level. Presenting each as a second user selectable target on the interface;
Responsive to detecting a user selection of a portion of the extracted text in the first block, initiating an action based on the content of the user selected text. And a system comprising. 32. The system of claim 31, wherein initiating the action comprises initiating communication to a destination address identified by the user-selected text. Initiating the action
Generating a search query based on the content of the user-selected text;
32. Sending the search query to a search engine. The operation is
Extracting context from the image,
Determining that the image belongs to one of a plurality of predefined categories using the extracted context of the image;
34. The system of claim 33, further comprising adjusting the search query based on the extracted context in response to determining that the image belongs to one of the predefined categories. .. The operation further comprises receiving from the search engine search result content selected based on the search query, the search engine searching based on the one predefined category to which the image belongs. 35. The system of claim 34, ranking results. The operation is
Extracting context from the image,
Further using the context of the extracted image to determine that the image does not belong to one of a plurality of predefined categories,
34. The system of claim 33, wherein generating the search query comprises generating a general search query based solely on the user-selected text. Initiating the action
Generating two or more candidate search queries based on the content of the user-selected text;
Displaying the candidate search query in the user interface;
32. Sending the selected candidate query to a search engine in response to a user selection of one of the candidate queries. The action is prior to the user selecting any of the blocks,
Generating for the first block one or more search queries based on the extracted text of the first block;
Submitting the one or more search queries to a search engine;
32. The system of claim 31, further comprising: 38. The data processing device generates and sends the one or more search queries prior to presenting the plurality of blocks as respective first user-selectable targets on the user interface. The system described in. The operation is
Receiving search result content from the search engine based on the one or more search queries;
39. The system of claim 38, further comprising storing the received search result content in a local memory of the data processing device. Generating the one or more search queries based on the extracted text of the first block,
Parsing the extracted text of the first block into a plurality of first sets of text at a first level text-based granularity;
39. Generating a search query for each first set of text. The operation is
In response to detecting the user selection of the first block,
Parsing the extracted text of the first block into a plurality of second sets of text at a second level text-based granularity greater than the first level text-based granularity;
Before receiving the user selection of the portion of the extracted text in the first block,
Generate additional search queries for each second set of text, and
Sending each additional search query to the search engine;
Receiving additional search result content based on the additional search query from the search engine;
42. The system of claim 41, further comprising storing the additional search result content in a local memory of the data processing device. 32. The system of claim 31, wherein dividing the extracted text into the blocks is based at least in part on a semantic analysis of the extracted text. 32. The system of claim 31, wherein the operation further comprises displaying content received from a search engine on the user interface. 45. The system of claim 44, wherein displaying content received from the search engine on the user interface comprises displaying the content in a viewport of the data processing device with at least a portion of the image. When executed by the data processing device, the data processing device:
Extracting the text from the image,
Splitting the extracted text into a plurality of blocks,
Presenting the plurality of blocks as a respective first user selectable target on a user interface at a first zoom level;
Detecting a user selection of a first block of the plurality of blocks;
In response to detecting the user selection of the first block, the user extracts a portion of the extracted text in the first block at a second zoom level that is greater than the first zoom level. Presenting each as a second user selectable target on the interface;
Storing instructions for performing an action in response to detecting a user selection of a portion of the extracted text in the first block, initiating an action based on the content of the text of the user selection. A non-transitory computer-readable storage medium. 47. The non-transitory computer-readable storage medium of claim 46, wherein initiating the action comprises initiating communication to a destination address identified by the user-selected text. Initiating the action
Generating a search query based on the content of the user-selected text;
47. Submitting the search query to a search engine, the non-transitory computer readable storage medium of claim 46. The operation is
Extracting context from the image,
Determining that the image belongs to one of a plurality of predefined categories using the extracted context of the image;
The method of claim 48, further comprising adjusting the search query based on the extracted context in response to determining that the image belongs to one of the predefined categories. Temporary computer-readable storage medium. The operation further comprises receiving from the search engine search result content selected based on the search query, the search engine searching based on the one predefined category to which the image belongs. 50. The non-transitory computer readable storage medium of claim 49, which ranks results. The operation is
Extracting context from the image,
Further using the context of the extracted image to determine that the image does not belong to one of a plurality of predefined categories,
49. The non-transitory computer-readable storage medium of claim 48, wherein generating the search query comprises generating a general search query based solely on the user-selected text. Initiating the action
Generating two or more candidate search queries based on the content of the user-selected text;
Displaying the candidate search query in the user interface;
47. In response to user selection of one of the candidate queries, sending the user selected candidate query to a search engine. The action is prior to the user selecting any of the blocks,
Generating for the first block one or more search queries based on the extracted text of the first block;
Submitting the one or more search queries to a search engine;
47. The non-transitory computer readable storage medium of claim 46, further comprising: 54. The data processing device generates and sends the one or more search queries before presenting the plurality of blocks as respective first user-selectable targets on the user interface. A non-transitory computer-readable storage medium as described in. The operation is
Receiving search result content from the search engine based on the one or more search queries;
54. The non-transitory computer readable storage medium of claim 53, further comprising storing the received search result content in a local memory of the data processing device. Generating the one or more search queries based on the extracted text of the first block,
Parsing the extracted text of the first block into a plurality of first sets of text at a first level text-based granularity;
The non-transitory computer-readable storage medium of claim 53, comprising generating a search query for each first set of text. The operation is
In response to detecting the user selection of the first block,
Parsing the extracted text of the first block into a plurality of second sets of text at a second level text-based granularity greater than the first level text-based granularity;
Before receiving the user selection of the portion of the extracted text in the first block,
Generate additional search queries for each second set of text, and
Sending each additional search query to the search engine;
Receiving additional search result content based on the additional search query from the search engine;
57. The non-transitory computer readable storage medium of claim 56, further comprising storing the additional search result content in a local memory of the data processing device. 47. The non-transitory computer readable storage medium of claim 46, wherein dividing the extracted text into blocks is based at least in part on a semantic analysis of the extracted text. 47. The non-transitory computer-readable storage medium of claim 46, wherein the acts further comprise displaying content received from a search engine on the user interface. 60. The non-temporary of claim 59, wherein displaying content received from the search engine on the user interface comprises displaying the content in a viewport of the data processing device with at least a portion of the image. Computer readable storage medium.

Images