JP2011081556A - Information processor, method of processing information, program, and server - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve visibility of information by outputting additional information based on hierarchy information of objects included in images. <P>SOLUTION: An information processor 10 includes: a storage section 210 for relating feature information indicating features of objects within images, hierarchy information indicating a hierarchy relationship among the objects, and additional information added to the objects and stores them; an image acquisition section 202 for acquiring real space images; a recognition section 204 for recognizing objects included in the real space images acquired by the image acquisition section, based on the feature information indicating the features of the objects within the images; and a determination section 206 for determining propriety of output of the additional information added to the objects to a display section, based on the hierarchy information of the objects related to the objects recognized by the recognition section. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an information processing apparatus, an information processing method, a program, and a server.

  Recently, some additional information is superimposed on a captured image as part of augmented reality technology (Augmented Reality). For example, a technique for displaying information around a current position by superimposing it on a real video screen using augmented reality technology is disclosed (for example, Patent Document 1).

  In Patent Document 1, visual feature information is extracted from video data captured by an imaging device such as a camera, and the similarity is calculated by comparison with visual feature information stored in the device. Based on the calculated similarity, an object in the video data is specified, and virtual information is overlaid on the video as annotation (annotation) and output.

Special Table 2004-095374

However, if there are multiple objects in the video data and one object contains multiple objects, and additional information is set for each object, multiple virtual information will be overlaid simultaneously. End up. In this case, there is a problem that a plurality of pieces of additional information are overlapped and information visibility is lowered.
Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to output additional information based on hierarchical information of objects included in an image to improve information visibility. It is an object of the present invention to provide a new and improved information processing apparatus, information processing method, program, and server.

  In order to solve the above-described problem, according to an aspect of the present invention, feature information indicating a feature of an object in an image, hierarchical information indicating a hierarchical relationship of the object, and additional information added to the object are provided. An object included in the real space image acquired by the image acquisition unit based on feature information indicating the characteristics of the object in the image, an image acquisition unit that acquires the real space image, and a storage unit that stores the association A recognizing unit that recognizes, and a determining unit that determines whether to output additional information added to the object to the display unit based on the hierarchical information of the object associated with the object recognized by the recognizing unit. An information processing apparatus is provided.

  A display control unit configured to superimpose the additional information on the object included in the real space image and display the additional information on the display unit when the determination unit determines that the additional information added to the object is output to the display unit; Also good.

  The determining unit determines to output additional information of an object whose hierarchical information is the highest layer among the plurality of objects when the recognizing unit recognizes the plurality of objects in the real space image. May be.

  The determining unit determines to output additional information of an object whose hierarchical information is the lowest layer among the plurality of objects when the recognizing unit recognizes the plurality of objects in the real space image. May be.

Further, the storage unit further stores object priority information in association with each other.
The determination unit may determine to output the additional information of the object based on the priority information when a plurality of objects are recognized in the real space image by the recognition unit.

  Further, the storage unit further stores attribute information of the object by the user in association with each other, and the determination unit, based on the attribute information when the recognition unit recognizes a plurality of objects in the real space image. You may decide to output the additional information of an object.

  The determination unit may determine to output additional information of an object within a predetermined threshold among the plurality of objects when the recognition unit recognizes the plurality of objects in the real space image.

  In addition, when the recognition unit recognizes a plurality of objects in the real space image, the determination unit outputs additional information of a predetermined number of objects among the plurality of objects according to the display size of the display unit. You may decide.

  Further, the recognition unit may not recognize an object that is a lower hierarchy than the object recognized in the real space image.

  In order to solve the above-described problem, according to another aspect of the present invention, feature information indicating characteristics of an object in an image, hierarchical information indicating a hierarchical relationship of the object, and an addition added to the object The information is stored in the storage unit in association with the information, and the object included in the acquired real space image is recognized based on the step of acquiring the real space image and the feature information indicating the characteristics of the object in the image. There is provided an information processing method including a step and a step of determining whether to output additional information added to the object to the display unit based on the hierarchical information of the object associated with the recognized object.

  Furthermore, when the output of the additional information added to the object to the display unit is determined, a step of superimposing the additional information on the object included in the real space image and displaying the additional information on the display unit may be included.

  In order to solve the above problem, according to another aspect of the present invention, a computer adds feature information indicating characteristics of an object in an image, hierarchical information indicating a hierarchical relationship of the object, and the object. Based on feature information indicating features of objects in the image, a storage unit that stores the additional information to be associated with each other, an image acquisition unit that acquires a real space image, and an actual image acquired by the image acquisition unit. A recognition unit that recognizes an object included in the spatial image, and a determination that determines whether to output additional information added to the object to the display unit based on hierarchical information of the object associated with the object recognized by the recognition unit A program for causing the information processing apparatus to function.

  In order to solve the above problem, according to another aspect of the present invention, a receiving unit connected to an information processing apparatus via a network and receiving a real space image captured by the information processing apparatus, Based on the feature information indicating the characteristics of the object, the recognition unit for recognizing the object included in the real space image acquired by the image acquisition unit, and the hierarchical information of the object associated with the object recognized by the recognition unit A determination unit that determines whether or not the additional information added to the object can be output to the display unit; and the determination unit determines whether to output the additional information added to the object to the display unit. And a transmission unit that transmits image information obtained by superimposing additional information on the object included in the information processing apparatus.

  As described above, according to the present invention, it is possible to improve the visibility of information by outputting additional information based on hierarchical information of objects included in an image.

It is explanatory drawing explaining the outline | summary of the information processing apparatus concerning one Embodiment of this invention. It is a block diagram which shows the hardware constitutions of the information processing apparatus concerning the embodiment. It is a block diagram which shows the function structure of the information processing apparatus concerning the 1st Embodiment of this invention. It is explanatory drawing explaining the hierarchy information concerning the embodiment. 3 is a flowchart showing an overall operation of the information processing apparatus according to the embodiment. It is a flowchart which shows the recognition process of the object concerning the embodiment. It is a flowchart which shows the detail of the selection process of the object concerning the embodiment. It is a flowchart which shows the whole operation | movement of the information processing apparatus concerning the 2nd Embodiment of this invention. It is a flowchart which shows the recognition process of the object concerning the embodiment. It is a flowchart which shows the detail of the selection process of the object concerning the embodiment. It is a block diagram which shows the function structure of the information processing apparatus concerning the 3rd Embodiment of this invention. 3 is a flowchart showing an overall operation of the information processing apparatus according to the embodiment. It is a flowchart which shows the determination process of the semantic hierarchical relationship of an object concerning the embodiment. It is explanatory drawing explaining the semantic hierarchical relationship determined by the hierarchical relationship determination part concerning the embodiment.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

Further, the “best mode for carrying out the invention” will be described in the following order.
[1] Purpose of this embodiment [2] Outline of information processing apparatus [3] Hardware configuration of information processing apparatus [4] First embodiment [4-1] Functional configuration of information processing apparatus [4-2] Information Details of Operation of Processing Device [5] Second Embodiment [5-1] Functional Configuration of Information Processing Device [5-2] Details of Operation of Information Processing Device [6] Third Embodiment [6-1] Information Processing Functional configuration of apparatus [6-2] Details of operation of information processing apparatus [7] Determination of availability of additional information output at server

[1] Purpose of this Embodiment First, the purpose of this embodiment will be described. Recently, some additional information is superimposed on a captured image as part of augmented reality technology (Augmented Reality). For example, using augmented reality technology, a technology is disclosed in which information around a current position is displayed superimposed on a real video screen.

  In the above technique, visual feature information is extracted from video data captured by an imaging device such as a camera, and the similarity is calculated by comparison with visual feature information stored in the device. Based on the calculated similarity, an object in the video data is specified, and virtual information is overlaid on the video as annotation (annotation) and output.

  However, if there are multiple objects in the video data and one object contains multiple objects, and additional information is set for each object, multiple virtual information will be overlaid simultaneously. End up. In this case, there is a problem that a plurality of pieces of additional information are overlapped and information visibility is lowered.

  For example, when a far-away building is photographed with the imaging device, the entire photographed building and stores in the building or stores around the building are simultaneously recognized. For this reason, the virtual information for the entire building and the virtual information for each individual store are simultaneously overlaid on the captured image. When the user does not need virtual information for individual stores, the information overlaid on the captured image is excessive and the visibility is lowered.

  Therefore, the information processing apparatus 10 according to the embodiment of the present invention has been created with the above circumstances as a focus. According to the information processing apparatus 10 according to the present embodiment, it is possible to improve the visibility of information by outputting additional information based on hierarchical information of an object included in an image.

[2] Overview of Information Processing Device Next, an overview of the information processing device 10 will be described with reference to FIG. FIG. 1 is an explanatory diagram for explaining an overview of the information processing apparatus 10. Examples of the information processing apparatus 10 include an information processing terminal including a display device such as a mobile phone, a PDA (Personal Digital Assistant), and a small PC (Personal Computer). In the information processing apparatus 10, virtual information 500 added to a real image is registered. As the virtual information 500, an actual image of a building or a store is associated with additional information added to the actual image.

  For example, as illustrated in FIG. 1, additional information “B building” is registered as virtual information 501 of a building corresponding to a real image of the entire view of the building. Further, as store virtual information 502, additional information of each store such as “cafe C”, “restaurant D”, “drug store E”, and “F clinic” is registered corresponding to the actual image of each store. Yes. In addition, as virtual information 503 of the apartment, additional information such as “parking full” and “recruiting vacant residents” is registered for the real image of the apartment.

  The image 550 is an image in which virtual information is overlaid on the captured image by the conventional information processing apparatus 50. In the image 550, virtual information is added to all the real images recognized in the captured image. For example, when stores such as “Cafe C”, “Restaurant D”, “Drug Store E”, “F Clinic” are included in “B Building”, these are located at substantially the same position in the image 550. Additional information will be displayed. Therefore, in the image 550, a plurality of additional information is displayed in an overlapping manner, and the visibility is lowered.

  On the other hand, an image 510 in FIG. 1 is an image in which virtual information is overlaid on a captured image by the information processing apparatus 10 according to the present embodiment. In the image 510, the additional information is not added to all the real images recognized in the captured image, and some additional information selected from the plurality of additional information is displayed. For example, since B building can recognize the entire view of the building, only the building name “B building” is superimposed on the real image, and additional information of a plurality of stores included in B building is not displayed. In this way, rather than superimposing all the additional information of the real image recognized in the captured image, some additional information is selected and displayed, thereby improving the visibility as compared with the image 550. be able to.

  Hereinafter, an additional object of additional information such as a building or a store recognized in a real image is referred to as an object. In the present embodiment, hierarchical information indicating the hierarchical relationship between objects is held. The hierarchical relationship between objects is a semantic hierarchical relationship. For example, when a building includes a store, the building is an upper layer and the store is a lower layer. In the present embodiment, which object additional information is to be displayed is determined from a plurality of additional information based on the hierarchical information of the object included in the image. For example, when the building B and a plurality of stores are in the upper and lower hierarchy relationship, it is determined to display only the hierarchy information of the object in the upper hierarchy.

[3] Hardware Configuration of Information Processing Device The outline of the information processing device 10 has been described above. Next, the hardware configuration of the information processing apparatus 10 will be described with reference to FIG. FIG. 2 is a block diagram illustrating a hardware configuration of the information processing apparatus 10. The information processing apparatus 10 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a host bus 104, a bridge 105, an external bus 106, an interface 107, and the like. , Input device 108, output device 110, storage device (HDD) 111, drive 112, and communication device 115.

  The CPU 101 functions as an arithmetic processing device and a control device, and controls the overall operation in the information processing device 10 according to various programs. Further, the CPU 101 may be a microprocessor. The ROM 102 stores programs and calculation parameters used by the CPU 101. The RAM 103 primarily stores programs used in the execution of the CPU 101, parameters that change as appropriate during the execution, and the like. These are connected to each other by a host bus 104 including a CPU bus.

  The host bus 104 is connected via a bridge 105 to an external bus 106 such as a PCI (Peripheral Component Interconnect / Interface) bus. Note that the host bus 104, the bridge 105, and the external bus 106 are not necessarily configured separately, and these functions may be mounted on one bus.

  The input device 108 is, for example, an input means for a user to input information, such as a mouse, keyboard, touch panel, button, microphone, switch, and lever, and an input that generates an input signal based on the input by the user and outputs it to the CPU 101 It consists of a control circuit. A user of the information processing apparatus 10 can input various data and instruct a processing operation to the information processing apparatus 10 by operating the input device 108.

  The output device 109 includes, for example, a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Display) device and a display device such as a lamp, and an audio output device such as a speaker and headphones. . The output device 109 outputs the reproduced content, for example. Specifically, the display device displays various information such as reproduced video data as text or images. On the other hand, the audio output device converts reproduced audio data or the like into audio and outputs it.

  The storage device 110 is a data storage device configured as an example of a storage unit of the information processing device 10 according to the present embodiment. The storage device 110 is a storage medium, a recording device that records data in the storage medium, and reads data from the storage medium. A reading device and a deleting device for deleting data recorded in the storage medium can be included. The storage device 110 is composed of, for example, an HDD (Hard Disk Drive). The storage device 110 drives a hard disk and stores programs executed by the CPU 101 and various data. The storage device 110 also stores items, identification numbers, etc., which will be described later.

  The drive 111 is a storage medium reader / writer, and is built in or externally attached to the information processing apparatus 10. The drive 111 reads information recorded on a removable storage medium 24 such as a mounted magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs it to the RAM 103.

  The communication device 112 is a communication interface configured with, for example, a communication device for connecting to the communication network 50. The communication device 112 may be a wireless LAN (Local Area Network) compatible communication device, a wireless USB compatible communication device, or a wire communication device that performs wired communication.

[4] First Embodiment The hardware configuration example of the information processing apparatus 10 has been described above with reference to FIG. Next, the functional configuration of the information processing apparatus 10 according to the first embodiment will be described with reference to FIG. FIG. 3 is a block diagram illustrating a functional configuration of the information processing apparatus 10 according to the present embodiment.

[4-1] Functional Configuration of Information Processing Device As illustrated in FIG. 3, the information processing device 10 includes an image acquisition unit 202, a recognition unit 204, a determination unit 206, a display control unit 208, a storage unit 210, and the like.

  The image acquisition unit 202 has a function of acquiring a real space image. The real space image is an image such as a landscape of the information processing apparatus 10 captured by an imaging apparatus (not shown). The imaging device may be configured integrally with the information processing device 10 or may be configured as a separate device. For example, an image captured by an imaging device separate from the information processing device 10 may be stored in a storage device such as a memory card and captured by the image acquisition unit 202. The image acquisition unit 202 provides the acquired real space image to the recognition unit 204 and the display control unit 208.

  The recognition unit 204 has a function of recognizing an object included in the real space image acquired by the image acquisition unit 202 based on feature information indicating the feature of the object in the image. Specifically, feature information of each area of the image divided into the whole or an arbitrary part of the landscape image provided from the image acquisition unit 202 is extracted. Then, the extracted feature information is compared with the feature information of the object stored in the storage unit 210 described later, and the similarity is calculated. If the similarity is greater than or equal to the threshold value, it is determined that the object stored in the storage unit 210 exists in the real space image.

  The recognition unit 204 determines whether or not an object exists in the real space image for all objects that may exist. The recognition unit 204 can use, for example, the technique disclosed in Japanese translations of PCT publication No. 2004-095374 for the object recognition process. The recognition unit 204 provides information on the recognized object to the determination unit 206.

  The determination unit 206 has a function of determining whether to output additional information added to the object to a display unit (not shown) based on the hierarchical information of the object associated with the object recognized by the recognition unit 204. Have. Specifically, it is determined whether or not the object recognized by the recognition unit 204 is stored in the storage unit 210. When the object recognized by the recognition unit 204 exists in the storage unit 210, an object that matches a predetermined condition is determined based on the hierarchical information of the object, and additional information associated with the object is determined. Superimpose on the object.

  Here, the hierarchical information stored in the storage unit 210 will be described with reference to FIG. The hierarchical information stored in the storage unit 210 indicates a semantic hierarchical relationship. As illustrated in FIG. 4, for example, the building group A is the upper hierarchy, the buildings a and b are the intermediate hierarchy, the store 1, 2 and 3 are in a lower hierarchy. The storage unit 210 stores feature information indicating the characteristics of the object, hierarchical information indicating the hierarchical relationship of the object, and additional information added to the object in association with each other. Therefore, when the object recognized by the recognition unit 204 is the building b, the building b is a lower hierarchy of the building group A and the upper hierarchy of the stores 1 to 3 based on the hierarchical information associated with the feature information of the object. It can be seen that it is.

  For example, when the recognition unit 204 recognizes the building a and the store 2 at the same time, the building a becomes the highest layer among the recognized objects. When the determination unit 206 outputs the additional information of the object of the highest hierarchy among the plurality of objects, the building a which is the highest hierarchy among the recognized objects is selected as a target to which the additional information is added. To do. On the other hand, the store 2 is not selected as a target to which additional information is added because the building a that is a higher hierarchy than the store 2 is recognized.

  Further, the determination unit 206 may output additional information of an object in the lowest hierarchy among a plurality of objects. In this case, the store 2 which is the lowest hierarchy among the recognized objects is selected as a target to which additional information is added. On the other hand, since the store 2 which is a lower hierarchy than the building a is recognized, the building a is not selected as a target to which additional information is added.

  Further, when the priority information of the object is associated with the object and stored in the storage unit 210, whether or not the additional information of the object can be output may be determined based on the priority information. For example, the priority information may be determined based on word-of-mouth popularity, registration date / time and update date / time, frequency of referring to an object at the location in the past, and the like. In this case, the determination unit 206 determines to output additional information of an object having a high priority when a plurality of objects are recognized.

  When the attribute information of the object is stored in the storage unit 210 in association with each other, whether or not the additional information of the object can be output may be determined based on the attribute information. For example, the attribute information is information such as when and where to output additional information of an object. For example, the additional information of the object may be output when the current state information of the user matches the attribute information of the object. The current state of the user is, for example, the date / time such as night or noon, birthday or anniversary, the user's service purchase status, the user's moving speed, the community to which the user belongs, and the user's normal life And the like, the presence / absence of hope for new arrival information, user preference information, and the like.

  The storage unit 210 stores attribute values corresponding to the current state of the user as object attribute information. For example, when the value of the user's birthday is stored as attribute information in the object and the object recognized by the recognition unit 204 matches the imaging date and time, the additional information of the object may be superimposed. Good. The storage unit 210 may store setting information for an upper limit value of the number of objects on which additional information is superimposed. The determination unit 206 provides the display control unit 208 with output determination information.

  The display control unit 208 has a function of superimposing additional information associated with the object stored in the storage unit 210 on the object determined by the determination unit 206 and displaying the superimposed information on the display unit. The display control unit 208 generates an image obtained by superimposing the additional information of the object determined by the determination unit 206 on the target object of the real space image on the real space image provided by the image acquisition unit 202 and displays the generated image on the display unit. .

[4-2] Details of Operation of Information Processing Apparatus The functional configuration of the information processing apparatus 10 has been described above. Next, details of the operation of the information processing apparatus 10 will be described with reference to FIGS. FIG. 5 is a flowchart showing the overall operation of the information processing apparatus 10. First, the image acquisition unit 202 acquires an image such as a landscape around the information processing apparatus 10 via the imaging device (S102). Next, the recognition unit 204 recognizes all the objects on the image acquired in step S102 (S104). The object recognition process in step S104 will be described in detail later.

  Then, the determination unit 206 selects an object that overlays (superimposes) the virtual information (additional information) among the objects recognized in step S194 based on the hierarchical information stored in the storage unit 210 (S106). . The object selection process in step S106 will be described in detail later.

  Then, the display control unit 208 generates an image in which virtual information is overlaid on the object and outputs the generated image to the display unit (S108). The overall operation of the information processing apparatus 10 has been described above.

  The operation shown in FIG. 5 is periodically executed according to, for example, the frame rate of the moving image. Further, the object recognition step S104 shown in FIG. 5 and the object selection step for overlaying the virtual information may be executed only once out of several overall operations. As a result, it is possible to reduce the processing load of the object recognition step S104 and the object selection step S106 for overlaying virtual information, and to realize image output at a high frame rate that does not give the user a sense of incongruity.

  Next, the details of the object recognition process will be described with reference to FIG. FIG. 6 is a flowchart showing object recognition processing. As illustrated in FIG. 6, the recognition unit 204 first obtains the visual feature amount of the object that is a recognition candidate stored in the storage unit 210 (S202). The visual feature amount of the object acquired by the recognition unit 204 in step S202 is an example of feature information indicating the feature of the object.

  Next, the recognition unit 204 extracts a partial image of a real space image such as a landscape image acquired by the image acquisition unit 202 (S204). Then, the visual feature amount of the partial image extracted in step S204 is calculated (S206). Then, using the visual feature amount of the object as the recognition candidate acquired in step S202 and the visual feature amount of the partial image acquired in step S206, the similarity between the two is calculated (S208).

  Then, it is determined whether or not the similarity calculation in step S208 has been completed for all regions of the real space image (landscape image) (S210). If it is determined in step S210 that the similarity calculation has not been completed for all regions of the real space image, the processing from step S204 to step S208 is repeated.

  If it is determined in step S210 that the similarity calculation has been completed for all regions of the real space image, it is determined whether or not the maximum similarity calculated in step S208 is equal to or less than a predetermined threshold. (S212). The value indicating the similarity calculated in step S208 becomes a smaller value as the similarity is higher. That is, in step S212, if the maximum value of the similarity is equal to or less than a predetermined threshold, the image extracted from the partial image is similar to the object stored in the storage unit 210 that is a recognition candidate. Become.

  If it is determined in step S212 that the maximum similarity is not more than a predetermined threshold, an object in the image is specified (S214). If it is determined in step S212 that the maximum similarity is greater than a predetermined threshold, the process of step S216 is executed. In step S216, it is determined whether or not processing has been completed for all objects that are recognition candidates acquired in step S202 (S216).

  If it is determined in step S216 that the process has been completed for all objects that are recognition candidates, the object recognition process is terminated. On the other hand, if it is determined in step S216 that the processing has not been completed for all objects that are recognition candidates, the process returns to step S202.

  The details of the object recognition process have been described above. Note that the recognition processing shown in FIG. 6 is an example, and other processing methods can be used if the object on the real space image is recognized using the object feature information stored in the storage unit 210. There may be.

  Next, with reference to FIG. 7, the details of the process of selecting an object that overlays virtual information (additional information) will be described. FIG. 7 is a flowchart showing details of the object selection process. As described above, in the process of selecting an object that overlays virtual information, a selection is made from a plurality of objects recognized using hierarchical information associated with the object. In the present embodiment, when a plurality of recognized objects have a hierarchical relationship, an object that meets a predetermined condition is selected as a virtual information overlay target.

  Here, the predetermined condition includes, for example, a case where the hierarchical information is a semantically highest hierarchy or a semantically lowest hierarchy. In FIG. 7, the predetermined condition is described as being semantically the highest hierarchy.

  First, among the objects recognized in step S104, one object for which no display / non-display mark is set is selected (S222). The display / non-display mark is a mark indicating whether or not to display the additional information of the corresponding object of the real space image. Next, using the hierarchical information associated with the object stored in the storage unit 210, the highest object among the objects recognized in the real space image that is higher than the object selected in step S222. The upper one is extracted (S224).

  Then, the “display” mark is set as the object whose top level is extracted in step S222 as an object for displaying additional information (S228). Then, it is determined whether display / non-display is marked for all objects recognized in the real space image (S230). If display / non-display is marked for all objects in step S230, the process ends. On the other hand, if it is determined in step S230 that display / non-display is not marked for all objects, the processing in steps S222 to S228 is repeated.

  The object selection process has been described above. According to this embodiment, the real space image is obtained by comparing the feature information of the partial image of the acquired real space image with the feature information of the object stored in the storage unit 210 and calculating the similarity between the two. Recognize objects inside. Then, based on the recognized hierarchy information of the object, it is determined whether to superimpose additional information on the object in the real space image. Thereby, for example, only the additional information of the object of the highest hierarchy is superimposed, or only the additional information of the object of the lowest hierarchy is superimposed, and the additional information superimposed on the real space image is appropriately displayed. Visibility can be improved.

[5] Second Embodiment The first embodiment has been described above. Next, a second embodiment will be described. Since the information processing apparatus 10 ′ in the present embodiment has substantially the same configuration as the information processing apparatus 10 according to the first embodiment, functions different from those in the first embodiment will be described in detail. In the first embodiment, recognition processing is executed for all the objects stored in the storage unit 210. However, in this embodiment, the object to be recognized and the object not to be recognized are determined. This is different from the first embodiment.

[5-1] Functional Configuration of Information Processing Device In this embodiment, as in the first embodiment, the determination unit 206 selects an object on which additional information is superimposed based on the hierarchical information of the object. In the present embodiment, the determination unit 206 selects an object on which additional information is to be superimposed, determines an object that is not a recognition target from the hierarchy information and a predetermined condition, and notifies the recognition unit 204 of it.

  For example, it is assumed that the building a is recognized by the recognition unit 204 when the predetermined condition is a condition that the additional information of the highest object in the hierarchy is superimposed. in this case. The determination unit 206 instructs the recognition unit 204 not to recognize an object in a lower hierarchy than the building a.

[5-2] Details of Operation of Information Processing Device Next, details of the operation of the information processing device 10 ′ will be described with reference to FIGS. 8 to 10. FIG. 8 is a flowchart showing the overall operation of the information processing apparatus 10 ′. First, the image acquisition unit 202 acquires an image such as a landscape around the information processing apparatus 10 ′ via the imaging device (S302). Next, the recognition unit 204 recognizes one object on the image acquired in step S202 (S304). The object recognition process in step S104 will be described in detail later.

  Then, the determination unit 206 determines an object to execute object recognition processing from the hierarchical information stored in the storage unit 210 and the object recognition result in step S204 (S306). The object selection process in step S306 will be described in detail later.

  Then, it is determined whether or not the processing has been completed for all recognition candidate objects (S308). If it is determined in step S308 that the processing has not been completed for all recognition candidate objects, the processing in steps S304 to S308 is repeated. If it is determined in step S308 that the processing has been completed for all recognition candidate objects, the determination unit 206 notifies the display control unit 208 of the display / non-display mark result set for the object.

  Then, the display control unit 208 generates an image in which virtual information is overlaid on the object and outputs the generated image to the display unit (S310). The overall operation of the information processing apparatus 10 ′ has been described above.

  Next, the details of the object recognition process will be described with reference to FIG. FIG. 9 is a flowchart showing object recognition processing. As shown in FIG. 9, the recognition unit 204 first acquires one of the visual feature amounts of an object that is a recognition candidate stored in the storage unit 210 (S402). The determination unit 206 specifies which object's visual feature value is to be acquired in step S402. If not specified by the determining unit 206, the recognizing unit 204 has not yet acquired the visual feature amount, and the object is not instructed by the determining unit 206 not to perform the recognition process. Select any one.

  Next, the recognition unit 204 extracts a partial image of a real space image such as a landscape image acquired by the image acquisition unit 202 (S404). Then, the visual feature amount of the partial image extracted in step S404 is calculated (S406). Then, using the visual feature amount of the object as the recognition candidate acquired in step S402 and the visual feature amount of the partial image acquired in step S406, the similarity between the two is calculated (S408).

  Then, it is determined whether or not the similarity calculation in step S408 has been completed for all regions of the real space image (landscape image) (S410). If it is determined in step S410 that the similarity calculation has not been completed for all regions of the real space image, the processing from step S404 to step S408 is repeated.

  If it is determined in step S410 that the similarity calculation has been completed for all regions of the real space image, it is determined whether the maximum similarity calculated in step S408 is equal to or less than a predetermined threshold. (S412). If it is determined in step S412 that the maximum similarity is not more than a predetermined threshold, an object in the image is specified (S414). If it is determined in step S412 that the maximum similarity value is greater than the predetermined threshold value, the process ends.

  The details of the object recognition process have been described above. Next, with reference to FIG. 10, the details of the process of selecting an object to overlay virtual information will be described. FIG. 10 is a flowchart showing details of the object selection processing. In the following, a case will be described in which when a plurality of recognized objects have a hierarchical relationship, an object having the highest hierarchical relationship is selected as a target for overlaying virtual information. However, the present invention is not limited to this example, and the object having the lowest hierarchical relationship may be selected as a virtual information overlay target.

  First, it is determined whether or not an object as a recognition candidate is in the real space image (S422). If it is determined in step S422 that an object that is a recognition candidate is in the real space image, an object that is semantically higher than the object that is a recognition candidate is extracted from the storage unit 210. In the extraction process of step S424, it is determined whether or not an object that is semantically higher than the recognition candidate object is stored in the storage unit 210 (S426). If it is determined in step S426 that there is a semantically superior object, the semantically superior object is set as a new recognition candidate object.

  On the other hand, if it is determined in step S422 that the recognition candidate object is not in the real space image, or if it is determined in step S426 that there is no semantically superior object, the last recognized object Are marked as display objects on which additional information is superimposed (S430). In step S430, an object that is semantically lower than the object marked as a display object is marked as a non-display object. Here, an object marked as a non-display object is an object for which recognition processing is not performed.

  The details of the object selection process have been described above. According to the present embodiment, it is possible to limit the target to be recognized using the hierarchical information associated with the object to objects that may superimpose additional information. Thereby, it is possible to reduce the calculation time of the entire recognition process as compared with the first embodiment in which the recognition process is performed on all objects stored in the storage unit 210.

[6] Third Embodiment The second embodiment has been described above. Next, a third embodiment will be described. The information processing apparatus 15 according to the present embodiment is different from the information processing apparatus 10 according to the first embodiment and the information processing apparatus 10 ′ according to the second embodiment in that a function of inputting virtual information is added. Yes. Hereinafter, differences from the first embodiment and the second embodiment will be described in detail.

[6-1] Functional Configuration of Information Processing Device The functional configuration of the information processing device 15 according to the present embodiment will be described with reference to FIG. As illustrated in FIG. 11, the information processing apparatus 15 includes an image acquisition unit 202, a recognition unit 204, a determination unit 206, a display control unit 208, a storage unit 210, an additional information input unit 212, a hierarchy relationship determination unit 214, and the like. .

  Since the image acquisition unit 202, the recognition unit 204, the determination unit 206, the display control unit 208, and the storage unit 210 have the same functions as those in the first embodiment, detailed description thereof is omitted.

  The additional information input unit 212 has a function of inputting additional information in response to a user operation. The additional information is virtual information such as character information superimposed on the real space image. The additional information input unit 212 is an example of the input device 108 described above, and is realized using a keyboard, voice input, a touch panel, or the like.

  In addition to the information described in the first embodiment, the storage unit 210 also stores visual information of the divided partial areas of the object. The visual information of the partial area of the object is used by the recognition unit 204.

  In addition to the functions described in the first embodiment, the recognition unit 204 includes a part of an object stored in the storage unit 210 in a real space image such as a landscape image input from the image acquisition unit 202. It also has a function of recognizing that Specifically, this is realized by calculating the similarity between the entire or partial area of the landscape image and the entire or partial visual information stored in the storage unit 210. The recognition result by the recognition unit 204 is output to the determination unit 206 and simultaneously to the hierarchy relationship determination unit 214.

  The hierarchical relationship determining unit 214 has a function of determining the hierarchical relationship of the object to which the additional information input by the additional information input unit 212 is added. Additional information input by the user from the additional information input unit 212 is provided, and the recognition result of the object is provided from the recognition unit 204. The hierarchical relationship determination unit 214 uses the recognition result provided from the recognition unit 204 to determine the hierarchical relationship of objects.

  Specifically, first, the hierarchical relationship determination unit 214 determines whether or not a recognized object exists in the real space image. When a recognized object exists in the real space image acquired by the image acquisition unit 202, the area ratio of the recognized object in the real space image and the visual in the storage unit 210 that matches the object are displayed. A semantic hierarchical relationship is determined using the area ratio of the information to the entire object. A method of determining the hierarchical relationship by the hierarchical relationship determining unit 214 will be described in detail later. The hierarchical relationship of the objects determined by the hierarchical relationship determining unit 214 and additional information input by the user are stored in association with the storage unit 210.

[6-2] Details of Operation of Information Processing Device The functional configuration of the information processing device 15 has been described above. Next, details of the operation of the information processing apparatus 15 will be described with reference to FIGS. 12 and 13. In the following, processes different from those in the first embodiment and the second embodiment will be described in detail, and detailed description of similar processes will be omitted.

  FIG. 12 is a flowchart showing the overall operation of the information processing apparatus 15. First, the image acquisition unit 202 acquires an image such as a landscape around the information processing device 15 via the imaging device (S502). Next, the recognition unit 204 recognizes all objects on the image acquired in step S502 (S504).

  Then, the hierarchical relationship determining unit 214 determines the semantic hierarchical relationship of the object to which the additional information input by the user is added (S506). The process of determining the semantic hierarchical relationship of objects in step S506 will be described in detail later. Then, the semantic hierarchical relationship determined in step S506 is added or updated to the storage unit 210, and the input additional information is added (S508).

  The overall operation of the information processing apparatus 15 has been described above. Next, with reference to FIG. 13, a process for determining the semantic hierarchical relationship of objects will be described. FIG. 13 is a flowchart showing a process for determining the semantic hierarchical relationship of objects. As shown in FIG. 13, first, it is determined whether or not an object recognized by the recognition unit 204 exists in the real space image (landscape image) (S602).

  If it is determined in step S602 that the recognized object exists, the area ratio of the recognized object and the area of the visual information in the storage unit that matches the object with respect to the entire object in the processing after step S604. The semantic hierarchical relationship shown in FIG. 14 is determined using the rate. FIG. 14 is an explanatory diagram for explaining the semantic hierarchical relationship determined by the hierarchical relationship determining unit 214. By executing the processing after step S602, the semantic hierarchical relationship of FIG. 14 is determined.

  On the other hand, if it is determined in step S602 that the recognized object does not exist, it is determined that there is no semantic hierarchical relationship with the object stored in the storage unit 210 (S606).

  In step S604, when the recognition unit 204 recognizes a plurality of objects in the real space image, the object having the highest area ratio in the real space image is extracted (S604). The object extracted in step S604 is referred to as object X and will be described below.

  Next, it is determined whether or not the screen occupation ratio of the object X is equal to or greater than a predetermined threshold (S608). If it is determined in step S608 that the screen occupation ratio of the object X is equal to or greater than a predetermined threshold value, the process of step S612 is executed. In step S612, the visual information having the maximum image occupancy rate (hereinafter referred to as reference Y) is extracted from the visual information in the storage unit 210 referred to when the object X is recognized (S612). Here, the image occupancy is an area ratio of visual information with respect to an image used for recognition when visual information is stored in the storage unit 210.

  On the other hand, if it is determined in step S608 that the screen occupancy rate of the object X is not equal to or greater than the predetermined threshold value, the process of step S610 is executed. In step S610, the object to which the user intends to add additional information is determined to be unregistered in the storage unit 210 and semantically higher than the recognized object.

  After executing the process of step S612, it is determined whether or not the screen occupancy rate of the reference Y is equal to or greater than a threshold value (S614). If it is determined in step S614 that the screen occupancy rate of reference Y is equal to or greater than the threshold value, the process of step S618 is executed. In step S618, it is determined that the object to which the user is to add additional information is the object X and is the same as the reference Y.

  On the other hand, if it is determined in step S614 that the screen occupancy rate of reference Y is not equal to or greater than the threshold value, the process of step S616 is executed. In step S616, the object to which the user intends to add additional information is the object X, and is determined to be a hierarchically lower layer than the reference Y.

  The processing for determining the semantic hierarchical relationship of objects has been described above. According to the third embodiment, in addition to the additional information held in advance in the information processing apparatus 15, it is possible to arbitrarily add unique additional information to the object in accordance with user input. Thereby, it is possible to superimpose information expected by the user on the real space image, and the convenience for the user is improved.

[7] Determination of availability of additional information output in server The third embodiment has been described above. The recognition unit 204, determination unit 206, storage unit 210, and hierarchical relationship determination unit 214 in the first to third embodiments described above are provided in a server connected to the information processing apparatuses 10, 10 ', and 15 via a network. It may be. By providing the recognition unit 204, the determination unit 206, the storage unit 210, and the hierarchy relationship determination unit 214 in the server, it is possible to perform a process such as recognition of an object with a high load on the server having a high calculation capacity and improve the response. It becomes. Further, it is easy to increase the capacity of the storage unit 210, and it becomes possible to refer to additional information of an object among a plurality of users, thereby improving user convenience.

  The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

  In addition, each step in the processing of the information processing apparatuses 10, 10 ', and 15 in this specification does not necessarily have to be processed in time series in the order described as a flowchart. That is, each step in the processing of the information processing apparatuses 10, 10 ′, and 15 may include processing executed in parallel or individually (for example, parallel processing or object processing).

  In addition, hardware such as a CPU, ROM, and RAM incorporated in the information processing apparatuses 10, 10 ', 15 and the like is used to exhibit functions equivalent to those of the above-described configurations of the information processing apparatuses 10, 10', and 15 Computer programs can also be created. A storage medium storing the computer program is also provided.

10, 10 ', 15 Information processing device 202 Image acquisition unit 204 Recognition unit 206 Determination unit 208 Display control unit 210 Storage unit 212 Additional information input unit 214 Hierarchy relationship determination unit

Claims (13)

A storage unit that stores feature information indicating features of an object in the image, hierarchical information indicating a hierarchical relationship between the objects, and additional information added to the object in association with each other;
An image acquisition unit for acquiring a real space image;
A recognition unit for recognizing the object included in the real space image acquired by the image acquisition unit based on feature information indicating a feature of the object in the image;
A determination unit that determines whether to output the additional information added to the object to the display unit based on the hierarchical information of the object associated with the object recognized by the recognition unit;
An information processing apparatus comprising:   A display control unit that causes the display unit to display the additional information superimposed on the object included in the real space image when the output of the additional information added to the object to the display unit is determined by the determination unit The information processing apparatus according to claim 1, comprising:   The determining unit outputs additional information of an object whose hierarchical information is the highest layer among the plurality of objects when the recognizing unit recognizes the plurality of the objects in the real space image. The information processing apparatus according to claim 1, which determines this.   The determining unit outputs additional information of an object whose hierarchical information is the lowest hierarchy among the plurality of objects when the recognizing unit recognizes the plurality of objects in the real space image. The information processing apparatus according to claim 1, which determines this. In the storage unit, the priority information of the object is further stored in association with each other,
The determination unit determines to output the additional information of the object based on the priority information when a plurality of the objects are recognized in the real space image by the recognition unit. The information processing apparatus described in 1. In the storage unit, attribute information of the object by the user is further stored in association with each other,
The determination unit determines to output the additional information of the object based on the attribute information when the recognition unit recognizes a plurality of the objects in the real space image. The information processing apparatus described.   The determination unit determines to output the additional information of the object within a predetermined threshold among the plurality of objects when the recognition unit recognizes the plurality of objects in the real space image. The information processing apparatus according to claim 1.   The determination unit, when the recognition unit recognizes a plurality of the objects in the real space image, according to a display size of the display unit, the additional information of the predetermined number of the objects among the plurality of objects. The information processing apparatus according to claim 1, wherein the information processing apparatus determines to output the information.   The information processing apparatus according to claim 1, wherein the recognition unit does not recognize the object that is in a lower hierarchy than the object recognized in the real space image. Feature information indicating the characteristics of the object in the image, hierarchical information indicating the hierarchical relationship of the object, and additional information added to the object are stored in association with each other,
Obtaining a real space image;
Recognizing the object included in the acquired real space image based on feature information indicating a feature of the object in the image;
Determining whether to output the additional information added to the object to the display unit based on the hierarchical information of the object associated with the recognized object;
Including an information processing method.   And a step of superimposing the additional information on the object included in the real space image and displaying the additional information on the display unit when output of the additional information added to the object to the display unit is determined. Item 11. The information processing method according to Item 10. Computer
A storage unit that stores feature information indicating features of an object in the image, hierarchical information indicating a hierarchical relationship between the objects, and additional information added to the object in association with each other;
An image acquisition unit for acquiring a real space image;
A recognition unit for recognizing the object included in the real space image acquired by the image acquisition unit based on feature information indicating a feature of the object in the image;
A determination unit that determines whether to output the additional information added to the object to the display unit based on the hierarchical information of the object associated with the object recognized by the recognition unit;
A program for causing an information processing apparatus to function. Connected to an information processing device via a network,
A receiving unit that receives a real space image captured by the information processing apparatus;
A recognition unit for recognizing the object included in the real space image acquired by the image acquisition unit based on feature information indicating a feature of the object in the image;
A determination unit that determines whether to output the additional information added to the object to the display unit based on the hierarchical information of the object associated with the object recognized by the recognition unit;
When the determination unit determines output of additional information added to the object to the display unit, image information in which the additional information is superimposed on the object included in the real space image is transmitted to the information processing apparatus. A transmitter to
Comprising a server.

Images