JP2017211720A - Image retrieval device, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately retrieve the same object robustly against brightness variations of the object from a query image imaging the object having a texture.SOLUTION: A correspondence establishment part 12 performs positioning between a first query image and a second query image, and makes a pixel of the second query image correspond to one or more pixels of the first query image. An image synthesizing part 13 selects a pixel of interest from the first query image. When the corresponding pixel of the second query image corresponding to the selected pixel of interest exists, a brightness value of the selected pixel of interest and the brightness value of the corresponding pixel of the second query image are compared. When the brightness value of the corresponding pixel is lower than the brightness value of the pixel of interest, the pixel value of the corresponding pixel is updated by the pixel value of the corresponding pixel, thereby generating a synthesized image. A retrieval part 14 outputs a reference image within the reference image database which is closest to the synthesized image.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image search apparatus, method, and program, and more particularly, to an image search apparatus, method, and program for accurately searching for the same object from a query image obtained by photographing a textured object.

  The progress of object recognition technology is remarkable. Until now, the key points of use, such as face / fingerprint authentication and factory automation, where the recognition target and environment are limited have been the focus. Recently, with the spread of small imaging devices such as smartphones, there has been an increase in industrial demands for recognition of objects from free-shot images such as images of arbitrary objects taken by general users in free places and environments. ing. Expectations are especially high for O2O services that connect products in the real world and the web world, and information guidance / navigation services that recognize various landmarks in the real environment and provide information.

  There can be several forms of object recognition technology for such new applications, but one of the typical ones is object retrieval. The following outlines a typical procedure for object retrieval. First, by analyzing the luminance value of each image, a large number of minute key points (called key points) having a characteristic luminance distribution are extracted, and each key point is expressed by the amount of change in luminance ( Called local features). Next, by measuring the distance between the local feature quantities included in two different images, the correspondence between key points between different images is taken, and the pairs with many correspondences show the same object. Assume it is an image.

  Assume that a database of images (reference images) obtained by capturing an object to be recognized in advance is constructed. At this time, an object present in the query image is specified by searching for a reference image in the database that contains the same object as the captured query image.

  One of the greatest features of object search is to express an image as a set of one or more minute key points (and local feature values describing it). Even if it is just an image of the same object, it is not captured in the same position, posture (small key point angle), and size in every image, but is captured in various ways depending on the image. Is normal. In addition, it is almost impossible to know in advance how to capture an object in an image that a general user freely shoots. However, it is desirable that the feature vector describing the image has invariance independent of the position, orientation, and size.

  It is difficult to obtain the desired invariance with a global feature amount in which an entire image is represented by a single vector. For example, the arrangement of the color (RGB value) of each pixel in a vector is not invariant with respect to position, orientation, and size. On the other hand, an abstraction of some information, such as a color histogram, can have invariance to position and orientation, but is not invariant to size. In addition, the accuracy is easily lowered, for example, the object is weak even when a part of the object is missing.

  On the other hand, in object search, an image is expressed by a set of minute key points. Since these are sets of key points, they are invariant to position. In addition, local feature quantities that describe key points have been invented that have invariance with respect to posture and size. For example, Scale Invariant Feature Transform (SIFT) described in Non-Patent Document 1 is a representative example.

  As described above, according to a typical procedure for object search, an image is represented by a set of one or more key points. You can search.

JP-A-2005-70026

D.G.Lowe, “Distinctive Image Features from Scale-Invariant Keypoints”, International Journal of Computer Vision, pp.91-110, 2004 J. Philbin, O. Chum, M. Isard, Josef Sivic and Andrew Zisserman.Object retrieval with large vocabularies and fast spatial matching 1470-1477, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2007.

  However, there is a problem with object search. Usually, an object in real space is placed under various light source environments, and a captured object image is naturally affected by reflection by the light source. In particular, it is known that an object having a smooth surface causes specular reflection. Since specular reflection is often observed as very strong luminance, the degree of luminance change inherent in the object is often changed or obscured. As a result, in the object search, an incorrect correspondence occurs, and a different object may be searched.

  In view of such problems, several inventions have been made.

  Non-Patent Document 2 discloses an image search method based on geometric verification of key points. This is a method based on a reasonable assumption that the same object has the same spatial distribution of key points, except for changes in shooting viewpoint. First, after taking correspondences between keypoints between different images, only correspondences in which the spatial geometric relationship when multiple correspondences are viewed as a set are constrained by a specific linear transformation are effective. Deletes the correspondence that is not valid. As a result, images with a large number of effective correspondences are ranked higher in the search results.

  Patent Document 1 discloses an apparatus that removes specular reflection due to the influence of illumination light. In advance, a reference image and an illumination light component when the reference image is photographed are measured (using an image obtained by photographing a white plate). When actually shooting an object, both the query image that captures the object and the image that captured the white plate are acquired, and the query image is projected onto a space orthogonal to the illumination light component to eliminate specular reflection. Generated images.

  As disclosed in Non-Patent Document 1 and Non-Patent Document 2, the existing techniques are both image search techniques based on simple keypoint matching, but components that avoid erroneous correspondence due to specular reflection. I do not have. As described above, the key points are determined based on the change in luminance, and the local feature value describes the change in luminance. Therefore, the key points are directly affected by specular reflection, resulting in a large deterioration in accuracy. The point is to do.

  Moreover, if the specular reflection removal method as disclosed in Patent Document 1 is applied, an image can be converted into an image having no specular reflection. On the other hand, the technique described in Patent Document 1 must include a camera that always captures a white plate when capturing both a reference image and a query image. In addition, it is necessary to always accumulate images obtained by photographing a white plate, and the number of necessary images increases. From a global perspective, existing specular reflection removal methods cannot be used unless the light source or imaging device is equipped with special equipment that is not available in general-purpose cameras, or requires a very large number of images. If the image search is based on the inquiry image freely photographed by the general user as described above due to reasons such as processing time, etc., it is not necessarily suitable. .

  As described above, until now, there has not been invented a technique capable of effectively retrieving an image of an object image that causes specular reflection that is freely photographed.

  The present invention has been made to solve the above problems, and searches for the same object with high accuracy and robustness against a change in luminance of an object from a query image obtained by photographing an object having a texture. An object of the present invention is to provide an image search apparatus, method, and program that can be used.

  In order to achieve the above object, an image search apparatus according to a first invention includes a reference image database in which at least one reference image is accumulated, and a first query image obtained by photographing the same object from different viewpoints, Image search for receiving at least two query images and at least one second query image and outputting a reference image in the reference image database including the same object as the object appearing in the first query image An apparatus for performing alignment between the first query image and the second query image, and for the one or more pixels of the first query image, the second query image A correspondence determination unit that associates the selected pixel, and a target pixel is selected from the first query image, and if there is a corresponding pixel of the second query image corresponding to the selected target pixel, the selection is performed. The luminance value of the target pixel is compared with the luminance value of the corresponding pixel of the second query image. When the luminance value of the corresponding pixel is lower than the luminance value of the target pixel, the pixel value of the target pixel Is updated with the pixel value of the corresponding pixel, and includes an image composition unit that generates a composite image, and a search unit that outputs a reference image in the reference image database that is closest to the composite image. ing.

  Further, in the image search device according to the first invention, the correspondence determination unit is configured to perform the first operation based on the key points extracted from the first query image and the key points extracted from the second query image. The key points of one query image and the key points of the second query image are associated with each other, and based on the correspondence between the key points, the corresponding key point pixels of the first query image and the second query image A linear transformation matrix for transforming from one of the key image pixels of the query image to the other is obtained, and the image composition unit calculates the coordinates of the second query image based on the linear transformation matrix. In the region where the first query image and the second query image overlap on the converted coordinates, the target pixel from the first query image Select and compare the luminance value of the selected pixel of interest with the luminance value of the corresponding pixel of the second query image, and if the luminance value of the corresponding pixel is lower than the luminance value of the pixel of interest, A composite image may be generated by updating the pixel value of the target pixel with the pixel value of the corresponding pixel.

  An image composition method according to a second invention comprises a reference image database storing at least one reference image, a first query image obtained by photographing the same object from different viewpoints, and at least one second image. An image search method in an image search apparatus that receives a query image and at least two query images and outputs a reference image in the reference image database including the same object as the object reflected in the first query image. The correspondence determination unit performs alignment between the first query image and the second query image, and the second query is performed on one or more pixels of the first query image. A step of associating the pixels of the image, and the image composition unit selects a target pixel from the first query image, and there is a corresponding pixel of the second query image corresponding to the selected target pixel If the luminance value of the corresponding pixel is lower than the luminance value of the target pixel, the luminance value of the selected target pixel is compared with the luminance value of the corresponding pixel of the second query image. Updating a pixel value of the target pixel with a pixel value of the corresponding pixel, and generating a composite image; and a step of outputting a reference image in the reference image database closest to the composite image by the search unit And executing.

  In the image search method according to the second invention, the step of causing the correspondence determining unit to correspond is based on key points extracted from the first query image and key points extracted from the second query image. The key points of the first query image are associated with the key points of the second query image, and based on the correspondence between the key points, the corresponding key point pixels of the first query image and A step of obtaining a linear transformation matrix for transformation from any one of the key point pixels of the second query image to the other, and the step of synthesizing the image synthesis unit, based on the linear transformation matrix, The coordinates of the query image are converted into the coordinates of the first query image, and the first query image and the second query image overlap with each other on the converted coordinates. The target pixel is selected from the first query image, the brightness value of the selected target pixel is compared with the brightness value of the corresponding pixel of the second query image, and the brightness value of the corresponding pixel is When the luminance value is lower than the luminance value of the target pixel, the synthesized image may be generated by updating the pixel value of the target pixel with the pixel value of the corresponding pixel.

  A program according to the third invention is a program for causing a computer to function as each part of the image search device according to the first invention.

  According to the image search device, method, and program of the present invention, alignment is performed between the first query image and the second query image, and one or more pixels of the first query image are detected. , When the corresponding pixel of the second query image is selected, the target pixel is selected from the first query image, and there is a corresponding pixel of the second query image corresponding to the selected target pixel, When the luminance value is compared with the luminance value of the corresponding pixel of the second query image and the luminance value of the corresponding pixel is lower than the luminance value of the target pixel, the pixel value of the target pixel is set to the pixel value of the corresponding pixel. To generate a composite image, and output the reference image in the reference image database that is closest to the composite image, so that robustness against changes in the brightness of the object can be obtained from the query image obtained by capturing a textured object. Or , It is possible to accurately find the same object, the effect is obtained that.

It is a figure explaining the principle of the position alignment and image composition which concern on embodiment of this invention. It is a block diagram which shows the structure of the image search device which concerns on embodiment of this invention. It is a flowchart which shows the image search process routine in the image search device which concerns on embodiment of this invention. It is a figure explaining the image composition process which concerns on embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Principle according to the embodiment of the present invention>

  First, the principle in the embodiment of the present invention will be described with reference to FIG.

  It is known that the radiance of an object can be modeled as the sum of a diffuse reflection component that usually represents the object's original appearance and a specular reflection component that occurs slightly outside the object surface, and is called a dichroic reflection model. . In the case of complete diffuse reflection, the diffuse reflection component is observed as a constant intensity regardless of the viewpoint (direction) with respect to the parallel light from the point light source, but the latter changes in intensity depending on the viewpoint. Also, if it is not parallel light, the position at which specular reflection appears also changes depending on the light source position and the viewpoint. Therefore, if at least two query images having different viewpoints are obtained, the specular reflection areas have different positions and intensities on the object appearing in the two images, such as image 1 and image 2 shown in FIG. Appears at

  Here, the two images of image 1 and image 2 are aligned, that is, one of the images (here, image 2) is converted so that the objects appearing in the two images are exactly overlapped. Think. For alignment of two different images, for example, alignment based on key points described in Non-Patent Document 2 can be performed. Often, the specular reflection region often has a uniformly high luminance value, and since the inside of the specular reflection region is poor in luminance change, it is difficult to detect key points from within the region. Therefore, when the alignment based on the key points is applied to the two images as described above, it is detected mainly from the diffuse reflection area (that is, the original object appearance) as shown in FIG. Alignment is performed based on the key points. In FIG. 1, a broken line across image 1 and image 2 is illustrated by connecting corresponding key points.

  Considering that the image 2 is converted and aligned with the image 1 based on the result of this alignment, the pixel value of the image 1 and the corresponding value of the image 2 are associated with each pixel of the aligned composite image. ) Arbitraryness arises as to which pixel value to take, so we want to take the pixel value with the weaker specular reflection. Based on the previous dichroic reflection model, the radiance of the image can be expressed by the sum of the diffuse reflection component and the specular reflection component, and it can be said that the portion having the specular reflection component has higher luminance. If the pixel having the lower luminance among the pixels of the image 2 is employed, the image 3 in which the specular reflection component is suppressed can be synthesized.

  By searching the image 3 synthesized in this way as a query image, an inquiry without ambiguity due to specular reflection becomes possible, and a more accurate search can be executed.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

<< Overall structure >>

  FIG. 2 is a block diagram showing an example of the configuration of the image search apparatus 11 according to the embodiment of the present invention. The image search apparatus 11 illustrated in FIG. 2 includes a correspondence determination unit 12, an image composition unit 13, and a search unit 14.

  The image search device 11 is connected to the reference image database 15 via a communication unit and communicates information with each other, and can register or read a feature amount of an arbitrary image in the reference image database 15. I take the. The reference image database 15 can be configured by, for example, a file system mounted on a general general-purpose computer. Assume that an identifier (for example, an ID based on a serial number or a unique image file name) that uniquely identifies each image is given. In general, at the time of image search, each image is usually expressed by some feature quantity, but a file describing these is also stored in association with the identifier of the image. The database may be implemented and configured by RDBMS (Relational Database Management System) or the like. In addition, the metadata may include, for example, data representing the content of the image (image title, summary text, or keyword), data related to the image format (image data amount, thumbnail size, etc.), and the like. Although not required, it is not essential in the practice of the present invention.

  The reference image database 15 is a database in which reference images including at least one object are accumulated. The reference image database 15 may be inside or outside the image search apparatus 11, and any known communication means can be used. In the present embodiment, the reference image database 15 is outside. It is assumed that the communication means is connected to communicate via the Internet or TCP / IP.

  Further, each unit provided in the image search device 11 and the reference image database 15 may be configured by a computer or server provided with an arithmetic processing device, a storage device, and the like, and the processing of each unit may be executed by a program. This program is stored in a storage device included in the image search device 11, and can be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or provided through a network. Of course, any other components need not be realized by a single computer or server, but may be realized by being distributed to a plurality of computers connected by a network.

  << Processor >>

  Each processing unit of the image search apparatus 11 in the present embodiment will be described. Here, description will be made on the assumption that two images, that is, a first query image 16 and a second query image 17 obtained by photographing the same object from different viewpoints are input. As will be described later, the present invention can also be applied when two or more second query images 17 are input. In this case, a plurality of second query images 17 are input.

  When the first query image 16 and the second query image 17 are given from the outside, the correspondence determination unit 12 performs alignment between the first query image 16 and the second query image 17. The pixels of the second query image 17 are associated with one or more pixels of the first query image 16, and the correspondence result is output to the image composition unit 13. Here, based on the key points extracted from the first query image 16 and the key points extracted from the second query image 17, the key points of the first query image 16 and the keys of the second query image 17. A linear transformation matrix for associating points with each other and converting the key point pixels of the second query image 17 to the corresponding key point pixels of the first query image 16 based on the correspondence between the key points. Ask. Note that a linear transformation matrix for converting key point pixels of the first query image 16 to corresponding key point pixels of the second query image 17 may be used.

  When three or more query images are input, one of the query images (first query image 16 in the example of the present embodiment) is used for all other second queries. The image 17 may be registered based on key points, and the query image having the largest number of corresponding key point pairs may be employed.

  The image composition unit 13 selects a target pixel from the first query image 16, and if there is a corresponding pixel of the second query image 17 corresponding to the selected target pixel, the brightness value of the selected target pixel, The brightness value of the corresponding pixel in the second query image 17 is compared, and if the brightness value of the corresponding pixel is lower than the brightness value of the target pixel, the pixel value of the target pixel is updated with the pixel value of the corresponding pixel. Then, a composite image is generated and output to the search unit 14. Here, the first query image 16 and the second query image 16 are converted into the coordinates of the first query image 16 by converting the coordinates of the second query image 17 based on the linear transformation matrix obtained by the correspondence determination unit 12. Although the query image 17 is pasted on the same coordinates, the target pixel is selected in an area where the first query image 16 and the second query image 17 overlap on the converted coordinates.

  The search unit 14 makes an inquiry to the reference image database 15 using the generated composite image as a new query image, and outputs a reference image closest to the composite image among the reference images in the reference image database 15 as a search result 18. To do.

<< Process overview >>

  Next, processing of the image search apparatus 11 in the present embodiment will be described. FIG. 3 is a flowchart showing the flow of processing.

  First, in step S301, when two or more query images are given from the outside, the correspondence determination unit 12 sets the first query image 16 as the first query image and the second query image 17 as the first query image. The query image 16 is aligned based on the key points, the pixels of the second query image 17 are associated with the pixels of the first query image 16, and the correspondence result is displayed as the image composition unit 13. Output to.

  Subsequently, in step S302, the image composition unit 13 determines that the brightness value of each pixel of the first query image 16 is lower than the brightness value of the pixel of the second query image 17 corresponding to the pixel. By replacing the value of the pixel of one query image 16 with the pixel value of the corresponding pixel of the second query image 17, a composite image is generated and output to the search unit 14.

  Subsequently, in step S303, the search unit 14 inquires the reference image database 15 using the composite image as a query image, and outputs the matched reference image as a search result.

  Through the above processing, a reference image including the same object can be searched for the input query image.

<< Details of each process >>

  Hereinafter, an example of the detailed processing of each processing will be described in the present embodiment.

[Correspondence decision processing]

  First, a process for obtaining a correspondence between two query images in the correspondence determination unit 12 will be described.

  In order to obtain the correspondence between query images, for example, a method based on key point matching described in Non-Patent Document 1, Non-Patent Document 2, or the like is employed. Not limited to these, other known key point matching methods may be adopted.

Here, a certain key point extracted from the first query image 16 is expressed as Q _i , and a key point extracted from the second query image 17 is expressed as R _j . Each key point is described by a feature vector. An arbitrary feature quantity vector may be used, but a local feature quantity such as SIFT described in Non-Patent Document 1 is preferably used.

The feature vector describing the key points _{Q i v} i, a feature vector that describes the _{R j} and expressed as _{w j.} At this time, the distance dist (Q _i , R _j ) between the feature points of the key points is obtained by the following equation.

Subsequently, based on the obtained feature distance between key points, it is determined whether or not each set of key points corresponds. When focusing on a certain key point R _j , it is assumed that the closest key point is Q _k and the next closest key point is Q _l . At this time, when the following conditions are satisfied, it is determined that R _j corresponds to Q _k .

  Here, T is a parameter determined in advance, and may take an arbitrary value of 0 <T ≦ 1. For example, T = 0.8 may be set.

  By executing the above calculation for all keypoint sets, the corresponding keypoints can be obtained.

In addition, the correspondence obtained in this way allows duplication. That is, when focusing on R _j , there is a possibility that a plurality of key points R _j of the query image on the focused side correspond to a certain key point Q _k . Conversely, if attention is paid to Q _k, for a given keypoint R _j, a plurality of key points Q _k side of the query image by focusing might correspond. Naturally, it is unlikely that a plurality of key points of an object with different appearance correspond to one key point of the object even though the objects are the same. Therefore, post-processing may be introduced so as not to allow duplicate correspondence. For example, after all correspondence is obtained by the above method, the key points of the first query image 16 corresponding to the plurality of key points of the second query image 17 are listed. Subsequently, among the key points of the second query image 17 corresponding to the key points of the first query image 16, only the closest one is determined to be an effective correspondence, and the other groups The response will be rejected. By introducing the processing as described above, it is possible to constrain all key points to always correspond one-to-one. In this way, correspondence between key points can be taken between two query images.

  Subsequently, alignment is performed by associating corresponding pixels of the second query image 17 corresponding to the target pixel on the first query image 16 from the obtained correspondence relationship between the key points.

  If the object in the image is a rigid body, the object in the query image and the object in the reference image are only taken from different viewpoints, and this viewpoint variation can be modeled by linear transformation under realistic assumptions. . In other words, as long as the key points exist on the same object, the coordinates of the key points of the first query image 16 and the coordinates of the key points of the second query image 17 can be expressed by linear transformation. Such a linear transformation can be obtained uniquely if there is a set of key points on three sets of objects, assuming that the linear transformation is an affine transformation, and if it is assumed to be a projective transformation. Can be uniquely determined if there are a set of key points on four sets of objects.

  However, in reality, not all keypoints are present on the object in any of the two query images, so linear transformation is obtained while accurately sampling the set of keypoints on the object. Must be configured. Fortunately, there are known and effective methods for obtaining linear transformation under such conditions. For example, the RANSAC algorithm described in Reference 1 and the LO-RANSAC algorithm described in Reference 2 are suitable.

[Reference 1] MA Fischler and RC Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” Comm. ACM, vol. 24, no. 6, pp. 381-395, 1981 .

[Reference 2] O. Chum, J. Matas, and S. Obdrzalek, “Enhancing RANSAC by generalized model optimization,” Proceedings of Asian Conference on Computer Vision, pp. 812-817, 2004.

  By such a method, a linear transformation matrix (matrix size) that gives a transformation from the key point pixel on the object of the second query image 17 to the target pixel of the key point on the object of the first query image 16. 3 × 3) can be obtained. Note that by using this linear transformation matrix, it is possible to take correspondence between arbitrary pixels of two query images only for pixels existing on the object.

  As described above, the correspondence between two different query images in the form of a linear transformation matrix can be determined.

Note that by using the obtained linear transformation matrix, it is possible to more accurately count the number of keypoint sets on the object from the corresponding set of keypoints. If the coordinate of the key point of the second query image 17 is expressed as (x, y), the estimated value (x _e , y _e) of the key point coordinate of the first query image 16 corresponding to this key point. )

It can be asked. Here, H is the linear transformation matrix obtained previously. On the other hand, since the true coordinates of the key points of the first query image 16 are known, if the true coordinates are represented as (x ′, y ′), (x, y), (x ′, y) If ') is a set of keypoints present on both objects, (x', y ') should be accurately estimated by (x _e , y _e ). Therefore, whether or not the correspondence is between the key points extracted from the object is determined depending on whether or not these distances (errors) are within a certain threshold. The threshold value may be an arbitrary value, but may be set to a small value such as 1, 9, 36, 64, for example. In particular, when three or more query images are input, it is preferable to determine a pair having a large number of keypoint groups left in this way and proceed with the subsequent processing.

[Image composition processing]

  Next, a process of generating a composite image from two query images based on the association by the correspondence determination unit 12 in the image composition unit 13 will be described.

  First, based on the previous linear transformation matrix, the second query image 17 is transformed into coordinates on the first query image 16, and these are pasted together. Then, as in the example shown in FIG. 4, two images can be bonded at the object position. If this combined image is regarded as a single image, the pixel value of the original first query image 16 and the original second query image 17 of the area where the two images overlap are displayed. Two pixel values of the pixel value can be taken. Preferably, the pixel value of the image with no specular reflection or weaker image is desired to be adopted.

  In the embodiment of the present invention, selection of a pixel value that meets this condition is realized by selecting a pixel value based on a dichroic reflection model. The dichroic reflection model is a model expressing the assumption that the radiance of an image can be expressed by the sum of a diffuse reflection component and a specular reflection component. If it is the same object that is completely diffusely reflected, it can be assumed that the diffuse reflection component at the same position under the same light source takes almost the same value regardless of the viewpoint, so if there is a large luminance difference between the two query images If this occurs, it can be considered that it is largely due to the specular reflection component. Therefore, when attention is paid to a certain pixel, the pixel value having the lower luminance of the two query image pixels is adopted.

  In this way, a composite image in which the specular reflection component is suppressed as shown in FIG. 4 can be obtained.

[Search section]

  Finally, the search unit 14 makes an inquiry to the reference image database 15 using the synthesized image generated by the image synthesizing unit 13 as a query image, and searches only for a reference image including the same object as the query image.

  There are many known methods for executing such a search. Preferably, a method based on object search is employed that can search for the same object image regardless of how the object is captured (position, posture, and size). For example, it is preferable to use methods such as Reference 3 and Reference 4.

[Reference 3] Japanese Patent Application Laid-Open No. 2016-18444

[Reference 4] G. Tolias, Y. Avrithis, and H. J´egou, “Image search with selective match kernels: aggregation across single and multiple images,” International Journal of Computer Vision, vol. 116, pp. 247- 261, 2016.

  As described above, according to the image search device according to the embodiment of the present invention, the first query image 16 and the second query image 17 are aligned, and the first query image 16 is registered. The pixel of the second query image 17 is made to correspond to one or more of the pixels, the pixel of interest is selected from the first query image 16, and the pixel of the second query image 17 corresponding to the selected pixel of interest is selected. When the corresponding pixel exists, the luminance value of the selected target pixel is compared with the luminance value of the corresponding pixel of the second query image 17, and the luminance value of the corresponding pixel is lower than the luminance value of the target pixel. Shoots a textured object by generating a composite image by updating the pixel value of the target pixel with the pixel value of the corresponding pixel and outputting the reference image in the reference image database that is closest to the composite image Query image Et al, robust to luminance change of an object, accuracy can search the same object.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 11 Image search device 12 Correspondence determination part 13 Image composition part 14 Search part 15 Reference image database 16 1st query image 17 2nd query image

Claims (5)

A reference image database storing at least one reference image is provided, and at least two query images of a first query image obtained by photographing the same object from different viewpoints and at least one second query image are provided. An image search device that receives and outputs a reference image in the reference image database, including the same object as that shown in the first query image,
Alignment is performed between the first query image and the second query image, and pixels of the second query image are made to correspond to one or more pixels of the first query image. A response determination unit;
When a target pixel is selected from the first query image, and there is a corresponding pixel of the second query image corresponding to the selected target pixel, the luminance value of the selected target pixel and the second query When the luminance value of the corresponding pixel is lower than the luminance value of the target pixel, the pixel value of the target pixel is updated with the pixel value of the corresponding pixel. An image composition unit for generating a composite image;
A search unit that outputs a reference image in the reference image database that is closest to the composite image;
Image search device including The correspondence determination unit is configured to determine the keypoint of the first query image and the second query based on the keypoint extracted from the first query image and the keypoint extracted from the second query image. The key points of the image are associated with each other, and the key point pixels of the first query image and the key point pixels of the second query image are associated with each other based on the correspondence between the key points. Find a linear transformation matrix to convert to
The image composition unit converts the coordinates of the second query image into the coordinates of the first query image based on the linear transformation matrix, and the first query image and the coordinates on the converted coordinates In a region where the second query image overlaps, the target pixel is selected from the first query image, and the luminance value of the selected target pixel is compared with the luminance value of the corresponding pixel of the second query image. The composite image is generated by updating the pixel value of the target pixel with the pixel value of the corresponding pixel when the luminance value of the corresponding pixel is lower than the luminance value of the target pixel. Image search device. A reference image database storing at least one reference image is provided, and at least two query images of a first query image obtained by photographing the same object from different viewpoints and at least one second query image are provided. An image search method in an image search apparatus for receiving and outputting a reference image in the reference image database including the same object as the object shown in the first query image,
The correspondence determining unit performs alignment between the first query image and the second query image, and the second query image is applied to one or more pixels of the first query image. Matching the pixels of
When the image composition unit selects a target pixel from the first query image and there is a corresponding pixel of the second query image corresponding to the selected target pixel, a luminance value of the selected target pixel; The brightness value of the corresponding pixel of the second query image is compared. If the brightness value of the corresponding pixel is lower than the brightness value of the target pixel, the pixel value of the target pixel is set to the pixel value of the corresponding pixel. Generating a composite image by updating with:
A search unit that outputs a reference image in the reference image database that is closest to the composite image;
Image search method including The step of causing the correspondence determination unit to correspond is based on the keypoint extracted from the first query image and the keypoint extracted from the second query image, and the keypoint of the first query image and the keypoint The key points of the second query image are associated with each other, and based on the correspondence between the key points, any of the corresponding key point pixels of the first query image and the key points of the second query image Find a linear transformation matrix to convert from one to the other,
The step of combining by the image combining unit converts the coordinates of the second query image into the coordinates of the first query image based on the linear conversion matrix, and the first coordinate image on the converted coordinates. In a region where the query image and the second query image overlap, the target pixel is selected from the first query image, the luminance value of the selected target pixel, and the luminance value of the corresponding pixel of the second query image When the luminance value of the corresponding pixel is lower than the luminance value of the target pixel, the synthesized image is generated by updating the pixel value of the target pixel with the pixel value of the corresponding pixel. Item 4. The image search method according to Item 3.   The program for functioning a computer as each part of the image search device of Claim 1 or Claim 2.