JP6091217B2 - Image search device, image search method, search source image providing device, search source image providing method, and program - Google Patents

Description

  The present invention relates to an image feature extraction technique, and more particularly to a technique related to local features used for comparison of similar images.

  There has been proposed a method of searching for a similar image using a local feature amount (local feature amount) of an image. In this method, first, characteristic points (local feature points) are extracted from an image (Non-Patent Document 1). A technique for calculating a feature amount (local feature amount) corresponding to the local feature point based on the local feature point and surrounding image information is also known (Non-Patent Document 2). Image retrieval is performed by matching local feature amounts.

  In the technique using the local feature quantity, the local feature quantity is defined as information composed of a plurality of elements that are invariant to rotation, enlargement, and reduction. Thereby, even when the image is rotated, enlarged or reduced, the search can be performed. The local feature amount is generally expressed as a vector. However, it is a theoretical story that the local feature quantity is rotation invariant and expansion / reduction invariant. In actual digital images, calculation errors occur when image rotation or enlargement / reduction processing is performed, so there is a slight variation between these local feature values before processing and the corresponding local feature values after processing. Occurs.

  In order to extract the rotation-invariant local feature value, for example, in Non-Patent Document 2, the main direction is calculated from the pixel pattern of the local region around the local feature point, and the local region is rotated based on the main direction when calculating the local feature value. To normalize the direction. Further, in order to calculate the local feature quantity that does not change in size, the image of different scales is generated internally, and local feature points are extracted from the images of the respective scales and the local feature quantities are calculated. Here, a series of image sets of different scales generated internally is generally called a scale space.

  In Patent Document 1, in an image feature point extraction method for extracting feature points from image data, a plurality of pieces of image data are acquired by changing an image, and the feature points are narrowed down to stable feature points. Further, in Patent Document 2, the reproducibility of local feature points is evaluated, and the number of local feature amounts is controlled by describing the features with high reproducibility.

JP 2011-43969 A Japanese Patent Application No. 2011-23762

C. Harris and M.M. J. et al. Stephens, “A combined corner and edge detector,” In Album Vision Conference, pages 147-152, 1988. David G. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints,” International Journal of Computer Vision, 60, 2 (2004), pp. 199 91-110.

  In an image search using local feature amounts, matching is performed by extracting a plurality of local feature points from one image and comparing the local feature amounts calculated from the respective local feature points. When many local feature points are extracted from the query image, the number of comparisons between local feature amounts increases and the search speed decreases, so the number of local feature points does not increase too much, and the rareness unique to query images It is desirable that feature points having high characteristics (high discrimination performance) are included.

  In addition, mobile devices such as mobile phones, smartphones, and digital cameras equipped with a photographing function and a communication function are increasing. Images taken with these portable devices are used as queries, searched on a PC / server, and the search results are taken as portable devices. The use of displaying with is also considered. As an implementation method thereof, there is a method of calculating a local feature amount in a portable device and transmitting the local feature amount to a PC / server. In this case, if there are many local feature values calculated in the mobile device, it takes time to transmit to the PC / server, and it may take a long time to display the search results, so the number of local feature points is large. It is desirable not to become too much. In addition, there may be an upper limit on the size of the local feature amount due to restrictions on the mobile device, the PC / server, the communication line, and the like.

  For these reasons, it is considered that not all of the extracted local feature points are used, but only some of the local feature points are used as in Patent Document 1 and Patent Document 2. However, these methods are methods for narrowing down local feature points that are stably extracted from local feature points extracted from the query image. For example, consider a case where there are a plurality of landmarks that can be photographed from a certain place and one of them is searched as a query image.

  In that case, if there is a local feature amount calculated from the local feature points that are stably extracted in common with those landmarks, the feature amount becomes useless for distinguishing the landmarks from each other. High search accuracy cannot be obtained. There is a limit to the optimization of the feature amount closed by one image like this. If only the feature amount is narrowed down mainly using information obtained from one image, the feature amount as a result of narrowing down does not necessarily become a feature amount with high discrimination performance in the search.

  The present invention has been made in view of the above problems, and intends to provide a technique for obtaining local feature points and local feature quantities that are highly distinguishable from other images and that are significant for search.

  The image search apparatus according to the present invention includes an input unit for inputting an image, an acquisition unit for acquiring attribute information when the image is captured, an extraction unit for extracting an image feature of the image, the image and the attribute information. And an image extracted by the extraction unit for a search source image input by the input unit based on the number of image features co-occurring in the image for each attribute information Screening means for selecting image features to be used for searching the search source image and image features to be restricted from being used, and image features of the search source image selected to be used for searching the search source image And comparison means for comparing the image characteristics of the registered image stored in the storage means.

  According to the present invention, it is possible to obtain local feature points and local feature amounts that are highly distinguishable from other images and that are significant for search.

It is a block diagram which shows the function structural example of the image search device in 1st Embodiment. 6 is a flowchart illustrating an example of an image registration processing procedure in the first embodiment. FIG. 6 is a diagram illustrating an example of a reduced image generation process in the first embodiment. In a 1st embodiment, it is a figure showing an example of a feature list. In a 1st embodiment, it is a figure showing an example of a database registered so that a feature list and a registered image may be associated with each other. 6 is a flowchart illustrating an example of a differentiated image feature list generation processing procedure in the first embodiment. It is a processing flowchart which calculates the co-occurrence value of a feature-value. 5 is a flowchart illustrating an example of an image search processing procedure in the first embodiment. In the first embodiment, it is a flowchart showing a processing flow for preferentially adding a differentiated image feature amount from the first feature extraction list to the second feature extraction list. 5 is a flowchart illustrating an example of an image feature comparison processing procedure in the first embodiment. It is a block diagram which shows the function structural example of the image search system in 2nd Embodiment. It is a flowchart which shows an example of the image search process sequence in the image search system of 2nd Embodiment. 14 is a flowchart illustrating an example of a process for updating a differentiated image feature list in the second embodiment. In 2nd Embodiment, it is an example of the user interface for the update setting of a differentiated image feature list. Example of differentiated image feature list Example of image attribute data

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. The first embodiment is an example of an image search system that performs image feature extraction from a search query image, performs a search using the image feature group, and displays a search result.

  FIG. 1 is a block diagram illustrating a functional configuration example of an image search apparatus according to the present embodiment. Details of the operation in each configuration shown in FIG. 1 will be described later.

  In FIG. 1, an image input unit 101 inputs a registered image and a query image (search source image). The image feature calculation unit 102 calculates the image feature group of the registered image and the query image input from the image input unit 101.

  The differentiated image feature list generation unit 103 prepares, in advance, an image of a subject object group that matches position information by GPS, for example, attribute information such as time information and season. For these images, the co-occurrence of image features in the subject object unit is obtained. Then, by adopting in order from image features having low co-occurrence, a differentiated image feature list having high discrimination performance between images corresponding to the same attribute information is generated.

  The image feature selection unit 104 refers to an image feature group having a high discrimination performance in the differentiated image feature list, and selects an image feature group in the differentiated image feature list from the image feature groups of the query image. The image feature group is selected so as to be used preferentially for processing.

  The image feature comparison unit 105 performs image feature comparison using the image feature group of the query image selected by the image feature selection unit 104. The image feature comparison result display unit 106 displays the image feature comparison result of the image feature comparison unit 105. The storage unit 107 is a memory, HDD, or the like used for storing image characteristics in addition to storing data being processed.

  Each of these components is centrally controlled by a CPU (not shown).

  The CPU can function as various means by executing a program. Note that a control circuit such as an ASIC that operates in cooperation with the CPU may function as these means. These means may be realized by cooperation between the CPU and a control circuit that controls the operation of the image processing apparatus. Further, the CPU need not be a single one, and may be a plurality. In this case, a plurality of CPUs can perform processing in a distributed manner. The plurality of CPUs may be arranged in a single computer, or may be arranged in a plurality of physically different computers. Note that the means realized by the CPU executing the program may be realized by a dedicated circuit.

[Image registration process]
FIG. 2 is a flowchart illustrating an example of an image registration processing procedure in the image search apparatus according to the first embodiment. The flowchart is realized when the CPU executes a control program.

  First, in step S <b> 201, a registered image is input via the image input unit 101. The input image is stored in the storage unit 107.

  Next, the image feature calculation unit 102 performs the processing from step S202 to step S207. First, in step S202, a luminance component is extracted from the input registered image, and a luminance component image is generated based on the extracted luminance component.

  Next, in step S203, it repeatedly repeats the reduction of the luminance component image in accordance with the magnification (reduction rate) p, and n reduced images including the original image, which are reduced stepwise from the original size image, are generated. . Here, it is assumed that the magnification p and the number n of reduced images are determined in advance.

  FIG. 3 is a diagram illustrating an example of a reduced image generation process. The example shown in FIG. 3 is a case where the magnification p is “2 to the power of − (1/4)” and the number n of reduced images is “9”. Of course, the magnification p is not necessarily "2 to the power of-(1/4)". In FIG. 3, reference numeral 301 denotes the luminance component image generated in step S202. Reference numeral 302 denotes a reduced image obtained by recursively reducing the luminance component image 301 four times according to the magnification p. Reference numeral 303 denotes a reduced image obtained by reducing the luminance component image 301 eight times according to the magnification p.

  In this example, the reduced image 302 is an image obtained by reducing the luminance component image 301 to ½, and the reduced image 303 is an image obtained by reducing the luminance component image 301 to ¼. In the first embodiment, the method for reducing an image is to generate a reduced image by a reduction method using linear interpolation. The image may be reduced by other methods.

  Next, in step S204, local feature points (local feature points) that are robustly extracted even if there is image rotation in each of the n reduced images are extracted. As a method for extracting local feature points, a Harris operator is used in the first embodiment (Non-patent Document 1: C. Harris and MJ Stephens, “A combined corner and edge detector,” In Alvery Vision Conference, pages. 147-152, 1988.).

  Specifically, with respect to the pixel on the output image H obtained by applying the Harris operator, the pixel values of the pixel and pixels in the vicinity of the pixel (eight pixels in total) (total nine pixels) are examined. Then, a point at which the pixel becomes a local maximum (a pixel value of the pixel becomes the maximum among the nine pixels) is extracted as a local feature point. Here, even when the pixel reaches a local maximum, it is not extracted as a local feature point if the value of the pixel is less than or equal to the threshold value.

  Note that any feature point extraction method is applicable as long as it is a method capable of extracting local feature points, not limited to the feature point extraction method using the Harris operator described above.

  Next, in step S205, for each of the local feature points extracted in step S204, a feature quantity (local feature quantity) defined so as to remain unchanged even when the image is rotated is calculated. As a method for calculating this local feature amount, the first embodiment uses a local jet and a combination of derivatives thereof (JJ Koenderink and AJ van Doorn, “Representation of local geometry in the visual system, "Riologic Cybernetics, vol. 55, pp. 367-375, 1987.).

  Specifically, the local feature amount V is calculated by the following equation (1).

  However, the symbols used on the right side of the equation (1) are defined by the following equations (2) to (7). Here, G (x, y) on the right side of Expression (2) is a Gaussian function, I (x, y) is a pixel value at image coordinates (x, y), and “*” is a symbol representing a convolution operation. is there. Equation (3) is a partial derivative of variable L defined by equation (2) with respect to x, and equation (4) is a partial derivative of variable L with respect to y. Equation (5) is the partial derivative of variable Lx defined in equation (3) with respect to y, equation (6) is the partial derivative of variable Lx defined in equation (3) with respect to x, and equation (7) is the equation. It is a partial derivative with respect to y of Ly defined in (4).

  It should be noted that any feature amount calculation method is applicable as long as it is a method capable of calculating a local feature amount, not limited to the above-described feature amount calculation method.

  Next, in step S206, each local feature amount calculated in step S205 is quantized, and a label value is assigned. An N-dimensional local feature value is calculated from one local feature point by a combination of local feature values used in the first embodiment, that is, Local Jet and their derivatives. Here, K quantization is performed for each dimension. Here, N and K are assumed to be determined in advance.

  Specifically, quantization is performed by the following equation (8).

Qn = (Vn * K) / (Vnmax−Vnmin + 1) (8)
Here, Qn is a value obtained by quantizing the feature quantity Vn of the nth dimension among the N dimensions. Vnmax and Vnmin are the maximum value and the minimum value that can be taken by the feature quantity of the nth dimension, respectively.

  After quantization for each dimension, labeling is performed by the following equation (9).

  Note that any calculation method capable of quantization and labeling is not limited to the calculation method as described above, and any quantization and labeling method can be applied.

  Next, in step S207, those feature amounts and quantization labels are collected as an image feature list. FIG. 4 is an example of an image feature list. A list is generated using the quantization label as a key. A plurality of image feature lists may be associated with one quantization label.

  In the first embodiment, the feature amounts and the quantization labels are collected as an image feature list. However, only the quantization labels can be collected as an image feature list. Also, other information such as the coordinates of feature points can be included in the image feature list.

  Finally, in step S208, the image feature list is registered in the storage unit 107 in association with the registered image. FIG. 5 is an example of a database in which an image feature list is registered in association with a registered image. Register the quantization label as a key in the database. Also, an image ID is assigned to the registered image, and the ID is registered in the database. A plurality of image IDs may be associated with one quantization label, for example, when the same image is registered multiple times or when similar images are registered. There may also be quantization labels that are not used.

[Different image feature list generation processing]
FIG. 6 is a flowchart illustrating an example of a differentiated image feature list generation processing procedure in the differentiated image feature list generation unit 103 of the image search apparatus. The flowchart is realized when the CPU executes a control program.

  The differentiated image feature list is, for example, determining the co-occurrence of image features in units of subject objects corresponding to the attribute information such as position information by GPS, time information, season, etc. These are image feature lists that have high discrimination performance and can be differentiated. It is a partial list of group feature groups included in the image feature list created in step S207. Image features with low co-occurrence values may be sorted in order, and the portion of co-occurrence values below the threshold may be used as a differentiated image feature list.

  The differentiated image feature list generation unit 103 prepares in advance an image of the subject object group corresponding to each attribute information, and obtains the co-occurrence of the image feature for each subject object with reference to the storage information of the image feature. Then, by adopting in order from image features having low co-occurrence, an image feature list having high discrimination performance between images corresponding to the same attribute information is generated.

  The image feature selection unit 104 refers to the image feature group in the differentiated image feature list, and selects the image feature group having high discrimination performance included in the differentiated image feature list from the image feature group of the query image.

  In the present embodiment, an example is shown in which GPS is used to acquire information about the shooting environment. Position attribute information is used. In this example, a target for generating a differentiated image feature list is a landmark. A knowledge base is constructed by pairing the GPS information of the position where the landmark exists and the landmark image.

  First, in step S601, a nearby landmark group is estimated based on the GPS information of the query image using the knowledge base. In step S602, image feature groups of neighboring landmark groups are obtained. In step S603, a feature co-occurrence value is calculated in units of landmarks as subject objects that are targets for generating differentiated image features.

  FIG. 7 is a flowchart illustrating an example of a processing procedure for calculating the co-occurrence value in step S603. The flowchart is realized when the CPU executes a control program.

In step S701, GPS information serving as a reference when generating differentiated image features is obtained.
This GPS information can correspond to many landmarks by obtaining a list of GPS information with famous landmarks in advance and performing the processing of FIG. 7 for each GPS information. is there. Of course, GPS information may be automatically generated at appropriate intervals.
In step S702, landmarks in the vicinity of the reference GPS value are obtained, and the number thereof is M. A knowledge base in which landmarks, GPS values, image data, and image feature groups are associated in advance is prepared, and landmarks whose distance from a reference GPS value is within a predetermined value may be obtained. The GPS information includes height information, but the term may or may not be included.

In step S703, the following values such as the number of image features corresponding to the M landmark images are read from the knowledge data into the memory.
The number of images corresponding to each landmark i is expressed as NumP (i) i = 1,.
An image corresponding to the landmark i is represented by Pic [i] [j] j = 1,... NumP (i)
NumFeat [Pic [i] [j]] for the number of features in each image
The k-th feature of each image is Feat [Pic [i] [j]] [k] k = 1, numFeat [Pic [i] [j]]
Also, i is set to 1 and initialization is performed with a co-occurrence value CO [] = 0 for each feature.
The co-occurrence value CO array portion [] contains the quantization label value of the feature quantity.

  In step S704, it is determined whether the counter representing the image serving as the comparison reference is equal to or less than M, that is, whether there is a remaining landmark image serving as the comparison reference.

  In the processing from step S704 to step S713, Feat [Pic [i] [j]] for switching the image group for the landmark while referring to M landmarks for comparison, and for referring to the feature amount of the image. [K] is generated.

  This is a process for determining how much the other images contain the same feature quantity in units of landmarks based on the feature quantities of all the images corresponding to the landmarks.

All feature quantities are quantized and handled by quantization labels. (In this embodiment, the co-occurrence is obtained not by the feature quantity identity but by the quantization label identity.)
In step S705, j indicating the number of the image in the image group corresponding to the landmark i is initialized to 1.

  In step S706, it is determined whether j does not exceed the number of image groups corresponding to the landmark i. If it exceeds, i is incremented by 1 in the next landmark image and changed in step S704. Return to.

  If it is determined in step S706 that j does not exceed the number of image groups corresponding to the landmark i, a counter k for referring to the feature value is initialized to 1 in step S709.

  In step S709, it is determined whether k does not exceed the number of feature quantities of the j-th image associated with the landmark i. If not, i2 representing the landmark image of the comparison destination in step S711 is initialized to 1, and the reference comparison target feature amount Feat [Pic [i2] [j2]] [k2] from step S712 to step S721. ] Go to processing to refer to.

  If it is determined in step S709 that k has exceeded the number of features of the j-th image associated with the landmark i, j representing the associated image is incremented by 1 in step S710, and the process returns to step S706 again.

  In step S712, it is determined whether i2 representing the landmark to be compared does not exceed the number M of landmarks.

  If it exceeds, in step S713, k is incremented by 1, and the process returns to the process of updating the reference landmark feature quantity.

  If all the image feature groups of the landmark i have been compared, the next landmark is updated to the comparison reference in step S707 through steps S713, 709, 710, and 706.

  In the processing from step S712 to step S724, the image group for the landmark is switched while switching up to M landmarks as comparison destinations, and Feat [Pic [i2] [j2]] [ k2]. Then, the reference feature quantity is compared in step S722.

  This is a process for determining how much the other images contain the same feature quantity in units of landmarks based on the feature quantities of all the images corresponding to the landmarks.

In step S714, j2 indicating which image in the image group corresponding to the landmark i2 is initialized to 1, and in i2, the co-occurrence value is already incremented in an image related to any i2. Flag indicating whether or not there is initialized to 0 (not incremented).
In step S715, it is determined whether j2 does not exceed the number of image groups corresponding to the landmark i2, and if so, it is determined in step S716 whether Flag is 1, and if it is 1. The co-occurrence value corresponding to the feature quantity quantization value is increased by one.

  In step S718, i is increased by one to the next landmark image, and the process returns to step S712.

  If it is determined in step S715 that j2 does not exceed the number of image groups corresponding to the landmark i2, a counter k2 for referring to the feature value is initialized to 1 in step S719.

In step S720, it is determined whether k2 exceeds the number of feature quantities of the j2th image related to the landmark i2. If not, it is determined in step S722 whether the feature values Feat [Pic [i] [j]] [k] and Feat [Pic [i2] [j2]] [k2] are the same. If they are the same, Flag is set to 1 in step S723 and it is assumed that the same feature already exists. If it is determined in step S722 that the features are not the same, k2 is incremented by 1 in step S724, and the process returns to step S720.
If all the image feature groups of the landmark i2 have been compared, the next landmark is updated to the comparison reference in 1621 through steps S724 and 1620.

  As a result, the co-occurrence value col [] stores the co-occurrence value for the feature value quantized value indicated by [] as a value from 0 to M * M.

  Further, col [] is divided by M * M and normalized to a value from 0 to 1.0.

  Since 0 means that the feature quantity does not appear, the co-occurrence values larger than 0 are sorted from the smallest, and the feature quantity quantized values corresponding to the co-occurrence values are also sorted. The result of sorting the feature quantity quantization values becomes an information source for generating differentiated image features.

  When this co-occurrence value is large, or 1.0 in an extreme case, it is a feature that always exists in all landmarks, which means that it is difficult to distinguish the landmarks. Conversely, when the co-occurrence value is small, it means that it is useful for discriminating landmarks. In this embodiment, a feature with a small co-occurrence value is positively adopted as a differentiated image feature.

  Therefore, in step S604, the quantized labels are sorted in ascending order of co-occurrence values, that is, in ascending order.

  In step S605, a list of features (quantization labels) of co-occurrence values less than the threshold is created, and in step S606, a list of differentiated image features is output and accumulated. A co-occurrence value that can be determined to have discrimination performance is set as a threshold value.

  FIG. 15 is an example of a differentiated image feature list.

  Of course, a predetermined number may be selected from those having a small co-occurrence value without using a threshold value in the process of step S605. Of course, when there are many landmarks around the reference GPS information, it is preferable to use at least the number of landmarks. Further, a predetermined number can be changed from the number of landmarks, for example, a number multiplied by a certain number.

  As the timing of generating the differentiated image feature list, on-demand generation may be performed using the current GPS information when the power is turned on, the shooting mode, or the image search mode.

  Of course, there is a method in which a plurality of differentiated image feature lists are generated in advance and the differentiated image feature lists are selected according to GPS information. It is possible by appropriately quantizing the GPS information of the shooting position in a 100 m square considering the limit reflected in the size of the camera, and generating a differentiated image feature list in that quantization unit It is.

  Further, there is a method for increasing the accuracy of the differentiated image feature list and reducing the size of each differentiated image feature list. For each landmark, an image is prepared for each shooting direction. The direction to be photographed is obtained from the GPS information of the photographing position and the GPS information of the position of each nearby landmark, and the image for calculating the co-occurrence value is switched. This makes it possible to generate a differentiated image feature list in accordance with the composition of landmarks that can be taken from the shooting position. Since the number of landmarks that can be photographed is also reduced, the size of each differentiated image feature list can be reduced.

  However, if the GPS information of the position where the landmark exists and the landmark image cannot be prepared, the combination of the GPS information and the shooting direction information by the compass is used instead. In other words, 360 degrees from all GPS directions may be classified into, for example, 45-degree units, that is, captured images in eight directions, and feature co-occurrence values may be obtained for these classifications.

  In step S603, the co-occurrence value of the feature is calculated in the unit of the landmark. However, in the case of performing simply, the co-occurrence value in the entire image including the landmark may be used instead. However, in this case, the accuracy may decrease.

  In addition, the image of the landmark is presented to the person, the area having the characteristic color of the landmark is designated, and the feature point is detected by the image feature extraction process, and only the feature point existing in the area designated by the person is used. By performing the processing of FIG. 6, it is possible to generate a differentiated image feature amount list with higher accuracy. In addition, the feature points and their co-occurrence values corresponding to the differentiated image feature list can be visualized and displayed by a person to create a differentiated image feature list that is more compact and close to the human sense. It becomes. Differentiate by displaying the number of stored image or co-occurrence values of image feature groups associated with image feature groups calculated from the feature points related to the parts that show the features of landmarks An image feature list may be generated.

[Image search processing]
FIG. 8 is a flowchart illustrating an example of an image search processing procedure in the image search apparatus according to the first embodiment. The flowchart is realized when the CPU executes a control program.

  First, in step S <b> 801, a query (search source) image is input via the image input unit 101. The input image is stored in the storage unit 107.

  Next, the image feature calculation unit 102 performs the processing from step S802 to step S807. Since this process is equivalent to the process from step S202 to step S207 in the image registration process, detailed description thereof is omitted.

  However, two types of image feature extraction lists are used for the image feature extraction list. A first feature extraction list that stores features extracted from the query image as they are and a feature that reflects the image features that match the differentiated image feature list to be used in preference to the comparison processing by the image feature comparison unit 105 are stored. It is a 2nd feature extraction list.

  That is, the image feature extraction list in the image registration process corresponds to the first feature extraction list in the image search process. The second feature extraction list is used for storing features used in the final search comparison process.

  Next, in step S808, if the number of features of the extracted query image is larger than the maximum number of features that can be used at the time of search, the process proceeds to step S809, and if it is less than the maximum number, the process proceeds to step S814.

  The reason why the number is not more than the maximum number is to control the upper limit of the number of image features used for the search and reduce the variation in the search processing time. Of course, good results may be obtained even if the number is less than the maximum number or even if the process proceeds to step S809. However, if the number of features acquired from the query image is very small in the first place, care must be taken that the number of features does not decrease too much. It is.

  In the first embodiment, the determination is made based on whether or not the number of feature points of the query image is larger than the maximum number of feature points that can be used during the search. However, the determination may be made based on whether or not the size when the image feature list is stored in the storage unit is larger than the maximum size that can be used during the search.

  In step S809, the same quantization label as that in the differentiated image feature list is added to the second feature extraction list. Processing may be performed so that quantization labels that are not included in the differentiated image feature list are deleted from the first feature extraction list, and image features with high discrimination performance are left in the first feature extraction list. In addition, an identifier may be added to the first feature extraction list as to whether or not the quantization label is on the differentiated image feature list.

  Here, FIG. 9 is a flowchart illustrating an example of a process of adding the same quantization label as that in the differentiated image feature list in step S809 to the second feature extraction list. The flowchart is realized when the CPU executes a control program. This process is a process performed by the image feature selection unit 104.

  First, in step S901, the first quantization label in the differentiated image feature list is set as a processing target quantization label. Next, in step S902, if the processing target quantization label exists in the image feature list of the query image, the process proceeds to step S903, and if not, the process proceeds to step S905.

  In step S903, the processing target quantization label is added to the second feature extraction list. In step S904, if the number of features of the query image is equal to or less than the maximum number of features that can be used at the time of search, the process proceeds to step S905, and if it reaches the maximum number, the process ends.

  In the first embodiment, the determination is made based on whether or not the number of feature points of the query image is larger than the maximum number of feature points that can be used during the search. However, as in step S808, the determination may be made based on whether or not the size when the image feature list is stored in the storage unit is larger than the maximum size that can be used during the search.

  In step S905, if an unprocessed quantization label exists in the differentiated image feature list, the process proceeds to step S906, and if not, the process ends.

  In step S906, the next quantization label in the differentiated image feature list is set as a processing target quantization label, and the process returns to step S902.

  After the process of step S809, if the number of features in the second feature extraction list is smaller than the minimum number of features that can be used at the time of search in step S810, the process proceeds to step S811, and if it is greater than the minimum number, the process proceeds to step S812. .

  In image search based on feature points, it has been found from experience that information is insufficient if at least about 10 feature points are not included in the landmark area in voting matching as shown in FIG. Therefore, if the number is less than the minimum number in the differentiated image feature list, the image features are supplemented in step S811.

  In the first embodiment, the determination is made based on whether or not the number of feature points of the query image is larger than the maximum number of feature points that can be used during the search. However, as in step S808, the determination may be made based on whether or not the size when the image feature list is stored in the storage unit is larger than the maximum size that can be used during the search.

  In step S811, the stability of feature quantities that have not yet been included in the second feature extraction list in the first feature extraction list of the query image is evaluated. Then, by adding a feature quantity with high stability to the second feature extraction list, the number of feature points of the query image is replenished to a number that does not exceed the maximum number of features that can be used during the search. Of course, when the number of image features in the second feature extraction list does not satisfy the minimum number, other image features are used. A method for selecting the number of feature points using the stability of the feature amount of the query image can be achieved by using the method according to Patent Document 2.

   That is, the position of the feature point and the distance between the features of the feature amount are obtained by performing image processing such as rotating or reducing the image to be analyzed, and this is used as a stability index. If the variation of the position of the feature point is large, the feature amount obviously changes. Therefore, first, the feature point is narrowed down to a feature point that falls within the predetermined threshold range. Then, a change in feature quantity before and after image processing such as rotation or reduction, that is, a distance between feature quantities is used as an index of stability. The feature points with smaller distances are considered to be more stable, and are sorted in ascending order of distance, and the feature points with higher stability are prioritized until the number of feature points reaches the number of feature points that can be used during search. Create an image feature list for the query image.

  In step S812, if the number of features in the second feature extraction list is greater than the maximum number of features that can be used during the search, the process proceeds to step S813. If the number is less than the maximum, the process proceeds to step S815.

  In step S813, features from the second feature extraction list including only differentiated image features are selected from features having the smallest co-occurrence value up to the maximum number that can be used in the comparison processing by the image feature comparison unit 105. , It is stored again in the second feature extraction list. Of course, although the performance is lowered, up to the maximum number of features that can be used in the comparison processing by the image feature comparison unit 105 may be selected at random.

  In step S814, since the number of features in the first feature extraction list is less than the maximum number, all features are adopted and all are copied to the second feature extraction list.

  In step S815, an image feature group is read from the storage unit 107, and image feature comparison processing is performed. FIG. 10 is a flowchart illustrating an example of the image feature comparison process in step S815. The flowchart is realized when the CPU executes a control program.

  First, in step S1001, ballot boxes for the number of registered images are prepared and reset at 0. Next, in step S1002, the first quantization label in the second feature extraction list is set as the processing target quantization label.

  In step S1003, if there is an image ID registered in association with the processing target quantization label, the process proceeds to step S1004, and if not, the process proceeds to step S1005. In step S1004, one vote is cast in the ballot box of the image ID registered in association with the processing target quantization label.

  In step S1005, if an unprocessed quantization label exists in the second feature extraction list, the process proceeds to step S1006, and if not, the process proceeds to step S1007.

  In step S1006, the next quantization label in the second feature extraction list is set as the processing target quantization label, and the process returns to step S1003.

  Next, in step S1007, the voting results are sorted in descending order of the number of votes, and in step S1008, a predetermined number of image IDs from the one with the largest number of votes are output as an image feature comparison result list.

  Finally, the image feature comparison result is displayed in step S816. When displaying the image feature comparison result, the image corresponding to the image ID is displayed together.

  If only the landmark is searched, the landmark does not move, and therefore the GPS information is unique. With reference to the image attribute data in FIG. 16, GPS information in a range that can be photographed from the GPS information of the current search device. The process of narrowing down to the image having the image may be performed in step S1008. Of course, the same processing is possible as long as it does not move like a landmark and can be associated with unique GPS information.

  As described above, in the first embodiment, priority is given to image features with a high degree of differentiation in the differentiated image feature list in which landmark-specific image features using GPS information are described from the image feature group extracted from the search query image. Adopted. Image features that are not included in the differentiated image feature list and are not stable in the image feature group extracted from the search query image are not used. This makes it possible to narrow down to feature quantities with high discrimination performance. The search is performed using the selected image feature group. Thereby, even when there is a limit to the number of feature points that can be used at the time of search, it is possible to perform a search with reduced deterioration in search accuracy.

(Second Embodiment)
Hereinafter, a second embodiment of the present invention will be described with reference to the drawings. In the second embodiment, a search query image is generated by photographing with a search client (request source) that is an image search terminal, an image feature group is extracted, and the image feature group is sent to an image search server to perform an image search. It is an example of the image search system which searches in a server and displays a search result on a search client. The image search client is a search source image providing device that provides a search source image. Image registration processing to the image search server is performed by the same method as in the first embodiment.

  FIG. 11 is a block diagram illustrating a functional configuration example of the image search system according to the present embodiment. Details of the operation in each configuration shown in FIG. 11 will be described later.

  The image search system in FIG. 11 includes a search client 1101 and an image search server 1102. The search client 1101 and the image search server 1102 have communication units 1112 and 1120, respectively. The communication units 1112 and 1120 are connected by a network (wired or wireless), and information communication is performed via the network. .

  In the search client 1101, the image acquisition unit 1103 acquires a query (search source) image by camera photography or the like. In this embodiment, the shooting environment information transmission unit 1104 transmits GPS information to the image search server 1102.

  The differentiated image feature list receiving unit 1105 receives the distinguishable image feature list generated by the image search server 1102 via the communication unit 1112.

  For example, it is assumed that there are a plurality of landmarks that can be photographed from the current shooting location, and that feature quantities useful for distinguishing these landmarks are received in the differentiated image feature list. For this purpose, the search client 1001 transmits GPS information to the image search server 1102 and receives a differentiated image feature list associated with the GPS information via the communication unit 1112. In the differentiated image feature list, for example, GPS information and date / time information are used, a group of subjects that can be photographed at that location, that season, and at that time are assumed. It can be greatly improved.

  The image feature calculation unit 1106 calculates an image feature group of the query image acquired by the image acquisition unit 1103. The image feature selection unit 1107 selects an image feature group with high discrimination performance from the image feature group of the query image. The image feature transmission unit 1108 transmits the image feature group selected by the image feature selection unit 1107 to the image search server 1102 via the communication unit 1112. The image feature comparison result acquisition unit 1109 receives the image feature comparison result executed by the image search server 1102 via the communication unit 1112. The image feature comparison result display unit 1110 displays the image feature comparison result acquired by the image feature comparison result acquisition unit 1109. The storage unit 1111 is a memory, HDD, or the like that stores data being processed in the search client.

  In the image search server 1102, the differentiated image feature list generation unit 1114 has an appearance frequency that is low with respect to the same attribute information among image feature groups calculated from a plurality of registered images, and corresponds to the same shooting environment information. An image feature list with high discrimination performance is generated.

  The differentiated image feature list transmission unit 1115 transmits to the search client 1101 via the communication unit 1120 a differentiable image feature list that matches the shooting environment information obtained from the shooting environment information reception unit 1113. . The image feature reception unit 1116 receives the image feature group of the query image calculated by the search client 1101 and preferentially adopted by the image feature selection unit 1107 via the communication unit 1120. The image feature comparison unit 1117 performs image feature comparison using the image feature group of the query image received by the image feature reception unit 1116. The image feature comparison result transmission unit 1118 transmits the result of the image feature comparison executed by the image feature comparison unit 1117 to the search client 1101 via the communication unit 1120. The storage unit 1119 is a memory, HDD, or the like that stores data being processed in the image search server.

  The image feature group selection unit 1107 may be on the image search server side. In that case, the image feature group transmission unit 1108 transmits the image feature group of the query image extracted by the image feature extraction unit 1106 to the image search server 1102 via the communication unit 1112. Then, an image feature group selection unit 1107 in the image search server 1102 selects an image feature group with high discrimination performance from the image feature group of the query image. The image feature group comparison unit 1117 performs image feature group comparison using the image feature groups selected by the image feature group selection unit 1107 on the image search server 1102 side. Even in the flowchart of the image search processing procedure, the processing when the image feature group selection unit 1107 is on the image search server 1102 side is performed.

  Each of these components is centrally controlled by a CPU (not shown).

  The CPU can function as various means by executing a program. Note that a control circuit such as an ASIC that operates in cooperation with the CPU may function as these means. These means may be realized by cooperation between the CPU and a control circuit that controls the operation of the image processing apparatus. Further, the CPU need not be a single one, and may be a plurality. In this case, a plurality of CPUs can perform processing in a distributed manner. The plurality of CPUs may be arranged in a single computer, or may be arranged in a plurality of physically different computers. Note that the means realized by the CPU executing the program may be realized by a dedicated circuit.

[Different image feature list generation processing]
The differentiated image feature list generation processing in the differentiated image feature list generation unit 1114 of the image search server of the image search system of the second embodiment is performed by the differentiated image feature list generation unit 103 of the image search device of the first embodiment. This is equivalent to the differentiated image feature list generation process in FIG. Therefore, detailed description is omitted.

  Note that receiving a differentiated image feature list each time a search increases the processing load, so it is preferable to cache the differentiated image feature list in the search client 1101 in the case of the same GPS information.

[Image search processing]
FIG. 12 is a flowchart illustrating an example of an image search processing procedure in the image search system according to the second embodiment. The flowchart is realized when the CPU executes a control program.

  First, in step S1201, the search client 1101 acquires a query (search source) image via the image acquisition unit 1103 by camera shooting or the like. The acquired image is stored in the storage unit 1111.

  Next, the image feature calculation unit 1106 performs the processing from step S1202 to step S1207. This process is the same as the process from step S202 to step S207 in the image registration process of the first embodiment or the process of step S802 to step S807 in the image search process of the first embodiment, and thus detailed description thereof is omitted. To do.

  Next, in step S1208, if the number of feature points of the extracted query image is larger than the maximum number of feature points that can be used at the time of search, the process proceeds to step S1209, and if it is less than the maximum number, the process proceeds to step S1213.

  In the second embodiment, the determination is made based on whether the number of feature points of the query image is larger than the maximum number of feature points that can be used at the time of search. However, the determination may be made based on whether or not the size when the image feature list is stored in the storage unit is larger than the maximum size that can be used during the search.

  In step S1209, the differentiated image feature list is updated.

  Here, FIG. 13 is a detailed flowchart of step S1209 showing an example of the update processing of the differentiated image feature list. The flowchart is realized when the CPU executes a control program.

  First, in step S1301, the search client 1101 confirms the update setting of the differentiated image feature list. The update setting of the differentiated image feature list sets whether to update at the time of search. In addition, in the case of the setting to be updated at the time of search, it is set which network connection is to be updated. FIG. 14 is an example of a user interface for the update setting of the differentiated image feature list. A button 1401 in FIG. 14 sets whether to update at the time of search, and a button 1402 sets which network connection to update.

  In step S1302, if there is a change in GPS information at the time of the previous search, the process proceeds to step S1303. If there is no change, the differentiated image feature list currently held is used as a cache as it is, and the update process ends. To do.

  In step S1303, the currently connected network is checked. If it is determined in step S1304 that the network is an update target network, the process proceeds to step S1305. If the network is not an update target network, the differentiated image feature list update process ends.

  In step S <b> 1305, the differentiated image feature list receiving unit 1105 acquires the differentiated image feature list from the image search server 1102 via the communication unit 1112. Finally, in step S1306, the differentiated image feature list received in step S1305 is stored in the storage unit 1111 and the differentiated image feature list update process is terminated.

  After the update processing of the differentiated image feature list in step S1209 is completed, in step S1210 to step S1212, the second feature extraction list is supplemented by giving priority to the stable feature from the first feature extraction list. Since these processes are equivalent to the processes from step S809 to step S811 in the image search process of the first embodiment, detailed description thereof is omitted.

  In step S1213, the image feature list extracted and selected from the query image is transmitted to the image search server 1102 via the communication unit 1112.

  In step S1217, the image search server 1102 performs image feature comparison processing. Since this process is equivalent to the process of step S815 in the image search process of the first embodiment, detailed description thereof is omitted. In step S1218, the image feature comparison result obtained in step S1212 is transmitted to the search client 1101 via the communication unit 1120.

  Finally, an image feature comparison result is displayed in step S1219. This process is equivalent to the process of step S816 in the image search process of the first embodiment.

  As described above, in the second embodiment, the image that has been selected from the image feature group extracted from the search query image by the search client so that the differentiated image feature is preferentially used for the comparison process of the image search. The feature group is transmitted to the image search server, and the search is performed by the image search server. Thereby, even when there is a limit to the number of feature points that can be used at the time of search, it is possible to perform a search with reduced deterioration in search accuracy.

(Other embodiments)
The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

  By using GPS information (category of shooting location) and date and time, it is possible to generate a differentiated image feature list that can accurately search for animals and plants that can be shot at that location and time. It is also possible to generate a differentiated image feature list of nocturnal insects and animals whose appearance probability changes depending on weather information and time information. It is necessary to process not only time information but also local time.

  Furthermore, it is also possible to generate a differentiated image feature list of insects and animals whose appearance frequency changes depending on weather, temperature, humidity, and the like.

  Even in buildings, differentiated image features are useful when moving objects such as a dome that opens and closes due to the weather change the appearance.

  For alpine animals and plants, the shooting altitude information of GPS information may be used, but atmospheric pressure may be used for estimation of the altitude.

  Differentiated image feature list that can be searched with high accuracy by obtaining the image feature group with low co-occurrence between well-captured subjects using attribute settings such as camera shooting mode dial settings such as presence of flash and macro shooting Can also be generated.

  In telephoto shooting, it is possible to generate a differentiated image feature list that can be searched with high accuracy by obtaining image feature groups with low co-occurrence between birds, assuming that birds that are difficult to approach are shot. Is possible. Of course, when a camera shooting mode for specifying the subject attribute of the shooting subject can be specified, a differentiated image feature list can be generated from an image having the attribute.

DESCRIPTION OF SYMBOLS 101 Image input part 102 Image feature calculation part 103 Differentiated image feature list generation part 104 Image feature selection part 105 Image feature comparison part 106 Image feature comparison result display part 107 Storage part 1101 Search client 1102 Image search server 1103 Image acquisition part 1104 Photographing Environment information transmission unit 1105 Differentiated image feature list reception unit 1106 Image feature calculation unit 1107 Image feature selection unit 1108 Image feature transmission unit 1109 Image feature comparison result acquisition unit 1110 Image feature comparison result display unit 1111 Storage unit 1112 Communication unit 1113 Imaging environment Information reception unit 1114 Differentiated image feature list generation unit 1115 Differentiated image feature list transmission unit 1116 Image feature reception unit 1117 Image feature comparison unit 1118 Image feature comparison result transmission unit 1119 Storage unit 1120 Communication unit

Claims (20)

An input means for inputting an image;
Acquisition means for acquiring attribute information at the time of photographing the image;
Extracting means for extracting image features of the image;
Storage means for storing the image, the attribute information, and the image feature in association with each other;
Based on the number of image features co-occurring in the image for each attribute information, the search source image input by the input unit is used for searching the search source image among the image features extracted by the extraction unit. A sorting means for sorting into image features and image features whose use is restricted;
Comparison means for comparing the image features of the search source image selected for use in searching the search source image with the image features of the registered image stored in the storage means;
An image search apparatus comprising: Further comprising: generating means for generating, for each attribute information, an image feature list in which the number of image features co-occurring in the image for each attribute information is less than a predetermined number;
The selection unit is present in the image feature list corresponding to the attribute information of the search source image acquired by the acquisition unit among the image features extracted by the extraction unit for the search source image input by the input unit. 2. The image search apparatus according to claim 1, wherein an image feature used for searching a search source image and an image feature whose use is restricted are selected based on the image feature to be searched.   3. The image search apparatus according to claim 1, wherein a total size of image features or the number of image features of the search source image used by the comparison unit is predetermined. 4. The attribute information according to claim 1, wherein the attribute information is at least one of information related to a place and a shooting direction , information related to time , information related to weather, and information related to a shooting mode. The image search apparatus described.   The generating means has a number of image features determined from the order in which the co-occurrence values of the image features stored in the storage means with respect to the registered image to which at least one of the attribute information matches is smaller than the threshold or the co-occurrence values are small. The image search device according to claim 2, wherein the image feature list is described in an image feature list.   The generation means has a number of co-occurrence values stored in association with an image feature for a registered image to which at least one of the attribute information is matched is smaller than a threshold value or a predetermined number of co-occurrence values. The image search apparatus according to claim 2, wherein the image feature is described in an image feature list.   The generation means describes the image feature selected by the user in the image feature list based on the co-occurrence value of the image stored in association with the image feature for the registered image to which at least one of the attribute information is matched. The image search device according to claim 2.   3. The image generation list according to claim 2, wherein the generation unit generates, for each direction, an image feature list in which the number of image features that co-occur in the image for each direction at the shooting location of the image is smaller than a predetermined number. Image search device.   3. The image search apparatus according to claim 2, wherein the selecting means selects image features by extracting image features that match the image feature list from image features of the search source image.   3. The image search apparatus according to claim 2, wherein the selecting means selects image features by deleting image features that do not exist in the image feature list from image features of the search source image.   3. The image search apparatus according to claim 2, wherein the selecting means selects an image feature that matches the image feature list by identifying the image feature in the image feature of the search source image. The image search apparatus according to claim 4, wherein the information related to the location and the shooting direction is at least one of GPS information, direction information, and shooting altitude information. The image search apparatus according to claim 4, wherein the time- related information is local time information at the shooting location. 5. The image search apparatus according to claim 4, wherein the weather information is at least one of weather information , temperature information , humidity information , and atmospheric pressure information . The information related to the shooting mode is at least one of information related to the dial setting of the camera shooting mode, information on the presence / absence of flash , information on whether to perform macro shooting , information on whether to perform telephoto shooting , or information on subject attributes. The image search apparatus according to claim 4, wherein the image search apparatus is provided. Storage means for storing image features corresponding to an image and attribute information when the image is captured;
Transmitting means for transmitting information on the number of image features co-occurring in the image for each attribute information to a request source in response to a request;
Receiving means for receiving an image feature selected by a request source as an image feature to be used for searching a search source image based on information on the number of image features co-occurring in the image for each attribute information;
Comparing means for comparing the image feature of the received search source image with the image feature stored in the storage means;
An image search apparatus comprising: An input process for inputting an image;
An acquisition step of acquiring attribute information at the time of capturing the image;
An extraction step of extracting image features of the image;
The step of storing the image, the attribute information, and the image feature in association with each other in the storage unit, and the search source input in the input step based on the number of image features that co-occur in the image for each attribute information A selection step of selecting an image feature to be used for search of a search source image and an image feature to be restricted among the image features extracted in the extraction step for the image,
A comparison step of comparing the image features selected to be used for searching the search source image with the image features of the registered image stored in the storage means;
An image search method characterized by comprising: Storing image characteristics corresponding to an image and attribute information at the time of capturing the image in a storage unit;
A transmission step of transmitting information on the number of image features co-occurring in the image for each attribute information to a request source upon request;
A receiving step of receiving an image feature selected by a request source as an image feature to be used for searching a search source image based on information on the number of image features co-occurring in the image for each attribute information;
A comparison step of comparing the image feature of the received search source image with the image feature of the registered image stored in the storage means;
An image search method characterized by comprising: Computer
An input means for inputting an image;
Acquisition means for acquiring attribute information at the time of photographing the image;
Extracting means for extracting image features of the image;
Storage means for storing the image, the attribute information, and the image feature in association with each other;
Based on the number of image features co-occurring in the image for each attribute information, the search source image input by the input unit is used for searching the search source image among the image features extracted by the extraction unit. A sorting means for sorting into image features and image features whose use is restricted;
Comparison means for comparing the image features of the search source image selected for use in searching the search source image with the image features of the registered image stored in the storage means;
Program to function as. Storage means for storing an image feature corresponding to an image and attribute information when the image is captured;
Transmitting means for transmitting information on the number of image features co-occurring in the image for each attribute information to a request source in response to a request;
Receiving means for receiving an image feature selected by a request source as an image feature to be used for searching a search source image based on information on the number of image features co-occurring in the image for each attribute information;
Comparing means for comparing the image feature of the received search source image with the image feature stored in the storage means;
Program to function as.

Images