JP2000242648A - Searching image spatial display method and apparatus, and storage medium storing program for this method - Google Patents

Abstract

(57) [Summary] [PROBLEMS] To provide a method and apparatus for arranging and displaying a set of search images in space, intuitively displaying the arrangement of search requests, and making it easy to grasp the relevance between the search requests and the search images. In addition, the judgment result of the relevancy is made easy to understand, and the search time is shortened. An image search unit (30) searches for an image having a high degree of association between a search request and a feature amount of an image on an image database (10). The image arrangement unit 40 arranges a set of searched images in an intuitive expression space with a part of a feature amount used in the search as an axis and the position of an image to be searched as an origin. The image selection unit 50 selectively displays only a set of images in a partial space designated by the user. Thus, the arrangement image indicates the feature amount in which the direction from the origin is focused, and the distance from the origin indicates the degree of relevance, so that the relevance can be easily grasped. Further, since images with similar feature amounts are arranged on the extension line of the origin and images having high relevance are gathered near the origin, the search time can be reduced by specifying one space portion.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a method and an apparatus for displaying video search results.

[0002]

2. Description of the Related Art Conventionally, in a method of displaying a video search result, color characteristics of an image, shape information of an object, linguistic information such as keywords, telops, and voices, and movements of a camera and an object are extracted as feature amounts. By calculating the degree of association with the search request input by the user, an image related to the image desired by the user is displayed. For example, the document "Qu
ery Image and Video Conte
nt: The QBICSystem "(M. Flic
Kner et al., IEEE, Vol. 28, no. 9, p
p. 23-32, Sept. 1995), a sample image or an image drawn by hand is input as a search request, and the similarity of the image on the database is calculated using the color information of the image, the shape information of the object, the layout, etc. Make a decision. When displaying the search result, about eight images are sequentially displayed from the image judged to have a high degree of relevance, so that the time required to search for the relevant image is reduced.

[0003] Also, the document "Information V"
isuization With a Dig
ital Video Library ”(Chris
tel, M .; Other, Submitted for pub
ligation in asesial issue
e of Journal of Intellige
nt Information Systems de
aling with information vi
sualization, acceptedJune
1998), a plurality of search requests input by the user
The image is arranged on the extension on the dimensional plane, and an image related to the search request is arranged in a polygon formed by these search requests. The position of the image is determined by the degree of relevance to the search request, and it is determined that the closer the distance to the search request is, the higher the image is related to the search request. In this method, the user can grasp the relevance in more detail by enlarging (zooming) the partial space of interest.

[0004]

In the former prior art, the ranking is performed using the sum of the similarities of the feature amounts, and the image is displayed one-dimensionally as shown in FIG. The images are arranged without a sense of unity, for example, the first place has a high similarity of the feature amount A, and the second place has a high similarity of the feature amount B. Further, there is a problem that it is difficult to intuitively understand how much each feature amount is reflected in the degree of association.

In the latter prior art, in which the relevance is expressed by spatially arranging the search image in a polygonal space in which the search request is extended, the relevance of the search request arranged on the screen is reduced. The position is not intuitive, the more search requests, the more difficult it is to grasp the space between the image and the search request, the more difficult it is to determine the relevance, and the spatial arrangement does not reflect which image was determined to be the most relevant There is a problem that it is difficult to understand. In addition, since the distance from a search request placed outside the space indicates the relevance, in the initial state, all the searched images must be placed in the space, and the user needs to grasp the entire space. Therefore, there is a problem that it takes time to display a screen and a problem that it is necessary to search for a target image by zooming a partial space, and the number of steps required for the search increases.

It is an object of the present invention to provide a one-dimensional display method based on ranking, in which it is difficult to grasp the relevance between each search request and a searched image, and in a polygon space in which search requests are extended. In the display method where images are arranged, the search request arrangement is not intuitive, the more search requests, the more difficult it is to grasp the relevance to the image, the more difficult it is to determine the relevance judgment result, and the search method is used to arrange the search requests. It is an object of the present invention to provide a method and an apparatus for displaying a video search result, which solves the above-mentioned drawbacks of the prior art that time is increased.

[0007]

In order to solve the above problems, a search image display method according to the present invention obtains a degree of association between an input search request and an image feature amount on a database,
A step of retrieving an image having a high degree of relevance; and focusing on at least one or more of a plurality of image feature amounts input as the search request, determining a value of the feature amount and a feature in each image of the searched image set. The direction of the feature amount vector of each image is determined by comparing with the value of the amount, and the feature amount vector of each image is determined by the relevance between the search request and each image derived at the stage of searching for the image. Determining the size, and expressing the features of the image in a direction from the origin of the expression space on an expression space corresponding to the feature amount of interest by the direction and size of the determined feature amount vector. Arranging the images so as to indicate the degree of relevance by the distance from the image.

Alternatively, in addition to the above steps, in order to reduce the search time, a partial space in the expression space is designated,
An operation of moving or deforming the subspace is performed, and only an image arranged in the subspace is selected and displayed.

Alternatively, a program for causing a computer to execute the steps in the method for displaying a search image spatially is stored in a storage medium readable by the computer.

[0010] Similarly, the search image display device of the present invention comprises:
An image search means for obtaining a degree of relevance between the input search request and the image feature amount on the database and searching for an image having a high degree of relevance; and at least one of a plurality of image feature amounts input as the search request And the direction of the feature vector of each image is determined by comparing the value of the feature with the value of the feature in each image of the searched image set, and the image is searched by the image search means. The magnitude of the feature vector of each image is determined according to the degree of association between the search request and each image derived at that time, and the magnitude and direction of the determined feature vector are determined in accordance with the feature quantity of interest. Image arranging means for arranging each image on the expression space in such a manner that the characteristic of the image is represented by a direction from the origin of the expression space and the degree of relevance is represented by a distance from the origin. And

Alternatively, in order to shorten the search time,
An image selecting means for designating a partial space in the expression space, and performing an operation of moving or deforming the partial space, selecting and displaying only an image arranged in the partial space. I do.

According to the present invention, an image search step (/ means) for obtaining a degree of relevance between an input search request and a feature amount of an image on a database, and searching for and ranking images having a high degree of relevance.
And an image arranging step (/ means) for arranging the feature amounts of the arranged images in an intuitive space using at least one feature amount of interest, and a partial space in a space designated by the user. Image selection step (/ means) for selecting a set of images
And

That is, in the present invention, attention is paid to a part of the feature amount used in the search, and the image is arranged in an intuitive expression space according to the feature amount. The arrangement position is determined by a feature amount whose direction from the origin is noted, and the distance from the origin is determined by the relevance used in the search stage (/ means). The relevance between the image, the degree of relevance, and the feature amount can be easily grasped.
Also, by adopting such an arrangement method / apparatus,
Since images with similar feature amounts are arranged on the extension line from the origin and images with high relevance are gathered at the origin, the user can grasp the space and search from images with high similarity without displaying the entire space. I can do it. Therefore, as an initial state, it is only necessary to display a partial space centered on the origin which is considered to be most relevant.

In the conventional space arrangement method, since the user needs to display the entire space and zoom in on a subspace having a highly relevant image or an interesting subspace,
The method / apparatus can reduce the search time.

[0015]

Next, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing an embodiment of the apparatus of the present invention. An image database 10 storing images to be searched, a search request input unit 20 for inputting a search request or operation from a user to the apparatus. An image search unit 30 that obtains a relevance of each feature amount from a search request and an image in a database, searches for a relevant image from the total value, and arranges images according to the degree of the relevance, , An image arrangement unit 40 for arranging the images arranged by the image search unit 30 on an expression space corresponding to the feature amount which is most noticed, and an image selection unit 50 for selecting a part of the expression space according to a user's request A display 60 for displaying an image on the selected expression space,
The control unit 70 controls these units.

The present invention particularly relates to the processing of the image arrangement unit 40 and the image selection unit 50. Specifically, the image arrangement unit 40 compares the value of the feature amount of interest input by the search request input unit 20 with the value of the feature amount of the image registered in the image database 10 to extract the feature amount vector. By doing so, the image set is rearranged on an intuitive expression space corresponding to the feature amount.

For example, if the most noticeable feature amount is the color information of the entire image, the expression space is an expression space having an origin centered on each of the RGB colors.
It is rearranged using a feature amount vector corresponding to the most characteristic color compared to the value of the feature amount of interest input by the user.

The length of the feature vector is determined by the image search unit 30
Is determined based on the degree of relevance used when searching for an image, and is determined such that the lower the degree of relevance, the farther from the origin. In this way, in the expression space, the origin corresponds to the image to be searched, and the lower the relevance, the farther the image is located. Also, to show how the direction of the vector to each image centered on the origin differs from the image value to be searched for in the feature amount of interest, that is, the feature amount of the image to be searched, Are arranged on the same line. Therefore, the user need only determine the image close to the image only in the direction from the origin, and there is no need to grasp the entire space.

The image selecting section 50 selects a partial space requested by the user from the images arranged in the expression space in the image arranging section 40. For example, in the expression space using the above color information, if the user wants an image in which red is more emphasized among the searched images, the image selecting unit 5
“0” selects an image in the direction of the red axis viewed from the origin and close to the origin, and displays it on the display 60.

Hereinafter, an embodiment of an image spatial arrangement method according to the method of the present invention will be specifically described.

<Embodiment 1> This is an embodiment in which images are rearranged with respect to the color information of the entire image in a search request for an object, color information, or the like input by the user.

FIG. 2 is a flowchart showing a processing procedure of an embodiment of the method of the present invention.

First, as a step 100, a search request input unit 20 receives a feature amount of an image as a search request from a user.

The search request from the user is sent to the image search unit 30 through the control unit 70. At step 110, the image database 10 is searched with the data of the search request to obtain the corresponding image data. When acquiring image data, the image data can be ordered and arranged by determining the degree of relevance indicating how much the search request is satisfied. For example, an image having a degree of relevance equal to or greater than a certain value can be obtained as the relevant data by calculating the degree of similarity for each feature amount, and weighting and summing the similarities to obtain a degree of relevance. Since the calculation of the similarity of each feature and the calculation of the degree of association between the images can be achieved by the conventional technique, the details are omitted.

Next, in step 120, the search request input unit 20 receives which of the feature amounts of the image input by the user in step 100 to display the image by focusing on the feature amount.

The features specified by the user are stored in the control unit 70.
Is transmitted to the image arranging unit 40, and a comparison value between each image and the input value of the user in the feature amount designated in step 130 is calculated as a feature amount vector of each image. When the color information of the entire image is represented by a (R, G, B) three-dimensional vector using the color information of the entire image as a feature amount, the color specified by the user is (Ry, Gy, By), and the color of the entire image of the image A is Assuming that the information is represented as (Ri, Gi, Bi), the feature amount unit vector EA of the image _A is N = (Ri−Ry) ＾ 2
If + (Gi-Gy) ＾ 2 + (Bi-By) ＾ 2, it is expressed as follows.

[0028]

(Equation 1)

In this case, the feature quantity unit vector is a difference from the feature quantity that the user wants to search, and when compared with the image image that the user wants to search, the color direction in which the image A is most different in the image A is characteristically shown. For example, search strong image B red than the picture image should has the greatest value R _E from the calculation method of the feature quantity unit vector, the direction of the vector is characteristically exhibiting red axis. Step 1
The relevance between the image A obtained in step 10 and the user's search request
_{When A} is set, the feature amount vector of the image A is defined as a vector whose unit vector is E _A and whose size is C _A. By definition, the feature amount vector is expressed as a vector having a smaller magnitude in the (R, G, B) axis direction characteristic of the user-specified color information as the degree of relevance increases.

Next, in step 140, the image arranging unit 40 arranges the image in the expression space corresponding to the feature using the feature vector. The expression space when the color information is used as the feature amount is (R, G, B) as shown in FIG.
It is a representation space in which each axis is arranged in a direction at an angle of 120 degrees from the center. FIG. 3 shows an example in which an image is arranged in this space using the feature amount vector obtained in step 130. The image is arranged on the color space most characteristic of the image to be searched by the feature amount vector. Images with high relevance are located closer to the origin, and images with lower relevance are located farther away.

Next, in step 150, the search request unit 20 receives a partial space to be displayed from the user. The input subspace passes through the control unit 70, and in step 140, an image existing on the subspace is selected from the expression space arranged by the image selection unit 50. Examples of the input method include enlarging or reducing the partial space, designating a desired portion with a rectangle, and moving the partial space in a desired direction. As an initial state, n around the origin
It is appropriate to specify a partial space having a range of one image, and it is not always necessary to prompt the user to input in the initial search. The variable n can be determined on the system side, or an input can be obtained from the search request input unit 20.

The selected image is displayed on the display 60 as step 170. FIG. 4 is an image when a partial space centered on the origin in the expression space in FIG. 3 is displayed on the display 60 as n = 4. The outer rectangle indicates a display portion on the display 60. When the image is displayed on the display 60, the space is enlarged and reduced in consideration of the display range of the screen and the area of the selected partial space so that the image is arranged in the screen.

Next, in step 180, when a desired image is not found, another subspace is selected (step 190), and another image is set as the center of the expression space (step 200). The amount is selected again (step 210), and the search is repeated (step 2)
Perform 20).

If another subspace is selected in step 190, the process returns to step 150. In step 150, operations such as enlargement / reduction and movement of the area are acquired from the search request input unit 20 with respect to the partial space displayed on the display 60, and the process proceeds to step 160.

In step 200, when another image is set as the center of the expression space, an operation of selecting an image to be set as the center is acquired from the search request input unit 20, and the process proceeds to step 130. By retrieving the feature vector between the selected image and the other images in step 130, the search image can be rearranged, and the process proceeds to step 140.

When reselecting a feature value of interest as step 210, the process proceeds to step 120.

When the search is redone at step 220, the process proceeds to step 100.

If the image to be searched for is found or none of these processings is found, the processing is terminated.

Further, in addition to the above, the present invention focuses on camera operation such as panning and tilting of the camera in the vertical and horizontal directions, and also focuses on the position and movement of the object of interest. As in the case of being arranged in a space, a feature amount vector is calculated according to the feature amount of interest,
By preparing an expression space corresponding to the feature amount, it becomes easy to intuitively grasp the relevance between the search request and the actually searched image.

<Embodiment 2> FIG. 5 shows an embodiment focusing on camera operation. The image 500 corresponds to the image image to be searched, and it can be understood that the image 501 is an image in which the operation direction of the camera is relatively large to the upper right with respect to the image 500. It can also be seen that the image 501 has a lower similarity to the image 500 than the image 502 due to the distance from the center. An image 503 in which the camera operation is performed to the upper left relative to the image 500 is arranged at the position shown in the figure.
From the distance from the center, it can be intuitively grasped that the similarity is lower than that of the image 501.

<Embodiment 3> FIG. 6 is an example showing how the position of the house as an object is relatively different from the position of the house of the image to be searched. The image 600 corresponds to the image to be searched, and the image 601 is arranged at the position shown in the figure because the position of the object is relatively higher than the image 600. The image 602 is located at the right position in the figure because the position of the object is on the right side of the image 600. From the distance from the center, it can be intuitively grasped that the images 601 and 602 have substantially the same similarity as the image 600 as a whole image, but differ in the position of the object of interest.

It is needless to say that a part or all of the means shown in FIG. 1 can be made to function using a computer, or that the processing procedure shown in FIG. 2 or the like can be executed by a computer. A program for causing a computer to function as the means or a program for executing the processing procedure by the computer is stored in a storage medium readable by the computer, for example, FD (floppy (registered trademark) disk), MO, ROM, memory card, CD, DV
D, stored in a removable disk or the like, provided, and distributed.

[0043]

As described above, according to the present invention,
Attention is paid to a specific feature amount in the search request, a feature amount vector is calculated for each searched image based on the feature amount, and the feature amount is used according to the feature amount using the feature amount vector and the degree of association used at the time of the search. Images can be placed in the expression space, and necessary subspaces can be displayed according to the user's requirements, so that the user can accurately grasp and search for images according to his or her preferences. , And can be searched in a short time.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an embodiment of the device of the present invention.

FIG. 2 is a flowchart showing a processing procedure of an embodiment of the method of the present invention.

FIG. 3 is a diagram illustrating Embodiment 1 of image arrangement in an expression space for describing the present invention in an expression space corresponding to color feature amounts of the entire image.

FIG. 4 is a diagram showing a partial space centered on the origin on the display in the color expression space of FIG. 4;

FIG. 5 is a diagram showing Embodiment 2 of image arrangement in an expression space for describing the present invention in an expression space corresponding to a camera operation feature amount.

FIG. 6 is a diagram showing Embodiment 3 of image arrangement in an expression space for describing the present invention in an expression space corresponding to a position feature amount of an object (house).

FIG. 7 is a diagram showing a method of displaying search results by a conventional one-dimensional ranking method.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 ... Image database 20 ... Search request input part 30 ... Image search part 40 ... Image arrangement part 50 ... Image selection part 60 ... Display 70 ... Control part

Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat II (reference) G06F 15/62 P 15/70 460B F term (reference) 5B050 AA08 AA09 AA10 BA11 CA07 DA04 EA03 EA07 EA12 EA13 EA18 FA02 5B075 ND12 PP23 PP28 PQ02 PQ23 PQ32 PQ46 PQ48 PQ49 PQ74 PR06 QM08 QP05 5E501 AA13 AC15 AC33 BA05 CA02 EB05 FA14 FA23 FA47 FB04 FB43 5L096 AA02 DA04 EA23 EA31 EA35 EA39 FA15 GA40 GA41

Claims (5)

[Claims] 1. A step of obtaining a degree of relevance between an input search request and an image feature amount on a database, and searching for an image having a high degree of relevance, at least one of a plurality of image feature amounts input as the search request Determining the direction of a feature vector of each image by comparing the value of the feature with the value of the feature in each image of the set of searched images, and retrieving the image; Determining the magnitude of the feature vector of each image based on the degree of relevance between the search request and each image derived in the above; and determining the magnitude of the feature vector by the direction and magnitude of the determined feature vector. Arranging each image on a corresponding expression space in such a manner that the characteristics of the image are represented by a direction from the origin of the expression space and the degree of relevance is represented by a distance from the origin. Spatial display of search images Law. 2. In addition to the steps described above, in order to reduce a search time, a partial space in an expression space is designated, and an operation of moving or deforming the partial space is performed. 2. The method according to claim 1, further comprising the step of selecting and displaying only the search image. 3. An image search means for obtaining a degree of relevancy between an input search request and an image feature amount on a database, and searching for an image having a high degree of relevance, and a plurality of image feature amounts input as the search request. Attention is paid to at least one of the images, and the direction of the feature vector of each image is determined by comparing the value of the feature with the value of the feature in each image of the searched image set. The magnitude of the feature vector of each image is determined based on the relevance between the search request and each image derived at the time of searching for an image by the method, and the direction of interest and the magnitude of the determined feature vector are determined. On an expression space corresponding to the obtained feature amount, an image arrangement means for arranging each image so as to represent a feature of an image by a direction from the origin of the expression space and to express a degree of association by a distance from the origin. Characterized by having Space display device of the search image. 4. In order to shorten a search time, when a partial space in an expression space is designated and an operation of moving or deforming the partial space is performed, only an image arranged in the partial space is selected. 4. The search image spatial display apparatus according to claim 3, further comprising an image selection means for displaying. 5. A method for displaying a search image according to claim 1, wherein a program for executing the steps in the method for displaying a space of a search image by a computer is stored in a storage medium readable by the computer. Storage medium that stores the program of.