JP2006164008A - Image search apparatus and image search method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image search device capable of easily finding one favorite image while enjoying it from a large number of images.
An image search apparatus includes a display unit 106, a database 104 that stores an image, an image feature amount extracted from the image, and meta information that indicates image capturing conditions in association with each other, and a type of the image feature amount. Information generating means (input unit 101) for generating first information including the second information including the type of meta information, and searching for an image having an image feature amount based on the first information And a search key for generating a second search key for searching for an image having meta information that matches the meta information based on the second information. A generation unit 105 and a search unit 103 that searches the database 104 using at least one of a first search key and a second search key and sequentially displays a plurality of searched images on the display unit 106 are provided.
[Selection] Figure 1

Description

  The present invention relates to an image retrieval apparatus and method for storing and managing a large number of images such as digital photographs and retrieving the images.

  In recent years, digital cameras and mobile phones with digital cameras have become popular, and the digitization of photographs has been remarkable. Then, from the cycle of printing out all photos taken like a conventional silver halide camera to paper media, saving them in an album, and viewing them, the digital data is saved in a personal computer and viewed on a monitor. , The cycle is changing to print out only the digital photos you like on paper.

  In order to perform these operations on a personal computer, application software generally called digital camera software is sold. However, with any kind of digital camera software, when managing a large number of images, basically follow the basic concept of the file system of the operation system, and build folders in a tree structure and place them under each folder. Manage by putting digital photos. The user can distinguish the folder name by assigning a unique name depending on the shooting date, event name, shooting location, and the like. That is, the classification based on the folder and the file name is assumed.

  It is important to be able to quickly and easily reach a desired photograph from digital photographs managed by such a tree structure. However, the management by the tree structure has the following three problems.

  First, it is a very hard task for users to name each folder. The widespread use of digital cameras has changed taking pictures from the act of “fixing special events” to the act of “fixing casual everyday life”. As a result, the number of photographs to be taken is enormous, and the frequency of taking pictures is frequent, and it is almost impossible to name each group.

  Second, it is difficult to trace the relationship between folders. In the file management of the tree structure, there is usually some relationship between the child folders arranged under the parent folder. Still, if you are looking at the photos under the child folder and then see the photos in the next child folder, you can see the photos only after you have gone up the tree structure and reflected in the next child folder. Since a completely irrelevant work is sandwiched in a series of actions of seeing, the work efficiency is poor.

  The last third is that you can't remember what photos are in which folder. If the number of photos becomes enormous, the number of folders will become enormous, and it is actually impossible to remember the structure and name of all of them. Also, even if the folder name or file name is automatically given according to the shooting date and time, it is impossible to remember what the photograph was taken to fix that day, what happened, etc. It cannot be distinguished only by the folder name.

  Now, one of the technologies that form the basis of the Internet that has been remarkably popular in recent years is hyperlinks. A hyperlink is position information such as another document or an image embedded in a document. When a system = hypertext in which a plurality of documents and objects such as related images are associated using the hyperlink is constructed, Other documents and images can be easily browsed while following the position information.

  If image management using hyperlinks is performed, the above-described problems based on the tree structure can be solved, but new problems also arise. For example, link disorder. Since information is handled as the same information group without being classified at all (flat file management), when a large amount of information is handled, there is a problem that links are disordered and the user is confused when browsing information. Has occurred. In addition, when a part of information is updated, there is a problem that it is difficult to update related information. Furthermore, when information to be handled becomes large, there is a problem that it becomes impossible to determine what information can be obtained before following the link.

  In order to solve these problems, the inventions disclosed in Patent Document 1 and Patent Document 2 have been conventionally proposed.

FIG. 15 shows a conventional image search apparatus and method described in Patent Document 2.
In FIG. 15, the link information management unit 1 manages link information 2 for associating a specific area in the image with the image. The image search information management unit 3 manages image search information 4 that is information used to read an image from the data storage unit 9 based on the link information 2. The group management unit 5 manages the link information 2 and the image search information 4 described above for each group. The link search unit 6 traces the positional information of the area in the image instructed by the input unit 8 with respect to the image displayed on the display unit 7, reads the image associated with the area from the data storage unit 9, Displayed on the display unit 7.

  The link destination determination unit 11 determines what the link destination image group is based on the link information 2, and based on the determination result, the link destination visualization unit 12 visualizes the link destination image group, The visualized information is displayed on the display unit 7 together with the previously displayed image.

  Hereinafter, the operation of the conventional image search apparatus will be described with reference to the flowchart of FIG.

  It is assumed that “map” is displayed as an image on the display unit 7. When the image is displayed, the link destination determination unit 11 extracts the region information and inquiry information of the display image from the link information management unit 1, and compares the keyword attribute of each region information with the region information of the inquiry information.

  If they match, the link destination visualization unit 12 displays the link destination group on the screen as “with link”.

If they do not match, no link is established from the area in the display image. Since the above link destination group guide processing is performed every time an image is displayed, each time the searched image is displayed, the corresponding link destination is compared based on the flowchart of FIG. 16, and the display content of the link destination group is updated. . As a result, the user can confirm in advance the link destination from the display image at the stage of image display, and can efficiently perform direct search while recognizing the search target.
JP-A-1-233670 Japanese Patent No. 2506987

However, even the conventional configuration does not solve all the problems.
That is, since the link is static, the name of the link destination image or image group must be registered in advance. This is a very heavy routine work in a world where a large amount of images are produced every day, and raises the threshold for storing or managing images.

  Furthermore, if the link construction is arbitrary and the link creator and the viewer are different, the intention of the link creator is not understood, and an arbitrary image cannot be reached. In addition, if a large number of links are created, it is often said that the link creator forgets what kind of image the link has built and cannot reach any image. sell.

  The present invention has been devised in view of the above-mentioned problems, and even in a world where a large amount of images are produced, there is no burden of storage / management work for the user, and it seems to be interesting. To provide an image search apparatus and method capable of finding a favorite piece while repeating transitions from one image to another, and tracing and viewing images while enjoying the transition between images itself. Objective.

  That is, an object of the present invention is to provide an image search apparatus and method that can easily find a favorite image while enjoying a large number of images.

  In order to solve the above-described conventional problems, in the image search apparatus according to the present invention, an image search apparatus that extracts a desired image from a plurality of images, and display means for displaying the extracted image And a database that stores the image, the image feature amount extracted from the image, and meta information indicating the shooting condition of the image in association with each other, and first information including the type of the image feature amount And an information generating means for generating second information including the type of the meta information, and a first for searching for an image having the image feature amount based on the first information generated by the information generating means. And a second search key for searching for an image having meta information matching the meta information, based on the second information generated by the information generation unit Do Search key generation means; Search means for searching the database using at least one of the first search key and the second search key, and sequentially displaying the searched plurality of images on the display means. It is characterized by that.

  As a result, the user understands the tree structure, moves from folder to folder, displays images placed there, searches for images to be imaged, and creates link destinations in advance. It becomes unnecessary, and it is possible to find one favorite image while repeating the transition from an image that seems to be of interest to the image, and to trace and view the image while enjoying the transition between the images.

  In the image search device according to the present invention, the information generation unit includes the type of the image feature amount of the image specified by the user for the image displayed on the display unit and the type of meta information of the image. It is characterized by comprising input means for inputting.

  As a result, the user can specify an image of interest by a simple operation, and the related image group is extracted from the image as a starting point. Therefore, the user can move from one image group to the next. It is easy, and by repeating it, a desired image can be reached like surfing the net.

  In the image search device according to the present invention, the input unit accepts a designation of a pair of two or more images and the type of the image feature amount, and the logical operation type of the two or more image feature amount The image feature amount extraction unit extracts the specified image feature amount from two or more images, and performs the specified logical operation on the extracted image feature amount, thereby obtaining the first search key. It can also be characterized by generating.

Thereby, it is possible to extract an image that widens the range of association of the user.
Further, in the image search device according to the present invention, the input means detects the overlap when the display means displays two or more images in duplicate, and designates the overlap area as a designated area. And the input means accepts designation of the type of image feature quantity to be extracted from the overlap area and the type of logical operation based on the image feature quantity extracted from the overlap area. It can also be.

This also makes it possible to extract an image that broadens the range of user associations.
In the image search apparatus according to the present invention, the display means may select and reproduce music popular in the year of shooting date and time of the image group selected by the user.

  This not only enables the extraction of images that broaden the range of associations of the user, but also allows the user to recall the memories more clearly by displaying / playing music at the same time as the user's photos, and further traces the images. The entertainment nature of actions can be improved.

  In the image search device according to the present invention, when the search unit searches the database using the first search key, the search unit simultaneously acquires the degree of coincidence of the extracted image with respect to the first search key, and the degree of coincidence It is also possible to display the image on the display means in either descending order or ascending order.

  Thus, according to this rule, images can be arranged from the image group having the highest similarity with the region designated by the user first, and thus the image group extracted in the order of the image group having the highest user's interest is provided to the user. be able to.

  In the image search device according to the present invention, the search means first searches the database using the second search key, and then searches the database using the first search key. It can be.

  This also allows the user to specify the image of interest and to move from one image group to the next because the relevant image group is extracted from that image. By repeating this, it is possible to reach a desired image like surfing the net.

  In the image search device according to the present invention, the search key generation unit may generate two or more search keys based on two or more types of meta information.

  As a result, an image group having a plurality of relevances can be presented to the image extracted by the first search key, so that the user can jump to an image group that is more interesting from among them, and has a wider range of associations. To find unexpected images.

  In the image search device according to the present invention, the image search device further includes a recognition unit that identifies an object in the image feature quantity and outputs the name as a first search key. The database may be searched using at least one of the first search key and the second search key generated by the recognition means.

  Thus, by adding a recognition means, not only images containing the same partial image but also all images related to the object can be extracted by similarity search using key images, so images with different shooting angles that cannot be handled by the previous similarity search. In addition, past images that do not change with time can also be extracted, and the range of user associations can be further expanded.

  In the image search device according to the present invention, the image search device further includes an input device for acquiring environmental information from the outside, such as date, position, and temperature, and outputs the environmental information acquired by the input device. An acquisition unit may be provided, wherein the search unit searches the database using at least one of the first search key and the second search key and the environment information.

  As a result, if the season is early summer, a group of photographs taken in the early summer season is preferentially extracted. It is possible to extract a photograph and induce the user to recall the memory.

  In the image search device according to the present invention, the information generation means reads music data, analyzes the feature amount, and generates and outputs a scenario for performing a search along a characteristic sound of music. Scenario generation means, scenario reproduction means for controlling the search of the database and the display of the extraction result by issuing the first and second information according to the scenario while reading the scenario and starting the music reproduction It can be characterized by comprising.

  As a result, as long as the user selects BGM, the image search device switches the image to be selected according to the change point, and selects the image according to the sound feature amount. You can enjoy the transition of the image. That is, even if the user does not perform any operation, the images are switched as in the case of surfing the net. Furthermore, since it does not change according to a user's liking, an unexpected image is dug up and displayed, and a new memory can be aroused.

  In the image search device according to the present invention, the information generation means presents a plurality of themes as options to the user, selects one of them, and outputs the selected theme. , A scenario reading means for selecting and reading a scenario corresponding to the theme, and a scenario reproducing means for issuing the first and second information according to the scenario and controlling the search of the database and the display of the extraction result It is characterized by comprising.

  Even in this case, as long as the user selects a theme, the image transition process is displayed according to the scenario selected according to the theme, and the transition process of the image is displayed. Can enjoy. Furthermore, since the transition does not change according to the user's intention, an unexpected image is dug up and displayed, thereby arousing a new memory.

  Note that the present invention can be realized not only as such an image search apparatus, but also as an image search method using steps characteristic of the image search apparatus as a step. It can also be realized as a program to be executed. Needless to say, such a program can be distributed via a recording medium such as a CD-ROM or a transmission medium such as the Internet.

  As is apparent from the above description, according to the image search apparatus and method of the present invention, the second image is extracted from each image group extracted based on the image feature amount / meta information serving as the first search key. An image group is extracted using the search key, and the extracted image group is displayed in a matrix form, whereby an image related to the user can be presented. Therefore, the user can trace the related images by selecting one from the related images and repeating the search and display. At this time, there is no need for the user to worry about where an image related to a certain image is stored, so it is possible to find a favorite image by tracing the related image without being hindered by the act of tracing the image. .

  Therefore, according to the present invention, it is easy to arrive at a target image from a large amount of images, and the practical value of the present invention in the present day when digital cameras and the like have become widespread is extremely high.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
(Embodiment 1)
FIG. 1 is a block diagram showing a functional configuration of the image search apparatus according to Embodiment 1 of the present invention.

  As shown in FIG. 1, the image search apparatus includes an input unit 101, a feature amount extraction unit 102, a search unit 103, a database 104, a search key generation unit 105, and a display unit 106.

  The input unit 101 is information indicating the region designated by the user for the image displayed on the display unit 106 to generate the first search key and the type of feature quantity to be extracted therefrom. Key type (key1.type) and a second key type (key2.type) that is information indicating the type of meta information to be read from the image extracted by the first search key in order to generate the second search key. type) is input / input.

  The feature amount extraction unit 102 extracts key1... From the region in the image instructed by the input unit 101. An image feature amount of the type specified by type is extracted. Then, the feature amount extraction unit 102 converts the extracted feature amount into key1. data, which is output as the first search key. It should be noted that the feature quantity extracted from the image is extracted at the same time when it is registered in the database 104 in addition to the on-demand extraction format for the designated area, and is stored in association with the image data. A format of reading from the database 104 in accordance with an instruction of the input unit 101 may be used.

  The search key generation unit 105 calculates the key2.com from the meta information of the image extracted by the first search key (key1.data). The information specified by type is read, and the read information is read as key2. Set to data and output as the second search key. For example, key2. If type is the shooting date and time, key2. data acquires the actual shooting date and time of the extracted image from the meta information, and the key2. Set to data.

  The search unit 103 uses key1. data and key2. The database 104 is searched using data, and an image having a matching feature amount is extracted. That is, the search unit 103 uses key1. The database 104 is searched using data, and key1. Extract an image having a feature amount that matches the data, and use key2. data is obtained, and the obtained key2. Further search the database 104 by data, key2. An image having meta information that matches data is extracted.

  The database 104 is configured by storage means such as a hard disk, and can store a large amount of images. At this time, the image file is managed in the storage means depending on the file system that controls the image file, but there is no need to construct a tree structure, and all images are stored flat in one folder. . Of course, a folder may be constructed in a tree structure and set so that all subfolders under the root are referenced.

  The storage means may be a single disk drive such as a CD / DVD, a combination of a plurality of disk drives, or a semiconductor storage device.

  In addition to the image data itself, the data stored in the storage means indicates shooting conditions (shooting date, shooting location, shooting conditions (aperture, brightness, focal length, etc.), shooting equipment name, etc.) added during shooting. The meta information and the image feature amount extracted from the image data are included. The data may be stored in one file or another file, but it is important that the data is stored in association with each other.

  The image data itself and the meta information are generally stored in a format defined in the EXIF format. Of course, the TIFF / EP format, NSK-TIFF of the Japan Newspaper Association, and the standard TIFF-FX for FAX of Xerox are also candidates. The image feature amount is a physical feature amount that can be extracted by image processing of image data such as the position / size / tilt / representative color / average color of the face or object, and the representative color and average color of the region of interest in the image. It is extracted when it is stored in the database, and is stored in association with the image data.

The display unit 106 displays the image group extracted by the search unit 103.
In the image search apparatus configured as described above, there are search key pairs (key1, key2), and only their types are determined in the initial state. key1. “type” is a type of feature amount extracted from a specified area. The type is the type of information included in the meta information, such as date / time or location. The user applies key1.. To a region in the image displayed at random on the display unit 106. type, key2. Specify type. The feature quantity extraction unit 102 extracts the specified type of feature quantity from the area designated via the feature quantity extraction unit 102, so that the key 1. data is generated. The search unit 103 uses key1. The database 104 is searched by data to obtain an image group having the same feature amount. From each image group, key2. The search key generation unit 105 acquires information indicated by “type”, and stores the acquired data in key2. As data, the search unit 103 searches the database. Then, the display unit 106 causes the search unit 103 to execute key1. data, key2. An image group extracted by data is displayed.

  In the present embodiment, “search for an image” assumes an act of reaching a favorite one by tracing an image related to the displayed image. Fig. 2 illustrates how the image is traced, and for an image of interest from a series of images, a related image is traced to reach a favorite image. Or, the purpose is to trace itself.

  In this respect, there is a target image from the beginning, which is fundamentally different from an act in which a user searches for a specific image from a large number of image groups.

  FIG. 3 is a flowchart showing the operation of the embodiment of the present invention, in which the user displays the related image group from one image in the image search apparatus of the present invention. Hereinafter, details of the process of tracing an image will be described with reference to FIGS. 2 and 3.

  First, in a steady state before searching for an image, images stored in the database 104 are selected at random, sequentially displayed on the display unit 106, and presented to the user. In this state, the image search has not started yet, and there is a possibility that there is no relationship between the display images. Of course, it is possible to input a certain condition and display only the photos that meet the condition. For example, even if the year and month are limited and only the photos taken in that month and month are displayed, the location is limited and only the photos taken at the specified location are displayed. It doesn't matter. Further, the image extraction rule in the initial state is not limited to random, and may be other rules such as round robin.

  When the user happens to be interested in one of the images displayed on the display unit 106 and wants to see an image related thereto, the user can use the image search device of the present invention from here. The search for the selected image starts.

  First, the user designates a region in which the user is interested in an image and the type of information extracted from the region in the input unit 101 (S301). The designated area may be a geometrical shape such as a rectangle, triangle, or circle, the entire image, or a partial area such as a subject / object in the image.

  At this point, key1. type and key2. In type, only the type is specified, and the search key itself (key1.data, key2.data) is not determined. That is, if the combination is (key1.type, key2.type) = (representative color, date and time), then key1. The feature amount extracted as data is designated as a representative color, but it is not yet determined whether it is actually red or green. key2. The same applies to data, and only the type of date and time is specified, and the actual date is not determined.

  Next, the feature quantity extraction unit 102 performs key1. A feature amount of the type specified by type is extracted (S302). This feature amount is expressed as key1. data and output as the first search key.

  Key1 .. determined in step S302. Using the data, the search unit 103 searches the database 104 and extracts an image group having the same feature amount (referred to as an A group image) (S303). Group A images are key1. It is a set of images having a common term called data.

  Next, the search key generation unit 105 calculates the key2.. Key from the meta information of each image included in the group A. The information specified by type is acquired, and this is replaced with key2. Data is set (S304).

  Key2 determined in step S304. Using the data, the search unit 103 searches the database 104 again, and the key2. An image having meta information of data is extracted (S305).

  If there are n images of group A extracted in step S303, the search unit 103 repeats steps S304 and S305 n times (S306). As a result, for each image in group A, key2. An image having the same meta information as the data search key (an image group of 1 to n groups) is extracted.

  Finally, the search unit 103 displays the image groups from the first group to the n group on the display unit 106 in a two-dimensional matrix with all the extracted images as the representative images of the group A images.

  FIG. 4 shows a display example of the extracted image on the display unit 106. The images are displayed side by side on a two-dimensional array, and the horizontal axis is key1. This is a group of images extracted by data, and the vertical axis is key2. An image extracted by data is shown. In FIG. 3, group A images (A1, A2, A3,...) Are key1. Although the image group has a common feature amount of data, the group B image (B1, B2, B3,...) and the group C image (C1, C2, C3,...) have no common point. On the other hand, the first group (A1, B1, C1,...), The second group (A2, B2, C2,...), And the third group (A3, B3, C3,. Read key2. Since it is an image extracted by data, key2. There is a common point called data.

  Steps S301 to S307 in FIG. 3 show one cycle of the operation of tracing the image by the image search apparatus according to the present invention. By repeating this cycle, the user himself / herself while viewing the related image The user's interest is shifted, and a favorite image is reached from among a large number of images in the database 104.

  At this time, an important point is that no user work other than selecting an area is included when transitioning from an image to an image related thereto. Therefore, it is not necessary for the user to understand the tree structure, move from folder to folder, display the images placed there, and search for images to be imaged. It is possible to find a favorite piece while repeating the transition. Alternatively, it is possible to trace and view images while enjoying the transition between the images.

  Now, a scene in which a user uses an image search device will be described as an example of a situation in which a photo used for a New Year's card is selected from a large number of images stored in the image search device according to the present invention.

  When a user uses an image search device, from the beginning to the end, the user does not look for a single target image in the head and search for it in the database. In the process of selecting a photo that seems suitable for New Year's cards.

  First, a large amount of digital photographs taken by the user are stored in the database 104, which are New Year's photographs taken by the whole family in January 2004, photographs taken when skiing with the family in February 2004, 2004. Taken when child went to a theme park (for example, Universal Studios Japan (USJ)) in March 2004, photo of child 1's birthday, child 2 graduation ceremony, April 2004 child 2 entrance ceremony Photograph taken when traveling to Okinawa with family + grandparents in May 2004, wedding anniversary photo of a couple in June 2004, photograph taken when bathing with family + grandparents in July 2004, 2004 Included are photos of the September Sports Day and photos taken at the October 2004 Zoo.

  When not searching, in the initial state, the image search device displays an image on the display unit 106 while randomly selecting an image from all the images.

  The user is interested in a single image that happens to be noticed, and wants to see a related image from here.

  First, the user is interested in the child 1 shown in the image, and designates an area in which the child 1 is shown in the image.

  Then, the image search device extracts an image group in which the same face as the face in the area is extracted as a group A photo group, and each photo included in the group A photo group (three images) is extracted. From the shooting date, 1st to 3rd group photos are extracted and displayed on the display unit 106 in the format shown in FIG.

  Here, it is assumed that a group of photographs of Group 1 to New Year, group of photographs of Group 2 to ski, group of photographs of Group 3 to USJ are extracted, and photographs of Group A are displayed as representative images. When two groups of photographs (a group of ski photographs) are selected from these, images included therein are displayed. Needless to say, this includes a photograph of a person other than the child 1 (for example, the child 2).

  While looking at them, the user changes their interest in the child 2 and instructs the region where the face of the child 2 is present from the photograph of the child 2. The image search device repeats the same steps as described above, extracts an image group having the same face as the face in the designated area, further extracts a photo group of the same shooting date, and these are similarly used. Is displayed on the display unit 106. Here, it is assumed that a graduation ceremony photo group, a New Year photo group, and a USJ photo group are extracted.

  When you select a graduation photo group from these, the photos taken on that day are displayed. While looking at them, it became a topic about the clothes of the mother who attended together, so the area including the clothes of the mother was indicated, the image was extracted using the color as a feature, and A group of photographs having the same shooting date as that of the extracted image is extracted and displayed on the display unit 106 in the same format. Here, it is assumed that graduation ceremony, entrance ceremony, and wedding anniversary photos have been extracted.

  When I selected the entrance ceremony photo, there was a photo of my mother and two children, so I decided to use it for a New Year's card.

  As described above, the user designates the area of the image to be worried about, and at the same time, the related image group is extracted, so that the movement from the image group to the next image group is easy, and it is repeated. Thus, a desired image can be reached.

  If the image search apparatus according to the present invention is used, the user can extract a group of images of interest from the storage means of the database 104 in which all the captured images are simply stored. There is no need to make and classify. Since the user can trace an image that attracts interest as a starting point and trace an image related to the image based on the feature amount and the meta information, an action that prevents associative work such as changing a folder does not occur.

  Also, since viewing is not disturbed, it is possible to view a large number of images in the same way as surfing the Internet while changing the association key.

  It should be noted that when displaying 1 to n image groups side by side, key2. In addition to arranging the data in ascending or descending order, key1. A score may be given according to the degree of coincidence when searching for feature quantities using data, and the score may be arranged in ascending order of the scores.

  According to this rule, images can be arranged from the image group having the highest similarity with the area designated by the user first, that is, key1. The images extracted by data can be arranged in the order of their coincidence (score). Accordingly, it is possible to provide the user with an image group extracted in the order of the image group with the highest interest of the user.

  In the above-described embodiment, the case where one area is specified has been described as an example. However, the input unit 101 may be capable of specifying two or more areas. Of course, when there are two or more regions, the designation may be made not only from one image but also from two or more images. At this time, in addition to the region and the type of feature quantity to be extracted therefrom, the input unit 101 can further input two or more types of feature quantity logical operations.

  When two or more areas are specified, the search unit 103 performs a specified logical operation using the feature values extracted from the two or more areas as a key, and extracts an image group worth the A group. It will be.

  In other words, a plurality of areas may be specified in one or a plurality of photographs, and the search may be performed by performing a logical operation on the feature amounts of the areas.

FIG. 5 is a block diagram showing how an image group is extracted in the present embodiment.
Assume that six images are displayed on the display unit 501, the user is interested in two of them (image 502, image 503), and designates the areas of the images 502 and 503, respectively. The feature amount extracted from the image 502 is a representative color, and the feature amount extracted from the image 503 is a face. In this case, the search unit 103 performs a logical product operation on both image feature amounts to extract an image having both feature amounts. As a result, a photograph including face A and a red area is extracted as A group 505. For example, one of them (image 507) is a photograph of face A in the sunset (red) background. Selected.

Thereby, it is possible to extract an image that widens the range of association of the user.
In the present embodiment, it is described that the user explicitly specifies an area using the input unit 101, but the present invention is not limited to this format. In a steady state, when an image is selected at random and displayed on the display unit 106, when two images overlap each other, an overlapping determination unit that extracts the overlapping portion as a designated region is included. It does not matter. That is, in one or a plurality of photos, duplication of photos may be detected and searched by a logical operation of the feature values of the region.

  Therefore, in this case, the user only inputs the type of logical operation and the type of feature value to be extracted. Of course, the type of feature quantity may be automatically selected from a list registered in advance according to a rule such as random or round robin.

This also makes it possible to extract an image that broadens the range of user associations.
In the first embodiment, key1 is assigned a feature amount, and key2 is assigned meta information. First, the search is performed using the feature amount, and then the meta information of the extracted image is read to determine key2. It is not necessary to limit to the combination. In other words, the search key is replaced, and key1. type = meta information, key2. It is good also as type = feature-value.

  For example, the types of search keys may be switched so that key1 is the shooting date of the meta information and key2 is the shooting location of the meta information. In this case, instead of limiting the shooting date to August 1, 2004, it is possible to extract an image that expands the range of user associations if only the month is specified. Will.

  In the first embodiment, the image search apparatus has one first search key and one second search key. However, the number of second search keys need to be limited to one. Alternatively, two or more search keys such as key3, key4,... May be prepared, and an image may be searched using two or more search keys. In this case, since a plurality of related image groups can be presented to the image extracted by the first search key, the user can jump to a more interesting image group from among them. Similar to the effect described above, the range of associations can be expanded and unexpected images can be found.

  In addition, key2. When opening the image group extracted by data, if the key is the date, the music popular in that era may be adopted as the BGM. Specifically, when developing images of the first group to the n group, key2. If type is date, the music registered in the database is searched, and the music popular in that year is adopted as BGM. In addition, key2. If type is a place, it is also interesting to obtain an image related to that place and display it mixed with the image taken by the user. In any case, by acquiring the related second information by the meta information and displaying / playing back the music at the same time as the user's photo, the user can more clearly recall the memories, and further, the entertainment property of the action of following the image Will improve.

(Embodiment 2)
Next, an image search apparatus according to Embodiment 2 of the present invention will be described.

  FIG. 6 is a block diagram showing a functional configuration of the image search apparatus according to Embodiment 2 of the present invention.

The configuration is the same as that of the first embodiment except that a recognition unit 107 is added.
That is, the image search apparatus includes an input unit 101, a feature amount extraction unit 102, a search unit 103, a database 104, a search key generation unit 105, a display unit 106, and a recognition unit 107. Is done.

  The recognizing unit 107 recognizes an object (subject in the image data) extracted by the feature amount extracting unit 102 if it is a “face”, and if it is a “thing”, the name of the object. Identify the name as follows.

  In other words, in the image search apparatus, a recognition unit 107 is added so that the recognition unit 107 can recognize an object and search for an image by its name.

  FIG. 7 is a flowchart showing the operation of the embodiment of the present invention. Hereinafter, the operation will be described with reference to FIGS.

  First, the user designates an arbitrary image area from the input unit 101. At the same time, key1. type = specifies an object. In addition, the second search key type (key2.type) is also designated (S701).

  Next, the feature quantity extraction unit 102 extracts an object from the designated area (S702), and the recognition unit 107 recognizes the name of the extracted object (S703).

And the recognized name is key1. Data is output (S704).
The search unit 103 uses key1. Based on data, the database 104 is searched, and an image including a matching object is extracted (S705).

  The following steps S707 to S709 are the same as steps S304 to 307 (see FIG. 3) of the first embodiment.

  In the present embodiment, by adding the recognition unit 107, not only images including the same partial image but also all images related to the object can be extracted by the similarity search using the key image. Since images with different shooting angles and past images that do not change with time can also be extracted, the range of user associations can be expanded.

(Embodiment 3)
Next, an image search apparatus according to Embodiment 3 of the present invention will be described.

  FIG. 8 is a block diagram showing a functional configuration of the image search apparatus according to Embodiment 3 of the present invention. This image search apparatus has the same configuration except that the environment information acquisition unit 108 is added to the first embodiment.

  That is, the image search apparatus includes an environment information acquisition unit 108 in addition to the input unit 101, the feature amount extraction unit 102, the search unit 103, the database 104, the search key generation unit 105, and the display unit 106. Configured.

  The environment information acquisition unit 108 is a block that outputs one or more search keys to the search unit 103, includes an input device, and stores a memorial such as a birthday / marriage date registered in advance from the date of the month. It is possible to determine that it is a day, to determine the season, and to acquire position information by GPS.

  The search unit 103 performs a logical operation on the image feature amount and the environment information, and extracts an image from the database 104.

  Next, the operation in the embodiment of the present invention will be described with reference to the flowchart of FIG. In this flowchart, description will be made on the assumption that an image is extracted by determining a season from a date and performing a search that puts a weight on the season.

  First, the user selects an appropriate image from the displayed images and designates an area of interest in the image (S901).

  A feature quantity of the designated type is extracted from the designated area and is set as key1 (S902).

  The environmental information acquisition unit 108 determines the season from today's date and outputs it as environmental information (key-e) (S903). In this embodiment, if today's date is early July, the season is output as early summer.

  The search unit 103 searches the database 104 by a logical operation (logical product) of key1 and key-e, and is a photograph taken in the early summer. An image group having data (n images) is extracted as an image group of group A (S904).

  Subsequently, the search key generation unit 105 determines whether the key2. The meta information of the type of type is acquired, and this is stored in key2. Data is output (S905).

  Then, this time, the search unit 103 performs key2. The database 104 is searched only by data and an image is extracted (S906).

  The above steps S904 and S905 are repeated for n images in the A group, and image groups from the first group to the n group are obtained (S907).

  Finally, the search unit 103 displays images from the first group to the nth group with the group A images as the representative images (S908).

  Through the above steps, if the season is early summer, a group of photographs taken in the early summer season is preferentially extracted, so that it is possible to guide the user to select an image that appeals to the seasonality.

  Note that environmental information is not limited to the season, but if Memorial Day is output, it is possible to extract photos back to the past Memorial Day. Can be induced to do.

  Furthermore, the present invention is not limited to the determination of the date, and it may be configured to include an input device such as an acceleration sensor or a mouse and accept a user's positive input from those input devices. If an acceleration sensor is used, it is possible to reflect the user's emotions by shaking the input device, and if it is irritated, it is communicated by shaking, and the search unit 103 heals mainly on landscape images. It can be used as a key to select an effective photo.

(Embodiment 4)
Next, an image search apparatus according to Embodiment 4 of the present invention will be described.

  FIG. 10 is a block diagram showing a functional configuration of the image search apparatus according to Embodiment 4 of the present invention.

  Unlike Embodiments 1-3, this embodiment focuses on showing the process of going from image to image, like a demo screen of game software, rather than reaching a favorite piece. Is placed. That is, the present invention aims to select BGM, switch and display an image to be selected based on a scenario generated based on the feature amount, and enjoy the transition itself. In other words, BGM is selected, the search key is changed according to the music feature value, and the displayed image is changed.

  In contrast to the first embodiment, a scenario generation unit 109 and a scenario reproduction unit 110 are added instead of the input unit 101, and the other configuration is the same. That is, the image search apparatus includes the feature generation unit 102, the search unit 103, the database 104, the search key generation unit 105, the display unit 106, and the scenario generation unit 109 and the scenario in place of the input unit 101. A playback unit 110 is provided.

  The scenario generation unit 109 reads the music feature value, extracts the characteristic sound, and generates a scenario by determining a search key set according to the sound feature value.

  FIG. 11 shows an example of the scenario, and defines the type of search key for searching for an image to be shown from the time when a characteristic sound occurs as a change point.

  (1) indicates that at the time of music playback start (time: 00: 00: 000 milliseconds), key1. type = shooting date, key2. The type of search key to be acquired from an image of type = representative color is defined. In this example, the shooting date is acquired from the image (randomly selected here) that became the key image at the start of music playback, the still images with the same date are selected, and then each selected image It shows that the color which becomes the representative color is acquired from the image and the image is searched again by the color.

  As for (2) to (5), as in the case of (1), the key 1. type = shooting date, shooting location, object feature color, shooting date, and key2. type = a person's name, shooting date, shooting date, and shooting location definitions are acquired, and still images matching the conditions are searched.

  FIG. 12 is a flow chart for explaining the steps of showing the image following the music as BGM. Hereinafter, steps for tracing an image using the image search apparatus according to the present invention will be described with reference to FIGS.

  First, the scenario generation unit 109 reads a music feature amount corresponding to the selected music (S1201). The music feature amount is data previously extracted from the music data, and is stored in the database 104 in association with the music data. Note that the music feature amount is not limited to a format for reading pre-extracted data, but the scenario generation unit may include a music feature amount extraction unit, and a feature amount may be extracted in real time for the selected music. Absent.

Next, a scenario is generated based on the music feature amount (S1202).
After the music reproduction is started, the following steps S1203 to S1210 are repeated until the music reproduction is finished, that is, until the scenario is finished.

  First, the scenario reproducing unit 110 reads a scenario, and outputs (key1.type), (key2.type), and (image, area) when the time of the designated change point is reached (S1204).

  Next, the feature quantity extraction unit 102 receives key1. The feature amount of type is extracted, and the result is expressed as key1. Data is output (S1205).

  And the search part 103 is key1. The database 104 is searched by data (S1206).

  From the meta information of each extracted image, key2. The information specified by type is read, and this is read as key2. Data is output (S1207).

key2. The database 104 is searched with data (S1208).
The above steps S1207 and S1208 are repeated for all the images extracted in step S1206 (S1209).

  The image group extracted in step S1206 and step S1208 is displayed on the display unit 106 (S1210). At this time, key1. A single image is selected at random from the images selected by data, the image is focused, and still images are displayed in a slide show format in which the images selected from the image (key2.data) are switched one by one. . Of course, the image display is not limited to the slide show format, and various images can be added to dynamically display the image.

  According to the present embodiment, as long as the user selects BGM, the image search device switches the image to be selected according to the change point, and selects the image according to the sound feature amount. The user can enjoy the transition of the image. That is, even if the user does not perform any operation, the images are switched as in the case of surfing the net. Furthermore, since it does not change depending on the preference of the user, an unexpected image is dug up and displayed, and there is a possibility that a new memory is aroused.

(Embodiment 5)
Next, an image search apparatus according to Embodiment 5 of the present invention will be described.

  FIG. 13 is a block diagram showing a functional configuration of the image search apparatus according to Embodiment 5 of the present invention.

  In the present embodiment, as in the fourth embodiment, the main point is to show the process of tracing from image to image. The configuration is the same as that of the fourth embodiment except that a scenario reading unit 111 and a theme input unit 112 are added instead of the scenario generation unit 109.

  The theme input unit 112 includes a means for presenting a plurality of predetermined themes and selecting any one of them.

  The scenario reading unit 111 reads a scenario corresponding to the selected theme, and inputs this to the scenario reproducing unit 110.

  Thereby, when the user selects a theme, the search key is changed according to a predefined keyword string, and the displayed image is changed.

  FIG. 14 is a flowchart showing a process from theme selection to image display. Hereinafter, the manner in which an image is traced by the image search apparatus according to the present invention will be described with reference to FIGS.

  First, the theme input unit 112 presents selectable themes to the user and asks the user to select an arbitrary one from them (S1401).

  Next, the scenario reading unit 111 reads a scenario corresponding to the theme (S1402). At this time, there is a one-to-one correspondence between the theme and the scenario, and even if it is a format that does not need to be selected, a plurality of scenarios are associated with one theme so that it can be used in seasons, times of day, etc. You may select automatically according to it.

  Then, the following steps S1404 to S1410 are repeated until scenario reproduction ends.

  First, the scenario playback unit 110 starts playback of a scenario at 00:00:00 (= minute: second: millisecond), and when the time described in each line in the scenario comes, (key1.type), ( key2.type), (image, area) are output (S1404).

  Next, the feature quantity extraction unit 102 receives key1. The feature amount specified by type is extracted, and the result is expressed as key1. The data is output as data (S1405).

The search unit 103 uses key1. An image matching the data is searched (S1406).
From the meta information of each extracted image, key2. The information specified by type is read, and this is read as key2. Data is output (S1407).

  The search unit 103 further includes key2. The database 104 is searched with data (S1408).

  The above steps S1407 and S1408 are repeated for all the images extracted in step S1406 (S1409).

  Also in the present embodiment, as in the fourth embodiment, as long as the user selects a theme, the image group to be selected is switched according to the scenario selected according to the theme, and the image transition process is performed. Is displayed, the user can enjoy the transition. Furthermore, since the transition does not change according to the user's intention, an unexpected image is dug up and displayed, which leads to a new memory being aroused.

  The image search device according to the present invention can be applied as a personal computer (PC) having a device for storing a large amount of images, as a device such as a digital camera, a mobile phone with a camera, a car navigation system, a DVD recorder, a DVD TV, The image search method according to the present invention is also useful for searching digital photographs on a PC.

It is a block diagram which shows the function structure of the image search device in Embodiment 1 of this invention. It is a figure which shows the example of a transition of the image of the interest object in Embodiment 1 of this invention. It is a flowchart which shows operation | movement of the image search device in Embodiment 1 of this invention. 6 is a diagram illustrating a display example of an extracted image on the display unit 106. FIG. It is a figure which shows the image in the case of designating the several area | region in Embodiment 1 of this invention. It is a block diagram which shows the function structure of the image search device in Embodiment 2 of this invention. It is a flowchart which shows operation | movement of the image search device in Embodiment 2 of this invention. It is a block diagram which shows the function structure of the image search device in Embodiment 3 of this invention. It is a flowchart which shows operation | movement of the image search device in Embodiment 3 of this invention. It is a block diagram which shows the function structure of the image search device in Embodiment 4 of this invention. It is a figure which shows an example of the scenario which the image search device in Embodiment 4 of this invention processes. It is a flowchart which shows operation | movement of the image search device in Embodiment 4 of this invention. It is a block diagram which shows the function structure of the image search device in Embodiment 5 of this invention. It is a flowchart which shows operation | movement of the image search device in Embodiment 5 of this invention. It is a block diagram which shows the function structure of the conventional image search apparatus. It is a flowchart which shows operation | movement of the conventional image search apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Input part 102 Feature quantity extraction part 103 Search part 104 Database 105 Search key generation part 106 Display part 107 Recognition part 108 Environment information acquisition part 109 Scenario generation part 110 Scenario reproduction part 111 Scenario reading part 112 Theme input part

Claims (19)

An image search device for extracting a desired image from a plurality of images,
Display means for displaying the extracted image;
A database that stores the image, the image feature amount extracted from the image, and meta information indicating the shooting condition of the image in association with each other;
Information generating means for generating first information including the type of the image feature amount and generating second information including the type of the meta information;
Feature quantity extraction means for generating a first search key for searching for an image having the image feature quantity based on the first information generated by the information generation means;
Search key generating means for generating a second search key for searching for an image having meta information matching the meta information based on the second information generated by the information generating means;
Searching means for searching the database using at least one of the first search key and the second search key, and sequentially displaying a plurality of searched images on the display means. Search device. The information generating means includes input means for inputting the type of the image feature amount of the image designated by the user with respect to the image displayed on the display means and the type of meta information of the image. The image search apparatus according to claim 1, wherein the image search apparatus is an image search apparatus. The input means accepts designation of a pair of two or more images and image feature quantities, and accepts designation of a type of logical operation of two or more image feature quantities,
The image feature amount extraction unit extracts a specified image feature amount from two or more images, and generates a first search key by performing the specified logical operation on the extracted image feature amount. The image search device according to claim 2. The input unit includes an overlap determination unit that detects the overlap when two or more images are displayed in the display unit and outputs the overlapped region as a designated region.
4. The image according to claim 2, wherein the input unit receives designation of a type of an image feature amount to be extracted from the overlapping region and a type of logical operation based on the image feature amount extracted from the overlapping region. 5. Search device. The image search apparatus according to any one of claims 2 to 3, wherein the display unit selects and reproduces music popular in the year of shooting date and time of the image group selected by the user. When the search means searches the database using the first search key, it simultaneously obtains the degree of coincidence of the extracted image with respect to the first search key, and the image is stored in either the descending order or the ascending order of the coincidence degree. The image search apparatus according to claim 2, wherein the image search apparatus is displayed on a display unit. The image search apparatus according to claim 2, wherein the search means searches the database with the first search key after first searching the database with the second search key. The image search apparatus according to claim 2, wherein the search key generation unit generates two or more search keys based on two or more types of meta information. The image search device further includes a recognition unit that identifies an object in the image feature and outputs the name as a first search key.
The image search apparatus according to claim 1, wherein the search means searches the database using at least one of the first search key and the second search key generated by the recognition means. The image search apparatus further includes an input device that acquires environmental information from the outside such as date, position, temperature, and the like, and includes environmental information acquisition means that outputs environmental information acquired by the input device,
The image search apparatus according to claim 1, wherein the search unit searches the database using at least one of the first search key and the second search key and the environment information. The information generating means
Scenario generation means for reading music data, analyzing the feature value, and generating and outputting a scenario for performing a search along the characteristic sound of the music;
The scenario playback means is configured to read out the scenario and start playback of music, and at the same time issue the first and second information according to the scenario to control the search of the database and display of the extraction result. The image search device according to claim 1. The information generating means
Theme input means for presenting a plurality of themes as options to the user, selecting one of them, and outputting the selected theme,
A scenario reading means for selecting and reading a scenario corresponding to the theme;
2. The image search apparatus according to claim 1, further comprising scenario reproducing means for issuing the first and second information according to the scenario and controlling the search of the database and the display of the extraction result. . An image search method for extracting a desired image from a plurality of images,
A display step for displaying the extracted image;
A database that stores the image, the image feature amount extracted from the image, and meta-information indicating the photographing condition of the image in association with each other;
An information generation step of generating first information including a type of image feature amount extracted from the image and generating second information including a type of meta information indicating a shooting condition of the image;
A feature amount extraction step of generating a first search key for searching for an image having the image feature amount based on the first information generated in the information generation step;
A search key generating step for generating a second search key for searching for an image having meta information matching the meta information based on the second information generated in the information generating step;
Using at least one of the first search key and the second search key, a database that stores the image, the image feature amount, and the meta information in association with each other is searched, and a plurality of searched images And a search step of sequentially displaying in the display step. The information generation step includes an input step of inputting the type of the image feature amount of the image specified by the user with respect to the image displayed on the display means and the type of meta information of the image. The image search method according to claim 13, wherein the image search method is an image search method. The image search method according to claim 14, wherein, in the search step, the database is first searched with the second search key, and then the database is searched with the first search key. The image search method according to claim 14, wherein in the search key generation step, two or more search keys are generated based on two or more types of meta information. The information generation step includes
A scenario generation step of reading music data, analyzing the feature amount, generating a scenario for performing a search along the characteristic sound of the music, and outputting the scenario,
A scenario playback step for controlling the search of the database and the display of the extraction result by reading the scenario and starting the playback of music, and simultaneously issuing the first and second information according to the scenario. The image search method according to claim 13. The information generation step includes
A theme input step of presenting a plurality of themes as options to the user, selecting one of them, and outputting the selected theme;
A scenario reading step of selecting and reading a scenario corresponding to the theme;
14. The image search method according to claim 13, further comprising a scenario reproduction step of issuing the first and second information according to the scenario and controlling the search of the database and the display of the extraction result. .   A program for causing a computer to execute the steps included in the image search method according to claim 13.

Images