JP6647039B2 - Image processing apparatus, image processing method, and computer program - Google Patents

Description

  The present invention relates to an image display technique.

  In recent years, in addition to the spread of digital cameras, the spread of smart devices and the improvement of camera performance of smart devices have led to a rapid increase in the number of photographed photos by users. Photo books are attracting attention as a means of organizing photographed pictures. However, creating a photo book is troublesome because good photos are selected from a large number of photos and the layout is determined by trial and error. In recent years, in order to solve such a problem, a service for automatically analyzing a photograph and proposing a layout has appeared.

  In Patent Literature 1, as a method of automatically selecting a cover image of an album from a large number of images in an electronic album, image data is divided into groups based on shooting frequency and resolution of image data, and images are selected for each group. An approach has been proposed.

JP 2003-58867 A

  However, in the image selection described in Patent Literature 1, image data is grouped, and a cover image is automatically selected for each group. Therefore, it takes time to display the cover layout, and it takes time to complete the cover layout including editing. It cannot be done efficiently.

  The present invention has been made in view of such a problem, and provides a technique for efficiently performing a cover layout in creating a photo album.

  One embodiment of the present invention is a generation unit that generates layout image data in which an image is laid out based on image data selected from an image data group, and a display unit that displays a layout image based on the layout image data generated by the generation unit. Display generating means for generating layout image data corresponding to the cover and layout image data corresponding to the book body, and the generating means generating layout image data corresponding to the cover. The display control means displays a layout image of the cover on the display unit based on the layout image data corresponding to the cover, and the generating means sets the main body in a background of the display of the layout image of the cover. Is generated.

  According to the configuration of the present invention, in creating a photo album, it is possible to efficiently perform until the cover layout is completed.

FIG. 2 is a block diagram illustrating an example of a hardware configuration of the image processing apparatus. FIG. 2 is a block diagram illustrating a configuration example of a functional unit related to an automatic layout function. FIG. 3 is a diagram illustrating a configuration example of a GUI. 5 is a flowchart of a process performed by the image processing apparatus 100. The figure which shows an example of image analysis information. The figure which shows an example of grouping. The figure which shows an example of scene classification reference data. The figure explaining step S410. The figure explaining step S421. The figure which shows a template and a picked-up image. FIG. 3 is a block diagram showing a functional configuration example of a selection unit 210. 5 is a flowchart of an image selection process performed by a selection unit 210. 9 is a flowchart illustrating a correction timing according to the third embodiment. FIG. 4 is a diagram for explaining operations of a determination unit 2206 and a trimming unit 2207. 9 is a flowchart of screen transition. FIG. 14 is a view showing a display example of a GUI displayed in S1503. FIG. 9 is a diagram illustrating a display example of a GUI 1701. FIG. 9 is a view showing a display example of a GUI. The figure explaining the selection method of the captured image for covers. The figure which shows a template and a picked-up image. FIG. 9 is a view showing a display example of a window 2101 indicating the degree of progress of processing. FIG. 3 is a diagram illustrating a configuration example of a layout unit 217. 9 is a flowchart showing the operation of the layout unit 217.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. The embodiment described below shows an example in which the present invention is specifically implemented, and is one specific example of the configuration described in the claims.

[First Embodiment]
In the present embodiment, an example of an image processing device will be described. The image processing apparatus generates a cover layout image by laying out an image based on image data selected from an image data group on a cover of a photo album. Further, a layout image of the main body is generated by laying out an image based on the image data selected from the image data group in the main body of the photo album. The generation of the main body layout image operates in the background of the display of the cover layout image.

  In the following, a case will be described in which the image processing apparatus executes application software for automatically laying out images on the front and back covers of an electronic album (photo album) and on each page of a book.

  First, an example of a hardware configuration of the image processing apparatus 100 according to the present embodiment will be described with reference to the block diagram of FIG. The CPU 101 executes processing using computer programs and data stored in the ROM 102 and the RAM 103. Accordingly, the CPU 101 controls the operation of the entire image processing apparatus 100 and executes or controls various processes including the display control described later as the processing performed by the image processing apparatus 100. The number of CPUs 101 may be one or more. The ROM 102 stores setting data of the apparatus that does not require rewriting, and computer programs (such as boot programs) of the apparatus that do not require rewriting.

  The RAM 103 has an area for storing computer programs and data loaded from the HDD (hard disk drive) 104 and the ROM 102, and data received from the outside via the data communication device 108. Further, the RAM 103 has a work area used when the CPU 101 executes various processes. As described above, the RAM 103 can appropriately provide various areas.

  The HDD 104 is an example of a large-capacity information storage device. The HDD 104 stores an OS (Operating System) and computer programs and data for causing the CPU 101 to execute or control each process to be described later that is performed by the image processing apparatus 100. This computer program includes the above application software. The application software is downloaded from, for example, an external device and stored in the HDD 104 or the like. Further, this data includes information appearing as information (known information) created in advance in the following description. The computer programs and data stored in the HDD 104 are loaded into the RAM 103 under the control of the CPU 101, and are processed by the CPU 101.

  The display 105 is an example of a display unit, and displays a processing result of the CPU 101 as an image, characters, or the like. The keyboard 106 and the mouse 107 are examples of a user interface, and various instructions can be input to the CPU 101 by operating the user. Note that a touch panel screen may be used instead of the display 105, the keyboard 106, and the mouse 107.

  The data communication device 108 is for performing data communication with an external device. For example, the data communication device 108 performs data communication with a terminal device, a printer, and a multifunction peripheral via the Internet or a LAN.

  The CPU 101, the ROM 102, the RAM 103, the HDD 104, the display 105, the keyboard 106, the mouse 107, and the data communication device 108 are all connected to the bus 109. Note that, for example, a PC (personal computer), a tablet terminal device, a smartphone, or the like can be applied to the image processing apparatus 100 having the above-described configuration.

  Next, a functional unit related to an automatic layout function described later in the above application software will be described with reference to the block diagram of FIG. Although this application software has various functions, FIG. 2 particularly shows functional units related to the automatic layout function. As shown in FIG. 2, the application software includes a designation unit 201 and an automatic layout processing unit 218. The automatic layout function is a function for creating a photo album in which a predetermined number of photos are selected from a plurality of captured photos, laid out, and displayed on the display 105.

  As shown in FIG. 2, the automatic layout processing unit 218 includes a plurality of functional units, and the functions of the automatic layout processing unit 218 are realized by the plurality of functional units. In FIG. 2, the specifying unit 201 is shown as a functional unit separate from the automatic layout processing unit 218, but the specifying unit 201 may be included in the automatic layout processing unit 218.

  The application software is activated when, for example, the user double-clicks the application icon displayed on the display 105 using the mouse 107.

  In the following description, the functional unit illustrated in FIG. 2 may be mainly described as a processing entity. However, in practice, the CPU 101 executes a computer program corresponding to the functional unit to execute the function of the functional unit. Is realized.

  In addition, some or all of the functions of each of the functional units described below may be left to another functional unit, or may be realized by performing a cooperative operation with another functional unit. In other words, as long as the function of the automatic layout processing unit 218 is realized by a plurality of functional units, the operation of each functional unit is not limited to a specific form.

  The specifying unit 201 acquires “various information constituting the conditions for creating a photo album” and supplies the acquired information to the automatic layout processing unit 218. The “various information constituting the conditions for creating a photo album” is, for example, information set on a GUI (graphical user interface) displayed on the display 105 by the user operating the keyboard 106 and the mouse 107. When the user inputs the completion instruction after setting these pieces of information on the GUI, the specifying unit 201 acquires “various information constituting the conditions for creating a photo album” set on the GUI. Of course, the designation unit 201 may acquire, from the HDD 104, “various information forming the conditions for creating a photo album” that has been created in advance on the GUI and stored in the HDD 104. The conditions to be set include, for example, designation of a target image or a hero ID, the number of facing pages of an album, template information, ON / OFF conditions for image correction, and the like. The specification of an image may be, for example, a specification based on accompanying information or attribute information of an individual image, such as an imaging date and time, or may be a specification based on a file system structure including an image, such as a device or a directory. In the present embodiment, the “spread pages” correspond to, for example, one display window in display, and correspond to a pair of adjacent pages printed on different sheets in printing.

  The image acquisition unit 202 reads a group of captured image data (group of captured image data that is a candidate to be laid out on each page when one photo album is created) from the HDD 104. When the information acquired by the specifying unit 201 includes a condition of the captured image data read by the image acquiring unit 202, the image acquiring unit 202 converts the captured image data according to the condition. Read from HDD 104. For example, it is assumed that the information acquired by the specifying unit 201 includes a condition that “the imaged date and time is a captured image from January 1, XX to December 31, XX”. In this case, the image acquisition unit 202 determines that the captured image data of the imaging date and time within the period from January 1, XX to December 31, XX (for example, Image data) from the HDD 104.

  Examples of the captured image data stored in the HDD 104 include still image data and cutout image data cut out from a moving image. The still image data and the cut-out image are obtained from an imaging device such as a digital camera or a smart device. The imaging device may be included in the image processing apparatus 100 or may be included in an external device. When the imaging device is an external device, the image data is acquired via the data communication device 108. Still image data and cutout image data may be image data acquired from a network or a server via the data communication device 108. Examples of image data acquired from a network or a server include social networking service image data (hereinafter, referred to as “SNS image data”). The CPU 101 determines the storage source of each image data by analyzing data accompanying the image data. The SNS image data is acquired from the SNS via the application software, and the acquisition destination may be managed under the management by the application software. The image data is not limited to the above-described image data, but may be other types of image data.

  The image conversion unit 203 generates reduced image data of the captured image data (for example, the short side has a size of 420 pixels, and each pixel has color information of sRGB for each captured image data acquired by the image acquisition unit 202). Is generated as analysis image data.

  The image analysis unit 204 performs analysis processing such as feature extraction, face recognition, individual recognition, and facial expression recognition on each of the analysis image data generated by the image conversion unit 203. In addition, the image analysis unit 204 acquires the date and time of imaging of the captured image data from supplementary information (for example, Exif information) of the captured image data from which the analysis image is generated.

  The image classification unit 205 divides the captured image data group acquired by the image acquisition unit 202 into a plurality of scenes (scene classification) based on the result of analysis processing by the image analysis unit 204 and the date and time of imaging. The scene is an imaging scene such as a trip, a daily life, and a wedding. A scene can be, for example, a collection of image data of one imaging target that has been imaged at one time imaging opportunity.

  The input unit 206 obtains information (for example, identification information of the main character) that specifies the main character from the information obtained by the specifying unit 201.

  The image score unit 207 obtains a score for each captured image data acquired by the image acquisition unit 202. The method of obtaining the score by the image score unit 207 will be described later, but the information used at that time is not limited to the following information.

  The input unit 208 acquires the number of facing pages of the photo album from the information acquired by the designation unit 201.

  The spread allocation unit 209 allocates each group classified by the image classification unit 205 or each group rearranged based on each group to each spread of the photo album. The spread here is, for example, two pages of a spread when a photo album is created. Here, the description will be made with a spread, but a page may be used.

  The selection unit 210 allocates each of the captured image data to the photo slot in the spread based on the score obtained by the image score unit 207 for each of the captured image data in the group assigned to the spread. Image data to be captured is selected. The image arranged in the photo slot is a captured image based on the captured image data. Hereinafter, an image based on the captured image data is referred to as a “captured image”.

  The input unit 211 acquires a cover template group and a main body template group from the information acquired by the designation unit 201. The cover template group is a template group representing the layout of the front cover and the back cover (the layout of each photo slot in the front cover and the layout of each photo slot in the back cover). The template group for the main body is a template group representing the layout of each spread in the main body (the layout of each photo slot in the spread). The template referred to here is a template of a facing spread page (a cover page and a back cover page) and a template of a book facing page.

  The layout unit 212 selects one template from the templates input by the input unit 211 based on the captured image data selected by the selection unit 210, and determines the arrangement of the captured image data in the template.

  The input unit 213 acquires, from the information acquired by the designation unit 201, a flag value indicating whether to perform image correction on the captured image. ON or OFF of the image correction may be specified for each type of correction, or may be specified for all types at once.

  If the flag value obtained by the input unit 213 is a value indicating that image correction is to be performed, the image correction unit 214 performs correction processing such as dodging correction, red-eye correction, and contrast correction on the captured image. Is applied. Of course, if the flag value acquired by the input unit 213 is a value indicating that image correction is not to be performed, the image correction unit 214 does not execute the correction processing.

  The number of pixels of the image data input from the image conversion unit 203 to the image correction unit 214 can be changed according to the size of the layout image data determined by the layout unit 212. In the present embodiment, the image correction is performed on each image after the layout image data is generated. However, the present invention is not limited to this, and each image may be corrected before layout (arrangement). .

  The output unit 215 outputs layout image data (images of the main body) in each of the photo slots on the template selected by the layout unit 212 by resizing the captured image data assigned to the photo slot to the size of the photo slot. Data). Further, the output unit 215 resizes the image data (the cover / image) in which the captured image data assigned to the photo slot is resized to the size of the photo slot and arranged in each photo slot on the template selected by the layout unit 217. (Back cover image data). The layout image data is integrated image data obtained by arranging the image data in the template, and is, for example, bitmap data. Then, the output unit 215 outputs the generated layout image as layout information.

  The selection unit 216 is configured to, based on the score obtained by the image score unit 207 for each of the captured image data acquired by the image acquisition unit 202, select a captured image to be arranged in a photo slot on the front cover and the back cover of the respective captured image data. Select data.

  The layout unit 217 selects one cover template from among the templates input by the input unit 211 based on the captured image data selected by the selection unit 216, and determines the arrangement of the captured image data in the template.

  In the above description, the image to be corrected or placed in the photo slot is a captured image, but may be an analysis image generated from the captured image. Of course, another image generated from the captured image may be used. The same applies to other processes for handling a captured image, and another image such as an analysis image may be used instead of the captured image.

  Here, an example of the configuration of a GUI for inputting the “various information constituting the conditions for creating a photo album” to the image processing apparatus 100 will be described with reference to FIG. The GUI 301 is activated and displayed on the display screen of the display 105 by, for example, double-clicking the application icon displayed on the display 105 by the user using the mouse 107. Note that the CPU 101 executes or controls any processing performed in response to user input to all GUIs described below, including the GUI 301 in FIG.

  The pass box 302 is used to input a path of a folder in the HDD 104 in which a group of captured images to be laid out in the photo album is stored. For example, the user can operate the keyboard 106 to input a path. it can. When the user operates the keyboard 106 or the mouse 107 to specify the folder selection button 303, a window in which the tree structure of the folder / data in the HDD 104 is displayed on the display 105 separately. When the user specifies a desired folder by operating the keyboard 106 or the mouse 107 on this window, the path of the folder is displayed in the path box 302. As described above, any one of the path box 302 and the folder selection button 303 can specify a group of captured images to be laid out in the photo album.

  A list of face images of various people is displayed in the area 304, and the user can operate the keyboard 106 or the mouse 107 to specify any one of the face images. Each face image displayed in the area 304 is extracted from the captured image and registered in the HDD 104 in association with unique identification information for each face image (person). When the user operates the keyboard 106 or the mouse 107 to specify one of the face images, the identification information corresponding to the specified face image is registered in the HDD 104 as the identification information of the main character (main character ID). The method of specifying the identification information of the hero may be manual in this way, or may be specified according to the analysis result of the image as described later.

  The facing page number box 305 is used to input the facing page number in the photo album. For example, the user can operate the keyboard 106 or the mouse 107 to enter the facing page number. The method of inputting the number of facing pages is not limited to this. For example, even if a list in which numerical values are arranged is displayed, and a desired numeric value as the number of facing pages is designated by the user operating the keyboard 106 or the mouse 107 therefrom. good.

  In the area 306, illustration images showing the taste (pop tone, chic tone, etc.) of the template are displayed as a list of icons, and the user can operate the keyboard 106 and the mouse 107 to specify an arbitrary icon. . In this embodiment, one or two or more templates are assigned to each taste, and when one taste is selected, all templates assigned to that taste are set. Then, as described later, a template that best matches the selected image is selected from the set templates. By specifying one icon in this manner, a cover template and a body template (templates for the number of facing pages) can be selected. The user may designate a plurality of icons in the area 304 to designate a plurality of templates for the cover and the body as candidates. In the processing according to the flowchart of FIG. 4 described below, it is described that the cover template group and the body template group are selected by the user selecting one taste. In the present embodiment, the template has one or more image arrangement frames (photo slots) for arranging images.

  The check box 307 is for designating ON / OFF of the image correction, and the user can operate the keyboard 106 or the mouse 107 to turn on (check) or off (uncheck) the check box 307. it can. When the check box 307 is turned on, the image correction is set to ON, and when the check box 307 is turned off, the image correction is turned OFF. The method for setting ON / OFF of the image correction is not limited to this. Also, a check box may be provided for each type of image correction.

  The radio button 310 is used to select a product of a photo album to be created. When the user operates the keyboard 106 or the mouse 107 to specify the upper radio button, the A4 size is set as the product, and when the lower radio button is specified, the A5 size is set as the product. Note that the method of designating the merchandise is not limited to this method. For example, the merchandise may be specified from a list using a pull-down menu.

  When the user operates the keyboard 106 and the mouse 107 to specify the OK button 308, the content set on the GUI 301 in FIG. 3 is input to the automatic layout processing unit 218 via the specifying unit 201. More specifically, the path input to the path box 302 is notified to the image acquisition unit 202, and the identification information corresponding to the face image specified in the area 304 is notified to the input unit 206. Also, the number of facing pages input in the number-of-pages box 305 is notified to the input unit 208, and template information (information that can specify the template) corresponding to the icon specified in the area 306 is notified to the input unit 211. You. Further, information of the check box 307 (whether or not there is a check) is notified to the input unit 213.

  On the other hand, when the user operates the keyboard 106 or the mouse 107 to specify the reset button 309, the content set on the GUI 301 in FIG. 3 is discarded, and the GUI 301 is hidden on the display 105.

  Next, processing performed by the CPU 101 executing the automatic layout processing unit 218 will be described with reference to the flowchart in FIG.

<Step S401>
The image conversion unit 203 sequentially focuses (selects) on the image data included in the image data group acquired by the image acquisition unit 202, and generates the analysis image data from the focused image data. The analysis image data is generated based on “generation parameters such as the number of vertical and horizontal pixels of the analysis image and the color space of the pixel value” included in the information acquired by the specifying unit 201.

<Step S402>
The image analysis unit 204 extracts the feature amount from the analysis image data generated in step S401, and extracts the imaging date and time of the imaging image from the supplementary information (for example, Exif information) of the imaging image data from which the analysis image data was generated. get. Various features can be used as the feature to be extracted. In the following, as an example, the focus amount is extracted as the feature.

  As a method for detecting an edge in an image, a known Sobel filter can be used. An edge is detected from an image based on the image data using a Sobel filter, and the inclination (luminance gradient) of the edge can be calculated by dividing the luminance difference between the start point and the end point of the detected edge by the distance between the start point and the end point. . Then, the average value of the inclination of the edge in the image is calculated. It can be considered that an image having a large average inclination is in focus than an image having a small average inclination. Therefore, a plurality of different thresholds are set in advance for the inclination, and it is possible to calculate a focus amount in the image by determining which threshold is equal to or more than the average value of the inclination obtained from the image. In the present embodiment, two different threshold values θ1 and θ2 (θ1 <θ2) are set, and if the average value A of the obtained inclination is A <θ1, the focus amount = × (unacceptable), and θ1 ≦ A <θ2. If there is, the focus amount = △ (acceptable), and if A ≧ θ2, it is determined that the focus amount = ○ (preferred). Focus amount = O indicates the inclination of the focus desired to be adopted in the photo album. The focus amount = △ indicates an allowable inclination of the focus. The focus amount = x indicates an unacceptable slope.

<Step S403>
The image analysis unit 204 detects a human face from an analysis image based on the analysis image data generated in step S401. As a method of detecting a face from an image, for example, a known technique can be used. For example, there is Adaboost which creates a strong classifier from a plurality of prepared weak classifiers. In the present embodiment, a human face is detected from an image using a strong classifier created by Adaboost. Through this face detection processing, the coordinate position and size of the human face area (face area (rectangular area)) on the image are obtained. Instead of the size, the coordinate position of the upper left corner and the coordinate position of the lower right corner of the face area may be obtained. A strong classifier can be created by Adaboost using objects such as animals such as dogs and cats, flowers, food, buildings, and figurines as detection targets instead of faces. This makes it possible to detect an object other than a face.

<Step S404>
The image analysis unit 204 performs, on each of the face areas detected in step S403, individual recognition processing that is processing for recognizing who is the face in the face area. For example, for images of various human faces registered in the HDD 104 in advance, the similarity with the image in the face area detected in step S403 is determined. Then, among images of various human faces registered in advance in the HDD 104, an image that is equal to or more than a threshold and for which the maximum similarity is obtained is specified, and identification information ( The identification information (personal ID) of the individual having the face in the image is acquired. If any of the obtained similarities is less than the threshold, new identification information is issued, and the image in the face area detected in step S403 is registered in the HDD 104 in association with the issued identification information. .

  In this step, any personal recognition processing may be employed as long as identification information of a person having a face in the face area detected in step S403 can be obtained.

  Then, the image analysis unit 204 registers the information collected in steps S402 to S404 in the HDD 104 as image analysis information. FIG. 5 shows an example of the image analysis information. The image analysis information includes, for each piece of captured image data, an ID (image ID) unique to the captured image data, the date and time of imaging of the captured image data, the amount of focus, the number of human faces detected from the captured image, It includes the coordinate positions of the upper left corner and lower right corner of each person's face area detected from the image. For example, the following information is registered for the captured image of image ID = 1.

Date of imaging = “July 1, 2015, 10:11:12”
Focus amount = "○"
Number of detected human faces = "6"
The coordinate position of the upper left corner and the lower right corner of the face area of the person with the personal ID = 1 The coordinate position of the upper left corner and the lower right corner of the face area of the person with the personal ID = 2 The person with the personal ID = 3 The coordinate position of the upper left corner and the coordinate position of the lower right corner of the area of the face of the person The coordinate position of the upper left corner and the coordinate position of the lower right corner of the area of the face of the person with the personal ID = 4 The coordinate position of the corner and the coordinate position of the lower right corner The coordinate position of the upper left corner and the coordinate position of the lower right corner of the area of the face of the person with the personal ID = 6. No information is registered, and the imaging date and time and the focus amount acquired in step S402 are stored.

<Step S405>
If all the captured image data groups acquired by the image acquisition unit 202 have been selected in step S401, the process proceeds to step S406 via step S405. On the other hand, if unselected captured image data remains in the captured image data group acquired by the image acquisition unit 202, the process returns to step S401.

<Step S406>
The image classifying unit 205 acquires the image capturing date and time of each captured image data acquired by the image acquiring unit 202 from the image analysis information registered in the HDD 104. Then, the image classification unit 205 divides the captured image data group acquired by the image acquisition unit 202 into a plurality of groups (sub-image groups) (scene division) based on the image capturing date and time of each of the acquired captured image data.

  An example of the scene division method includes the following method. For example, when each of the captured image data acquired by the image acquisition unit 202 is viewed in the order of the image capturing date and time, the image capturing date and time of the N (N is an integer of 1 or more) captured image data and the (N + 1) th captured image data It is assumed that the difference between the imaging date and time of the image data is two days or more. At this time, a group boundary is set between the Nth captured image data and the (N + 1) th captured image data.

  If the difference between the imaging date and time of the Nth captured image data and the imaging date and time of the (N + 1) th captured image data is one day or less in the order of older imaging date and time, the following is performed. If the difference between the imaging date and time of the N-th captured image data and the imaging date and time of the (N + 1) -th captured image data is 16 hours or more in the order of older imaging date and time, the N-th captured image data and the (N + 1) -th A group boundary is set between the image data and the captured image data. Further, it is assumed that the difference in the imaging date between adjacent captured image data items in the order of older imaging date and time is less than 16 hours in any captured image data. In this case, if the difference between the imaging date and time of the first captured image data and the imaging date of the last captured image data in the group of captured image data captured in one day is less than 4 hours, the imaging of the one day is performed. The image data group is defined as one group. On the other hand, it is assumed that the difference between the imaging date and time of the first captured image data and the imaging date and time of the last captured image data in the group of captured image data captured in one day is 4 hours or more. In this case, if the number of captured images captured in one day is less than 50 (the number of captured images captured in one day is less than 50), the group of captured image data for one day is regarded as one group. FIG. 6A shows an example of such grouping of captured image data. 6A to 6C, the horizontal axis represents the date and time of imaging (older toward the left and newer toward the right), and the vertical axis represents the number of captured image data. In FIG. 6A, the captured image data group is divided into eight groups of groups 601 to 608. In FIG. 6A, arrows indicate the boundaries between groups. It should be noted that the threshold values for the difference between the imaging dates and times and the threshold values for the number of captured image data used in the above description of the scene division method are merely examples, and are not limited to these values.

<Step S407>
The image classification unit 205 selects one of the groups divided in step S206 as a group of interest, and determines to which scene the group of interest is classified. In the present embodiment, it is determined whether the group is classified into a trip, a daily life, or a ceremony scene. Of course, the types of scenes are not limited to these types.

  For this step, data serving as a reference (scene classification reference data) for determining to which scene a group is classified is registered in the HDD 104 in advance. An example of a method for creating scene classification reference data will be described below.

  For example, an image data group (travel image data group) determined in advance as a travel image, an image data group (daily image data group) determined in advance as a daily image, and an image determined in advance as a ceremony image A plurality of data groups (ceremony image data groups) are prepared in advance. Then, for each of the travel image data group, the daily image data group, and the ceremony image data group, the average and standard deviation of the imaging period, the average and standard deviation of the number of images, and the average and standard deviation of the number of imagers are obtained. For example, a description will be given of a case where the average and standard deviation of the imaging period, the average and standard deviation of the number of images, and the average and standard deviation of the number of imaged persons are obtained for the travel image data group.

  The imaging period of the image data group is the difference between the imaging date and time of the most recently captured image in the image data group and the imaging date and time of the most recently captured image. Accordingly, for each travel image data group, the difference between the imaging date and time of the most recently captured image data and the imaging date and time of the most recently captured image data in the travel image data group is determined as the imaging period of the travel image data group. Further, an average and a standard deviation of the imaging periods obtained for the respective travel image data groups are obtained.

  Further, the number of captured images of the image data group is the number of image data constituting the image data group. Therefore, for each travel image data group, the number of captured image data constituting the travel image data group is obtained. In addition, an average and a standard deviation of the number of captured image data obtained for each of the travel image data groups are obtained.

  Also, the number of imagers of the image data group is the number of faces appearing in each image constituting the image data group. Therefore, for all the travel image data groups, the number of faces appearing in the captured images based on the respective captured image data constituting the travel image data groups is obtained by a known face recognition technique or the like. Further, the average and the standard deviation of the obtained number of faces are obtained.

  Similarly, the average and standard deviation of the imaging period, the average and standard deviation of the number of images, and the average and standard deviation of the number of images are obtained for the daily image data group and the ceremony image data group. Then, the travel image data group, the daily image data group, and the ceremony image data group were obtained in this manner, the average and standard deviation of the imaging period, the average and standard deviation of the number of images, the average and standard deviation of the number of imagers, Is registered in advance in the HDD 104 as scene classification reference data. FIG. 7 shows an example of the scene classification reference data.

  The image classification unit 205 calculates the average of the imaging period, the number of images, and the number of imagers for the group of interest in the same manner as described above. Then, the image classification unit 205 refers to the scene classification reference data, and calculates the score for the trip (scene 1), the score for the daily life (scene 2), and the score for the ceremony (scene 3) for the group of interest as follows. Ask for.

Score of imaging period of scene i = 50− | 10 × (average of imaging period of scene i−imaging period of selected group) / standard deviation of imaging period of scene i |
Score of the number of captured images of scene i = 50− | 10 × (average of number of captured images of scene i−number of captured images of selected group) / standard deviation of number of captured images of scene i |
Score of the number of imagers in scene i = 50− | 10 × (average of the number of imagers in scene i−average of the number of imagers in selected group) / standard deviation of the number of imagers in scene i |
Score of scene i = (score of imaging period of scene i + score of number of images captured of scene i + score of number of people who captured image of scene i) / 3
Where i = 1,2,3
Then, the image classification unit 205 determines (classifies) the scene with the highest score among the score of scene 1, the score of scene 2, and the score of scene 3 obtained for the group of interest as the scene of the group of interest. If there are a plurality of scenes with the highest score, the scene with the highest priority in the preset priority is adopted. For example, the order of daily>ceremony> travel is set in advance. Note that this priority is an example, and another priority may be adopted, or the user may change the priority by operating the keyboard 106 or the mouse 107. For example, assume that the imaging period of the group 605 in FIG. 6A is 36 hours, the number of captured images is 300, and the average number of captured images is 1.7. In this case, using the scene classification reference data of FIG. 7 and the above equation, the travel score is 45.32, the daily score is 18.38, the ceremony score is −29.92, and the scene of the group 605 is travel. are categorized. Note that the scene into which each group is classified is managed by the HDD 104, for example, by associating identification information unique to the scene of the classification destination for each group.

<Step S408>
If all of the groups divided in step S406 have been selected in step S407, the process proceeds to step S409. On the other hand, when an unselected group remains among the groups divided in step S406, the process returns to step S407.

<Step S409>
The input unit 206 acquires information for identifying the hero (here, the hero ID) from the information acquired by the designation unit 201. Note that the hero ID is an ID (personal ID in FIG. 5) unique to the person who is the hero among the people appearing in the captured image data group acquired by the image acquisition unit 202. The hero ID may be specified by the user using the GUI described above, or may be determined without the user's operation by the following processing.

  For example, by the processing in step S404 described above, it is possible to acquire information on which personal ID has appeared in each captured image data. Therefore, the number of appearances of each person in the captured image data group acquired by the image acquisition unit 202, the number of appearances of each person in each scene, the number of scenes in which each person appears, and the like can be obtained. From such information, it can be determined whether or not each person is the main character. For example, when the captured image data group acquired by the image acquisition unit 202 is composed of a plurality of scenes, the personal ID of a person who appears in the most scenes is set as the main character ID. When the captured image data group acquired by the image acquisition unit 202 is composed of a single scene, the personal ID of the person who appears most in the single scene is set as the main character ID. Note that when the user specifies a face image in the area 304 in the GUI 301 of FIG. 3, the input unit 206 does not perform the processing for determining the main character ID and responds to the specified face image. The identification information to be obtained is acquired as the main character ID.

<Step S410>
The image scoring unit 207 provides a score for a main photo slot (main slot) and a score for a sub-photo slot (sub-slot) with respect to one image data of interest in the captured image data group acquired by the image acquisition unit 202. Ask for. Here, the main photo slot and the sub photo slot will be described with reference to FIG. (1-1) to (4-4) in FIG. 10A are template groups input by the input unit 211. In FIG. 10, information other than the photo slot is omitted to explain the photo slot in the template. For example, in the cover template, an area (character input area) for arranging text information such as a title is provided in addition to the photo slot.

  In the (1-1) template 1001, reference numerals 1002 to 1004 denote photo slots, but the photo slot 1002 is larger in size than the other photo slots 1003 and 1004. In such a case, the photo slot 1002 is a main photo slot, and the photo slots 1003 and 1004 are sub-photo slots. Therefore, reference data used for obtaining a score for the main photo slot and a score for the sub photo slot for the image data of interest is created in advance as follows, for example.

  What is preferable as an image (image based on image data) to be arranged in the main photo slot and what is preferable as an image to be arranged in the sub-photo slot differs depending on the scene of the image. FIG. 8A shows an example of the relationship between the characteristics of an image preferable as an image to be arranged in the main photo slot, the characteristics of an image preferable as an image to be arranged in the sub-photo slot, and an image scene. The table in FIG. 8A is a table summarizing the features of the image in which the score increases for each of the travel, everyday, and ceremony scenes.

  As shown in FIG. 8 (a), the feature of increasing the score differs depending on the type of the photo slot. For example, for the scene "Travel", a scored image for a main photo slot has a higher score for a drawn image including a person and a landscape, and a score for a sub-picture slot has a higher score for a face-up or profile image. For example, in the scene “daily”, a close-up image or a face image has a higher score for the main photo slot, and a drawn image including a person and a landscape has a higher score for the sub-photo slot. For example, for the scene "Ceremony", an image where the distance between the two is short has a high score for the main photo slot, and an image of a large number of people has a high score for the sub photo slot. As described above, the feature of increasing the score, that is, the evaluation criterion differs according to the type of the scene and the photo slot. In the present embodiment, the feature of increasing the score according to the type of the scene and the photo slot, that is, the above-described reference data is registered in the HDD 104 in advance. Note that the features that increase the score are not limited to those described above.

  The above reference data is prepared as follows, for example. For the main photo slot, a travel image data group, a daily image data group, and a ceremony image data group that satisfy the conditions defined in FIG. 8A are prepared in advance. A group of travel image data, a group of daily image data, and a group of ceremony image data that satisfy the conditions defined in FIG. Then, for the image data group of each scene prepared for the main photo slot, the average and standard deviation of the number of faces appearing in the image data group, the appearance position of the face, and the face size are obtained. Similarly, for the image data group of each scene prepared for the sub-photo slot, the average and standard deviation of the number of faces, the appearance position of the face, and the face size appearing in the image data group are obtained. The average and standard deviation of the number of faces, the appearance position of faces, and the size of faces obtained for the main photo slot and the sub photo slot as described above are registered in the HDD 104 in advance as the above-mentioned reference data.

  Then, the image scoring unit 207 obtains a score (1) for the main photo slot and a score (2) for the sub photo slot for the image of interest I based on the image data of interest as follows.

<Score for main photo slot>
Score of the number of faces (1) = 50− | 10 × (average of the number of faces obtained for the scene of the image of interest I for the main photo slot−the number of faces of the hero appearing in the image of interest I) / scene of the image of interest I Standard deviation of the number of faces found for the main photo slot |
Score of face appearance position (1) = 50− | 10 × (average of face appearance positions calculated for main picture slot for scene of image of interest I−position of appearance of hero face appearing in image of interest I) / image of interest Standard deviation of face appearance position found for main photo slot for scene I |
Face size score (1) = 50− | 10 × (average face size obtained for main scene in the scene of interest image I−size of hero face appearing in interest image I) / scene of interest image I Standard deviation of face size found for main photo slot |
Score (1) = (Score of face number (1) + Score of face appearance position (1) + Score of face size (1)) / 3

<Score for sub photo slot>
Score of the number of faces (2) = 50− | 10 × (average of the number of faces obtained for the scene of interest image I for the sub-photo slot−the number of faces of the main character appearing in interest image I) / scene of interest image I Standard deviation of the number of faces obtained for sub-photo slot |
Score of face appearance position (2) = 50− | 10 × (average of face appearance positions obtained for sub-picture slots for scene of image of interest I−position of appearance of hero face appearing in image of interest I) / image of interest Standard deviation of face appearance position obtained for sub-photo slot for scene I |
Score of face size (2) = 50− | 10 × (average of face size obtained for sub-picture slot for scene of target image I−size of hero face appearing in target image I) / scene of target image I Standard deviation of face size obtained for sub-photo slot |
Score (2) = (Score of face number (2) + Score of face appearance position (2) + Score of face size (2)) / 3

Regarding the number of “the face of the hero appearing in the image of interest I”, in the line of the image ID corresponding to the image of interest I in the image analysis information of FIG. If the position of the lower corner is registered, it is "1", and if not, it is "0". The appearance position and size of “the face of the protagonist appearing in the image I of interest” are registered in the individual ID corresponding to the protagonist ID in the image analysis information row of FIG. It can be specified by referring to the position of the upper left corner and the position of the lower right corner.

  Since it is preferable that the image used for the photo album is in focus, the score may be further added to the captured image in which the focus = “○” shown in FIG. FIG. 8B shows an example of the score (1) for the main photo slot and the score (2) for the sub photo slot obtained for the captured images corresponding to the respective image IDs.

<Step S411>
If all the captured image data groups obtained by the image obtaining unit 202 have been processed in step S410, the process proceeds to step S412. On the other hand, if unprocessed captured image data remains in step S410 in the captured image data group obtained by the image obtaining unit 202, the process returns to step S410.

<Step S412>
The selection unit 216 selects, from the captured image data group acquired by the image acquisition unit 202, a specified number of captured image data to be laid out on a cover page that is one page of the front cover and the back cover. Hereinafter, the description will be made assuming that the number of selected image data (specified number) = 3, but the specified number is not limited to this number. For example, the number of selected image data may be changed according to the size of the product designated by the radio button 310 in the GUI 301 of FIG. For example, the number of selected image data corresponding to each of the A4 size and the A5 size may be set from a table that holds in advance. Further, the user may be able to specify the number of image data used for the front cover and the back cover on the GUI 301 or the like in FIG.

  A method of selecting the cover image data by the selection unit 216 will be described with reference to FIG. In FIG. 19, “Start” represents the date and time of imaging of the most recently captured image data in the group of captured image data acquired by the image acquisition unit 202. End represents the date and time of imaging of recently captured image data in the captured image data group acquired by the image acquisition unit 202. However, (Start-End) represents the imaging period of the captured image data group acquired by the image acquisition unit 202. That is, FIG. 19 shows a shooting period (designated shooting period) used for creating an album.

  First, from the captured image data group (FIG. 19A) acquired by the image acquisition unit 202, the captured image data to be arranged on the cover is selected. In the present embodiment, among the captured image data groups acquired by the image acquisition unit 202, the captured image data with the highest score for the main photo slot is selected as the captured image to be arranged on the cover. The captured image data selected as the captured image data to be placed on the cover is represented as (1) in FIG. 19B. Although the preferred image for the main photo slot differs depending on the scene, the score for the main photo slot obtained in step S410 is used here as it is as the “score for the main photo slot”.

  Next, two pieces of captured image data to be arranged on the back cover are selected from the captured image data group acquired by the image acquisition unit 202. In order not to concentrate on a part of the imaging period during image selection, the imaging period is subdivided as described below, and image data is acquired from each.

  First, the first captured image data to be arranged on the back cover is selected. For this purpose, first, as shown in FIG. 19C, the imaging period is divided into two division periods. Then, as shown in FIG. 19D, of the two divided periods, the sub-picture slot of the captured image group belonging to the remaining divided period (solid line) excluding the divided period (dotted line) to which the captured image data (1) belongs The picked-up image data with the highest score is selected as the picked-up image data to be arranged on the back cover. The captured image data selected as the first captured image data to be arranged on the back cover is shown as (2) in FIG. 19D.

  Next, the second captured image data to be arranged on the back cover is selected. For this purpose, first, as shown in FIG. 19E, the imaging period is divided into four divided periods by dividing each divided period into two. As shown in FIG. 19 (F), the sub-images in the captured image data group belonging to the remaining divided period (solid line) excluding the divided period (dotted line) to which the captured images (1) and (2) belong out of the four divided periods The captured image data with the highest score for the photo slot is selected as the captured image data to be arranged on the back cover. The captured image data selected as the second captured image to be arranged on the back cover is shown as (3) in FIG.

  The captured image data selected to be placed on the cover is the captured image data with the highest score for the main photo slot, so that it is used not only on the cover but also on any spread on the book itself.

  In the above description, the captured image data selected to be placed on the cover may be placed on the back cover, and the captured image data selected to be placed on the back cover may be placed on the cover. In the above description, the criterion for the score of the image data to be arranged on the front cover is different from the criterion for the score of the image data to be arranged on the back cover. Specifically, for the front cover, the score for the main photo slot was used, and for the back cover, the score for the sub photo slot was used. In either case, the same reference may be used. Further, the number of captured image data to be arranged on each of the front cover and the back cover is not limited to the above number (it may be 0).

  Instead of using the score for the main photo slot or the score for the sub photo slot, the captured image to be arranged on the front cover or the captured image to be arranged on the back cover may be determined by using the face detection result. For example, a captured image including a person may be arranged on the cover, and an image not including the person may be arranged on the back cover. In addition, here, the captured image data to be arranged on the front cover and the back cover is selected using the score for the main photo slot and the score for the sub photo slot, but is set in advance based on the classification result by the image classification unit 205. The image data of the scene may be preferentially selected. For example, when arranging travel image data on the cover, the image data with the highest score for the main photo slot is selected from the image data group determined to be travel based on the image classification result. After that, as described above with reference to FIG. 19, the imaging period may be divided, and the image data with the highest score for the sub-photo slot may be sequentially selected and used as image data for the back cover.

  Further, in the above description, the setting is made such that the back cover image data is selected from a division period different from the division period in which the cover image data is selected, but from the same division period as the division period in which the cover image is selected. The setting may be made so as to select the back cover image data.

<Step S413>
The layout unit 217 selects a template based on the image data selected in step S412. From the template group input by the input unit 211, a template to be applied to the cover page is selected based on the captured image data selected in step S412. In the following, it is assumed that the input unit 211 has input eight templates (1-1) to (2-4) in FIG. 20 in order to describe the process in step S413 with a specific example.

  In each of the eight templates, the layout of the captured image data for the back cover is defined on the left, and the layout of the captured image data for the cover is defined on the right. In the template (1-1) 2001, the photo slots 2003 and 2004 are for arranging captured image data for the back cover, and the photo slot 2002 is for arranging captured image data for the cover. In this embodiment, the photo slot 2002 for the front cover is larger in size than the photo slots 2003 and 2004 for the back cover. The title 2008 is a text area for inputting text information such as the title of a photo album.

  When one piece of captured image data for the cover and two pieces of captured image data for the back cover are selected in step S412, a template group having three photo slots as candidates as shown in FIG. 20 is set as a candidate. Here, it is assumed that the captured images based on the three captured image data selected in step S412 are a portrait (vertical) image 2005, a portrait (vertical) image 2006, and a landscape (horizontal) image 2007. . Note that images 2006 and 2007 are captured images to be arranged on the back cover, and image 2005 is a captured image to be arranged on the cover.

  First, since the image 2005 arranged on the cover is a vertically long image, the photo slot for the cover must also be vertically long. Therefore, of the templates (1-1) to (2-4), the templates (1-1) to (1-4) whose cover photo slots are vertically long are candidates. In the present embodiment, with respect to the back cover, an image with an older imaging date and time is laid out on the upper left of the template, and an image with a newer imaging date and time is laid out on the lower right of the template. An image based on image data older than the image capturing date and time is a vertically long image 2006, and an image based on image data newer than the image capturing date and time is a horizontal image 2007. Therefore, the template among templates (1-1) to (1-4) is used. (1-2) is the final candidate.

  However, in this case, the layout (1-2) is selected as the cover page template. As described above, in step S413, the template to be used for the layout and the image to be laid out in which slot of the template are determined. The determined information is input to the output unit 215.

  The method of selecting a cover page template is not limited to the above method, and another method may be adopted. Further, the template is not limited to the template shown in FIG. For example, the page of the back cover and the frame of the slot in which the image is arranged may be matched, or a part of the slot may overlap a part of another slot. When a part of the slots overlaps, the overlapping portion may be arranged such that one of the images is arranged in the foreground. For example, the image having a high score may be arranged in the foreground.

  In FIG. 20, the method of selecting a template for laying out three captured images has been described. However, even if the number of captured images is not three, the method of selecting a template can be similarly applied. It is assumed that a plurality of templates corresponding to the number of images to be arranged are registered in the HDD 104 as cover templates. In addition, a plurality of templates having different numbers of images to be arranged are prepared, layout images obtained by arranging the images in the respective templates are evaluated, and the one with the higher evaluation value is determined as the layout image of the cover. You may. For example, the trimming amount of the image to be arranged and the number of faces overlapping in the title area are used as evaluation criteria, and those with a small amount of trimming are evaluated as high, and those with a small number of faces overlapping in the title area are evaluated as high. Is also good.

<Step S414>
If the flag value acquired by the input unit 213 is a value indicating that image correction is to be performed, the image correction unit 214 applies dodging to each of the three pieces of captured image data selected by the selection unit 216. Correction processing such as correction, red-eye correction, and contrast correction is performed. Of course, if the flag value acquired by the input unit 213 is a value indicating that image correction is not to be performed, the image correction unit 214 does not execute the correction processing. The number of pixels of the image to be corrected is 1200 pixels on the short side, and image correction ON / OFF is performed on the image converted to the sRGB color space.

<Step S415>
The output unit 215 resizes the captured image data selected in step S412 for each of the photo slots on the template selected in step S413 to the size of the photo slot. As image data of the back cover page. " That is, the layout image data is generated as one image data. Then, the output unit 215 outputs the generated “image based on the image data of the front cover / back cover page” as layout information. The output destination may be the display 105 or the HDD 104. In the present embodiment, the “image of the front / back cover page” is displayed on the display 105 so as to allow the user to confirm the “image of the front / back cover page” and to enable editing if necessary. Output.

<Step S416>
As will be described in detail later, the processing after step S416 includes displaying an edit screen including “an image based on the image of the front cover / back cover page”, accepting an operation input to the edit screen, and executing a process corresponding to the operation input Work in the background.

  The input unit 208 acquires the number of facing pages of the photo album from the information acquired by the designation unit 201. The spread allocating unit 209 determines whether the number of groups divided in step S <b> 406 is the same as the number of spreads obtained by the input unit 208. If the result of this determination is that they are the same, the process proceeds to step S420; otherwise, the process proceeds to step S417. For example, in the case of FIG. 6A, since the number of group divisions = 8, if the number of facing pages acquired by the input unit 208 = 8, the process proceeds to step S420.

<Step S417>
The spread allocation unit 209 determines whether or not the number X of the groups divided in step S <b><the number Y of the spreads acquired by the input unit 208. As a result of this determination, if X <Y, the process proceeds to step S418, and if X ≧ Y, the process proceeds to step S419.

<Step S418>
The facing spread assignment unit 209 performs group redivision (subscene division) of the captured image data group acquired by the image acquisition unit 202 so that X = Y. In the following, in order to describe the processing of step S418 with a specific example, a case where Y = 10 in a state where X = 8 as shown in FIG. 6A is taken as an example. In this case, as shown in FIG. 6B, the groups 602 and 605 in FIG. 6A are divided at the portions indicated by the dotted lines to generate ten groups 601 to 610. The position of the dotted line (division reference line) is obtained, for example, as follows.

  First, of the groups 601 to 608 in FIG. 6A, a group having a large number of captured image data is specified. When X = 8 and Y = 10, it is necessary to provide division reference lines at two places. Therefore, of the groups 601 to 608, the groups having the largest number of captured image data and the group having the second largest number of captured image data Identify the two groups. In the case of FIG. 6A, the group having the largest number of captured image data is the group 605, and the groups having the second largest number are the groups 601 and 602. Here, the number of captured image data is the same in the group 601 and the group 602, but the imaging period of the group 602 is longer than that of the group 601. However, in this case, the group 602 is selected. As a result, each of the groups 605 and 602 is divided into two groups. If it is still not possible to decide, for example, a group that is earlier in time may be preferentially re-divided, and may be appropriately decided.

  First, a method of dividing the group 602 will be described. In FIG. 6A, in the group 602, a position between two peaks of the number of captured image data (a boundary between image capturing dates and times) is defined as a position of the division reference line. Next, a method of dividing the group 605 will be described. In FIG. 6A, there are three peaks in the number of captured image data in the group 605, but the difference in the number of captured image data between the two groups after division is made smaller. Therefore, here, the position on the right side among the positions between the three peaks (boundary of the imaging date and time) is set as the position of the division reference line. Note that, here, the image is divided at a portion different in the imaging date. However, if the portion having a large number of captured image data is a single day, the image is divided at a portion having the largest time difference in the single day. Note that various group division methods can be used as the group division method in this step, and the method is not limited to a specific group division method.

<Step S419>
The facing spread assignment unit 209 performs group redivision (scene integration) of the captured image data group acquired by the image acquisition unit 202 so that X = Y. In the following, a case where Y = 6 in a state where X = 8 as shown in FIG. 6A will be described as an example to describe the process of step S419 with a specific example. In this case, X = Y is achieved by reducing the number of groups by integrating several groups into one group. As shown in FIG. 6C, by integrating the groups 603 and 604 in FIG. 6A and the groups 607 and 608 in FIG. ~ 606 is generated. The position of the dotted line (integrated reference line) is obtained, for example, as follows.

  First, of the groups 601 to 608 in FIG. 6A, a group having a small number of captured image data is specified. In the case of X = 8 and Y = 6, it is necessary to provide an integrated reference line at two places. Therefore, of each of the groups 601 to 608, the group with the smallest number of captured image data and the group with the second smallest number Identify the two groups. In the case of FIG. 6A, the group having the smallest number of captured image data is the group 608, and the groups having the second smallest number are the groups 603 and 607. Here, the number of captured image data is the same between the group 603 and the group 607, but since the group 607 is adjacent to the group 608 to be integrated, the group 603 is selected in this case. However, as a result, each of the group 603 and the group 608 is to be integrated.

  First, a method of integrating the groups 603 will be described. The difference between the most recent imaging date and time among the imaging date and time of the captured image data group belonging to the group 603 and the oldest imaging date and time of the imaging date and time of the captured image data group belonging to the group 604 adjacent to the group 603 is defined as DA. The difference between the most recent imaging date and time among the imaging date and time of the captured image data group belonging to the group 602 adjacent to the group 603 and the oldest imaging date and time among the imaging date and time of the captured image data group belonging to the group 603 is stored in the DB. I do. At this time, since DA <DB, the group 603 is integrated with the group 604 to generate the group 603 in FIG. 6C.

  Since only the group 607 is adjacent to the group 608, the group 608 is integrated with the group 607 to generate the group 606 in FIG. 6C.

  Note that various group integration methods can be used for the group integration method in this step, and the method is not limited to a specific group integration method. For example, a group to be integrated may be integrated with a group having a small number of pieces of imaging data.

<Step S420>
The spread allocating unit 209 allocates each of the same number of groups as the number of spreads acquired by the input unit 208 to any one of the spreads. There are various methods for assigning which group to which spread. For example, for the first spread, a group whose imaging date / time (imaging date / time of any captured image data in the group) is earlier. Note that, as described later, in one facing page, the image data does not have to be lined up with the image capturing date and time.

<Step S421>
The selection unit 210 first selects one unselected spread from each spread, and among the captured image data groups belonging to the group assigned to the selected spread, captures captured image data to be displayed in the spread. select. A process of selecting four pieces of captured image data to be displayed in the spread from among the captured image data groups belonging to the group assigned to the selected spread will be described with reference to FIG. 9 as an example. Here, the double-page spread indicates two pages, and the first double-page spread and the last double-page spread are one page.

  In FIG. 9, “Start” indicates the date and time of imaging of the most recently captured image data in the group of captured image data belonging to the group (target group) assigned to the selected spread. End represents the date and time of imaging of recently captured image data in the group of captured image data belonging to the group of interest. However, (Start-End) indicates the imaging period of the target group.

  First, the captured image data to be arranged in the main photo slot is selected from the captured image data group (FIG. 9A) belonging to the group of interest. As the captured image data to be arranged in the main photo slot, the captured image data having the highest score for the main photo slot is selected from the respective captured image data belonging to the group of interest. The captured image data selected as the captured image data to be arranged in the main photo slot is represented as (1) in FIG. 9B.

  Next, three pieces of captured image data to be arranged in the sub-photo slot are selected from a captured image data group belonging to the group of interest. As will be described below, the image data is selected by subdividing the imaging period so that the image selection does not concentrate on a part of the imaging period.

  First, the first captured image data to be arranged in the sub-photo slot is selected. First, as shown in FIG. 9C, the imaging period is divided into two division periods. Then, as shown in FIG. 9D, of the two divided periods, the sub-photograph of the captured image data group belonging to the remaining divided period (solid line) excluding the divided period (dotted line) to which the captured image data (1) belongs. The captured image data with the highest score for the slot is selected as the captured image data to be arranged in the sub-photo slot. The captured image data selected as the first captured image data to be arranged in the sub photo slot is shown as (2) in FIG. 9D.

  Next, the second captured image data to be arranged in the sub-photo slot is selected. For this purpose, first, as shown in FIG. 9E, the imaging period is divided into four divided periods by dividing each divided period into two. As shown in FIG. 9F, the captured image data group belonging to the remaining divided period (solid line) excluding the divided period (dotted line) to which the captured image data (1) and (2) belong out of the four divided periods The captured image data with the highest score for the sub-photo slot is selected as the captured image data to be arranged in the sub-photo slot. The captured image data selected as the second captured image data to be arranged in the sub-photo slot is shown as (3) in FIG. 9 (F).

  Next, third captured image data to be arranged in the sub-photo slot is selected. The captured image data is selected from the divided period G to which the captured image data (1) to (3) does not belong, among the four divided periods of the captured image arranged in the sub-photo slot. Here, in the present embodiment, regardless of the number of image data, the imaging period is divided according to time, and therefore, as shown by hatching in FIG. A division period to which no data belongs also occurs. In such a case, as shown in FIG. 9H, each of the divided periods to which the captured image data (1) to (3) belongs is divided into two divided periods to be six divided periods. As shown in FIG. 9 (I), of the six divided periods, the captured image data belonging to the divided period (dotted line) to which the captured image data (1) to (3) belong and the remaining divided period (solid line) excluding the divided period G The captured image data with the highest score for the sub-photo slot in the group is selected as the captured image data for the sub-photo slot. The captured image data selected as the third captured image data to be arranged in the sub-photo slot is shown as (4) in FIG. 9 (I).

  An example of the functional configuration of the selection unit 210 that performs such an operation will be described with reference to the block diagram of FIG.

  The image group acquisition unit 1102 acquires a group of captured image data belonging to the group assigned to the target spread from among the group of captured image data acquired by the image acquisition unit 202. Further, the image group obtaining unit 1102 obtains the image capturing date and time of each captured image data belonging to the group, the score for the main photo slot, and the score for the sub photo slot.

  The counter 1103 counts the number C of times image data is selected from one group. The counter 1103 resets the value of C to 0 when the image data selection for one group is completed and the image data selection for the next group is performed.

The dividing unit 1105 divides the imaging period as described above. First, the dividing unit 1105 obtains the difference between the original imaging date and time and the latest imaging date and time, that is, the imaging period, at the imaging date and time of each captured image data acquired by the image group acquiring unit 1102. Then, the division unit 1105 equally divides the imaging period into ^2C division periods. Note that the division of the imaging period is not limited to the division into ^2C division periods evenly, and the imaging period may be equally divided so as to be the division period of the number of images to be selected.

  The setting unit 1106 manages information on each piece of captured image data belonging to the group.

  When the image selection unit 1108 selects the captured image data for the main photo slot, the score axis setting unit 1104 uses an evaluation criterion (for the main photo slot) to select the captured data for the main photo slot as the score axis. The evaluation criterion is supplied to the image selection unit 1108. In addition, when the image selection unit 1108 selects the captured image data for the sub-photo slot, the score axis setting unit 1104 uses the evaluation criterion (sub-photo slot) for selecting the imaging for the sub-photo slot as the score axis. Is supplied to the image selection unit 1108. For example, the score axis setting unit 1104 supplies the evaluation criterion for the main photo slot to the image selection unit 1108 if C = 0, and supplies the evaluation criterion for the sub photo slot to the image selection unit 1108 if C> 0. .

  The image selection unit 1108 selects captured image data for the main photo slot and captured image data for the sub photo slot from the captured image data group belonging to the group. When receiving the evaluation criteria for the main photo slot from the score axis setting unit 1104, the image selection unit 1108 selects the captured image data with the highest score for the main photo slot from the captured image data group belonging to the group. Perform processing. When receiving the evaluation criteria for sub-photo slot from the score axis setting unit 1104, the image selection unit 1108 selects the captured image data with the highest score for the sub-photo slot from the captured image data group belonging to the group. Process for As described above, the image selecting unit 1108 performs a process for selecting a captured image having the highest score corresponding to the score axis received from the score axis setting unit 1104 from the captured image data group belonging to the group. As described with reference to FIG. 9, no image data is selected from a divided period in which there is no image data. Further, in the present embodiment, the number of image data to be selected from the divided periods is one, but may be two or more.

  In the present embodiment, the score axis used for the main photo slot and the score axis used for the sub-photo slot have been described as examples. However, another score axis may be used. For example, in addition to the score axis for the main photo slot and the score axis for the sub photo slot, a score axis for the degree of person relevance may be used. Note that the person relevance can be scored based on, for example, the number of appearances of the person or the number of times the person has been imaged together. Specifically, when the number of appearances is large, it can be determined that the degree of person relevance is high, and when the number of appearances together is large, it can be determined that the degree of person relevance is high. Although the respective score axes are switched, an integrated score axis combining the score axis for the main photo slot, the score axis for the sub-photo slot, the score axis for the degree of person relevance, and the like may be used. At this time, the weight of each score when calculating the integrated score of the score for the main photo slot, the score for the sub-photo slot, and the score of the person relevance may be changed. The score axis to be used may be changed at random according to the number of times of processing, or the score axis may be switched by referring to a table prepared in advance.

  The similarity determination unit 1109 calculates the similarity between the captured image data selected from the group and the captured image data previously selected from the group. If the similarity is equal to or greater than the threshold, it is determined that the images are similar, and if it is less than the threshold, it is determined that the images are not similar. When it is determined that the images are not similar, the captured image data selected from the group is output. When it is determined that the images are similar, the captured image data selected from the group is discarded. For the similarity determination, a determination based on pattern matching, a determination using the SHIFT method, or the like can be used. The threshold value used for the similarity determination is changed according to the difference between the imaging date and time of the image data to be compared. For example, when the similarity determination result is output in the range of 0 to 100, it is determined that the similarity is closer to 100, the similarity is determined if the value is equal to or more than the threshold value, and the similarity is determined if the value is less than the threshold value. I do. If the difference between the imaging dates and times is short (for example, within 15 seconds) due to continuous shooting or the like, the threshold is set low (for example, set to 60). Otherwise, the threshold is set to 80. By lowering the threshold value, it is easy to determine that the images are similar. Even if the images are slightly similar, the images are determined to be similar images, and the selected image data is released. In this embodiment, the criterion is changed according to the difference between the imaging dates and times.

  The integration unit 1112 selects captured image data to be finally left from the captured image data determined to be dissimilar by the similarity determination unit 1109. When picked-up image data is selected from a plurality of divided periods, it is determined which divided period is to be used to select a picked-up image. In this determination, the image selection unit 1108 selects the image data with the highest score by using the score used for image selection. In the present embodiment, the image with the highest score is selected, but the present invention is not limited to this. For example, a section in which image data is not selected from an adjacent section among sections in which image data is selected may be selected.

  The number setting unit 1101 sets the total number of captured images based on the captured image data arranged in one spread. The image management unit 1113 adds the captured image data selected by the integration unit 1112 as the selected image data, and determines whether the number exceeds the required number set by the number setting unit 1101. If the required number has been reached, the image data selection processing ends. If the required number has not been achieved, the counter 1103 is instructed to count up, and the above-described image data selection processing is executed again. Image data selection processing performed by the selection unit 210 having the configuration shown in FIG. 11 will be described with reference to the flowchart in FIG.

<Step S1201>
The number setting unit 1101 sets the total number of pieces of captured image data to be selected from the group assigned to the spread to be processed.

<Step S1202>
The image group acquiring unit 1102 acquires a group of captured image data belonging to the group assigned to the spread of the processing target from the group of captured image data acquired by the image acquiring unit 202.

<Step S1203>
The counter 1103 counts the number C of image data selections by the image selection unit 1108. In the initial state, C = 0 since no image selection has been performed.

<Step S1204>
The score axis setting unit 1104 supplies the score axis to the image selection unit 1108 according to the number of image selections C. In the present embodiment, when the number of image data selections C = 0, the score axis for the main photo slot is set, and when the number of image data selections C = 1 or more, the score axis for the sub photo slot is set.

<Step S1205>
The dividing unit 1105 obtains, as an imaging period, a difference between the earliest imaging date and the latest imaging date and time in the captured images belonging to the group assigned to the spread of the processing target, and determines the obtained imaging period according to the number of image selections C. Is divided into divided periods (sections). Note that in the first step S1205, since the number of times of image selection C is 0, the imaging period is not divided.

<Step S1206>
The setting unit 1106 manages information on each piece of captured image data belonging to the group. Examples of the information related to the captured image data include the score of the image obtained in step S410, shooting time information, information on which divided period the image belongs to, and information on whether or not the image has been selected.

<Step S1207>
It is determined whether or not each of the divided periods divided by the dividing unit 1105 is unselected. If there is no unselected section, that is, if captured image data has already been selected from all divided periods, the process proceeds to step S1211. On the other hand, if there is an unselected section in which the captured image data has not been selected from the divided period, the process proceeds to step S1208.

<Step S1208>
The image selecting unit 1108 selects one piece of captured image data from the unselected division periods in step S1207 according to the score axis supplied from the score axis setting unit 1104.

<Step S1209>
The similarity determination unit 1109 obtains the similarity between the captured image data selected in the current step S1208 and the captured image data selected in the previous step S1208. If the similarity is equal to or larger than the threshold value, it is determined that they are similar, and the process proceeds to step S1210. On the other hand, if the similarity is less than the threshold, it is determined that there is no similarity, and the process proceeds to step S1211.

<Step S1210>
The similarity determination unit 1109 discards the captured image data selected in step S1208.

<Step S1211>
If an unselected divided period remains among the divided periods divided by the dividing unit 1105, the process returns to step S1206; otherwise, the process proceeds to step S1212.

<Step S1212>
The integration unit 1112 selects image data to be finally left from the selected image data from each of the divided periods determined as dissimilar by the similarity determination unit 1109. That is, when two or more pieces of captured image data remain in steps S1207 to S1211, sorting is performed to leave any one.

<Step S1213>
The image management unit 1113 adds the captured image data remaining after step S1212 as image data (selected image data) to be used for the spread to be processed.

<Step S1214>
The image management unit 1113 determines whether the number of currently selected image data does not exceed the required number set by the number setting unit 1101 as a result of the addition in step S1213. If the required number has been reached, the image data selection processing ends. On the other hand, if the required number has not been achieved, the process returns to step S1203. In this case, the counter 1103 increments the image data selection count C by one, and the processing of each subsequent step is performed. As a result, two times more divided periods than the previous period are set. As the division period is more finely divided, the rank of the captured image data according to the score in the divided period changes, so that the captured image data not selected previously is selected.

<Step S422>
The layout unit 212 selects a template based on the image data selected in step S421 to determine an image layout. From the template group input by the input unit 211, a template to be applied to the spread of interest is selected based on the captured image data selected for the spread of interest. In the following, it is assumed that the input unit 211 has input the 16 templates (1-1) to (4-4) in FIG. 10A in order to describe the process in step S422 with a specific example. In step S422, one piece of captured image data for the main photo slot and two pieces of captured image data for the sub photo slot are selected. In this case, as shown in FIG. 10A, a template group having three photo slots is set as a candidate. Here, when images based on the three pieces of captured image data selected in step S421 are arranged in descending order of image capturing date and time, as shown in FIG. 10B, a vertically long (portrait) image 1006, a horizontally long (horizontal) It is assumed that the image 1007 is a vertically long (vertically oriented) image 1005. Note that images 1006 and 1007 are captured images to be placed in the sub-photo slot, and image 1005 is a captured image to be placed in the main photo slot.

  In the present embodiment, it is assumed that an image with an older imaging date and time is laid out at the upper left of the template and an image with a newer imaging date and time is laid out at the lower right. As shown in FIG. 10B, since the image capturing date and time of the vertically long image 1005 arranged in the main photo slot is the latest, the layout (3) in which the image for the vertically long main photo slot of the latest image capturing date and time is arranged at the lower right. -1) to (3-4) remain as candidates.

  The image with the oldest imaging date and time is the image 1006 for the vertically long sub-photo slot, and the layout (3-1) and (3-2) in which the image for the vertically long sub-photo slot with the oldest imaging date and time is arranged at the upper left. ) Remain as candidates.

  The image with the second oldest shooting date and time is the horizontally long sub-photo slot image 1007, and the layout for arranging the second oldest horizontally long sub-photo slot image other than the upper left and lower right is as follows. Only (3-2).

  However, in this case, the layout (3-2) is selected as a template to be applied to the spread of interest. The method of selecting a template to be applied to the spread of interest is not limited to the above method, and another method may be adopted.

<Step S423>
If the flag value acquired by the input unit 213 is a value indicating that image correction is to be performed, the image correction unit 214 applies dodging to each of the three pieces of captured image data selected by the selection unit 210. Correction processing such as correction, red-eye correction, and contrast correction is performed. Of course, if the flag value acquired by the input unit 213 is a value indicating that image correction is not to be performed, the image correction unit 214 does not execute the correction processing. The number of pixels of the image to be corrected is 1200 pixels on the short side, and image correction ON / OFF is performed on the image converted to the sRGB color space.

<Step S424>
The output unit 215 resizes the captured image data selected in step S421 for each photo slot on the template selected in step S422 and arranges the captured image data in the photo slot in the size of the photo slot. Page image data "is generated. Note that the image to be arranged may be an analysis image generated from the captured image instead of the captured image. Then, the output unit 215 outputs the generated “image data of a double-page spread” as layout information. The output destination may be the display 105 or the HDD 104. In the present embodiment, the “spread page image” is output to the display 105 in order to allow the user to check “spread page image” and to enable editing if necessary.

<Step S425>
If all spreads have been selected in step S421, the processing according to the flowchart in FIG. 4 ends. On the other hand, if unselected spreads remain, the process returns to step S421.

  Next, a transition of a screen displayed on the display 105 during the above-described processing by the automatic layout processing unit 218 will be described with reference to a flowchart of FIG. As described above, for example, when the user double-clicks the application icon displayed on the display 105 using the mouse 107, the GUI 301 of FIG. 3 is displayed on the display 105 (S1501). A GUI 301 in FIG. 3 is a setting screen for setting album creation conditions.

  After performing various settings as described above on the GUI 301 in FIG. 3, if the user operates the keyboard 106 or the mouse 107 to specify the OK button 308, the processing according to the flowchart in FIG. 4 is started. Further, a window 2101 indicating the degree of the progress of the process is displayed on the display 105 as shown in FIG. 21 (S1502).

  The progress bar 2102 indicates 0% at the start of the process according to the flowchart of FIG. 4, and its length increases as the process proceeds, and indicates 100% when the process of step S415 is completed. That is, in the present embodiment, in the “album creation progress display”, the progress of the processing from the generation of the analysis image to the creation of the cover layout image data is displayed. The length of the progress bar 2102 during that time is updated so as to become the corresponding length each time each process set in advance in the flowchart of FIG. 4 is completed. In addition, since analysis processing such as face detection and individual recognition takes a long processing time, the display may be updated according to the number of images at the time of analysis execution.

  When the progress bar 2102 indicates 100%, that is, when the process of step S415 is completed, the GUI illustrated in FIG. 16 is displayed on the display 105 (S1503).

  In a region 1601, a preview screen as a result of laying out the captured images is displayed. FIG. 16 shows the GUI immediately after startup. At this time, only the image of the cover page (spread page of the front cover and the back cover) has been generated, and the image of the cover page is displayed in the area 1601. Have been.

  The photo slot 1602 is a photo slot for the back cover. In the photo slot 1602, a captured image assigned to the back cover is arranged. A photo slot 1603 is a photo slot for a cover, and a captured image assigned to the cover is arranged in the photo slot 1603.

  The title area 1604 is an area for inputting text information such as the title of the photo album, and nothing is displayed immediately after the GUI is started. For example, when the user operates the keyboard 106 and the mouse 107 to specify the title area 1604, characters can be input to the title area 1604. By operating input means such as the keyboard 106 and the mouse 107, characters can be input to the title area 1604. input. The position of the title area 1604 can be moved by a drag operation with the mouse 107 or the like.

  The user operates the keyboard 106 and the mouse 107 to specify a desired photo slot (one of the photo slots 1602 and 1603 in FIG. 16) among the photo slots on the cover page, and specifies the image exchange button 1605. Upon receiving this designation, the display 105 displays a GUI 1701 illustrated in FIG. The GUI 1701 is a GUI for receiving an operation for replacing a captured image currently arranged in a photo slot designated by a user with a desired captured image. That is, the GUI 1701 is an image replacement screen. This GUI 1701 will be described later.

  Returning to FIG. 16, the user operates the keyboard 106 or the mouse 107 to specify a desired photo slot (one of the photo slots 1602 and 1603 in FIG. 16) among the photo slots on the cover page, and presses the image edit button 1606. specify. Upon receiving this designation, a GUI illustrated in FIG. 18 is displayed on the display 105. The GUI illustrated in FIG. 18 is a GUI that accepts operations for resizing, rotating, and trimming a captured image currently arranged in a photo slot designated by a user. That is, the GUI in FIG. 18 is an image editing screen. This GUI will be described later.

  Referring back to FIG. 16, when the user operates the keyboard 106 or the mouse 107 to specify the button 1608, the layout result of the next page is displayed in the area 1601. When the user operates the keyboard 106 or the mouse 107 to specify the button 1607, the layout result of the previous page is displayed in the area 1601. That is, the buttons 1607 and 1608 are switching units for switching the layout image to be displayed.

  Here, when the GUI of FIG. 16 is displayed, the layout of the main body is being created. The GUI shown in FIG. 16 is displayed, and the main body layout processing is executed in the background while the user operates the GUI to input a title or adjust an image. Therefore, while the layout of the main body is being created, the buttons 1607 and 1608 may not be displayed so as not to function, or may be grayed out and cannot be pressed (invalidated). At this time, when the buttons 1607 and 1608 are pressed, a display such as “layout is being created” may be displayed.

  Next, it is determined whether the layout creation of the main body being executed in the background has been completed (whether the processing according to the flowchart of FIG. 4 has been completed) (S1504). If the layout creation has been completed, the process proceeds to S1505, and if not completed, the process returns to S1503. In the present embodiment, it is assumed that the process does not proceed to step S1505 until all layouts of the main body are completed. However, the present invention is not limited to this, and there is a spread page whose processing is completed among a plurality of spread pages of the main body. In this case, the process may proceed to step S1505.

  When the layout creation is completed, the buttons 1607 and 1608 are activated (instruction is possible), so that the user can also confirm and edit the layout results of pages other than the cover page by preview (S1505).

  In FIG. 16, when the user designates an order button 1609 by operating the keyboard 106 or the mouse 107, order processing of a photo album is performed. In the photo album ordering process, the image data of the page created at this time is transferred to a server device of a company that creates the photo album. Generally, the order button 1609 is designated when the front cover / back cover, body confirmation, and editing work are completed, so that a completed photo album can be ordered.

  Next, the GUI 1701 in FIG. 17 will be described. In a region 1702, a recommended captured image serving as a replacement candidate is displayed. The recommended captured image is determined, for example, as follows. At the time of image selection, the score of the captured image group belonging to the same section as the desired captured image that the user has requested to replace (the captured image arranged in the photo slot specified before specifying the image replacement button 1605) is The second and subsequent captured images in descending order are recommended captured images. The score at this time is the score of the score axis set in the photo slot (or the front and back covers) in which the desired captured image is arranged. Then, when the captured images up to N (an integer of 2 or more) are the recommended captured images, N captured images are displayed in the area 1702 as the recommended captured images. As a result, it is possible to easily replace an image having characteristics similar to the desired captured image with a high score. The present invention is not limited to this, and the captured images with the highest score in the shooting period used for creating the album, that is, the captured images with the second and subsequent scores may be set as the recommended captured images.

  In an area 1703, all captured images stored in the HDD 104 are displayed. When there are many captured images, the captured images are sequentially displayed in accordance with the user's operation of moving the scroll bar. Note that the condition may be specified so that only the captured image satisfying the desired condition is displayed in the area 1703.

  When the user operates the keyboard 106 and the mouse 107 to select a desired captured image from the area 1702 and the area 1703 and then specifies the OK button 1704, the GUI 1701 is deleted. Then, the captured image in the photo slot designated by the user in the GUI of FIG. 16 in response to the desire to replace the image is replaced with the captured image designated by the user in the GUI 1701. On the other hand, if the user operates the keyboard 106 or the mouse 107 to specify the cancel button 1705, the GUI 1701 is deleted, and the image is not replaced.

  Next, the GUI of FIG. 18 will be described. In the preview area 1801, a captured image (image to be edited) in the photo slot specified by the user in the GUI of FIG. 16 when the image editing button 1606 is specified is displayed in the same aspect ratio as the size of the photo slot. You.

  Each time the user operates the keyboard 106 or the mouse 107 to specify the enlarge button 1802, the image to be edited is enlarged. When the image to be edited is enlarged, its size becomes larger than the preview area 1801 and protrudes. However, the protruding part is not displayed, and only the part that fits in the preview area 1801 is displayed in the preview area 1801. That is, the portion displayed in the preview area 1801 is trimmed from the captured image. Each time the enlargement button 1802 is designated, the enlargement is performed, for example, by 5%.

  Each time the user operates the keyboard 106 or the mouse 107 to specify the reduction button 1803, the image to be edited is reduced. The reduction button 1803 is valid only when the image to be edited has a size larger than the preview area 1801, and is invalid when the image to be edited has the same size as the preview area 1801. Each time the reduction button 1803 is designated, the size is reduced by, for example, 5%.

  Each time the user operates button 1061 by operating keyboard 106 or mouse 107, the image in preview area 1801 is rotated 90 degrees counterclockwise. Each time the user operates the keyboard 106 or the mouse 107 to specify the button 1805, the image in the preview area 1801 is rotated 90 degrees clockwise. Note that the image in the preview area 1801 is appropriately resized so that a blank area does not occur in the preview area 1801 when rotating the image. In addition, by dragging an image on the preview area 1801 and moving the image in an arbitrary direction, another image portion not included in the preview area 1801 can be displayed in the preview area 1801.

  Here, both the enlargement ratio and the reduction ratio are 5%, but are not limited to these values. Further, the enlargement ratio and the reduction ratio may be appropriately changed by the user by inputting text or adjusting the length of the bar. Here, enlargement / reduction and rotation of an image are shown as an example of image editing. However, a GUI (operation unit) that enables other image editing such as color adjustment and gradation adjustment may be configured. I do not care.

  Also, for the sake of explanation, the method of selecting and laying out a cover / back cover image after image scoring has been described, but a high-score image is selected for each spread after spread allocation and used as the cover image. You may. At this time, the number of images to be adopted may be set to the maximum number of page spreads, and the cover may be created with a smaller number of images. The number of images to be used may be set according to the resolution of the maximum image size after layout, the face size within the minimum image size after layout, and the like.

  In the present embodiment, the layout process of the front and back covers is executed before the layout process of the main body, and the layout result is previewed when the layout of the front and back covers (the layout image in which the images are arranged in the template) is completed. . Thereby, the user can perform editing such as title input. By performing layout processing of the main body in the background while the user is editing the cover, and enabling previewing and editing at the time of completion, album creation can be performed efficiently. In addition, by previewing when the layout of the front cover / back cover is completed, it is possible to reduce the user's waiting time until the start of editing.

  Further, by changing the selection conditions of the images used for the front cover and the back cover, it is possible to perform a variety of image selections, and to create an album cover with high user satisfaction.

[Second embodiment]
Hereinafter, differences from the first embodiment will be mainly described, and unless otherwise noted below, it is assumed to be the same as the first embodiment. In the present embodiment, the captured image is trimmed so that the face region included in the image does not overlap the title region. In the present embodiment, the layout unit 217 has a configuration illustrated in FIG.

  The acquisition unit 2201 acquires a template group that is a candidate to be applied to the front cover / back cover from the template group input by the input unit 211. The template acquired by the acquisition unit 2201 has a title area.

  The acquisition unit 2202 acquires the coordinate position (included in the information attached to the template) of the title area in each template acquired by the acquisition unit 2201. The coordinate position may be the coordinate position of the four corners of the title area, the coordinate position of the upper left corner and the lower right corner, or the coordinate position of one corner and the vertical and horizontal sizes.

  The acquisition unit 2203 acquires the captured image data (covered image data) selected by the selection unit 216.

  The acquisition unit 2204 acquires the number of faces in the captured image and the coordinate position of the face included in the analysis result of the captured image data acquired by the acquisition unit 2203 by the image analysis unit 204. The coordinate position is the coordinate position of four corners or two opposite corners of the face area, or the coordinate position of one corner and the vertical and horizontal sizes.

  The selecting unit 2205 specifies the coordinate position of the face closest to the title area using the coordinate position of the title area acquired by the acquiring unit 2202 and the coordinate position of each face. For example, the coordinate position of each face is converted into the coordinate position on the template, and the coordinate position closest to the title area is specified from the converted coordinate positions.

  The determination unit 2206 specifies the coordinate position of the face overlapping with the title region using the coordinate position of the title region acquired by the acquisition unit 2202 and the coordinate position of each face. The trimming unit 2207 trims a partial image to be displayed from the captured image selected by the selection unit 216. Operations of the determination unit 2206 and the trimming unit 2207 will be described with reference to FIG.

  An area 2401 shown in FIG. 14 is an area (photo slot) for laying out a captured image for a cover, and FIG. 14 does not show the entire captured image but only faces (face areas 2402 and 2403) for the sake of explanation. , Except the face. The coordinate position of the upper left corner of the face area 2402 is (F1x, F1y). The vertical size of the face area 2402 is F1H, the horizontal size is F1W, the vertical size of the face area 2403 is F2H (> F1H), and the horizontal size is F2W (>). F1W). An area 2404 is a title area, and the coordinate position of the upper left corner is (Tx, Ty), vertical size = TH, and horizontal size = TW. Here, for the sake of explanation, the number of faces included in the cover image is two, the detected face is in a state where the in-plane rotation is 0 degree, and the width and height of the detected face are the same length. And

  It is assumed that the size of the captured image to be laid out in the area 2401 is sufficiently larger than the size of the area 2401, and that a part of the captured image is displayed in the area 2401 after being trimmed. In FIG. 14, the increasing direction of the x-axis is defined as the right direction, and the increasing direction of the y-axis is defined as the downward direction.

  In such a situation, first, the selecting unit 2205 specifies the maximum vertical size among the vertical sizes of the respective face regions. In the case of FIG. 14, since F2H> F1H, F2H is specified as the maximum vertical size. Then, the selection unit 2205 sets an area 2405 (horizontal size = JW, vertical / horizontal size = JH) obtained by expanding the vertical / horizontal size of the area 2404 by the maximum vertical size F2H, as a title area for performing the following overlap determination.

  Then, since the selecting unit 2205 determines that the face area 2402 is closer to the area 2404 than the face area 2403, the determining unit 2206 determines whether the face area 2402 and the area 2405 overlap. Various methods can be considered for the overlap determination. Here, it is determined whether or not the coordinate position (F1x, F1y) of the upper left corner of the face area 2402 is included in the area 2405. It should be noted that here, areas where a margin of a predetermined width is provided in the vertical and horizontal directions with respect to the face area 2402 determined as a face in the face detection result are set as trimming target areas. This is because the face detection result may erroneously determine the face size at the time of detection. In FIG. 14, the face is not actually applied to the area 2404, but since the trimming target area overlaps, it is determined that the face area 2402 overlaps the area 2404. Note that the margin may be a fixed value or may be appropriately set according to the size of the face area 2403. For example, a margin of 10% may be set for the face area 2403 in the up, down, left, and right directions.

  Here, a method of avoiding a face area over a title area using a uniform criterion has been described, but the criterion may be changed according to the face size. For example, when the face is small, the criterion may be loosened, and when the face is large, the criterion may be strict.

  If the determination unit 2206 determines that the partial region overlaps, the trimming unit 2207 moves the partial region displayed in the current region 2401 in the captured image in the y direction by an amount of {F1H- (Ty + TH + F2H)}. To identify. Then, the trimming unit 2207 arranges the image (partial image of the captured image) in the specified partial area in the area 2401.

  The operation of the layout unit 217 described above will be described with reference to the flowchart of FIG. In step S2301, the acquiring unit 2201 acquires a template group that is a candidate to be applied to the front cover / back cover from the template group input by the input unit 211. In step S2302, the acquisition unit 2202 acquires the coordinate position of the title area in each template acquired by the acquisition unit 2201. In step S2303, the acquisition unit 2203 acquires the captured image (captured captured image) selected by the selection unit 216. In step S2304, the acquisition unit 2204 acquires the number of faces in the captured image and the coordinate position of the face included in the analysis result of the captured image acquired by the acquisition unit 2203 by the image analysis unit 204. In step S2305, the selecting unit 2205 specifies the coordinate position of the face closest to the title region using the coordinate position of the title region acquired by the acquiring unit 2202 and the coordinate position of each face. In step S2306, the selection unit 2205 first specifies the maximum vertical size among the vertical sizes of the face areas, and sets an enlarged area obtained by expanding the title area by the specified vertical size. Then, the determination unit 2206 determines whether the face area specified by the coordinate position specified in step S2305 and the enlarged area overlap.

  If the result of this determination is that there is overlap, the process proceeds to step S2307; otherwise, the process according to the flowchart of FIG. 23 is completed. In step S2307, the trimming unit 2207 specifies a partial area in the captured image in which the currently displayed partial area has been moved in the y direction by the above amount, and arranges the image in the specified partial area on the cover.

  In this way, by controlling the face area of the cover image so as not to overlap the title area when creating the cover, it is possible to improve the degree of completeness of the cover of the album and the satisfaction of the user. Furthermore, by performing control so that the face area does not cover the title area before the display, it is possible to reduce the work required for editing by the user. It should be noted that the overlap determination with the original text area may be performed instead of the enlarged area.

[Third Embodiment]
Hereinafter, differences from the first embodiment will be mainly described, and unless otherwise noted below, it is assumed to be the same as the first embodiment. In the first and second embodiments, the image of the cover page laid out after correcting the image arranged on the front cover and the image arranged on the back cover is displayed. In the present embodiment, in order to display the image of the cover page faster, the image arranged on the front cover and the image arranged on the back cover are laid out without correction to generate and display the image of the cover page, and after the display, Update the display by performing image correction. This procedure will be described with reference to the flowchart of FIG.

  The selection unit 216 selects a specified number of pieces of captured image data to be laid out on the cover page, which is one page of the front cover and the back cover, as in the first embodiment (S2501). Next, the layout unit 217 selects a template to be applied to the front cover / back cover in the same manner as in the first embodiment (S2502). Next, the output unit 215 generates “cover page image data” in which captured images are arranged in the respective photo slots on the template in the same manner as in the first embodiment (S2503), and outputs “cover page image data”. Is output as layout information (S2504). As a result, the display 105 displays a GUI including a “cover page image” based on the cover page image data. Here, the “image of the cover page” is not corrected as in the first embodiment. Therefore, next, the image correction unit 214 corrects the captured image laid out in the “image of the cover page” in the same manner as in the first embodiment (S2505). Then, the output unit 215 generates “cover page image data” in which the corrected captured images are arranged in respective photo slots on the template, and converts the “cover page image data” into layout information in the same manner as in S2503. (S2506). Accordingly, the captured image included in the GUI displayed on the display 105 is updated from the captured image before correction to the captured image after correction. The order of the images to which the correction is applied is performed first for the cover image, and then for the back cover image. When there are a plurality of images, the correction processing is executed from the image having the smaller display size.

  When the image correction has been performed on all the images used for the front cover and the back cover, the processing according to the flowchart in FIG. 13 is completed, and the images that have not been subjected to the image correction remain. In this case, the processing of S2505 and S2506 is repeated.

  Here, for the sake of explanation, the order in which image correction is performed is set for the front cover and the back cover, but the processing order may be set according to the order of image scores used by the selection unit 216. Further, the image adopted for the front cover and the back cover may be simultaneously subjected to image correction by multi-threading, and the corrected images may be displayed in order.

  When displaying the layout result of the front cover / back cover as described above, the display is first performed without image correction. Thereafter, by updating the display from the image to which the image correction has been applied, the layout result of the front cover / back cover can be displayed more quickly, and the time until the user can start editing work can be reduced.

[Fourth embodiment]
In the above-described embodiment, a case has been described in which both the front cover and the back cover are created as the cover pages. However, only the front cover may be generated without generating the back cover.

  In the second embodiment, the captured image is trimmed so that the face area included in the selected image does not overlap the title area. However, the present invention is not limited to this. May be selected. That is, a plurality of cover templates having different title positions and / or slot positions may be prepared in advance, and a template in which the face area included in the selected image does not overlap the title area may be selected.

  In the above embodiment, the application for creating a photo album has been described, but the application to be created is not limited to the photo album. For example, the configuration of the above-described embodiment may be applied to an application or an image processing apparatus having an image selection process for automatically selecting a recommended image.

  Also, in FIG. 1, an image processing apparatus assuming a local environment has been described as an example, but the image processing apparatus is not limited to this. For example, it may be an image processing device on a server. In this case, the image data stored in the server may be used or the image data may be uploaded.

  Further, in the above-described embodiment, as shown in FIG. 9, the image data is selected and then divided sequentially, but the present invention is not limited to this. For example, the image data may be divided in advance in accordance with the number of image data to be selected. A method of selecting image data in this case will be described with reference to FIG. As an example, a case in which it is desired to select four pieces of captured image data will be described.

  First, the imaging period is divided into four equal parts as shown in FIG. Then, regardless of the imaging period, the first captured image data is selected. Next, the second captured image data is selected from three imaging periods in which the first captured image data is not selected. Next, the third captured image data is selected from two imaging periods in which neither the first captured image data nor the second captured image data is selected. The fourth captured image data is selected from an imaging period in which neither the first captured image data, the second captured image data, nor the third captured image data is selected. If no captured image data exists during the imaging period, similarly to FIG. 9H, an imaging period during which the first captured image data is acquired, an imaging period during which the second captured image data is acquired, What is necessary is to divide the imaging period in which the third captured image data is acquired, and select the captured image data.

  Further, in the above-described embodiment, the image is a photograph and the grouping is performed based on the shooting date and time. However, the grouping may be performed based on the date and time when the image data is created. In addition, each of the above embodiments may be used by combining some or all of them as appropriate.

(Other embodiments)
The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. This processing can be realized. Further, it can be realized by a circuit (for example, an ASIC) that realizes one or more functions.

  101: CPU 105: Display

Claims (11)

Generating means for generating layout image data in which an image based on image data selected from the image data group is laid out;
Display control means for displaying a layout image on a display unit based on the layout image data generated by the generation means,
With
The generating means generates layout image data corresponding to the cover and layout image data corresponding to the main body,
When the generating means generates layout image data corresponding to the cover, the display control means displays a cover layout image on the display unit based on the layout image data corresponding to the cover,
The image processing apparatus according to claim 1, wherein the generation unit generates layout image data corresponding to the book body in a background of the display of the cover layout image. The display control means displays the layout image of the cover on an editing screen for editing the layout image,
The image processing apparatus according to claim 1, wherein the editing screen includes an operation unit that receives a user's instruction.   The image processing apparatus according to claim 1, wherein the generation unit arranges an image based on the image data selected from the image data group in a photo slot on a cover template, and generates layout image data corresponding to the cover. 3. The image processing device according to 1 or 2.   The image processing apparatus according to claim 1, wherein the generation unit arranges an image based on the image data selected from the image data group in a photo slot on a template for a main body, and generates layout image data corresponding to the main body. The image processing device according to any one of claims 1 to 3. The display control unit displays a screen indicating the progress until the generation unit completes generation of the layout image data corresponding to the cover, and when the generation of the layout image data corresponding to the cover is completed, the screen is displayed. the image processing apparatus according to any one of claims 1 to 4, characterized in Rukoto to display the layout image of the cover sheet on the display unit is erased. The display control means,
After the display of the layout image of the cover sheet, the image processing apparatus according to any one of claims 1 to 5, characterized in that displaying the layout image corrected by correcting the layout image of said surface sheet. It said display control means, when said generating means generates the layout image data corresponding to the body, and characterized in that based on the layout image data corresponding to the main body, causes the display layout image of the book block on the display unit The image processing apparatus according to claim 1. The display control means displays a switching unit that instructs display of a layout image of the main body,
The switching unit is disabled until the generation of the layout image data corresponding to the main body by the generation unit is completed, and is enabled when the generation of the layout image data corresponding to the main body by the generation unit is completed. The image processing apparatus according to claim 1, wherein: The generation means,
When generating layout image data in which a part of an image based on image data selected from the image data group is overlapped with a character input area, a face included in an image based on the image data does not overlap the character input area. The image processing apparatus according to any one of claims 1 to 8, wherein the layout image data is generated as described above. A generation step of generating layout image data in which an image based on the image data selected from the image data group is laid out;
A display control step of displaying a layout image on a display unit based on the layout image data generated in the generation step,
With
In the generation step, layout image data corresponding to the cover and layout image data corresponding to the main body are generated,
In the case where layout image data corresponding to the cover is generated in the generation step, in the display control step, a layout image of the cover is displayed on the display unit based on the layout image data corresponding to the cover,
The image processing method, wherein in the generation step, layout image data corresponding to the book body is generated in a background of display of the cover layout image.   A computer program for causing a computer to function as each unit of the image processing apparatus according to claim 1.

Images