JP7350496B2 - Apparatus, method, and program - Google Patents

Description

TECHNICAL FIELD The present invention relates to an apparatus, method, and program, and more specifically, to an apparatus, method, and program for creating an album.

In addition to the spread of digital cameras, the number of photos taken by users has increased rapidly in recent years due to the spread of smart devices and improved camera performance of smart devices. As the number of photographed images increases, the variety of subjects photographed is also increasing.

2. Description of the Related Art Conventionally, a method of creating an album by laying out photographic images taken with a digital camera is known. Patent Document 1 proposes a method in which a plurality of users jointly create an album.

JP2016-66328A

When a plurality of users provide photos to each other to create an album, it is necessary to create the album in consideration of the photos each user wants to use. In Patent Document 1, each user can participate in album creation and create an album. However, Patent Document 1 does not take into consideration an album that has both content common to all users and content customized to individuals.

In view of the above-mentioned problems, the present invention aims to create an album having both content common to all users and content tailored to each individual.

The present invention is a device that generates an album including a plurality of pages, the plurality of pages being different for each album, a common page common to the plurality of albums, and a main subject set for each album. creating means for creating the plurality of pages including individual pages ; and specifying means for specifying the number of candidate images that are candidates to be laid out in the plurality of albums based on the main subject corresponding to each of the plurality of albums. , determining means for determining the number of individual pages that are the same in the plurality of albums based on the number of candidate images corresponding to each of the plurality of albums specified by the specifying means. , the creating means creates the individual pages for each of the plurality of albums so that the number of the individual pages is the same in the plurality of albums, so that the total number of pages and the common page number are the same in the plurality of albums. The plurality of pages for each of the plurality of albums are created such that the number of pages and the number of individual pages are the same , and the creation means determines the number of pages determined by the determination means for each of the plurality of albums. The method is characterized in that a number of the individual pages are created .

According to the present invention, it is possible to create an album having both content common to all users and content tailored to each individual.

1 is a block diagram showing the hardware configuration of an image processing device in a first embodiment. FIG. 1 is a block diagram showing a software configuration of an image processing apparatus in a first embodiment. FIG. FIG. 3 is a diagram illustrating the layout of images in an album created in the first embodiment. FIG. 3 is a diagram showing a GUI screen of an album creation application in the first embodiment. 7 is a flowchart of automatic layout processing in the first embodiment. FIG. 3 is a diagram illustrating image group division in the first embodiment. FIG. 3 is a diagram illustrating scene classification in the first embodiment. FIG. 3 is a diagram illustrating image selection in the first embodiment. FIG. 3 is a block diagram illustrating template determination in the first embodiment. It is a flowchart of template determination processing in a 1st embodiment. FIG. 3 is a diagram illustrating a method for determining an image layout in the first embodiment. FIG. 2 is a detailed block diagram of a spread joining section in the first embodiment. FIG. 3 is a flowchart of double-page spread combining processing in the first embodiment. FIG. FIG. 2 is a block diagram showing the software configuration of an image processing apparatus in a second embodiment. 7 is a flowchart of automatic layout processing in the second embodiment. FIG. 7 is a detailed block diagram of a facing-spread joining section in a second embodiment. 12 is a flowchart of double-page spread combining processing in the second embodiment. FIG. 3 is a block diagram showing the software configuration of an image processing apparatus in a third embodiment. 7 is a flowchart of automatic layout processing in the third embodiment. It is a flowchart of the number of two-page spread calculation process in 3rd Embodiment. FIG. 7 is a block diagram showing the software configuration of an image processing apparatus in a fourth embodiment. It is a flowchart of automatic layout processing in a 4th embodiment. FIG. 12 is a detailed block diagram of a facing-spread joining section in a fourth embodiment. 13 is a flowchart of double-page spread combining processing in the fourth embodiment. FIG. 3 is a diagram for explaining the effects of the first embodiment. FIG. 7 is a diagram for explaining the effects of the second embodiment. FIG. 7 is a diagram for explaining the effects of the third embodiment. FIG. 7 is a diagram for explaining the effects of the fourth embodiment.

[First embodiment]
In this embodiment, an application for creating an album (hereinafter also referred to as an "application" or simply "app") is operated in an image processing device, and an album in which a plurality of photo images are automatically laid out on the application is operated. Create multiple . In this embodiment, a double-page spread common to at least two albums and a separate double-page spread for each album are created. The total number of spreads (referred to as the total number of spreads) or the total number of pages (referred to as the total number of pages) constituting each album is the same for all albums according to user specifications. However, the number of common spreads and individual spreads for each album (that is, for each user) changes depending on the image group and settings. The two-page spread and the images arranged within the two-page spread are arranged in the order of shooting time.

<About the hardware configuration of the image processing device>
The hardware configuration of the image processing apparatus in this embodiment will be described below using FIG. 1. Note that although a case in which the image processing device is an information processing device (PC) will be described below, other devices such as a smartphone may be employed as the image processing device.

As shown in FIG. 1, the image processing device 100 includes a CPU 101, a ROM 102, a RAM 103, an HDD 104, a display 105, a keyboard 106, a mouse 107, and a data communication device 108. These parts are connected by a data bus 109 and can mutually send or receive data.

The CPU 101 controls the entire image processing apparatus 100. Further, the CPU 101 executes the image processing method described in this embodiment according to a program. Note that although the image processing device has one CPU in FIG. 1, the image processing device may have a plurality of CPUs.

The ROM 102 stores programs executed by the CPU 101. The RAM 103 provides a memory for temporarily storing various information when the CPU 101 executes a program. The HDD 104 stores a database that holds image files and processing results such as image analysis, and in this embodiment, an application program for creating an album is stored in the HDD 104. Note that this application program will be described later using FIG. 4 and the like.

The display 105 is a device for displaying the user interface (hereinafter referred to as UI) and image layout results of this embodiment and presenting them to the user. Display 105 may have touch sensor functionality. The keyboard 106 is one of the input devices included in the image processing apparatus 100, and is used, for example, to input predetermined information on the GUI displayed on the display 105. In this embodiment, the user inputs the number of spreads of an album via the keyboard 106. The mouse 107 is one of the input devices included in the image processing apparatus 100, and is used, for example, to click and press a button on the GUI displayed on the display 105.

The data communication device 108 is a device for communicating with external devices such as printers and servers. For example, automatic layout data is transmitted via the data communication device 108 to a printer or server connected to the image processing device 100. A data bus 109 connects each of the above-mentioned components and the CPU 101. The above is the content of the hardware configuration of the image processing apparatus in this embodiment.

<About the software configuration of the image processing device>
The software configuration of the image processing apparatus in this embodiment, in other words, the functional configuration realized by the album creation application installed in the image processing apparatus will be described below with reference to FIG. The album creation application is started when the user double-clicks the application icon displayed on the display 105 using the mouse 107. Although the album creation application has various functions, the automatic layout function provided by the automatic layout processing unit 202, which is a particularly important function in this embodiment, will be mainly explained below.

As shown in FIG. 2, this application includes an album creation condition setting section 201, an automatic layout processing section 202, and an album display section 213. The "automatic layout function" is a function that creates an album image by classifying or selecting images of photographed photographs based on their contents and attributes and then laying them out. The created album image is displayed on the display 105.

The album creation condition setting unit 201 sets album creation conditions in the automatic layout processing unit 202 according to the user's mouse operation. In this embodiment, as album creation conditions, a group of images used in an album, the total number of double-page spreads per album, merchandise for the album, the number of books in the album, and the main subject for each album are set. The image group settings may be set using the accompanying information or attribute information of individual image data, such as the shooting date and time, or may be set based on the structure of the file system in which the image data is stored, such as specifying the device or directory. You can also set it. Note that a "two-page spread" corresponds to, for example, one display window in display, and corresponds to a pair of adjacent pages printed on different sheets in printing, for example. Although the concept of double-page spreads will be described here, images may be laid out on a single page when printed. In this embodiment, the term "page" includes "two-page spread" and "single page." Furthermore, the main subject is a subject that is laid out with particular emphasis in the album. Examples include specific individuals, categories of objects such as dogs, cats, flowers, and food, and the color of the entire object or image. Here, for the sake of explanation, a person is used as the main subject.

Here, the album created in this embodiment will be explained using FIG. 3. In this embodiment, a plurality of different album variations are created based on the number of albums and the main subject of each album set by the album creation condition setting unit 201. In the example of FIG. 3, Album 1, Album 2, and Album 3 are created as albums with a plurality of different variations.

Each album consists of a "common spread" in which the same images are laid out in the same arrangement across multiple albums with different variations, and "individual spreads" in which the images or arrangement are different from those of other albums. Ru. In the example of FIG. 3, each of albums 1 to 3 includes a common spread 302. On the other hand, album 1 includes individual spreads 303, album 2 includes individual spreads 304, and album 3 includes individual spreads 305.

The image 301 is an image arranged in a double-page spread, and the characters in the image 301 represent the main subject in the image. As shown in FIG. 3, the common spread 302 is laid out to include images in which a plurality of main subjects are included. On the other hand, the individual spreads 303 to 305 are laid out so that they each include images in which a specific main subject is shown, such as the individual spread 303 including an image in which the main subject A is shown. In this way, when a plurality of albums are created in this embodiment, each of the plurality of albums includes spreads created based on different standards.

The image acquisition unit 203 acquires a group of images that satisfy the album creation conditions set by the album creation condition setting unit 201 from among the images stored in the HDD 104 . The image group here refers to a group of images that are layout candidates when creating an album. For example, if January 1, XX, to December 31, XX is specified as the shooting date and time, all images shot between January 1, XX, and December 31, XX will be included in the layout candidates. This corresponds to the image group. Examples of images stored in the HDD 104 include still images and cut-out images cut out from moving images. The still image and cutout image are obtained from an imaging device such as a digital camera or a smart device. The imaging device may be included in the image processing apparatus 100 or may be included in an external device of the image processing apparatus 100. Note that if the imaging device is an external device, the image acquisition unit 203 acquires the image via the data communication device 108. Furthermore, the still image or cutout image may be an image acquired from an external network or server via the data communication device 108. Examples of images acquired from networks and servers include social networking service images (hereinafter referred to as SNS images). Note that by executing the OS program, the CPU 101 analyzes data accompanying the image data for each image, and determines the storage source from which the image is acquired. However, by acquiring images from SNS via an application, the source from which the images are acquired may be managed within the application. Note that the images acquired by the image acquisition unit 203 are not limited to those described above, and may be other types of images.

The image analysis unit 204 analyzes the image data acquired by the image acquisition unit 203. In this embodiment, feature amounts of an image are derived, and object determination in the image, face detection, facial expression recognition of the detected face, and individual recognition of the detected face are performed. Furthermore, the image analysis unit 204 refers to data (for example, Exif information) accompanying the image data acquired from the HDD 104 and acquires information on the shooting date and time. Note that the information obtained as a result of the image analysis unit 204 analyzing the image data is referred to as "analysis information."

The main subject setting unit 205 sets a main subject to be emphasized for each album. In this embodiment, the main subject setting unit 205 acquires information about the main subject to be emphasized for each album, sent from the album creation condition setting unit 201.

The image scoring unit 208 scores each image based on the album creation conditions set by the album creation condition setting unit 201 and the analysis information derived by the image analysis unit 204. In this embodiment, each image is scored so that images suitable for the layout receive high scores. Image analysis information, image classification results, image information, and album creation conditions are used when scoring so that images suitable for the layout receive high scores. However, other information may be used in addition or instead. Images suitable for layout include images with high aesthetic appeal, such as high contrast and sharp edges, and images with a large main subject such as a person, animal, or building. can be mentioned. Furthermore, the image scoring unit 208 also scores each main subject set by the main subject setting unit 205. For example, even if the image is very aesthetically pleasing, if the set main subject is not included in the image, the score will be lowered. A higher score is given to an image that shows a larger image.

The scene dividing unit 206 uses the analysis information derived by the image analyzing unit 204 to divide the image group acquired by the image acquiring unit 203 into scenes. The term "scene" here means a group of images (sub-image group) grouped according to analysis information. For example, regarding two images taken at different times, the images for which the difference in the time taken is less than or equal to a predetermined threshold are determined to be the same scene, or images with the same person or background are determined to be the same scene.

The scene classification unit 207 determines the category corresponding to each scene obtained by the scene division unit 206. Scene categories include, for example, situations such as indoors and outdoors, and events at which photographs are taken, such as travel and weddings.

The double-page spread creation parameter generation unit 209 generates a plurality of parameters for creating a double-page spread based on the main subject set by the main subject setting unit 205 and the analysis information derived by the image analysis unit 204. The parameters here are set values that determine the standards for creating double-page spreads. Examples of such parameters include weights and thresholds that are used when selecting images to be used in a double-page spread and that determine the priority of scores obtained by the image scoring section 208. This determines whether priority is given to images containing only a specific main subject from among the images containing the main subject set in the main subject setting section 205, or whether images containing two or more main subjects are prioritized. This is a setting value that selects whether to give priority to By using multiple parameters, the creation criteria changes depending on the spread of each album.

The double-page spread creation unit 210 creates a double-page spread by selecting and arranging images to be used in the double-page spread according to the parameters generated by the double-page spread creation parameter generation unit 209.

The double-page spread combining section 211 combines the double-page spreads created by the double-page spread creation section 210. Specifically, the layout of the entire album is created by determining the order of each spread created by the spread creation unit 210.

The layout information output unit 212 outputs layout information according to the layout of the entire album created by the double-page spread combination unit 211. The layout information is, for example, image data in a bitmap format in which each image is arranged according to a predetermined layout.

The album display unit 213 displays the created album image on the display 105 based on the layout information output by the layout information output unit 212.

When the album creation application program in this embodiment is installed on the image processing device 100, the startup icon of this application is displayed on the top screen (desktop) of the OS (operating system) running on the image processing device 100. Ru. When the user double-clicks the startup icon on the desktop displayed on the display 105 with the mouse 107, the album creation application program stored in the HDD 104 is loaded into the RAM 103. Then, the program loaded into the RAM 103 is executed by the CPU 101, and the album creation application is activated. The above is the content of the software configuration of the image processing apparatus in this embodiment.

<About the GUI screen>
The GUI screen of the album creation application in this embodiment will be explained below using FIG. 4. FIG. 4 is a diagram showing a GUI screen 401 provided by the activated album creation application and displayed on the display 105. The user can set album creation conditions via the GUI screen 401.

The GUI screen 401 has a pass box 402 and a folder selection button 403 as a setting section for photo images to be included in the album. A path box 402 is a box for indicating a storage location (path) in the HDD 104 of a group of images to be created as an album. The folder selection button 403 is a button for selecting a folder containing a group of images for which an album is to be created. When the user clicks the folder selection button 403 with the mouse 107, a tree consisting of multiple folders is displayed. Then, when the user selects a folder containing a group of images for which an album is to be created, the folder path of the selected folder is displayed in the path box 402.

The number of double-page spreads box 404 is a box for specifying the total number of double-page spreads in one album. The user can use the keyboard 106 to directly input a number into the number of spreads box 404, or use the mouse 107 to enter a number from a list into the number of spreads box 404. The number of albums box 410 is a box for specifying the number of albums to be created.

The main subject designation section 405 is an element for specifying the main subject to be emphasized for each of the number of albums input in the number of albums box 410. For example, the user can specify the main subject by using the mouse 107 to drag and drop an image containing the main subject onto the main subject specification section 405.

The photo spread setting section 406 is a slider bar for setting the number of images to be placed in each spread of the album to be created. When the slider is moved to the "many" side, the number of images placed in each spread increases. On the other hand, when the slider is moved to the "fewer" side, the number of images arranged in each spread decreases.

The product specification section 407 is an element for setting the product of the album to be created. As specific items regarding the merchandise to be set, the size of the album and the paper type of the album can be adopted. The cover type and binding type may also be set.

The OK button 408 is a button that determines the selected condition as the album creation condition. When the user clicks the OK button 408 with the mouse 107, the album creation conditions are finalized, and the album creation conditions are transmitted to the automatic layout processing section 202 via the album creation condition setting section 201. Specifically, the information on the pass entered in the pass box 402 is transmitted to the image acquisition unit 203. Further, the value of the total number of double-page spreads input in the double-page spread number box 404 is transmitted to the double-page spread creation unit 210. The value of the number of albums input in the number of albums box 410 and the main subject information specified in the main subject specifying section 405 are transmitted to the two-page spread creation parameter generating section 209.

A reset button 409 is a button for resetting each setting on the display screen. The above is the content of the GUI screen of the album creation application in this embodiment.

<About automatic layout processing>
The automatic layout processing in this embodiment will be described below with reference to FIG. 5(a). FIG. 5A is a flowchart of a process for executing automatic layout of the album creation application according to this embodiment. Note that the flowcharts shown in FIGS. 5A to 5C are realized, for example, by the CPU 101 reading a program stored in the HDD 104 into the ROM 102 or RAM 103 and executing it.

In step S501, the album creation condition setting unit 201 sets album creation conditions. In the following, "step S~" will be simply abbreviated as "S~". In this embodiment, the album creation conditions include the image group to be used in the album, the number of spreads, the number of albums, the main subject for each album, the number of images to be placed in each spread, and the products for which the album is to be created (Fig. (see 4).

In S502, the image acquisition unit 203 reads from the HDD 104 images that satisfy the conditions regarding the image group used in the album set in S501. The image acquisition unit 203 then develops the read image in the RAM 103.

In S503, the image analysis unit 204 analyzes the image developed in the RAM 103 in S502. Here, the image analysis in this step will be explained using FIG. 5(b).

In S50301, the image analysis unit 204 acquires information on the shooting date and time corresponding to the image data acquired by the image acquisition unit 203. In this embodiment, information on the shooting date and time is acquired based on Exif information accompanying each image data.

In S50302, the image analysis unit 204 derives image quality feature amounts for the image data acquired by the image acquisition unit 203. An example of the feature amount of image quality is the amount of focus. An edge detection method can be used as a focus amount determination method, and a known Sobel filter can be used as an edge detection method. Edges of the image are detected using a Sobel filter, and the brightness gradient, that is, the slope of the edge, is calculated by dividing the brightness difference between the start and end points of the edge by the distance between the start and end points. By calculating the average slope of edges in an image, an image with a large average slope can be considered to be more in focus than an image with a small average slope. In this embodiment, a plurality of threshold values are set to measure the average slope of the calculated edge, and by determining which threshold the calculated edge slope is greater than or equal to, it is determined whether the focus amount is acceptable. Specifically, a first tilt threshold and a second tilt threshold (first tilt threshold > second tilt threshold) are set as two different tilt thresholds, and the focus is adjusted in three stages: ○△×. Determine the amount. If the average slope of edges in the image is equal to or greater than the first threshold value, the focus amount is determined to be suitable (represented by a circle). Also, if the average slope of edges in the image is less than the first threshold and greater than or equal to the second threshold, it is considered an acceptable focus amount (represented by △), and if the average slope of edges in the image is less than the second threshold If so, the amount of focus is considered unacceptable (represented by ×).

In S50303, the image analysis unit 204 performs object detection and classification of the detected object on the image data acquired by the image acquisition unit 203. In this embodiment, a face is detected as an object. Any known method can be used as the face detection method, and an example of such a known method is AdaBoost, which creates a strong classifier from a plurality of weak classifiers. In this embodiment, a person's face is detected using a strong classifier created by AdaBoost. In S50303, a face is detected and the upper left coordinate value and lower right coordinate value of the detected face area in the image are acquired. By determining these two types of coordinate values, the position and size of the face can be specified. Furthermore, by running AdaBoost, which detects animals such as dogs and cats as well as food, as well as human faces, it is possible to detect objects such as people, animals, and food, and at the same time detect objects in the image. or can be classified. Note that the objects to be detected are not limited to those described above, but may also be flowers, buildings, ornaments, etc. Furthermore, although a case has been described here in which objects are classified using AdaBoost, image recognition using a deep neural network may also be employed.

In S50304, the image analysis unit 204 performs personal recognition on the face detected by the image analysis unit 204. First, the degree of similarity between the extracted face image and the representative face image stored for each individual ID in the face dictionary database is derived. Then, the personal ID for which the derived degree of similarity is greater than or equal to a predetermined threshold and has the highest degree of similarity is determined as the ID corresponding to the extracted face image. In other words, the person whose degree of similarity is equal to or greater than a predetermined threshold and who corresponds to the individual ID with the highest degree of similarity is identified as the person of the extracted face image. Furthermore, if all of the similarities derived for each individual ID are less than a predetermined threshold, the person in the extracted face image is regarded as a new person, a new individual ID is assigned, and the person is registered in the face dictionary database. . The face dictionary database is stored in the HDD 104, for example.

Returning to the explanation of FIG. 5(a). In S504, the image analysis unit 204 determines whether the processing from S502 to S503 has been completed for all images in the image group that satisfy the conditions set in S501. If the determination result of this step is true, the process advances to S505, while if the determination result is false, the process returns to S502.

In S505, the scene division unit 206 performs scene division on the image group acquired by the image acquisition unit 203. Scene division refers to dividing an acquired image group into a plurality of sub-image groups. Specifically, the image group is divided into a plurality of images based on the time difference between the images, using the information on the photographing date and time already acquired in S50301. An example of actual division is as follows.

Among the images, the oldest (or newest) image is focused first, the time difference between it and the next oldest (or newest) image is calculated, and it is determined that the calculated time difference is greater than or equal to a predetermined threshold. Such processing is performed on all images while sequentially replacing the images of interest with images captured at newer (or older) times. Note that "dividing" in this embodiment means dividing the image group into two images, one with a newer shooting time and the other with an older shooting time. In this embodiment, if the difference between the shooting time of the image of interest and the shooting time of the next oldest (or newest) image is 16 hours or more, the image group is divided into different sub-images. Belong to a group. Note that the time difference threshold used for division is not limited to this. FIG. 6 shows the results of dividing a group of images using the scene division method described above.

Further, in this embodiment, scene division is performed using information on the shooting date and time, but the information used is not limited to this. For example, the image group may be divided using information on the photographing position so that images whose photographing positions are close to each other belong to the same scene. As another example, personal recognition results may be used. For example, in a graduation album, by registering images of students who belong to a certain group (class, club, etc.) in an application in advance, a group of images containing students belonging to the group can be combined into one image group. It may also be a scene (sub-image group). As yet another example, information other than image information may be used. Here, we will explain how to combine similar events into one scene. The name of the folder in which the image is saved and the tag information added by social networking services are acquired as meta information attached to the image. For example, based on a search word such as "athletic day" or "school trip," a group of images having meta information including the search word is treated as one scene. Regarding the designation of the search word, the user may select it from search words that are pre-installed in the application via the album creation condition setting section 201, or may input the search word into a text box.

In S506, the scene classification unit 207 performs scene classification on each of the sub-image groups obtained in S505. In the following, we will explain the case where each sub-image group (each scene) is classified into one of the three categories of "travel", "daily life", and "ceremony", but the items to be classified are these. It is not limited to.

First, a plurality of sub-image groups that have been determined in advance to be classified into travel, daily life, or ceremony are acquired, and the feature amount of the photographing is acquired for each of the acquired plurality of sub-image groups. The feature quantities of the photographing acquired here are, for example, the photographing period, the number of photographed images, and the number of photographing persons. The photographing period is the time difference between the photographing time of the oldest image and the photographing time of the newest image among the images included in the image group. The number of photographed images is the number of images (that is, the number of photographs) included in the sub-image group. The number of people photographed is the number of faces in an image that includes faces, and is the number of faces included in one image. Then, for the plurality of sub-image groups, the average value and standard deviation of the photographing period, the average value and standard deviation of the number of photographed images, and the average value and standard deviation of the number of people per image are determined. Note that in this embodiment, the number of faces per image is the number of people per image.

FIG. 7 shows an example of the average and standard deviation of the photographing period (time), the average and standard deviation of the number of photographed images, and the average and standard deviation of the number of people per image, calculated for a plurality of sub-image groups. These determined values are incorporated into the album creation application program in advance. That is, at the design stage of the application, parameters are generated by learning using images collected in advance, and the generated parameters are incorporated into the program. After the album creation application is started, for each sub-image group obtained as a result of dividing the image group specified by the user via the pass box 402 in S505, the shooting period, the number of images to be taken, and the number of people per image are displayed. Calculate the average value of each. Then, for each scene, parameters pre-installed in the above-mentioned application (specifically, the average value and standard deviation of the feature values of the shooting period, number of shots, and number of shots for each sub-image group) are used. , calculate the score according to the following formula.

Score for each scene and each feature amount = 50 - | 10 × (average value for each scene and each feature amount - feature amount for each sub-image group) / standard deviation for each scene and each feature amount | ... Formula (1 )

Average score for each scene = (Score for the shooting period for each scene + Score for the number of shots for each scene + Score for the number of people shooting for each scene) / Number of feature items...Equation (2)

Using equation (1), scores for each scene and each feature of the group of sub-images of interest are determined. For example, for a travel scene, scores are determined for the shooting period, the number of shots, and the number of shots. These scores are averaged using equation (2) to obtain a score for the travel scene. Similarly, scores can be obtained for other scenes such as daily life and ceremonies. By the method described above, the average score of the travel scene, the average score of the daily scene, and the average score of the ceremony scene are calculated for each sub-image group. Note that the number of feature quantity items in equation (2) is three.

Among the scores for each scene calculated for each sub-image group, the scene with the highest score is classified as the scene of that sub-image group. If two or more scenes have the same score, the scenes are classified according to a predetermined priority. For example, in this embodiment, it is predetermined that priority is given to daily life>ceremony>travel, and daily life is given the highest priority. Note that the order of priorities is not limited to this, and the user may be able to change the order of priorities.

Sub-image groups (1) to (8) in FIG. 6 show sub-image groups obtained by scene segmentation of the image group. Assume that for sub-image group (5) among these sub-image groups, the shooting period was 36 hours, the number of shots was 300, and the number of shots was 1.7 people. On the other hand, the average score for travel is 45.32, the average score for daily life is 18.38, and the average score for ceremony is -29.92.As a result, the scene of sub-image group (5) is classified as travel. The classified sub-image group is managed by associating it with a scene ID so that the scene can be identified. Note that, in this embodiment, a case has been described in which the photographing period, the number of photographed images, and the number of photographed persons are employed as the photographing feature quantities, but the photographing characteristic quantities are not limited to these. For example, in order to identify whether the image is indoors or outdoors, a flag indicating whether the sky is included in the image segmentation using machine learning may be used as the feature amount. Alternatively, ceremonies may be further categorized. For example, when classifying images into either a wedding or a reception, the two people who appear most frequently in all images are determined to be the bride and groom using the personal recognition results. Then, a scene in which there are many photographs in which only the bride and groom are photographed may be classified as a wedding ceremony, and a scene in which there are many photographs in which people other than the bride and groom are photographed together may be classified as a reception party. Furthermore, the feature values may be derived by machine learning. For example, scene classification may be performed by preparing a scene to be classified and a group of images representing the scene, and performing learning using a Convolution Neural Network (CNN) using an image as an input and a scene classification result as an output.

In S507, the main subject setting unit 205 sets the main subject for each album. In this embodiment, main subject information for each album transmitted from the album creation condition setting unit 201 is acquired. For the sake of explanation, it is assumed here that main subject A is set in album 1 and main subject B is set in album 2 for two types of albums.

In S508, the image scoring unit 208 scores each image. This process is called image scoring process. The "score" derived in this step is a parameter for evaluating suitability for use in an album, and each image data is given a score evaluated from the viewpoints described later, and is referred to when selecting an image described later. In this embodiment, in addition to the quality of the image, a score is given for each main subject set in S507. First, a score is given according to the image quality feature derived in S50302. Here, if the focus amount is ○ (suitable), it is given 1 point, if △ (acceptable), it is given 0.5 point, and if × (unacceptable), it is given 0 point. This makes it possible to increase the score of an image that is in focus. Note that in this embodiment, the amount of focus is used as the feature amount of image quality, but the feature amount of image quality is not limited to this. For example, the image size may be used, shooting information such as lens information used at the time of shooting may be used, or the compression format of the image input to the application may be used.

After assigning a score according to the feature amount of image quality, the image scoring unit 208 assigns a score for each set main subject. In other words, in this embodiment, one image is evaluated using three types of scores, specifically, whether the image quality is good, whether main subject A is included, and whether main subject B is included. Scores are given. The face size derived in S50303 and the personal recognition result in S50304 can be used to score each main subject. If the main subject is not included in the image, a score of 0 is given; if the main subject is included, the score for each main subject is determined by the ratio of the face of the individual set as the main subject to the image size. Note that in this embodiment, the face size is used to score each main subject, but things other than the face size may be used. For example, the facial expression of the person who is the main subject may be determined, and points may be added if the person is smiling. Furthermore, when an object other than a person is set as the main subject, scoring is performed in the same manner according to the object size by using the results of object detection and classification in S50303.

In S509, the image scoring unit 208 determines whether scoring in S508 has been completed for all images in the image group acquired by the image acquisition unit 203. If the determination result of this step is true, the process advances to S510, while if the determination result is false, the process returns to S508.

In S510, the two-page spread creation parameter generation unit 209 receives the album creation conditions from the album creation condition setting unit 201, receives main subject information for each album from the main subject setting unit 205, and generates two-page spread creation parameters. The two-page spread creation parameters are setting values used when selecting and laying out images constituting a two-page spread, which will be described later. For example, when selecting an image to be used for a double-page spread based on the score assigned to the image in S508, weights that control which score is prioritized between the score according to image quality and the score of each main subject, and image selection Generate threshold values that serve as standards. The double-page spread creation parameter generation unit 209 generates "common parameters" that are common to all albums and "individual parameters" that differ for each album as double-page spread creation parameters. That is, the double-page spread creation parameter generation unit 209 generates (number of albums+1) double-page spread generation parameters. This creates "common spreads" in which spreads with the same images and arrangement appear in multiple albums, and "individual spreads" in which the images and arrangement that make up the spreads appear only in a specific album. can. In this embodiment, a weight is set for the common parameter so that each score given to an image is equally evaluated. A score ^I score _S representing a certain image I when using the common parameter w _i is expressed by the following equation (3).

Here, ^I score _i represents the i-th score assigned to image I, and w _i represents the weight for each score. In this embodiment, by setting w _i =1/3, the image quality score, the score for the main subject A, and the score for the main subject B are evaluated equally. As a result, priority is given to images that have good image quality and include many people with higher scores.

On the other hand, the individual parameters are set with weights that give priority to images in which the main subject of each album is photographed. A score ^I score _I representing a certain image I when using the individual parameter w _o is expressed by the following equation (4).

Here, ^I score _q is the image quality score assigned to image I, ^I score _o is the score of main subject O assigned to image I, w _q is the weight for the image quality score, and w _o is the weight for main subject O represents. In this embodiment, by setting w _q =w _o =1, only images in which the main subject is photographed are evaluated, and the image to be given priority is determined based on the image quality and the size of the main subject. In this embodiment, weights for scores are used as double-page spread creation parameters, but the double-page spread parameters are not limited to this. For example, the spread parameter may include the number of spreads to be created. Further, although both ^I score _S and ^I score _I were determined so that the score would be high when the main subject was photographed, the present invention is not limited to this. For images such as landscapes that do not include the main subject at all, scores may be calculated separately based on image quality scores, image information, and the like. For example, for an image with a small difference in shooting time from an image with a high score, the image quality score may be used as the representative score. This makes it possible to give high scores to images that are likely to be related to the photo in which the main subject appears.

In S511, the double-page spread creation unit 210 creates a double-page spread of the album using the parameters generated in S510. Here, the two-page spread creation process in this step will be explained using FIG. 2(b) and FIG. 5(c). FIG. 2(b) is a detailed block diagram of the two-page spread creation section 210. FIG. 5C is a detailed flowchart of creating a two-page spread in S511.

In S51101, the two-page spread creation parameter setting unit 21001 reads one of the parameters generated in S510. In this embodiment, the individual parameters of album 1, the individual parameters of album 2, and the common parameters are read in this order.

In S51102, the double-page spread candidate image acquisition unit 21002 selects candidates to be used for double-page spreads from among all the image groups acquired by the image acquisition unit 203, based on the score assigned in S508 and the double-page spread creation parameters set in S51101. Get the image. In this embodiment, each image is scored again according to equation (3) or equation (4) based on the score obtained in S508 and the two-page spread creation parameters set in S51101 (image rescoring). . Then, images whose scores obtained as a result of rescoring are equal to or higher than a predetermined threshold are acquired as candidate images. If there are no candidate images, the image with the highest score is selected as the candidate image. Note that although candidate images are acquired from all the image groups here, the present invention is not limited to this. Images that have already been selected in the image selection process described later may be excluded.

In S51103, the double-page spread setting unit 21003 determines whether the double-page spread creation parameter set in the most recent S51101 is an individual parameter. If the determination result of this step is true, the process advances to S51104, while if the determination result is false, the process advances to S51105.

If YES in S51103 (that is, the individual parameters were set in the most recent S51101), in S51104, the number of spreads setting unit 21003 performs the following based on the scene division result performed in S505 and the candidate image acquired in S51102. Set the number of spreads. In this embodiment, the number of spreads is set according to the following equations (5) and (6).

Number of spreads for each scene = ceil (number of candidate images in a scene ÷ maximum number of images per spread)...Equation (5)

Number of individual spreads = Σ Number of spreads for each scene...Equation (6)

The number of double-page spreads created using the individual parameters is determined by equations (5) and (6). The maximum number of images per spread is set based on user input via the spread photo number setting unit 406. Note that if the number of individual double-page spreads is equal to or greater than the total number of double-page spreads set in S501, the following formula (7) is followed.

Number of individual spreads = total number of spreads - 1...Formula (7)

Using equations (6) and (7), the number of individual spreads can be made smaller by one or more than the total number of spreads. Thereby, double-page spreads created using different parameters, ie, individual parameters and common parameters, can be mixed in one album. Thereby, the number of individual spreads can be set for each scene according to the number of images in which the main subject is well captured. Note that an upper limit value and a lower limit value may be set for the number of individual spreads.

On the other hand, if NO in S51103 (that is, if the common parameter was set in the most recent S51101), in S51105, the number of spreads setting unit 21003 sets the number of spreads according to the following equation (8).

Number of common spreads = total number of spreads - MIN (total number of individual spreads for each album)...Formula (8)

Using equation (8), it is possible to set the number of common double-page spreads that matches both the number of individual double-page spreads and the total number of double-page spreads.

Although a case has been described here in which the number of double-page spreads is set according to the number of candidate images, the method for setting the number of double-page spreads is not limited to this. For example, the double-page spread creation parameter generation unit 209 may set the number of double-page spreads in the parameter, and the double-page spread number setting unit 21003 may read the number of double-page spreads set in the parameter. For example, the double-page spread creation parameter generation unit 209 may set the number of double-page spreads that is pre-installed in the program of the album creation application, or may set the number of double-page spreads based on the ratio of individual double-page spreads and common double-page spreads specified by the user. It may be calculated.

In S51106, the two-page spread allocation unit 21004 divides the candidate image acquired in S51102 into sub-image groups equal to the number of two-page spreads set in S51104 or S51105, and allocates the divided images. In this embodiment, the candidate images are arranged in the order of the photographing times acquired in S50301, and divided at locations where the time difference in the photographing times between adjacent images is large. Such processing is performed until the page is divided into the number of spreads set in S51104 or S51105. In other words, the image is divided (number of spreads - 1) times. As a result, it is possible to create an album in which images are arranged in the order of shooting time, with the image placement reference being the shooting time.

In S51107, the two-page spread image selection unit 21005 selects an image to be used for the layout from among the candidate images assigned to the two-page spread to be processed in S51106. Hereinafter, a case will be described using FIG. 8 as an example in which four images are selected from a group of candidate images assigned to a certain double-page spread. Note that here, a double-page spread refers to two pages, and the first double-page spread and the last double-page spread are one page.

In FIG. 8, (A) shows the time difference between the photographing date and time of the first image to the last image included in the candidate image group assigned to the two-page spread, in other words, the photographing period of the candidate image group. A method for selecting the first image when selecting four images will be explained using FIG. 8(B). Among all the images photographed during the photographing period of the candidate image group shown in FIG. 8B, the image with the highest score re-scored in S51102 is selected as the first image. In the second and subsequent selections, images are selected by subdividing the imaging period of the candidate image group, so that the selected images are not concentrated in a part of the imaging period of the candidate image group. First, as shown in FIG. 8C, the photographing period of the candidate image group is divided into two to form two photographing sections (periods). In other words, the imaging period is divided into two into two imaging sections (grouped). Next, as shown in FIG. 8(D), the image with the highest score from among the images belonging to the shooting section (solid line section) in which the first image has not been selected is selected as the second image. select. Next, as shown in (E) of FIG. 8, each imaging section in (D) is divided into two equal parts. In (F) of Fig. 8, the scores are among the images taken in the shooting sections indicated by solid lines, that is, the two shooting sections where the first and second images are not selected (images corresponding to the two shooting sections). Select the highest image as the third image. Next, a case in which an image cannot be selected because no image exists in the photographing interval of the image selection range will be described using selection of the fourth image as an example. In this embodiment, the shooting period is divided according to time, regardless of the number of images. Therefore, there may be cases where no image exists in the photographing section obtained by dividing. For example, as shown in FIG. 8G, it is assumed that the user wants to select the fourth image from a shooting section in which no images have been selected yet (the shooting section indicated by diagonal lines), but no image exists in this shooting section. In such a case, as shown in FIG. 8(H), each photographing section in which images have been selected is divided into two. Next, as shown in FIG. 8 (I), the image with the highest score is selected from among the images taken in the shooting interval where the first to third images shown by the solid line in the figure are not selected. Select as the image. Note that even when the number of images to be selected increases or decreases from four, the images are selected in the same manner as described above.

Returning to the explanation of FIG. 5(c). In S51108, the template setting unit 21006 selectively sets a plurality of templates to be used for layout from among the templates stored in the HDD 104. In this step, a template with a design that satisfies the album creation conditions set in S501 is set.

In S51109, the two-page spread layout unit 21007 determines the image layout for the two-page spread to be processed. Specifically, the spread layout unit 21007 determines a template suitable for laying out the image selected in S51107 from among the plurality of templates set in S51108. Here, the template determination method in this step will be explained using FIG. 9. FIG. 9 is a functional block diagram regarding template determination.

The selected image information acquisition unit 901 acquires the number of images selected by the spread image selection unit 21005 and information regarding the images. The image information to be acquired is the width and height of the image, the shooting date and time information, and the score calculated by the image scoring unit 208. The template selection unit 902 selects a template having the number of slots that matches the number of selected images from among the templates set by the template setting unit 21006. The selected image sorting unit 903 sorts the selected images in order of shooting time. The template selection unit 904 selects, from among the templates selected by the template selection unit 902, a template in which the main slot in which the main image is placed matches the order of the main images arranged in order of shooting time. Here, the "main image" is the image with the highest score obtained as a result of re-scoring in S51102 among the multiple images selected as the double-page image; An image that is not a main image is defined as a "sub image." When selecting this template, in addition to the order, the selection condition is that the aspect ratio of the main image and the aspect ratio of the slot in which the main image is placed must match. In this embodiment, an image with an older photographing time is laid out in the upper left of the template, and an image with a newer photographing time is laid out in the lower right. The template selection unit 905 selects templates among the templates selected by the template selection unit 904 in which the slots in which the sub-images are arranged match the order of the sub-images arranged in order of shooting time, and the aspect ratios of the images and slots match. Select. FIG. 10 is a flowchart of template determination processing.

In S1001, the selected image information acquisition unit 901 acquires selected image information. The selected image information acquired in this step includes the number of selected images. Here, for the sake of explanation, it is assumed that the number of selected images is three.

In S1002, the template selection unit 902 selects a template whose number of selected images matches the number of slots. Here, it is assumed that the number of selected images acquired in S1001 was 3, and that a template with the number of slots of 3 was selected. Hereinafter, a case in which templates (1-1) to (4-4) shown in FIG. 11 are selected in this step will be described as an example.

In S1003, the selected image sorting unit 903 arranges the selected images in order of shooting time. Here, it is assumed that the relationship shown in FIG. 11A is obtained as a result of arranging the selected images in the order of shooting time. Further, image 1105 is the main image, and image 1106 and image 1107 are sub-images.

In step S1004, the template selection unit 904 determines that the arrangement position of the main image matches the arrangement position of the main image slot (referred to as the main slot) when arranged in the order of shooting, and the aspect ratio of the main image and the main slot is the same. Select the template that matches. Here, since the image 1105 has the latest shooting time and is an image for the main slot, templates (3-1) to (3-4) in FIG. 11 are candidates.

In S1005, the template selection unit 905 selects a template in which the aspect ratio of the sub-image and the slot for the sub-image (referred to as a sub-slot) matches. In the example of FIG. 11, the older image 1106 for the subslot, that is, the image desired to be placed in the upper left, is a vertical image, and the newer image 1107 for the subslot is a horizontal image. Therefore, template (3-2) is determined as the most suitable template for the three selected images. In this manner, in S51109, a template to be used for layout and information that can identify which image is to be laid out in which slot of the template are determined.

Returning to the explanation of FIG. 5(c). In S51110, the two-page spread layout unit 21007 determines whether the processing in S51107 to S51109 has been completed for all the two-page spreads to be processed. If the determination result of this step is true, the process advances to S51111, whereas if the determination result is false, the process returns to S51107.

In S51111, the two-page spread creation parameter setting unit 21001 determines whether the processing in S51101 to S51110 has been completed for all parameters generated in S510. If the determination result of this step is true, the process advances to S512, while if the determination result is false, the process returns to S51101.

Returning to the explanation of FIG. 5(a). In S512, the double-page spread combination unit 211 determines the double-page spread number within the album for each double-page spread created by the double-page spread creation unit 210, and creates the layout of the entire album. In this embodiment, the two-page spread arrangement standard is the shooting time of the images arranged in the two-page spread, and the two-page spreads are arranged so that the images are arranged in the order of shooting time. Note that when setting the layout on a page-by-page basis, the page layout standard may be used instead of the two-page spread layout standard.

The two-page spread combination in S512 will be described in detail below with reference to FIGS. 12 and 13. FIG. 12 is a block diagram showing the detailed configuration of the double-page spread combining section 211 in this embodiment. FIG. 13 is a flowchart of the processing executed by the two-page spread combining unit 211.

In S1301, the two-page spread acquisition unit 1201 obtains a two-page spread related to one album from among the two-page spreads created by the two-page spread creation unit 210. In this embodiment, for example, the double-page spreads related to album 1 acquired in this step include double-page spreads created using common parameters and double-page spreads created using individual parameters of album 1.

In S1302, the sorting reference value calculation unit 1202 calculates a value serving as a sorting reference for each double-page spread acquired by the double-page spread acquisition unit 1201. In this embodiment, the standard value for sorting is the average photographing time of images arranged in a facing spread. Note that the reference value for sorting is not limited to the shooting time. For example, the average photographing position (latitude and longitude) of images arranged in a double-page spread may be used, the number of images may be used, or image analysis information such as a color histogram may be used.

In S1303, the sorting unit 1203 sorts the double-page spreads acquired by the double-page spread acquisition unit 1201 based on the sorting reference value calculated by the sorting reference value calculation unit 1202. In this embodiment, the spreads are arranged in ascending order from the earliest average imaging time.

In S1304, the double-page spread adjustment unit 1204 determines whether the number of double-page spreads in the album is the same as the total number of double-page spreads set in S501. If the determination result in this step is true, the process advances to S1306. On the other hand, if the determination result in this step is false, that is, if the number of double-page spreads in each album is set to be greater than or equal to the total number of double-page spreads in S51103, and the number of double-page spreads in the album exceeds the total number of double-page spreads, the process advances to S1305.

In S1305, the double-page spread adjustment unit 1204 adjusts the number of double-page spreads of the album to approach the total number of double-page spreads set in S501 by deleting unnecessary double-page spreads from among the double-page spreads acquired by the double-page spread acquisition unit 1201. In this embodiment, the number of double-page spreads is adjusted by deleting common double-page spreads. Specifically, among the common spreads in the album, the common spread whose shooting time calculated by the sort reference value calculation unit 1202 is closest to the shooting time of the individual spread is deleted. After deletion, the process moves to S1304. Note that although the common double-page spread is deleted in this embodiment, the present invention is not limited to this. The total number of double-page spreads may be adjusted by deleting individual double-page spreads. Note that in S1304, the number of double-page spreads acquired in S1301 may be less than the total number of double-page spreads. In this case, in S1305, processing for adding a common spread or an individual spread may be executed. For example, when adding an individual double-page spread, the double-page spread combination unit 211 or the double-page spread creation unit 210 reacquires the candidate images acquired in S51102 when the individual parameters for the album were set in S51101. Then, from the images that were not selected in S51007, an image (for example, an image with a higher score) to be placed in the individual spread to be added is selected. Then, the average shooting time of images included in the created additional individual spreads is calculated and added to the plurality of spreads sorted in S1303. For example, an additional individual double-page spread is added at a position such that the order of the plurality of double-page spreads after addition follows the order of average shooting times. Similarly, when adding a common double-page spread, the double-page spread combining unit 211 or the double-page spread creation unit 210 creates and adds an additional common double-page spread from the candidate images acquired in S51102 when the common parameters are set in S51101.

In S1306, the two-page spread acquisition unit 1201 determines whether the processes of S1301 to S1305 have been completed for all albums. If the determination result of this step is true, the process advances to S513, while if the determination result is false, the process returns to 1301. This allows images to be arranged in order of shooting time throughout the album. The above is the content of the automatic layout processing in this embodiment.

<About the effects of this embodiment>
FIG. 25 is a diagram for explaining the effects of this embodiment, and shows examples of albums with a plurality of variations created in this embodiment. To explain in detail, in FIG. 25, albums 1 to 3 are created as albums for subjects A to C, and the main subject in album 1 is subject A, the main subject in album 2 is subject B, and the main subject in album 3 is Cases set for subject C are shown.

Here, it is assumed that the total number of spreads is set to 4. Furthermore, when S51104 of FIG. 5(c) is executed for each of albums 1 and 3, the number of individual spreads is set to 1 according to formula (5), and when S51104 is executed for album 2, formula ( Assume that the number of individual spreads is set to 2 according to 5). In this case, when the common parameter is set in S51101 and S51105 is executed, "MIN (total number of individual spreads of each album)" in equation (8) becomes "1". Since the total number of double-page spreads is "4", the common number of double-page spreads is 3 according to equation (8).

Therefore, in S511 of FIG. 5A, three common spreads are created for albums 1 to 3, one individual spread for albums 1 and 3, and two individual spreads for album 2. Then, in steps S1301 to S1303 in FIG. 13, an album is created for each of albums 1 to 3, as shown in FIG. 25, in which a common spread and an individual spread are combined. For albums 1 and 3, the number of double-page spreads is 4, which is the same as the total number of double-page spreads, so S1304 in FIG. 13 is not executed. However, for album 2, the number of spreads is 5, so S1304 in FIG. 13 is executed.

FIG. 25 shows the result of executing the above process, and the common spread 2901 is the result of adjustment because the number of acquired spreads exceeds the total number of spreads (NO in S1304 → S1305 in FIG. 13), and the final It expresses the common spread that was not used in Album 2.

As shown in FIG. 25, for the variation albums created in this embodiment, the total number of double-page spreads does not change depending on the album and is always the same. In the example of FIG. 25, the total number of spreads is 4 regardless of the album.

Further, as described above, it is allowed that the number of individual spreads varies depending on the album. Furthermore, the position of the individual double-page spreads for each album is also allowed to change. That is, in S1303 of FIG. 13, sorting is performed in a state where common spreads and individual spreads are mixed. Therefore, depending on the average photographing time of the individual double-page spreads in each album, the positions where the individual double-page spreads are arranged in each album may differ.

In the example of FIG. 25, the number of individual spreads of album 1 and album 3 is one, while the number of individual spreads of album 2 is two. In each album, the individual spreads are arranged in chronological order.

Furthermore, since common double-page spreads are deleted depending on the album in S1305, the number of common double-page spreads is allowed to vary depending on the album. In the example of FIG. 25, the common number of double-page spreads between album 1 and album 3 is three, while the common number of double-page spreads between album 2 is two.

According to this embodiment, it is possible to automatically set the number of common double-page spreads and the number and position of individual double-page spreads for each album.

[Second embodiment]
In this embodiment, a double-page spread common to at least two albums and a separate double-page spread for each album are created. The total number of double-page spreads and the number of individual double-page spreads for each album, as well as the positions of the individual double-page spreads, change depending on the image group and settings. The two-page spread and the images arranged within the two-page spread are arranged in the order of shooting time.

<About automatic layout processing>
The automatic layout processing in this embodiment will be described below with reference to FIGS. 14(a), (b) and FIGS. 15(a) to (c). FIGS. 14A and 14B are block diagrams showing functional configurations related to automatic layout processing in this embodiment. FIGS. 15(a) to 15(c) are flowcharts for explaining automatic layout processing in this embodiment. It should be noted that processes denoted by the same reference numerals as those in the above-described embodiment (see FIG. 5) are the same as those in the above-described embodiment, and therefore will not be described here.

In S1501, the double-page spread creation unit 1401 creates a double-page spread of an album using the parameters generated in S510. Here, the creation of double-page spreads in this step will be explained using FIG. 15(c).

In S150101, the number of spreads setting unit 140101 sets the number of spreads based on the scene division result performed in S505 and the candidate image acquired in S51102. In this embodiment, the number of spreads is set according to the following equations (9) and (10).

Number of spreads for each scene = ceil (number of candidate images in a scene ÷ maximum number of images per spread)...Equation (9)

Number of spreads = Σ Number of spreads for each scene...Equation (10)

Using equations (9) and (10), the number of double-page spreads can be set for each scene according to the number of images in which the main subject is well captured.

The difference between the processing in FIG. 15(c) and FIG. 5(c) is that in FIG. 5(c), the number of common double-page spreads is calculated by equation (8) in S51105 in consideration of the number of individual double-page spreads. However, in FIG. 15(c), the number of common spreads is also determined by equation (10).

In S1502, the double-page spread combining unit 1402 determines the double-page spread number within the album for each double-page spread created by the double-page spread creation unit 1401, and creates the layout of the entire album. In this embodiment, the spreads are arranged so that the images are arranged in the order of shooting time based on the shooting times of the images arranged in the spread. The two-page spread combination in S1502 will be described in detail below with reference to FIGS. 16 and 17. FIG. 16 is a block diagram showing the detailed configuration of the double-page spread combining section 1402 in this embodiment. FIG. 17 is a flowchart of processing executed by the double-page spread combining unit 1402.

In S1301, the double-page spread acquisition unit 1201 acquires a double-page spread related to one album from among the double-page spreads created by the double-page spread creation unit 1401. In this embodiment, for example, the double-page spreads related to album 1 acquired in this step include double-page spreads created using common parameters and double-page spreads created using individual parameters of album 1.

In S1302, the sort reference value calculation unit 1202 calculates, for each double-page spread acquired by the double-page spread acquisition unit 1201, a shooting time representative of each double-page spread. In this embodiment, the photographing time representing a double-page spread is the average of the photographing times of images arranged in the double-page spread.

In S1303, the sorting unit 1203 sorts the double-page spreads acquired by the double-page spread acquisition unit 1201 based on the shooting time of the double-page spreads calculated by the sort reference value calculation unit 1202. In this embodiment, the double-page spreads are arranged in ascending order from earliest photographing time.

In S1701, the two-page spread acquisition unit 1201 determines whether the processes of S1301 to S1303 have been completed for all albums. If the determination result of this step is true, the process advances to S513, while if the determination result is false, the process returns to S1301. This allows images to be arranged in order of shooting time throughout the album. The above is the content of the automatic layout processing in this embodiment.

The difference between the processes in FIG. 13 and FIG. 17 is that in FIG. 17, the number of spreads in S1305 in FIG. 13 is not executed. That is, in this embodiment, the individual double-page spreads and the common double-page spreads are directly adopted as the final double-page spread. Therefore, it is allowed that the total number of double-page spreads differs from album to album due to the difference in the number of individual double-page spreads between albums.

<About the effects of this embodiment>
FIG. 26 is a diagram for explaining the effects of this embodiment, and shows examples of albums with a plurality of variations created in this embodiment. To explain in detail, in FIG. 26(a), albums 1 and 2 are created as albums for subjects A and B, and the main subject in album 1 is set to subject A, and the main subject in album 2 is set to subject B. This shows a case in which Further, in FIG. 26(b), albums 1 to 3 are created as albums for subjects A to C, and the main subject in album 1 is subject A, the main subject in album 2 is subject B, and the main subject in album 3 is subject A. shows cases set for subject C, respectively.

In this embodiment, the total number of double-page spreads is allowed to vary depending on the album of variations to be created. However, in the example of FIG. 26(a), the total number of double-page spreads is 4 regardless of the album, and in the example of FIG. 26(b), the total number of double-page spreads is 3 regardless of the album.

Further, the number of individual double-page spreads is allowed to change depending on the album, and the position of the individual double-page spreads for each album is also allowed to change. However, in the example of FIG. 26(a), the number of individual spreads between album 1 and album 2 is 1, and in the example of FIG. 26(b), the number of individual spreads between albums 1 to 3 is 1. In each album, the individual spreads are arranged in chronological order.

Further, the number of common spreads does not change depending on the album and is always the same. As shown in FIG. 26(a), the common number of double-page spreads between album 1 and album 2 is three. Further, as shown in FIG. 26(b), the common number of double-page spreads for albums 1 to 3 is two.

According to this embodiment, it is possible to automatically set the total number of double-page spreads in an album, as well as the number and position of individual double-page spreads for each album.

[Third embodiment]
In this embodiment, a double-page spread common to at least two albums and a separate double-page spread for each album are created. Align the number of common spreads and individual spreads for all albums. However, the position of the individual spreads is allowed to change depending on the image group and settings for each album. The double-page spread and the images arranged within the double-page spread are arranged in the order of shooting time.

<About automatic layout processing>
The automatic layout processing in this embodiment will be described below with reference to FIGS. 18(a), (b) and FIGS. 19(a) to (c). FIGS. 18A and 18B are block diagrams showing functional configurations related to automatic layout processing in this embodiment. FIGS. 19(a) to 19(c) are flowcharts for explaining automatic layout processing in this embodiment. It should be noted that processes denoted by the same reference numerals as those in the above-described embodiments are the same processes as in the above-described embodiments, and therefore description thereof will be omitted here.

In S1901, the double-page spread creation unit 1801 creates a double-page spread of an album using the parameters generated in S510. Here, the two-page spread creation process in this step will be explained using FIG. 19(c).

In S190101, the double-page spread number calculation unit 180101 calculates the number of double-page spreads to be created with each double-page spread creation parameter based on the parameters generated in S510, and updates each double-page spread creation parameter. Specifically, the calculated number of double-page spreads is set in each double-page spread creation parameter. The processing executed by the double-page spread calculation unit 180101 in this embodiment will be described below with reference to FIG. 20.

In S2001, the two-page spread creation parameter setting unit 21001 reads one of the parameters for aligning the number of two-page spreads among the parameters generated in S510. In this embodiment, the individual parameters of album 1 and the individual parameters of album 2 are read in order.

In S2002, the double-page spread candidate image acquisition unit 21002 acquires candidate images using the same method as in S51102, based on the score assigned in S508 and the double-page spread creation parameters set in S2001. This specifies the number of candidate images.

In S2003, the double-page spread number calculation unit 180101 calculates individual double-page spread number candidates based on the candidate images acquired in S2002. In this embodiment, the individual spread number candidates are calculated according to the following equation (11).

Candidate number of individual spreads = ceil (number of candidate images ÷ maximum number of images per spread)...Formula (11)

The number of individual spread candidates calculated using equation (11) increases or decreases depending on the number of candidate images.

In S2004, the double-page spread calculation unit 180101 determines whether the processes of S2001 to S2003 have been completed for all parameters for which the number of double-page spreads is to be equalized. If the determination result of this step is true, the process advances to S2005, while if the determination result is false, the process returns to S2001.

In S2005, the double-page spread number calculation unit 180101 determines the number of double-page spreads based on the individual double-page spread number candidates calculated in S2003. In this embodiment, the smallest number of spreads is selected from among the individual spread number candidates. With this, it is possible to create the same number of individual double-page spreads according to the parameters with fewer candidate images. In addition, the number of common spreads created using the common parameters is dependent on the number of individual spreads created using the individual parameters and the total number of spreads set in S501, as shown in equation (12) below. It's decided.

Number of common spreads = total number of spreads - number of individual spreads...Formula (12)

In S2006, the double-page spread number calculation unit 180101 adds the number of double-page spreads determined in S2005 to the parameters generated in S510, and updates the double-page spread creation parameters.

That is, according to the process in S190101 (FIG. 20), the number of common double-page spreads and the number of individual double-page spreads are set to be the same for all albums. Then, a common double-page spread and an individual double-page spread are created by the processing from S51101 onward in FIG. 19(c).

Furthermore, since the number of individual spreads and the number of common spreads are adjusted in S190101 before a spread is created in this manner, it is not necessary to adjust the number of spreads after the spread is created, as in S1304 and S1305 in S512.

Returning to the explanation of FIG. 19(c). In S190102, the double-page spread number setting unit 180102 reads the number of double-page spreads set in S190101 from the double-page spread creation parameters read in S51101.

In S190103, the two-page spread allocation unit 180103 allocates the candidate images acquired in S51102 to sub-image groups equal to the number of two-page spreads set in S190102. In this embodiment, further, from among the assigned sub-image groups, sub-image groups corresponding to the number of double-page spreads included in the double-page spread creation parameters set in the most recent S51101 are selected in descending order of the average image score. do.

Note that in this embodiment, the number of double-page spreads is set according to the number of candidate images, but the method for setting the number of double-page spreads is not limited to this. For example, the number of individual spreads and the number of common spreads may be set according to a ratio that is preset in the program of the album creation application. Specifically, when the number of individual spreads: the number of common spreads is set to 1:4, and the total number of spreads set in S501 is 10, the number of individual spreads is set to 2, and the number of common spreads is set to 8. As another example, the number of spreads may be determined based on the scene classification result in S506. For example, consider a case where the scene classification result is a wedding. Set the number of individual spreads according to the scene, such as 2 spreads with individual parameters with the bride and groom as the main subjects, 1 spread with individual parameters with relatives or friends as the main subjects, and 7 spreads with common parameters, for a total of 10 spreads. Multiple settings may be made. In the above example, in addition to the number of spreads, the spread number calculation unit 180101 may increase or decrease the threshold value for acquiring candidate images to match the number of spreads. For example, consider a case where only one candidate image can be obtained using the double-page spread creation parameters acquired in S2001, but it is desired to create two or more double-page spreads. The score threshold serving as the selection criterion is lowered until the two-page spread candidate image acquisition unit 21002 can acquire two or more candidate images. The changed threshold value is also added to the parameters generated in S510 in the same manner as the number of double-page spreads, and the double-page spread creation parameters are updated. Thereby, candidate images can be obtained according to the number of spreads requested by the user. The above is the content of the automatic layout processing in this embodiment.

<About the effects of this embodiment>
FIG. 27 is a diagram for explaining the effects of this embodiment, and shows examples of albums with a plurality of variations created in this embodiment. To explain in detail, in FIG. 27(a), albums 1 and 2 are created as albums for subjects A and B, and the main subject in album 1 is set to subject A, and the main subject in album 2 is set to subject B. This shows a case in which Further, in FIG. 27(b), albums 1 to 3 are created as albums for subjects A to C, and the main subject in album 1 is subject A, the main subject in album 2 is subject B, and the main subject in album 3 is subject A. shows cases set for subject C, respectively.

In this embodiment, the total number of double-page spreads does not change depending on the album and is always the same for the variation albums to be created. In the example of FIG. 27(a), the total number of double-page spreads is 3 regardless of the album, and in the example of FIG. 27(b), the total number of double-page spreads is 2 regardless of the album.

Further, the number of individual spreads does not change depending on the album and is always the same. In the example of FIG. 27(a), the number of individual spreads between album 1 and album 2 is two, and in the example of FIG. 27(b), the number of individual spreads between albums 1 to 3 is one. In each album, the individual spreads are arranged in chronological order.

Further, the number of common spreads does not change depending on the album and is always the same. In the example of FIG. 27(a), the common number of double-page spreads between albums 1 and 2 is 1, and in the example of FIG. 27(b), the common number of double-page spreads of albums 1 to 3 is 1.

That is, before creating a double-page spread, the number of individual double-page spreads and the number of common double-page spreads are set in S190101, and then the double-page spread is created. Therefore, the number of individual double-page spreads, the number of common double-page spreads, and the total number of double-page spreads can be made the same between albums, and there is no need to adjust the number of double-page spreads after creating double-page spreads.

[Fourth embodiment]
In this embodiment, a double-page spread common to at least two albums and a separate double-page spread for each album are created. Align the number and position of common spreads and individual spreads for all albums. The common double-page spread and the images arranged within the double-page spread are arranged in order of shooting time. The position of each individual spread is determined based on the image, regardless of the shooting time.

<About automatic layout processing>
The automatic layout processing in this embodiment will be described below with reference to FIGS. 21(a) and 21(b) and FIGS. 22(a) to 22(c). FIGS. 21A and 21B are block diagrams showing functional configurations related to automatic layout processing in this embodiment. FIGS. 22(a) to 22(c) are flowcharts for explaining automatic layout processing in this embodiment. It should be noted that processes denoted by the same reference numerals as those in the above-described embodiments are the same processes as in the above-described embodiments, and therefore description thereof will be omitted here.

In S2201, the double-page spread creation unit 2101 creates a double-page spread of an album using the parameters generated in S510. Here, the two-page spread creation process in this step will be explained using FIG. 22(c).

In S220101, the spread number setting unit 210101 reads the number of spreads set in S190101 from the spread creation parameters read in S51101. That is, as described in the third embodiment, the number of common double-page spreads, the number of individual double-page spreads, and the total number of double-page spreads are set to be the same between albums by the process in S190101.

In S220102, the two-page spread allocation unit 210102 divides the candidate image acquired in S51102 into sub-image groups equal to the number of two-page spreads set in S220101, and allocates the divided images. In this embodiment, if the spread creation parameter set in the most recent S51101 is a common parameter, candidate images are divided based on the shooting time acquired in S50301. On the other hand, if the two-page spread creation parameter set in the most recent S51101 is an individual parameter, the candidate image is divided based on the score assigned in S508.

Specifically, if the two-page spread creation parameter set in the most recent S51101 is a common parameter, candidate images are arranged in the order of the shooting times acquired in S50301, and divided at the location where the time difference between adjacent images is the largest. Such division processing is performed until the page is divided into the number of spreads set in S220101. In other words, the image is divided (number of spreads - 1) times. With this, it is possible to create an album in which images are arranged in order of shooting time. Note that, as described later, the images within one double-page spread do not have to be arranged in the order of shooting time.

Furthermore, if the double-page spread creation parameters set in the most recent step S51101 are individual parameters, candidate images with the highest scores are assigned to each double-page spread in descending order of score. For example, when the number of double-page spreads is 3, candidate images are assigned in ascending order of scores, such as double-page spread 1, double-page spread 2, double-page spread 3, double-page spread 1, double-page spread 2, and so on. As a result, images with high scores are evenly assigned to each spread, and images good for the main subject can be arranged in a well-balanced manner in each spread, regardless of the chronological order.

Returning to the explanation of FIG. 22(a). In S2202, the double-page spread combining unit 2102 determines the double-page spread number within the album for each double-page spread created by the double-page spread creation unit 2101, and creates the layout of the entire album. In this embodiment, common double-page spreads created based on shooting time are arranged in order of shooting time. On the other hand, the individual spreads created based on the scores are inserted at positions where the time difference between the common spreads is large.

The two-page spread combination in S2202 will be described below with reference to FIGS. 23 and 24. FIG. 23 is a block diagram showing the detailed configuration of the double-page spread combining section 2102. FIG. 24 is a flowchart of processing executed by the double-page spread combining unit 2102.

In S2401, the double-page spread acquisition unit 2301 acquires a double-page spread related to one album from among the double-page spreads created by the double-page spread creation unit 2101. In the present embodiment, for example, the two-page spreads related to album 1 acquired in this step include a common two-page spread created using common parameters and individual two-page spreads created using individual parameters of album 1.

In S2402, the sort reference value calculation unit 2302 calculates, for each double-page spread acquired by the double-page spread acquisition unit 2301, a shooting time representative of each double-page spread. In this embodiment, the photographing times representative of a two-page spread are set to the average, minimum, and maximum photographing times of images arranged in a double-page spread.

In S2403, the sorting unit 2303 sorts the double-page spreads acquired by the double-page spread acquisition unit 2301 based on the shooting time of the double-page spreads determined by the sort reference value calculation unit 2302. In this embodiment, common double-page spreads are arranged in ascending order of average shooting time.

In S2404, the individual spread insertion unit 2304 determines the spread number of the individual spread among the spreads acquired by the spread acquisition unit 2301. In this embodiment, an individual spread is inserted into the common spreads sorted in S2403 at a position where the photographing time difference between the common spreads is maximum. The photographing time difference between the N-th common spread and the N+1-th common spread is calculated according to the following equation (13).

N+Minimum shooting time for the 1st common spread - Maximum shooting time for the Nth common spread...Equation (13)

Using equation (13), an individual double-page spread can be inserted at a position where there is a large difference in shooting time (that is, there is a high possibility that the scene has changed). Here, as a result of the process in S190101, the number of common double-page spreads is the same between albums, and naturally, the content of the common double-page spreads is also the same between albums. Therefore, the position at which an individual album is inserted, which is specified by the photographing time difference determined by equation (13), can also be made the same between albums.

Although a case has been described here in which all the individual spreads are inserted between the common spreads with the largest imaging time difference, the method of inserting the individual spreads is not limited to this. For example, the individual spreads may be inserted at different positions, such as inserting the i-th individual spread between the common spreads with the i-th largest imaging time difference. Further, an individual spread may be inserted at a position where the common spread is not divided regardless of the time difference, such as before all the common spreads or after all the common spreads.

In S2405, the two-page spread acquisition unit 2301 determines whether the processing in S2401 to S2404 has been completed for all albums. If the determination result of this step is true, the process advances to S513, while if the determination result is false, the process returns to S2401. Thereby, the common spread and the individual spread can be arranged at the same spread number positions in all albums.

As described above, in S260201, common double-page spreads are sorted, but individual double-page spreads are not sorted. Then, depending on the album, whether or not common double-page spreads are deleted in S260202 and the number of double-page spreads to be deleted differ. Therefore, according to this embodiment, although the total number of double-page spreads is the same between albums, the number of common double-page spreads and the number of individual double-page spreads may differ between albums. Furthermore, the individual spreads may be arranged in an order that does not follow the shooting time.

The above is the content of the automatic layout processing in this embodiment.

<About the effects of this embodiment>
FIG. 28 is a diagram for explaining the effects of this embodiment, and shows examples of albums with a plurality of variations created in this embodiment. To explain in detail, FIG. 28 shows a case where albums 1 and 2 are created as albums for subjects A and B, and the main subject in album 1 is set to subject A, and the main subject in album 2 is set to subject B. It shows.

In this embodiment, the total number of double-page spreads does not change depending on the album and is always the same for the variation albums to be created. In the example of FIG. 28, the total number of spreads is 4 regardless of the album.

Further, the number of individual spreads does not change depending on the album and is always the same. In the example of FIG. 28, the number of individual spreads between album 1 and album 2 is 1. In each album, the individual spreads are not arranged in chronological order.

Further, the number of common spreads does not change depending on the album and is always the same. In the example of FIG. 28, the common number of spreads between album 1 and album 2 is three.

Here, FIG. 28 shows an example in which an individual spread is inserted after a common spread. However, if a method using formula (13) is adopted in which the insertion position of the individual spread is between the common spreads, the individual spread will be inserted at one of the two insertion positions in the three common spreads shown in Figure 28. be done.

As described above, according to the present embodiment, it is possible to automatically align the number of double-page spreads in an album and the positions of individual double-page spreads.

[Other embodiments]
In the embodiment described above, an example has been described in which images are laid out in a double-page spread corresponding to a pair of pages when printed. However, the present invention is not limited to this, and images may be laid out on a single page when printing. That is, the images may be laid out in either double-page spreads or on a single page when printed, and the above-described embodiments can be applied to any layout in which images are laid out on pages. For example, the common double-page spread and individual double-page spread in the above-described embodiments do not need to be double-page spreads, and can be read as a common page common between albums or an individual page unique to each album as a broader concept.

The present invention provides a system or device with a program that implements one or more of the functions of the embodiments described above via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions. Note that the configurations of the first to sixth embodiments described above may be combined as appropriate.

100 Image processing device 1204 Spread number adjustment unit

Claims (13)

A device for generating an album including multiple pages, the device comprising:
Creation means for creating the plurality of pages that are different for each album and include a common page common to the plurality of albums and an individual page corresponding to a main subject set for each album ;
Identification means for specifying the number of candidate images that are candidates to be laid out in the plurality of albums based on the main subject corresponding to each of the plurality of albums;
determining means for determining the number of individual pages that are the same in the plurality of albums based on the number of candidate images corresponding to each of the plurality of albums specified by the specifying means;
Equipped with
The creating means creates the individual pages for each of the plurality of albums so that the number of the individual pages is the same in the plurality of albums, thereby determining the total number of pages and the common page in the plurality of albums. creating the plurality of pages for each of the plurality of albums such that the number of pages and the number of individual pages are the same,
The creation means creates a number of the individual pages determined by the determination means for each of the plurality of albums;
A device characterized by: The determining means determines the minimum number of candidates for the number of individual pages of each of the plurality of albums, which is determined based on the number of candidate images, to determine the number of individual pages that are the same in the plurality of albums. 2. The device according to claim 1 , wherein the number of digits is determined as the number of . The creation means is
analysis means for analyzing the candidate images and calculating a first score for each of the candidate images;
A second score is calculated for each of the candidate images based on the first score and a common page creation standard common to all albums, and an individual page that is different for each album is created based on the first score. rescoring means for calculating a third score based on each of the criteria;
including;
The creation means creates a common page including an image selected from the candidate images based on the second score, and an individual page including an image selected from the candidate images based on the third score. The device according to claim 1 or 2, characterized in that: Further comprising generating means for generating a creation standard for the common page and a creation standard for the individual page,
The criteria for creating the individual page is at least the main subject to be focused on for each album,
4. The apparatus according to claim 3 , wherein the common page creation criterion is all of the main subjects. 5. The apparatus according to claim 3 , wherein the first score includes a score regarding image quality for each of the candidate images and a score regarding a main subject for each album. The rescoring means calculates the second score for each of the candidate images based on the score related to the main subject specified in the creation criteria of the common page and the score related to the image quality, and calculates the second score for each of the candidate images. 6. The apparatus according to claim 5 , wherein the third score is calculated based on a score related to the main subject and a score related to the image quality specified in page creation criteria. 7. The apparatus according to claim 3 , wherein the candidate images arranged on the common page and the individual page are arranged based on predetermined image arrangement criteria. 8. The apparatus according to claim 7 , wherein the predetermined image arrangement criterion is at least one of an image capturing time, the second score, and the third score. 9. The apparatus according to claim 7 , wherein the predetermined image arrangement standard is different between the common page and the individual page. According to any one of claims 1 to 9 , further comprising page arrangement means for arranging the common page and the individual page for each album based on a predetermined page arrangement standard. equipment. 11. The apparatus according to claim 10 , wherein the page arrangement means arranges the individual page between the common pages determined based on an average photographing time of the candidate images included in the common page. A method for generating an album including multiple pages, the method comprising:
creating the plurality of pages that are different for each album and include a common page common to the plurality of albums and an individual page corresponding to the main subject set for each album ;
identifying the number of candidate images that are candidates to be laid out in the plurality of albums based on the main subject corresponding to each of the plurality of albums;
determining the number of individual pages that are the same in the plurality of albums based on the number of candidate images corresponding to each of the plurality of albums specified in the identifying step;
has
In the step of creating, the individual pages are created for each of the plurality of albums so that the number of individual pages is the same in the plurality of albums, so that the total number of pages and the common page number are the same in the plurality of albums. creating the plurality of pages for each of the plurality of albums so that the number of pages and the number of individual pages are the same,
The creating step creates, for each of the plurality of albums, the number of individual pages determined by the determining step.
A method characterized by: A program for causing a computer to execute the method according to claim 12 .

Images