JP2015128259A - Image processing device, photographing device, photographing system, image processing method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device that enables good photographing even when the number of images is insufficient.SOLUTION: Provided is an image processing device receiving images from each of a plurality of photographing devices used at respective photographing points. Storage means stores a correspondence between the photographing device and the used photographing point, and the number of images that is required to be photographed. Specifying means adds a tag for each subject photographed in the received image, calculates the number of images in which the subject is photographed, and specifies a subject for which the number of photographed images is insufficient as a to-be-photographed subject. Generation means specifies at least one photographing point where the to-be-photographed subject will pass after the current time as an instruction transmission point, calculates a time when the to-be-photographed subject will arrive at the instruction transmission point, and generates a photographing instruction at least including at least a part of subject information on the to-be-photographed subject and the calculated arrival time. Transmission means transmits the photographing instruction to the photographing device used at the instruction transmission point.

Description

  The present invention relates to an image processing apparatus, a photographing apparatus, a photographing system, an image processing method, and a computer program that identify a person (subject) taken from a photograph at a competition or the like and distribute / sell it.

  In general participation-type marathon competitions, etc., a service is provided in which photographers take pictures of competition participants at multiple shooting locations and sell the participants' photographs of the participants themselves after the competition. Therefore, it is required to take a sufficient number of photographs that improve the willingness of purchase of the competition participants, for example, a photograph of the participant himself / herself in the center of the photograph and present it to the participants quickly.

For example, Patent Document 1 proposes a method of searching for participants in a moving image obtained by shooting a tournament from a predicted passing time to a measurement point in a tournament, and editing a video for the participant.
However, depending on the participants, there were situations where the number of photos in the photo was small, or there were few photos that were good, such as blurring, blurring, or hiding their faces. Conventionally, a technique has been proposed in which a person (subject) that needs to be photographed is identified and photographed. For example, Patent Document 1 proposes a method in which a participant is photographed at an event or the like, the shooting time or the size of the image is stored as points, and a participant with few points is selected to be photographed.

JP 2008-28764 A

However, in the conventional technology, since a person (subject) is selected from subjects that can be photographed, a sufficient number of images cannot be photographed when passing through photographing points such as a marathon event one after another.
The present invention has been made in view of the above problems, and provides a system and the like that can notify subject information of a participant whose number of shots is insufficient to a photographing device placed at a point where the participant will pass. The purpose is to do. The subject information of a participant with an insufficient number of shots is a bib number assigned to the participant, a time when the participant is expected to arrive, and the like.

In order to solve the above problems, an image processing apparatus according to the present invention provides:
An image processing device that receives images from each of a plurality of photographing devices used at each photographing point in order to photograph an event in which a plurality of subjects pass through a plurality of photographing points in a predetermined order,
Storage means for storing the correspondence between the photographing device and the photographing point where the photographing device is used, and the number of images that need to be photographed;
Tagging is performed for each subject appearing in the received image, and the number of images in which the subject is photographed is calculated for each tagged subject, and the number of photographed images is required to be photographed. A specifying means for specifying a subject that is less than the number of images as a subject to be photographed;
Specifying at least one of the shooting points through which the subject subject to be photographed passes after the current time as an instruction transmission point, and calculating a time at which the subject subject to photograph reaches the instruction transmission point;
Generating means for generating a shooting instruction including at least a part of the subject information of the subject to be shot and at least the calculated time;
Transmitting means for transmitting the photographing instruction to the photographing apparatus used at the instruction transmission point.

  When taking a photo during a competition, a participant (subject) with a small number of photos will be notified of the expected arrival time of the participant to the shooting device at the shooting point where the participant will pass, and the participant will be shot. By encouraging this, it is possible to reduce the bias in the number of photographs for each participant.

It is a figure which shows an example of a structure of the imaging | photography system using the image processing apparatus and imaging | photography apparatus of this invention. FIG. 2 is a functional block diagram illustrating a configuration example of functions of the image processing apparatus 100 and the imaging apparatus 110 in FIG. 1. It is a structural example of the data handled with an imaging | photography system. It is a flowchart which shows the example of the process in a to-be-identified part. It is a flowchart which shows the example of the face / person information link process in FIG. It is an example of the image and data structure which are processed in the subject specifying unit. It is an example of the image information after performing a face / person information link process. It is an example of the person information after performing the face / person information link process. It is an example of an output of a number detection and recognition process in FIG. It is an example of an execution object image in a subject specific part, and a data structure after execution. It is an example of an execution object image in a subject specific part, and a data structure after execution. It is an example of person information after performing processing in a subject specific part. 3 is an example of an image evaluation standard used in an image selection unit in the first embodiment. 4 is an example of a photo album generated by the image selection unit in the first embodiment. 3 is a table showing input data in the first embodiment. FIG. 3 is a flowchart showing processing in the first embodiment. 6 is a diagram illustrating a display example in Embodiment 1 or 2. FIG. 10 is a table showing input data in the second and third embodiments. FIG. 10 is a diagram illustrating a display example in the fifth embodiment. FIG. 10 is a flowchart showing processing in the fifth embodiment. 10 is a table showing input data in the third embodiment.

The present invention relates to a photographing system for photographing an event (for example, a marathon event) in which a plurality of participants pass a plurality of photographing points in a predetermined order. This photographing system requires a plurality of photographing devices used at each photographing point and an image processing device that receives an image from each photographing device.
The image processing apparatus includes a storage unit, a specifying unit, and a generation unit.
The storage means stores a correspondence relationship between the photographing apparatus and the photographing point where the photographing apparatus is used, and the number of images that need to be photographed.
The specifying unit attaches a tag such as a person ID to each subject in the received image, and calculates the number of images in which the person is photographed for each tagged person. Then, a person whose number of photographed images is less than the number of images that need to be photographed is identified as a person to be photographed (photographable subject).

The generating means specifies at least one shooting point through which the person requiring photographing passes after the current time as an instruction transmission point, and calculates a time when the person requiring photographing reaches the instruction transmission point.
For example, if the photographer is already passing a point 4 km away from the start point at the current time and is running toward a point 8 km away from the start point, the 8 km point passing after the current time is specified as the instruction transmission point To do. Then, the time at which the person requiring photographing reaches the 8 km point is calculated. Then, a photographing instruction including at least a part of the person information (subject information) of the person who needs to be photographed (for example, facial feature amount) and at least the calculated time is sent to the photographing apparatus C used at the 8 km point. Send.

[Embodiment 1]
FIG. 1 is a diagram showing an example of the configuration of a photographing system using the image processing apparatus and photographing apparatus of the present invention.
The image processing apparatus 100 includes a CPU 101, a memory 102, a hard disk 103, and a network I / F 104. The CPU 101 executes a program for performing control of a captured image and recognition processing. The memory 102 is used for a work memory or temporary storage of data when the program is executed. The hard disk 103 stores data such as the program and captured images. The network I / F 104 performs data input / output with the imaging apparatus 110.

The photographing apparatus 110 includes a CPU 111, a memory 112, a recording medium 113, a camera unit 114, a display 115, an operation unit 116, and a network I / F 117. The CPU 111 executes a program for performing shooting control, image encoding processing, and the like. The memory 112 is used for temporary storage of the program, a work memory when executing the program, data, and the like.
The recording medium 113 stores data such as photographed images. The camera unit 114 converts external light into digital data via a lens. The display 115 displays a shooting UI and an image. The operation unit 116 detects an operation such as a shutter. A network I / F 117 inputs and outputs data with the image processing apparatus.

  The image processing apparatus 100 and the photographing apparatus 110 are connected to a wired or wireless network 120 such as a LAN or the Internet via network I / Fs 104 and 117, respectively. A photographed image can be transmitted from the photographing device 110, received by the image processing device 100, and stored in the hard disk 103. Imaging instruction data is transmitted from the image processing apparatus 100 to the imaging apparatus 110. In addition, it is preferable that a plurality of photographing apparatuses 110 exist and are connected to the network 120 by wireless communication.

FIG. 2 is a functional block diagram illustrating a configuration example of functions of the image processing apparatus 100 and the imaging apparatus 110 in FIG. The configuration shown here can be realized by a combination of hardware and software, and one functional block does not correspond to one hardware.
The imaging unit 200, the imaging control unit 201, the transmission unit 211, the reception unit 212, the display unit 213, the subject detection unit 214, the automatic focus adjustment unit 215, the automatic exposure control unit 216, and the framing support unit 217 are included in the imaging device 110. It is a functional block.
The subject specifying unit 202, the image evaluation amount calculation unit 203, the image selection unit 204, the shooting instruction unit 205, and the storage unit 206 are functional blocks in the image processing apparatus 100.
The photographing unit 200 of the photographing device takes a picture of the competition and acquires an image. The acquired image is transmitted from the transmission unit 211, received by the reception unit 207 of the image processing apparatus 100 via the network 120, and stored in the hard disk of the image processing apparatus 100.

<Image processing device>
The subject specifying unit 202 of the image processing apparatus 100 specifies a person shown in the image obtained from the shooting control unit 201 and adds person information (subject information) to the image information. If a plurality of persons are shown in one image, a plurality of person information is added (linked) to one image information.

The image evaluation amount calculation unit 203 scores the image obtained from the shooting control unit 201 for each person information (subject information) obtained from the subject specifying unit 202 by a score calculation method described later. Since the position and size of the image are different for each target person, even if they are captured in the same image, the same score is not obtained if the target person is different.
The image selection unit 204 presents an image to the participant based on the score obtained by the image evaluation amount calculation unit 203. This presentation result can be used for photo sales.

The shooting instruction unit 205 determines a person who needs to be shot based on the person information (subject information) specified by the subject specifying unit 202 and instructs the shooting. The processing of the shooting instruction unit 205 requires a predetermined number or more of images with person information added in order to determine a person who needs to be shot. Therefore, it is performed when a predetermined number of processes are performed by the subject specifying unit 202. For example, it starts after the first participant in the competition has passed an intermediate point or when the number of pieces of personal information has been obtained for all participants.
The imaging instruction is transmitted from the transmission unit 208 and received by the reception unit 212 of the imaging device 110.
The imaging control unit 201 displays the content of the received imaging instruction on the display, or directly controls the imaging unit 200 according to the imaging instruction.
Details of internal processing of the subject specifying unit 202, the image evaluation amount calculation unit 203, the image selection unit 204, and the shooting instruction unit 205 will be described later.

<Photographing device>
The photographing device is used at each photographing point to photograph an event in which a plurality of participants pass a plurality of photographing points in a predetermined order. As illustrated in FIG. 2, the imaging apparatus 110 includes an imaging unit 200, an imaging control unit 201, a transmission unit 211, a reception unit 212, and a display unit 213. In addition, the photographing apparatus 110 preferably includes a subject detection unit 214, an automatic focus adjustment unit 215, an automatic exposure control unit 216, and a framing support unit 217.
The imaging control unit 201 controls the imaging unit 200, the transmission unit 211, the reception unit 212, the display unit 213, and the like.
The photographing unit 200 photographs a subject and outputs a video signal.
The transmission unit 211 transmits the captured image.
The receiving unit 212 receives person information (subject information) including a feature amount of a person who needs to be photographed (photographable subject) predicted to pass, and a time when the person is predicted to pass.
The display unit 213 is
The person information and the time predicted to pass or the time obtained by subtracting the current time from the time predicted to pass are displayed.

  The automatic exposure control unit 216 realizes an automatic exposure control (AE) function. The automatic exposure control unit 216 controls the exposure value based on the luminance of a predetermined area including the person to be photographed detected by the subject detection unit 214. The photographing apparatus 110 has an aperture priority AE mode, a shutter speed priority AE mode, and a program AE mode as automatic exposure control functions. Further, it is assumed that the user can set whether or not the photographing sensitivity can be determined by the automatic exposure control function.

A framing support function is realized by the framing support unit 217. Using the optical zoom function and electronic zoom function, it is possible to shoot a large subject far away, but if the subject is moving or the camera shake is significant, the subject will fall out of the field of view. It becomes easy. In such a case, a function for bringing the subject into the angle of view again is called a framing support function.
For example, when a person who needs to be photographed deviates from the angle of view, the optical zoom and / or the electronic zoom are driven to the wide angle side by a predetermined amount. When the angle of view becomes wider, it becomes easier to find a person who needs to be photographed that has been lost. When a person who needs to be photographed is found, the optical zoom and / or electronic zoom are driven to the telephoto side, and the original angle of view is restored.

FIG. 14B is an example of a photo album that is automatically generated by a competition photography system using the photographing apparatus of the present invention.
Each participant (subject) is a photo album in which a plurality of images showing participants in the competition are laid out on a screen or paper. Multiple images are arranged along the story.
As described above, in the present invention, an image showing a specific participant is extracted from a large group of images taken during the competition, and the most suitable photo is selected and arranged for each of the image positions of the template of the photo album. The purpose is to present images to participants.

FIG. 3 shows an example of a data structure processed in this embodiment.
This data structure is stored in the hard disk 103 of the image processing apparatus 100. The data has a tree structure, with the type name of the data itself at the top and the attribute blocks in the blocks below it.
FIG. 3A shows the structure of image information. Image information is created for each photographed image photographed by the photographing device 110. The image ID attribute is an ID uniquely set for each image information.

The image entity attribute is pointer information indicating a storage location of binary data of image information, or binary data itself.
The subject person attribute has a plurality of variable person attributes for each person shown in the image. The number of persons in the corresponding image is determined by the number of person attributes. Attributes set below the person attribute and their values are collectively referred to as subject person information.
The shooting position attribute is the position of the point at which the image was shot. Here, for the sake of simplicity, the distance from the starting point of the competition is used.
The shooting time attribute is the time when the image was taken. Here, in order to simplify the explanation, information on the elapsed time from the start of the competition is used.
The landmark attribute is information on a landmark included in the image.

The subject person information set below the person attribute includes six attributes: a person ID attribute, a face position attribute indicating a state in which the person is captured in the image, a face size attribute, a facial expression attribute, a blur attribute, and a blur attribute. Composed.
The person ID attribute is an ID associated with person information to be described later.
The face position attribute has, as an X attribute and a Y attribute, the X and Y coordinates of the upper left vertex of the rectangle when the face position is represented by a rectangle with the upper left vertex of the image as the origin.
The face size attribute has a rectangular width and a rectangular height when the position of the face is represented by a rectangle as a W attribute and an H attribute, respectively.
The expression attribute is a state such as whether the person is smiling or crying. The blur attribute is the degree of blur of the subject person. The blur attribute is the degree of focus on the subject.

FIG. 3B is a diagram showing the structure of the person information. The person information is created each time a competition participant is recognized from an image, and has a many-to-many relationship with the image information. The person ID attribute is an ID uniquely set for each person information. The feature amount attribute is information representing features such as a person's face and clothes. The bib number attribute is the bib number of the competition participant. The image attribute has a plurality of IDs of images of the person as image ID attributes.
This data structure is an example, and may be stored in other structures as long as each information can be searched / acquired as a key.

Details of each processing unit in the block diagram shown in FIG. 2 will be described below.
<Subject identification part>
FIG. 4 is a flowchart illustrating the internal processing of the subject specifying unit 202.
In step S401, an image photographed by the photographing control unit 201 is acquired.
In step S402, the facial feature amount in the image is associated (linked) with the person information.

<Face / person information linking process>
The face / person information linking process in step S402 will be described in detail with reference to FIG. Further, as a specific example, a case where the image shown in FIG. However, in this step, one piece of image information may be processed and a transition may be made to the next step, or a plurality of pieces of image information may be processed collectively.
In step S501, a region where a face exists is detected from the input image. A known technique may be used for face detection. In Japanese Patent Laid-Open No. 8-063597, a face candidate area corresponding to the shape of a human face is extracted, and the face area is determined from the feature amount in the area. Has been proposed. The face detection results 601, 602, and 603 in the image shown in FIG. 6B are examples of results of face detection in step S 501.

In step S502, a person information group is acquired. FIG. 6C is an example of the acquired person information group. Here, the description will proceed assuming that the personal information for two persons stored in the hard disk 103 has been acquired.
From step S503, the process is executed for each face in the input image detected in step S501.
In step S503, the feature amount of the face in the input image is calculated. Here, the feature amount used in step S501 may be used.

In step S504, a facial feature amount is acquired from each piece of person information in the person information group acquired in step S502.
In step S505, the facial feature value detected from the input image is compared with the facial feature value obtained from the person information to determine whether the difference between the two facial feature values is within a threshold.
If the determination is YES, that is, if it is determined that the difference between the facial feature value detected from the input image and the facial feature value acquired from the person information is smaller than the threshold value, the process proceeds to step S507 and the two are associated. Specifically, the image ID of the input image is added to the image ID attribute of the person information.

For example, when it is determined that the difference between the feature amount of the face detection result 601 of the image ID: 101 and the feature amount of the person ID A100 is smaller than the threshold, the ID 101 is added to the image ID of the person information of the person ID: A100. Is done.
If the determination is NO, that is, if it is determined that the difference between the facial feature value detected from the input image and the facial feature value obtained from the person information is large, the process returns to step S504, and the facial feature value is obtained from the other personal information. To get

The step S506 transitions to the case where no person information matching the facial feature amount acquired from the input image has been accumulated, and newly generates personal information from the facial feature amount acquired from the input image. .
For example, the face detection result 603 of the image ID: 101 is compared with the face feature amounts of all the stored personal information, but does not match any feature amount (the difference is determined to be smaller than the threshold value). Therefore, new person information is generated as person ID: A300.

FIG. 7 shows image information after the face / person information linking process in step S402, and FIG. 8 shows person information. Compared to FIG. 3A, an attribute with no item means a state with no attribute value. At this stage, since the feature amount of the face of the person information and the image ID are only associated with each other, the individual is not specified. That is, from this data, it is not possible to extract an image showing a specific player (subject) using the bib number.
The above is the details of the face / person information linking process in step S402. Hereinafter, returning to FIG. 4, the description of the subject specifying unit will be continued.

<Number detection / recognition processing>
After step S403, the process is executed for each image information, that is, for each image ID.
In step S403, a region where a race bib exists, that is, a character region is extracted from the input image, and character recognition is executed.
For the bib region extraction (character region extraction), a publicly known technique may be used, for example, a technique disclosed in Japanese Patent Application Laid-Open No. 2009-123205.
In Japanese Patent Laid-Open No. 2009-123205, character areas are extracted by the following method.

An edge image is generated from the input image, and a connected pixel block (Connected Component, hereinafter referred to as “CC”) is generated from the edge image. As a result of the Hough transform of the coordinate positions of candidate CCs that are presumed to be characters from the feature quantity such as the size of the CC, the candidate CCs arranged in a straight line are determined to be character regions.
A known character recognition technique is executed on the bib region (character region) extracted by the above method to obtain the bib number.

FIG. 9 shows an example of the number recognition result obtained after executing the number detection / recognition processing on the input image.
In the actual image during competition, the race bib is not successfully detected because the race bib is obstructed by the player's (subject) 's arm or the body of another player, or the race bib is distorted by the movement of the rider's body. There is also a possibility. For example, in FIG. 9A, two athletes are shown, but the number of bib detection is one. In FIG. 9B, three athletes are shown, but the number of the race bib detected is one.
In FIG. 9C, three athletes (subjects) are shown, but the number of bibs detected is zero. Searching for an image of a competitor using only a bib will not hit such an image. Therefore, the individual of the subject in the image is specified by combining the previously generated person information and the number recognition result.
In FIG. 9D, two athletes are shown, but the number of the race bib detected is one.

In step S404 and subsequent steps, processing is executed for each number number acquired in step S403.
In step S404, for each person ID included in the image information, the position of the face in the input image is compared with the position of the number number to determine whether the number corresponds to the face.
Specifically, there is a face area that satisfies the condition that the distance from the center of the race bib area to the center of the face area is equal to or less than a threshold value and the center of the face area is located vertically above the center of the race bib area. To determine. If there are a plurality of face areas that satisfy the condition, the face area closer to the bib area is selected. “Closer to the race bib region” is determined based on the straight line distance from the center of the race bib region to the center of the face region.

If the determination is YES, that is, if there is a combination of the corresponding face area and the number area, the process proceeds to step S405, and the number number item of the person information associated with the corresponding face area is recognized from the corresponding number area. Add the bib number that was assigned. After the process of step S405, another number number in the input image is acquired, and another person ID in the image information is acquired, and the process proceeds to step S404.
If the determination is NO, another person ID included in the image information is acquired, and the process proceeds to step S404.

The processing in steps S404 and S405 will be described again using FIGS. 10 and 11 as specific examples. The processing order is image ID: 100 and image ID: 101, but there is no particular intention in the processing order.
First, processing is performed on the image ID: 100.
Reference numeral 1003 in FIG. 10A denotes a race bib region extracted from the image with the image ID: 100 and extracted in step S403. Reference numerals 1001 and 1002 denote the face areas extracted from the image having the image ID: 100 and extracted in step S402.

In step S404, the positional relationship between the bib area 1003 and the face areas 1001 and 1002 is compared.
Since the face area 1001 is not above the bib area 1003, it is determined that it does not correspond. Since the face area 1002 is located above and adjacent to the race bib area 1003 (the distance is equal to or smaller than the threshold value), the process proceeds to step S405.

In step S405, the person ID corresponding to the face area 1002 is acquired by referring to the image information of the image ID: 100 shown in FIG.
In the example of FIG. 10, the person ID: A100 can be acquired. Next, the personal information of the acquired personal ID is acquired, and the bib attribute of the personal information is added.
Specifically, the number number Z001 recognized by the image ID: 100 is added to the number number of the person information of the person ID: A100 shown in FIG.
Information on the attribute value of the bib number of the person information of the person ID: A200 for which no corresponding bib is detected is not added.
Next, the process of image ID: 101 is executed.

Reference numeral 1104 in FIG. 11 is a bib number area extracted in step S403 with the image ID: 101 as an input. Reference numerals 1101, 1102, and 1103 denote the face areas extracted in step S402 with the image ID: 101 as an input.
In step S404, the positional relationship between the race bib area 1104 and the face areas 1101, 1102, and 1103 is compared. Since the face area 1101 is not above the bib area 1104, it is determined that the face area 1101 does not correspond. The face areas 1102 and 1103 are both above the bib area 1104. In this case, the face area 1102 located closer to the race bib area 1104 is set as the corresponding face area, and the process proceeds to step S405.

In step S405, the person ID corresponding to the face area 1102 is acquired with reference to the image information of the image ID: 101 shown in FIG.
In the case of FIG. 11, the person ID: A200 can be acquired. Next, the personal information of the acquired personal ID is acquired, and the bib attribute of the personal information is added.
Specifically, the bib number Z200 recognized by the image ID: 101 is added to the bib number of the person information of the person ID: A200 shown in FIG. If the number information already exists in the acquired personal information, a list in which the number numbers are sequentially added without being overwritten may be created and sorted based on the reliability of the character recognition result of the number number.
FIG. 11C shows the person information after all the bib number and person ID included in the image ID: 101 are processed in steps S404 and S405.

FIG. 12 shows personal information after the processing performed on the image IDs: 100 and 101 is also performed on the image IDs: 102 and 103.
Each person information in FIGS. 12A to 12C has a number in its bib attribute. Therefore, by first extracting the person information using the bib number as a key and then referring to the image ID held by the person information, it is possible to extract only an image showing a specific person.
The internal processing of the subject specifying unit 202 has been described above.

<Image evaluation amount calculation unit>
The image evaluation amount calculation unit 203 calculates an image evaluation amount for the image obtained from the imaging control unit 201.
First, a face size attribute, an expression attribute, a blur attribute, and a blur attribute, which are subject person information, are calculated for each piece of image information, and the calculated values are stored. A known technique may be used to calculate each attribute.

In addition, the information on the landmarks reflected in the image is stored in the landmark attribute value of the image information.
Here, the relationship between the shooting position attribute value and the landmark that is expected to be captured when shooting from this shooting position is prepared in a table first, and the landmark attribute value can be obtained by referring to this table. decide. In addition, if the position of the camera to be photographed is fixed during the competition, a value may be set on the assumption that a specific landmark is reflected in an image photographed by a specific photographing camera.

<Image selection part>
The image selection unit 204 targets an image group in which a specific person extracted based on the person information created by the subject specifying unit 202 is captured, and an image prepared in advance based on the image evaluation amount calculated by the image evaluation amount calculating unit 203. An image is selected according to the evaluation criteria.
FIG. 13A is an example of an image evaluation criterion stored in the storage unit 206. There are two major evaluation criteria. One is an evaluation point to be weighted. This weights which parameter is important for parameters existing in the subject person information such as facial expressions and face positions. The other is an essential condition, and only images satisfying the conditions set here are selected.

  For the evaluation point, the attribute value of the subject person information calculated by the image evaluation amount calculation unit 203 may be used directly, or another parameter or condition based on the attribute value of the subject person information is selected for a more suitable image selection. It may be set. For example, since the number of persons in the image can be known from the number of person attributes included in the image information, a condition for the number 1302 of persons to be photographed based on the attribute value is added to the image evaluation standard 1301. Yes.

Further, if weighting is simply performed using only the face size attribute, the face of the person 1304 in the foreground is sufficiently large as shown in FIG. 13B, so that the image shown in FIG. 13B is an image suitable for the person 1304. May be selected. However, since the person 1304 is located at the end of the photograph, the evaluation of the image shown in FIG. 13B in the image group in which the person 1304 is captured is considered to be low. Therefore, the distance from the center of the face area to the center of the image is separately calculated, and this distance is added to the image evaluation standard as “distance from the center of the face position” 1303.
The image evaluation criterion is switched for each image arrangement position in the photo album template.

FIG. 14A shows an example of the relationship between a photo album template and image evaluation criteria. In the arrangement position 1401, an image selected based on the evaluation 1 is arranged. Similarly, at the arrangement position 1408, an image selected based on the evaluation 3 is arranged.
Instead of setting an evaluation criterion for each arrangement position in the template, the basic evaluation standard may be the same for all arrangement positions, and an additional evaluation criterion may be provided for each arrangement position.

For example, the image evaluation standard of the arrangement positions 1402, 1403, and 1407 is set to evaluation 2. In the evaluation 2, it is an essential condition that there is a landmark attribute value. In addition to this, landmarks to be reflected are specified for each arrangement. For example, an additional reference is set such that landmark = “XX bridge” at the arrangement position 1402, landmark = “Marunouchi building group” at the arrangement position 1403, and landmark = “xx tower” at the arrangement position 1407.
In addition, the image evaluation standard of the arrangement positions 1404, 1405, and 1406 is set as an evaluation 4, and an additional standard for arranging the images in time order is provided in 1404, 1405, and 1406.

FIG. 14B shows an example in which an image group showing a specific person is selected based on the above image evaluation criteria and an image is arranged on a template.
This image placement process is premised on scoring and classifying / selecting a large number of images, but some participants may not have a sufficient amount of images to present the images. Hereinafter, the details of the processing of the imaging instruction unit 205 for eliminating the bias for each participant of the image will be described.

<Shooting instruction unit>
FIG. 15A shows an example of a photographing apparatus, a transmission destination address when a photographing instruction is transmitted to the photographing apparatus, and a distance from the start point of the photographing point where the photographing apparatus is arranged. . FIG. 15B illustrates an example of image information and person information 1 hour and 30 minutes after the start of the competition. The tables shown in FIGS. 15A and 15B are stored in the storage unit 206.
Information shown in a table 1501 in FIG. 15A is information regarding an imaging device and its address (for example, an IP address) and its installation position (arrangement position). This is static information that has been held before the competition.
The shooting device A is 0 km from the start, that is, the starting point, the shooting device B is 4 km point, the shooting device C is 8 km point, the shooting device D is 12 km point, the shooting device E is 16 km, the shooting device F is 20 km, and the shooting device Z is 42. It shows that it is a 195km point. The image information captured by the imaging apparatus A has a value of 0 in the imaging position attribute.

  Reference numeral 1502 in FIG. 15B is a table in which the person information one hour and thirty minutes after the start of the competition and the image information associated with the person information are collected. The first line is a person ID attached to the person information, and the second line is a bib number. In the third line, the number of image IDs having the image attribute of the personal information is described as the number of images. The fourth line is the image ID of the image with the newest shooting time among the image IDs possessed by the image attributes of the person information. The fifth line is the shooting time of the latest image. In this embodiment, it is the time when the start time of the competition is “0: 0” (that is, “elapsed time from the start time”).

The sixth line is the shooting position of the latest image. In this embodiment, it is the length of the journey from the starting point of the competition. For example, the person whose person ID is Z01 has the number number 02, the number of images is 25, and the latest image ID is I77. The image with ID I77 is 1 hour and 27 minutes after the start of the competition, and the shooting position is 20 km from the starting point. In this embodiment, for convenience of explanation, the number of participants is six, but it is of course not limited to this number. This system is intended for shooting competitions of tens, hundreds or more.
Hereinafter, the processing of the shooting instruction unit 205 will be described by taking data (data shown in Tables 1501 and 1502) at the time when the competition 1 hour and 30 minutes have passed as an input example.

FIG. 16 is a flowchart illustrating in detail the processing of the shooting instruction unit 205.
Steps S1601 to S1604 are performed by the image processing apparatus 100, and steps S1605 and S1606 are performed by the photographing apparatus 110.
In step S1601, the person who is the subject of the shooting instruction is specified. For example, the person with the smallest number of image IDs held by the person information, that is, the number of photographed images, is specified as the person who is the subject of the shooting instruction, and the person information of the person is acquired. In the input example of FIG. 15B, Z03 personal information with seven images is acquired as a processing target, and the process advances to step S1602. In the present embodiment, the number of persons is the smallest for the sake of explanation, but a plurality of pieces of person information may be acquired under the condition that the value of the number of captured images attribute is equal to or less than a predetermined value. In that case, the processes in and after step S1602 are repeated as many times as the number of acquired personal information.

  In step S1602, the image information is acquired from the image ID of the person information acquired in step S1601, and the image information with the newest shooting time is acquired. As for the person information identified and created from the image, at least one piece of image information is always obtained. From FIG. 15B, the latest image ID associated with the person ID of Z03 is I53. 1 hour and 12 minutes are acquired as the imaging time of the image I53 and 12 km are acquired as the imaging position, and the process proceeds to step S1603.

In step S1603, the current position (hereinafter also referred to as “current position”) is predicted. Specifically, information on the shooting position of the image information acquired in step S1602 is acquired, and the current position is predicted.
A shooting point (next shooting point) through which the shooting instruction target person passes next is specified from the predicted current position.
After that, the time for passing the next shooting point or the time for passing the next shooting point is predicted how many minutes and seconds after the current time.
In this embodiment, first, the average speed is acquired by dividing the shooting position of the latest image (distance from the start point) by the shooting time of the latest image (elapsed time from the competition start time).
Next, the current location is predicted by multiplying the acquired average speed by the elapsed time from the competition start time to the current time.
The estimated passing time is calculated by subtracting “the time elapsed from the competition start time to the current time” from “the time obtained by dividing the distance from the start point to the shooting point by the average speed obtained in step S1603”. do it. By dividing the shooting position 12 km of the latest image acquired in step S1602 by 1 hour 12 minutes (1.2 hours) of the shooting time, an average speed of 10 km is obtained. Then, by multiplying the average speed by the current time of 1 hour 30 minutes (1.5 hours), the current position of the person whose person ID is Z03 is predicted to be a 15 km point, and the process proceeds to step S1604.
This method is an example. For example, mapping is performed on two-dimensional coordinates from the shooting position and shooting time of all image information in which the person is photographed, a curve approximation formula is obtained, and the next shooting point is predicted. It may be used.

  In step S1604, based on the current location obtained by the image processing apparatus 100 in step S1603, the imaging device (hereinafter also referred to as “next imaging device”) at the imaging point where the person whose person ID is Z03 passes next is specified. Then, the person information is transmitted to the photographing apparatus. From the data shown in the table 1501 of FIG. 15A, it can be seen that the next photographing apparatus is the photographing apparatus E at the photographing point 16 km. Therefore, the person information of the person whose person ID is Z03 is transmitted to the imaging device E at the 16 km point, and the process proceeds to step S1605. Moreover, you may transmit to not only the following imaging | photography apparatus but all the imaging | photography apparatuses with which passage is predicted from now on.

In step S1605, the photographing apparatus E receives the person information transmitted in step S1604, and the process proceeds to step S1606.
In step S1606, the person information received by the photographing apparatus E in step S1605 is displayed on a UI (user interface), thereby prompting the photographer of the photographing apparatus E to photograph a person whose person ID is Z03, and processing. Complete.

FIG. 17 is a display example of a notification on the display 115 which is a UI of the photographing apparatus 110. 1701 in FIG. 17A is the personal information of Z03 received in step S1605. In this example, the number number 03, the current number of shots, and the estimated passage time of the shooting point are displayed from the top as the person information. The “predicted passage time” means the time from the current time to the predicted passage time, and the predicted passage time is “the person of Z03 passes through the photographing point where the photographing device that has received the personal information of the person of Z03 is arranged. Then, it means “predicted time”.
The predicted passage time can be calculated by subtracting the current time from the predicted passage time.

Reference numeral 1702 displays an image obtained by cutting out the target person from the image associated with the person information. By transmitting and receiving such image information in addition to the person information in steps S1604 and S1605, it becomes possible to visually notify the photographer of information about the object to be photographed. The photographer searches for participants based on this information and can focus on photographing if a target is found.
As described above, according to the first embodiment, even if there is a bias in the number of shots, the photographer is notified of the fact and the shooting is encouraged, and the photographer shoots according to the notification content to reduce the bias in the number of shots. It is possible to present a sufficient number of images to the user.

[Embodiment 2]
In the first embodiment, in step S1601, a person with a small number of shots is set as the shooting instruction target. However, the shooting instruction target may be selected using the image evaluation amount calculated by the image evaluation amount calculation unit 203.
For example, a participant who has the least number of successful images by calculating the number of images that are not blurred, that is, the focus on the participant's face, and that is not blurred by the movement of the camera or participant as a successful image. Is selected as an object that needs to be shot.

A table 1801 shown in FIG. 18A is obtained by adding the evaluation amount calculated by the image evaluation amount calculation unit 203 to the table of FIG. In the fourth line, the number of successful images is added. In the case of the table 1801, the person Z06 with the smallest number of successful images is selected as a shooting instruction target.
Since the processing from step S1602 to step S1606 after specifying the person who is the subject of the imaging instruction is the same as that in the first embodiment, the description thereof is omitted.

In addition, an index other than blurring and blurring, for example, the person with the smallest number of people in a sufficient size may be selected as the instruction target person.
The fifth line of the table 1801 in FIG. 18A is information on the number of people who are up, that is, a large image. Whether or not the image is a large copy may be determined based on the size information (width and height) of the face area at the time of face detection. In the case of Table 1801, the person whose person ID is Z04 has the smallest number of up images, and is therefore identified as the person who is the subject of the shooting instruction. After the person to be imaged is specified, the processing from step S1602 to step S1606 is performed as in the first embodiment.
FIG. 17B illustrates an instruction UI when a person whose person ID is Z04 is identified as a person who is a photographing instruction target because the number of up images is the smallest. In a region 1711, “Please take a close-up photo” is displayed as an instruction message for encouraging a close-up shot.
As described above, according to the second embodiment, when there is a deficiency / bias in the number of good-looking photos or large-sized photos, the photographer is notified so that the lacking photos can be taken. It is possible to prompt.

[Embodiment 3]
In the first embodiment, a person with a small number of captured images is set as a shooting instruction target. In the second embodiment, a person with a small number of predetermined images (such as an up image) is set as a shooting instruction target. However, even if the total number of images is large and there are more than a predetermined number of predetermined images (up images, etc.), the number of images satisfying a specific evaluation criterion among a plurality of evaluation criteria is small, and the photo album template image An image may not be arranged at a part of the position.
For this reason, in step S1601, a person whose number of images satisfying the indispensable condition for any one of the plurality of evaluation criteria is less than a predetermined value may be specified as an imaging instruction target.

A table 2101 shown in FIG. 21A is obtained by adding the number of images that satisfy the essential conditions for each evaluation to the table of FIG.
In FIG. 21A, the 10th line of the table 2101 is the number of images satisfying the essential condition of evaluation 1, the 11th line is the number of images satisfying the essential condition of evaluation 2, and the 12th line satisfies the essential condition of evaluation 3. The number of images and the 13th line indicate the number of images satisfying the essential condition of evaluation 4, respectively.

A table 2102 illustrated in FIG. 21B is an example of an image evaluation criterion. The required number of lines is the required number.
In the case of Table 2101, since the number of images satisfying the essential condition of evaluation 4 for the person whose person ID is Z01 is “2”, the required number of evaluations “3 or more” is not satisfied. For this reason, the person whose person ID is Z01 is identified as the person who is the subject of the shooting instruction in step S1601.
After the person to be imaged is specified, the processing from step S1602 to step S1606 is performed as in the first embodiment.

Thereby, in order to create a photo album as shown in FIG. 14 (B) using a template as shown in FIG. 14 (A), when an image satisfying various evaluation criteria is presented to a participant, It is possible to prevent the occurrence of a situation where there is a shortage of images that satisfy a specific evaluation criterion.
As described above, according to the third embodiment, it is possible to avoid that the number of images arranged at each image position of the photo album template is less than the required number.

[Embodiment 4]
In the first to third embodiments, the current location is determined, and a shooting instruction is given by the next shooting apparatus. However, for a shortage of images, the shooting apparatus that can perform suitable shooting is instructed to take a shooting instruction. Also good.
The sixth line of the table 1801 in FIG. 18A is information on the number of images in which landmarks are shown. In the processing in step S1601 in the first and second embodiments, the person with the smallest number of images in which the landmarks are photographed is selected as the photographing instruction target.

  A table 1802 in FIG. 18B is obtained by adding landmark information to the fourth row of the table 1501 in FIG. “◯” indicates that the landmark can be photographed, and “x” indicates that the landmark cannot be photographed. Since it can be determined whether or not the landmark is reflected depending on the shooting location, it is possible to determine whether the shot image includes the landmark by providing information for each shooting device in advance. Of course, a landmark may be recognized and information may be added using a known image object recognition technique or the like.

  In the example of the table 1801, the person Z05 having the smallest image in which the landmark is photographed is selected as the photographing instruction target. In step S1602, an image of the person Z05 is acquired as in the first embodiment, and in step S1603, the current position is predicted from the shooting time and shooting position of the newest image I65 associated with Z05. As a result of the calculation, it is predicted that the vehicle is at the 14.4 km point at 1 hour 30 minutes.

In step S1604, when the next photographing apparatus is obtained by the same method as in the first embodiment, the photographing apparatus E at the photographing point 16km is obtained. However, since there is no landmark at the E point, the next shooting point where the landmark can be shot is the shooting device F at the shooting point 20 km, and the person information is transmitted to the shooting device F.
The evaluation described in the present embodiment is an example, and based on the number of images scored by other methods, a photographing instruction may be issued to a suitable photographing device.
As described above, according to the fourth embodiment, it is possible to prompt photographing by notifying an appropriate photographing device of an image that can be photographed only by a specific photographing device such as a landmark.

[Embodiment 5]
In the first embodiment, the photographer is only notified by the UI, but the subject detection unit 214 of the photographing apparatus 110 may automatically detect and recognize a person using the feature amount of the person information.
FIG. 19 shows an example in which an image acquired by the camera unit 114 is displayed on the display 115 by the photographing device 110 during the competition. In the present embodiment, the camera unit 114 periodically acquires external light and is displayed on the display 115. When the photographer performs a shutter operation with the operation unit 116 while checking the display 115, the CPU 111 performs processing such as image encoding and the like and stores it in the recording medium 113.

FIG. 20 is a flowchart showing step S1605 of FIG. 16 and subsequent processing. The details of the processing will be described using the image of FIG. 19A as an input example and the flowchart of FIG.
In step S1605, the photographing apparatus receives the person information of the photographing instruction target, and proceeds to step S2001. In the present embodiment, it is assumed that a person whose person ID is Z06 is a shooting instruction target.

In step S2001, the live image 115 in FIG. 19A is acquired, and the process proceeds to step S2002.
In step S2002, face detection processing is performed, and as a result, three face areas 1901, 1902, and 1903 are detected from the live image 115, and the process proceeds to step S2003. In step S2003, each face area detected by the subject detection unit 214 is detected. Process. Processes are performed in the order of the face areas 1901, 1902, and 1903. First, whether the feature amount of the face area 1901 matches the feature amount of the person information of the shooting instruction target (person with the person ID Z06), with the face area 1901 as the processing target. Judging. If they do not match, the process of step S2003 is executed again with the next face area 1902 as the processing target. “The feature amounts match” means that the difference between the two feature amounts is within a predetermined threshold.
In the second step S2003, since the feature amount of the face area 1902 and the feature amount of the person information of the photographing instruction target (person with the person ID Z06) match, the process proceeds to step S2004.
In step S2004, UI highlighting, shooting settings, and the like are performed, and the process advances to step S2001 to perform the same processing for the next timing image.

FIG. 19B shows an example in which a character string “photographer required” indicating a subject to be photographed is displayed together with a guide 1911 in a face area 1902 that is determined to be a photographing instruction target. Thus, by automatically recognizing the photographing instruction target and displaying the guide, it is possible to assist the photographer to easily find the photographing target person.
Further, the guide display is an example, and other processing may be performed.
For example, when the shooting instruction target is recognized, the auto focus adjustment unit 215 focuses on the shooting instruction target, the automatic exposure control unit 216 performs exposure adjustment, or the continuous shooting mode is used to reduce failed photos. Or, the photographing apparatus may automatically perform photographing.
As described above, according to the fifth embodiment, it is possible to show that the person to be instructed to shoot can shoot and to easily focus on shooting.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

100 Image processing apparatus 101 CPU
102 Memory 103 Hard disk 104 Network I / F
110 Imaging device 111 CPU
112 Memory 113 Recording medium 114 Camera unit 115 Display 116 Operation unit 117 Network I / F
120 Network 200 Shooting Unit 201 Shooting Control Unit 202 Subject Identification Unit 203 Image Evaluation Amount Calculation Unit 204 Image Selection Unit 205 Shooting Instruction Unit 206 Storage Unit 213 Display Unit 214 Subject Detection Unit 215 Automatic Focus Control Unit 216 Automatic Exposure Control Unit 217 Framing Support Part

Claims (13)

An image processing device that receives images from each of a plurality of photographing devices used at each photographing point in order to photograph an event in which a plurality of subjects pass through a plurality of photographing points in a predetermined order,
Storage means for storing the correspondence between the photographing device and the photographing point where the photographing device is used, and the number of images that need to be photographed;
Tagging is performed for each subject appearing in the received image, and the number of images in which the subject is photographed is calculated for each tagged subject, and the number of photographed images is required to be photographed. A specifying means for specifying a subject that is less than the number of images as a subject to be photographed;
Specifying at least one of the shooting points through which the subject subject to be photographed passes after the current time as an instruction transmission point, and calculating a time at which the subject subject to photograph reaches the instruction transmission point;
Generating means for generating a shooting instruction including at least a part of the subject information of the subject to be shot and at least the calculated time;
An image processing apparatus comprising: a transmission unit configured to transmit the photographing instruction to the photographing apparatus used at the instruction transmission point. The storage means
Further storing at least one image evaluation criterion and the number of images that need to be photographed for each evaluation criterion;
The specifying means is:
The number of images satisfying the evaluation criterion is calculated for each of the evaluation criteria, and a subject that does not satisfy the number of images that need to be photographed for each of the evaluation criteria is identified as a subject to be photographed. The image processing apparatus described.   The image processing apparatus according to claim 1, wherein the generation unit generates the shooting instruction including an image of the subject requiring shooting. The storage means
An essential condition is that the correspondence between the first shooting point, the second shooting point, and the landmarks that can be shot from the first and second shooting points is stored, and that the landmark is shot. And storing the first evaluation criterion as a part of the evaluation criterion,
The specifying means is:
A subject that has passed through the first shooting point but does not satisfy the number of images required for shooting in the first evaluation criterion as an image that satisfies the first evaluation criterion is identified as a subject to be shot.
The generating means includes
A shooting instruction in which an image satisfying the first evaluation criterion includes subject information of a subject that does not satisfy the number of images required for photographing according to the first evaluation criterion, and the photographing device at the second photographing point is a transmission destination. The image processing apparatus according to claim 2, wherein the image processing apparatus generates the image processing apparatus. The generating means includes
The image processing apparatus according to claim 1, wherein a shooting instruction including a feature amount of the subject to be shot is generated as the subject information. An imaging device used at each imaging point to capture an event in which a plurality of subjects pass through a plurality of imaging points in a predetermined order,
Photographing means for photographing a subject and outputting a video signal;
A transmission means for transmitting the captured image;
Receiving means for receiving subject information including a characteristic amount of a subject requiring photographing that is predicted to pass, and a time at which the subject is predicted to pass;
An imaging apparatus comprising: the subject information; and a display means for displaying the time predicted to pass or the time obtained by subtracting the current time from the time predicted to pass. A subject for detecting a subject to be photographed from subjects in the live image by comparing the received feature amount of the subject to be photographed with the feature amount of the subject in the live image displayed on the display means. Detection means,
The imaging apparatus according to claim 6, further comprising: Automatic focus adjustment means for performing focus adjustment so as to focus on a subject to be detected detected by the subject detection means;
The photographing apparatus according to claim 7, further comprising: Automatic exposure control means for controlling the exposure value based on the luminance of a predetermined area including the subject to be detected detected by the subject detection means;
The photographing apparatus according to claim 7, further comprising: Framing support means for supporting framing for a subject requiring photographing detected by the subject detection means;
The imaging apparatus according to claim 7, further comprising:   An imaging system comprising: the image processing device according to any one of claims 1 to 5; and the imaging device according to any one of claims 6 to 10. An image processing method for processing images received from each of a plurality of photographing devices used at each photographing point in order to photograph an event in which a plurality of subjects pass a plurality of photographing points in a predetermined order,
A storage step of storing in advance the correspondence between the photographing device and the photographing point where the photographing device is used, and the number of images that need to be photographed;
Tagging is performed for each subject appearing in the received image, and the number of images in which the subject is photographed is calculated for each tagged subject, and the number of photographed images is required to be photographed. A specific step of identifying a subject that is less than the number of images as a subject to be photographed;
Specifying at least one of the shooting points through which the subject subject to be photographed passes after the current time as an instruction transmission point, and calculating a time at which the subject subject to photograph reaches the instruction transmission point;
Generating a shooting instruction including at least a part of subject information of the subject to be shot and at least the calculated time;
An image processing method comprising: a transmission step of transmitting the imaging instruction to the imaging device used at the instruction transmission point. A program for causing a computer to execute each step of the image processing method according to claim 12.

Images