JP2020107196A - Image processing device or image processing server - Google Patents

Abstract

To make it possible to prevent professional photographers and general spectators from losing sight of a subject when acquiring and observing images, and also enable a server to acquire image analysis results.SOLUTION: An image processing device is characterized by including display means for displaying images, selection means for selecting a specific object from the image displayed on the display means, designation information generation means for generating designation information relating to the specific object selected by the selection means, transmission means for transmitting the designation information generated by the designation information generation means to a server, acquisition means for acquiring, from the server, position information of the specific object based on the designation information and predetermined metadata created in the server, and control means for displaying, on the display means, additional information based on the position information of the specific object acquired by the acquisition means or the information on the metadata.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing device and the like for shooting and video monitoring.

In recent years, due to internationalization, many tourists have come to visit Japan. Also, in competitive sports, shooting opportunities for athletes from various countries have increased significantly.

However, it is relatively difficult for professional photographers as well as general photographers to find a specific player from a large number of players in a competitive sports scene, for example. In addition, during competition, especially in competitive sports, movements are fast and multiple athletes often cross each other and lose sight of where the athlete is. This is true not only in competitive sports, but also when shooting or monitoring a specific person or the like from among crowds.

Patent Document 1 describes a plurality of cameras for photographing a subject from a plurality of directions, and a plurality of image processing devices for extracting a predetermined region from an image captured by a corresponding camera among the plurality of cameras. Further, there is described an image generation device that generates a virtual viewpoint image based on image data of a predetermined area extracted from images captured by the plurality of cameras by the plurality of image processing devices.
Patent Document 2 describes an automatic focus detection device that drives a focus lens based on an AF evaluation value acquired from a captured image to perform automatic focus detection control.

JP, 2017-212828, A Patent No. 5322629 Specification

However, when many athletes are gathering, such as in a sports scene, the athletes may overlap and lose sight. It was also out of sight, making it more difficult to shoot the athlete at the right time.
Furthermore, recently, a photographer often leaves a memo when a photograph is taken in association with the photograph, for example, by voice. However, when the intention of shooting is left as metadata information, there is a drawback that it cannot be left in association with other metadata. Also, depending on how the game progresses, the result may differ from the memo contents taken by the photographer. In this case, there is also a problem that the memo of the metadata that is different from the photographer's intention of photographing remains as it is.

Further, in particular, a professional photographer needs to send the photographed image to the news agency immediately, but there is a drawback that it takes time to recognize the determination unless the determination result is known. Furthermore, even if I found the player I wanted to shoot, I had to focus on that player and then chase the player I wanted to shoot. This is very difficult for fast-moving sports, and if you concentrate on this chase, you will not be able to take good pictures.

In addition, the server side can grasp omnidirectional video, various information on the field of the game and competition, and can also obtain various valuable information inside and outside the ground, but the conventional system does not fully utilize the server. There was a problem not.
Similarly, a general user who monitors a game at a stadium or a terminal at home often loses sight of a particular player or loses the game situation. Similarly, in a car race, an airplane race, a horse race, or the like, a target such as a specific vehicle, an airplane, or a horse may be lost. Furthermore, even when a specific person is tracked on a street corner, the specific person may be lost in the crowded traffic. Further, if the user concentrates on visually following such a particular object of interest, there is a problem that it is not possible to smoothly perform shooting, focusing, exposure adjustment, and the like on the target.

In addition, professional photographers often add metadata to images, but the work is complicated and time-consuming, and the accuracy of the contents of the memo may be poor. Furthermore, although there is a case where the metadata at a certain timing should be corrected or changed over time, there is a problem that the metadata cannot be changed easily.
SUMMARY OF THE INVENTION It is an object of the present invention to provide an image processing device that solves the above problems, can add useful information in a timely manner, and can correct it appropriately.

In the image processing device,
Display means for displaying images,
Selecting means for selecting a specific object from the image displayed on the display means,
Designation information generation means for generating designation information relating to the specific target selected by the selection means,
Transmitting means for transmitting the designated information generated by the designated information generating means to a server;
Acquisition means for acquiring the position information of the specific target based on the designation information and the predetermined metadata created in the server from the server,
Control means for displaying on the display means additional information based on the position information of the specific target acquired by the acquisition means or information about the metadata;
It is characterized by having.

According to the present invention, it is possible to provide an image processing apparatus that can add useful information in a timely manner and can make appropriate corrections.

FIG. 1 is a block diagram of an entire system including an image processing device according to an embodiment. It is a block diagram of a server side. It is a detailed block diagram on the terminal side. It is a detailed block diagram on the terminal side. It is a block diagram which shows the functional structural example of the tracking part 371 by the side of a terminal. It is a figure which shows an example of an attention player display start sequence. It is a figure which shows the other example of an attention player display start sequence. It is a figure which shows the example of a structure of a file. 7 is a flowchart showing an example of control on the camera side. It is a flow chart which shows an example of attention player display tracking control by the side of a camera. It is a flow chart which shows an example of attention player display tracking control by the side of a camera. It is a flow chart which shows an example of detection control of an attention player on the server side. It is a flow chart which shows an example of detection control of an attention player on the server side. It is a flow chart which shows an example of detection control of an attention player on the server side. It is a flow chart which shows an example of detection control of an attention player on the server side. It is a flow chart which shows an example of detection control of an attention player on the server side. It is a flow chart which shows an example of detection control of an attention player on the server side. It is a flowchart which shows the example of the try determination control in the server side. It is a flowchart which shows the example of the try determination control in the server side. It is a figure which shows the referee operation|movement of a trial presence determination. It is a flowchart which shows the example of the try determination control in the server side. It is a flowchart which shows the example of the try determination control in the server side. 7 is a flowchart showing an example of try determination control on the side of a terminal such as a camera. It is a flow chart which shows an example of foul judgment control by the side of a terminal, such as a camera.

Hereinafter, modes for carrying out the present invention will be described using examples.

First, an overall image of a system using an image processing apparatus for supporting shooting and a video monitor in the first embodiment will be described with reference to FIG. FIG. 1 is a block diagram of the entire system including the image processing apparatus of the embodiment.

In FIG. 1, a server (image processing server) side having a plurality of cameras for a server (moving camera using a fixed camera or a drone, etc.) is located in the entire field of the stadium and the position of the player (specific target) or the game. Understand the latest situation in real time. Then, an example will be shown in which the server provides the information held by each spectator with necessary information in a timely manner when, for example, photographing with a camera or monitoring an image.

In general, there are many places where a professional photographer or a general photographer cannot understand or follow the angle or field of view taken by the camera. The same is true for spectators outside the stadium who are not filming. On the other hand, the system on the server side can grasp and map the omnidirectional image and the information of the entire field of the game (coordinate information of the field, etc.) in advance based on the images of the cameras for a plurality of servers.
Therefore, the service for the spectators can be significantly improved by the server grasping and delivering the incomprehensible information that cannot be seen by each user.

That is, it is possible to track the position, score, foul, judgment result of the referee, and other latest situation of each player by using a plurality of cameras (fixed camera and moving camera) for the server. It can also be analyzed by the server based on the information displayed on a large screen. As a result, the entire situation can be accurately recognized and timely transmitted to a camera terminal or the like owned by a professional photographer or an audience, or a terminal such as a smartphone or a tablet. As a result, the spectator can grasp the latest situation of the competition in a timely manner. In particular, a professional photographer needs to send the photograph taken immediately to the news agency, but it is difficult to accurately grasp the overall situation of the competition because the field of view is narrow only by looking at the screen of the camera. However, if the configuration of this embodiment is used, the situation of the competition and the like can be quickly known, so that it becomes possible to quickly select the photograph to be sent to the news agency.

As a terminal (image processing device) used by a professional photographer or an audience, a digital camera, a smartphone, a configuration in which a camera and a smartphone are connected, a tablet PC, a TV, or the like can be considered. Since the same service can be provided to spectators watching the competition at home via the Internet or TV broadcasting through terminals (image processing devices) such as PCs and TVs, it is possible to grasp the situation of the competition more accurately. You can enjoy the competition more.

In FIG. 1, 101 to 103 are server cameras, 101 (camera 1 fixedly installed), 102 (camera 2 fixedly installed), 103 (camera 3 fixedly installed), 104 (large screen). , 110 (server), 111 (input means), 112 (base station) perform image acquisition and audio acquisition for providing information to general professional photographers and spectators. In this embodiment, three server cameras 101 to 103 are used, but one or a plurality of server cameras may be used. Further, these server cameras may be cameras installed in, for example, a drone, instead of fixed cameras. In addition to video acquisition and audio acquisition, input information other than video (such as audio) can be captured from the input means, and services for general professional photographers and spectators can be expanded.

Reference numeral 105 is a wired/wireless LAN or the Internet, and 106 is a connection line for inputting the information output from the input unit 111 to the server 110. Reference numeral 107 is a connection line for transmitting/receiving a signal to/from the base station 112, and reference numeral 108 is an antenna unit for executing wireless communication of the base station.
That is, the blocks in the 100s are blocks for supporting video shooting by professional photographers and general spectators.

On the other hand, in FIG. 1, 401 (terminal 1 ), 402 (terminal 2 ), 403 (terminal 3) are terminals, such as cameras, smartphones, tablet PCs and TVs for shooting and monitoring by professional photographers and spectators. Is a video display terminal device such as. Here, 404 (antenna) 405 (antenna) and 406 (antenna) are antennas for 401 (terminal 1), 202 (terminal 2) and 203 (terminal 3) to perform wireless communication, respectively.

When the server detects the position of the player of interest, for example, the terminal sends ID information of the player of interest to the server side, and the server side sends various information such as position information regarding the player to the terminal. Since the players are moving and the competition situation is changing, it is necessary to detect the player of interest in a short time. Therefore, for example, 5G is used for the wireless communication here.

Note that the 401 (terminal 1), 402 (terminal 2), and 403 (terminal 3) may have a configuration in which a camera and a smartphone or the like are connected and combined. In the lower right part of FIG. 1, 301 is a smartphone, which mainly controls communication with the server. Also, by installing application software on this smartphone, various video acquisition services will be realized. Reference numeral 300 denotes a (digital) camera, which is an image processing apparatus mainly used by professional professional photographers and spectators to monitor images. Here, the camera 300 is connected to the smartphone 301 by USB or Bluetooth (registered trademark). 320 is an antenna for the smartphone 301 to wirelessly communicate with the base station 112.

If the terminal is a smartphone, etc., video and control signals are exchanged with the server wirelessly, but the connection for executing communication with the terminal may be adaptively selected from wireless communication and wired communication. good. For example, if the wireless communication environment is 5G, it is possible to perform wireless communication, and if the wireless communication environment is LTE, it is possible to send information with a large amount of data by wire and send a control signal with a small amount of data by wireless. Furthermore, it is possible to switch to wired communication depending on the congestion of wireless communication lines.

Next, FIG. 2 is a detailed block diagram of the server side, and FIG. 2 will be described. In FIG. 2, the same reference numerals as those in FIG.
Reference numeral 201 is an Ethernet (registered trademark) controller, and reference numeral 204 is a detection unit that detects a play position according to the role (so-called position) of the player. Here, the role (position) of the player is set in advance by registration or the like. For example, as for the role (position) in the case of rugby, 1 and 3 are props, 2 is hooker, 4 and 5 are rock, 6 and 7 are flanker, 8 is called number 8, 9 is scrum half, 10 is Called the standoff. Also, 11 and 14 are called wings, 12 and 13 are called centers, and 15 is called a fullback.
In the places where these players are, the forwards are often in front of the attack and the backs are often behind in the attack, such as in set play.

In other words, the position of a player is generally determined according to the role (position) of the player, so it is more effective and accurate to follow the player after understanding the role (position) of the noted player. You can
In general, the player's role can often be recognized from the uniform number. However, in some cases, the 10th player is injured, the 15th player stands off (enters the 10th position), and the reserve player enters the 15th player. Here, the reserve player's uniform numbers are 16 to 23. However, the position is not fixed only by the uniform number. Therefore, the detection unit 204 detects the play position of the player according to the preset role, and the information on the detected play position is fetched by the CPU 211 in the server 110. However, the preset role is the competition. It may change due to changes in players.

Reference numeral 205 denotes a contour information detection unit, for example, when a professional photographer or a spectator monitors an image on a terminal and shoots at a magnification of the camera from the position and angle, the server 110 determines the position of the attention player. Notify the terminals 401 to 403. Further, by notifying the terminals 401 to 403 and the like of the contour information of the focused player who is shooting the image, the terminals 401 to 403 can recognize the focused player more reliably. The contour information detected by the block 205 is fetched by the CPU 211.

Reference numeral 206 denotes a player's face recognition unit, which finds a player in an image based on AI, in particular, an image recognition technique such as Deep Learning based on pre-registered face photograph information of the player. Information on the face recognition result detected by the face recognition unit 206 is also fetched by the CPU 211.
A player's physique recognition unit 207 finds a player using the image recognition technique as described above based on the physique photo information of the noted player registered in advance.

Reference numeral 208 denotes a player's uniform number detection unit, which finds the player from the number (such as uniform number) of the target player registered in advance by using the image recognition technique as described above. Needless to say, when detecting the player's number, not only the number on the back side of the bibs but also the number written on the front side may be detected. A position information creation unit 209 recognizes the position, direction and angle of view of each camera from the position information of the cameras 101, 102, 103 and the like using GPS and the information on the direction and angle of view of the camera. Then, based on the video from each camera, absolute position information on the ground where the athlete is is acquired by the triangulation method. The position information creation unit 209 also acquires in advance the positions on the screen, such as poles and competition field lines (for example, side lines and end lines) as reference indices for detecting reference positions, which are installed in advance in the stadium. You may keep it.

Then, using these as reference coordinates, the absolute position of the player of interest on the field in the stadium may be acquired.
Reference numeral 210 denotes camera position information for detecting the position of each terminal and the direction and angle of view of the camera of each terminal from the position information, direction information, and angle information of each terminal sent from each terminal 401-403. It is a direction detector.
Reference numeral 211 denotes a CPU (Central Processing Unit) as a computer, which is a central processing unit that executes control shown in the following embodiments based on a control computer program stored in a program memory 712 as a storage medium. ..

It also serves as a display control unit and controls information displayed on the display unit 214 described later. A data memory 213 stores various data referred to by the CPU 211. The data memory 213 stores past match information, past player information, information about today's match (competition), information about the number of spectators, weather, etc., information about the player of interest, the current situation of the player, etc. .. The information about the player of interest includes information such as face, uniform number, and physique.
1101 is a data bus line in the server 110.

Next, details of the terminal side (camera) as the image processing apparatus will be described with reference to FIGS. 3 and 4. 3 and 4 are block diagrams showing a configuration example of a terminal, and show an overall configuration of a (digital) camera 500 using two drawings as an example of the terminal.
The terminal side (camera) as the image processing apparatus shown in FIGS. 3 and 4 is capable of shooting moving images and still images and recording shooting information. 3 and 4, a CPU (Central Processing Unit) 318, a program memory 319, and a data memory 320 are shown redundantly, but these are the same block and only one is incorporated therein.
In FIG. 3, 301 is an Ethernet (registered trademark) controller. A storage medium 302 stores moving images and still images (photos) taken by a digital camera in a predetermined format.

Reference numeral 303 denotes an image sensor as an image sensor such as CCD or CMOS, which converts an optical image from an optical signal into an electric signal, and further converts this information from analog information into digital data and outputs it. A signal processing unit 304 performs various corrections such as white balance correction and gamma correction on the digital data output from the image sensor 303 and outputs the digital data. A sensor driving unit 305 controls horizontal and vertical line driving for reading information from the image sensor 303, timing at which the image sensor 303 outputs digital data, and the like.

306 is an operation unit input means. Input is made in response to selection and setting of various conditions for shooting with a digital camera, a trigger operation for taking a picture, a selection operation for using a flash, an operation for replacing a battery, and the like. Further, in the operation unit input means 306, it is possible to select/set whether or not to perform AF (autofocus) on the player of interest based on the position information from the server. The selection/setting information as to whether the focused player is to be AF (autofocused) is output from the operation unit input unit 306 to the bus line 370.
Furthermore, the operation unit input means 306 can select/set whether or not to automatically track the player of interest based on the position information from the server. Information such as which player is designated as the target player (specific target) and whether or not the automatic tracking of the target player is executed based on the position information from the server is generated by the operation unit input unit 306 as the selection unit. It That is, the operation unit input unit 306 functions as a designation information generation unit that generates designation information regarding a specific target.

A wireless communication unit 307 functions as a transmitting/receiving unit, and is used by a camera terminal owned by a professional photographer or a general audience to wirelessly communicate with the server side. A magnification detection unit 308 detects the photographing magnification of the digital camera. An operation unit output unit 309 is for displaying UI information such as a menu and setting information on the image display unit 380 that displays image information captured by a digital camera or the like. Reference numeral 310 denotes a compression/expansion circuit, which develops digital data (RAW data) from the image sensor 303 by the signal processing unit 304 and then compresses it by the compression/expansion circuit 310 to form a JPEG image file or a HEIF image file, or The raw data is compressed as it is to form a raw image file. On the other hand, when a RAW image file is developed in the camera to generate a JPEG image file or a HEIF image file, a process of decompressing compressed information and returning it to RAW data is performed.

Reference numeral 311 is a face recognition unit, which refers to face photograph information registered in advance on the server for the player of interest and finds the player in the video by AI, in particular, image recognition using a technique such as Deep Learning. Is. Information regarding the face recognition result detected by the face recognition unit 311 is fetched by the CPU 318 via the bus line 370.

Reference numeral 312 denotes a physique recognizing unit, which refers to physique photo information registered in advance with the server for the player of interest and finds the player of interest in the video by the image recognition technique as described above.
A player's jersey number detection unit 313 finds a player from the player's jersey number of the player of interest (of course, the number on the front side) by the image recognition technique as described above. A direction detector 314 detects the direction in which the lens of the terminal is facing. A position detection unit 315 detects the position information of the terminal by using, for example, GPS.

A power management unit 316 detects the power state of the terminal, and supplies power to the entire terminal when the power switch is off and the pressing of the power button is detected. Reference numeral 318 denotes a CPU as a computer, which executes the control shown in the following embodiments based on a control computer program stored in a program memory 319 as a storage medium. It also serves as display control means and controls the image information displayed on the image display unit 380. The image display unit 380 is a display unit using liquid crystal or organic EL.
The data memory 320 is for storing the setting conditions of the digital camera, and for storing the photographed still image and moving image, and further, the attribute information of the still image and moving image.

In FIG. 4, reference numeral 350 denotes a taking lens unit, which includes a fixed first group lens 351, a zoom lens 352, an aperture 355, a fixed third group lens 358, a focus lens 359, a zoom motor 353, an aperture motor 356, and a focus motor 360. The fixed first group lens 351, the zoom lens 352, the diaphragm 355, the fixed third group lens 358, and the focus lens 359 form a photographing optical system. Although each of the lenses 351, 352, 358, and 359 is illustrated as a single lens for convenience, it may be composed of a plurality of lenses. Further, the taking lens unit 350 may be configured as an interchangeable lens unit that can be attached to and detached from the digital camera.

The zoom controller 354 controls the operation of the zoom motor 353 and changes the focal length (angle of view) of the taking lens unit 350. The diaphragm control unit 357 controls the operation of the diaphragm motor 356 and changes the aperture diameter of the diaphragm 355.

The focus control unit 361 calculates the defocus amount and the defocus direction of the taking lens unit 350 based on the phase difference between the pair of focus detection signals (A image and B image) obtained from the image sensor 303. Then, the focus control unit 361 converts the defocus amount and the defocus direction into the drive amount and the drive direction of the focus motor 360. The focus control unit 361 controls the operation of the focus motor 360 based on the drive amount and the drive direction, and drives the focus lens 359 to control the focus of the photographing lens unit 350 (focus adjustment).

In this way, the focus control unit 361 implements phase-difference detection-type automatic focusing (AF). The focus control unit 361 may perform contrast detection AF in which the peak of the contrast of the image signal obtained from the image sensor 303 is searched.
Reference numeral 371 is a tracking unit for tracking the player of interest with the digital camera itself. The tracking here means, for example, moving a frame display surrounding a focused player on the screen, and focusing or exposing the focused player tracked by the frame.

Here, the tracking unit 371 will be described with reference to FIG.
FIG. 5 is a block diagram showing a functional configuration example of the tracking unit 371 of the digital camera. The tracking unit 371 includes a collation unit 3710, a feature extraction unit 3711, and a distance map generation unit 3712. The feature extraction unit 3711 identifies the image area (subject area) to be tracked based on the position information sent from the server. Then, the feature amount is extracted from the image of the subject region. On the other hand, the collation unit 3710 refers to the extracted feature amount in the continuously supplied captured images of individual frames, and searches for a region having a high degree of similarity to the subject region of the previous frame as a subject region.

Note that the distance map generation unit 3712 can also acquire distance information from the pair of parallax images (A image and B image) from the image sensor to the subject, and increase the accuracy of identifying the subject region in the matching unit 3710. However, the distance map generator 3712 is not always necessary.
The matching unit 3710 uses, for example, template matching or histogram matching when searching a region having a high degree of similarity to the subject region as a subject region based on the feature amount of the subject region in the image supplied from the feature extraction unit 3711. ..

Next, an example of the attention player display start sequence will be described with reference to FIG. This is a sequence performed by the server 110 and the camera 500. FIG. 6A is a diagram showing an example of a sequence in which the server 110 side answers an inquiry (request) from the camera 500 side. Then, the server 110 side provides the camera 500 side with information regarding the absolute position of the player of interest.
The camera 500 notifies the server 110 of attention player designation information (ID information such as a uniform number and a player name). At that time, the user may touch the position of the player of interest on the screen of the terminal, or may surround the player of interest with his/her finger on the screen.

Alternatively, you can display a list of multiple players on the screen in the menu and touch the name of the player you are interested in, or you can display the character input screen on the screen and enter the player name or player's uniform number. You may. At this time, touching the face position on the screen of the player of interest may image-recognize the face or image of the uniform number to send the player name or uniform number. Alternatively, the face image itself may be sent to the server without image recognition, and the image may be recognized on the server side. At this time, if there is a predetermined password, it will also be sent to the server. On the server side, a block that supports capturing an image sends information about the absolute value position of the player to the camera based on the player specification information (ID information such as the player's number and player name). If the password is also sent from the camera, the content of the information sent to the camera will be changed accordingly.

Further, in FIG. 6(A), the camera sends information to the server such as position information of the camera photographed by the professional photographer or the audience, the direction of the camera, the magnification of the camera, and the like. On the server side, a free-viewpoint image is made in the position and direction the camera is looking at, and the image seen by the actual camera is recognized from the magnification of the camera. Position information about the position of the player in the video actually viewed by the camera, contour information of the player, and the like are sent to the camera. Based on the position information, contour information, etc. sent from the server, the camera displays the player of interest on the screen of the display unit of the camera so as to stand out with higher accuracy and performs AF and AE on the player of interest.

An example of the attention player display start sequence for finding the attention player is briefly explained, but there are many cases where it is desired to continue chasing the attention player. Therefore, next, a sequence for continuing the tracking of the player of interest will be described with reference to FIG. In FIG. 6B, the camera 500 serving as a terminal repeatedly inquires (requests) to the server 110, for example, periodically to continuously recognize the position of the player. In FIG. 6(B), the camera sends a focused player display start sequence (A1, B1,...) To the server periodically, and the server periodically sends a focused player display start sequence (A2, B2,. The operation of receiving and recognizing the position of the player of interest is repeated many times.

Further, FIG. 7 is a diagram showing another example of the focused player display start sequence on the camera side, and shows a method of automatically tracking the focused player by the camera itself. The camera 500 sends the ID information of the player of interest to the server 110, and temporarily acquires the position information of the player of interest from the server. After the acquisition, after narrowing down the position of the player of interest by referring to the position information, the camera 500 itself continuously tracks the player of interest by image recognition. At that time, tracking is performed by the tracking unit 371 shown in FIGS. In the focused player display tracking sequence of FIG. 7, the camera 500 itself tracks the focused player using the image recognition technology. However, when the focused player loses sight of the focused player in the middle (when the tracking fails), the camera server Then, request the position information of the noted player again. Specifically, when the camera loses sight of the attention player, the camera sends the attention player display start sequence (A1) again to the server, receives the attention player display start sequence (B2) from the server, and notifies the attention player of the attention player. Display position on screen. After that, the camera again tracks the player of interest by image recognition.

It should be noted that in the present embodiment, an example of a service for assisting shooting by a professional photographer or a spectator is described, but it may be used for remote camera control. By sending this information from the server, it is also possible to track the player and shoot the decisive moment with the remote camera mounted on the automatic pan-tilt pan head.
Further, although the present embodiment has been described by using the example of shooting assistance, the terminal may be a home TV. That is, when the spectator watching the TV specifies the player of interest, the server may send the position information of the player of interest to the TV and display the player of interest in a framed manner or the like. Note that not only the frame but also the cursor (for example, an arrow) may indicate the player of interest, and the color and brightness of the region of the player position of interest may be different from those of other portions. When the player of interest is outside the screen of the terminal, the direction in which the player is shifted from the screen of the terminal may be displayed by arrows or characters. Also, if you are outside the display screen, how far the terminal is from the angle you are currently looking at (deviation), how long the terminal must be rotated before the attention player can enter the display screen, etc. , May be displayed with numbers or scales.

Furthermore, when the player of interest is on the screen, additional information is controlled to be displayed on the screen, and when the player of interest moves outside the screen, the user should not display that the player is outside the screen with an arrow or the like. May be selectable.
Alternatively, if the player of interest moves to the outside of the screen even if the player of interest drops to the bench by automatically judging the competition situation, the fact that the player is outside the screen is not displayed with arrows etc. Good. If the user can select a mode in which the display of additional information is automatically erased and a mode in which it is not erased, the usability is further improved.
It should be noted that although the main purpose of the present embodiment is to assist the cameraman in shooting, this part may be a robot camera in the future. If you get this much information from the server, the robot camera mounted on the automatic platform can track the player and take a decisive moment on behalf of the cameraman.

The captured still image (photograph) becomes an image file compressed by JPEG in the compression/decompression circuit 310 after image processing.
In this embodiment, various metadata are added to a still image (photograph) and recorded. The metadata includes, for example, shooting date/time, shooting position information (latitude, longitude, altitude, direction), shooting information (aperture value, exposure time, ISO sensitivity, shutter speed, photometric method, etc.). , Lens focal length, presence/absence of flash, exposure correction value, camera shake correction, color space, etc., and camera information (camera model name, lens name), and the like included. These pieces of metadata are recorded in the image file in, for example, the Exif (Exchangeable Image File Format) format.

FIG. 8 is a diagram showing an example of a file structure. FIG. 8A shows a structure of a DCF (Design rule for Camera File system) thumbnail file in this embodiment, and FIG. 8B shows a structure of a DCF basic file. FIG.
As shown in FIG. 8A, the DCF thumbnail file is composed of SOI (Start Of Image), Exif information, thumbnail file, EOI (End Of Image), and the like. On the other hand, as shown in FIG. 8B, the DCF basic file is composed of SOI, Exif information, thumbnails, DCF basic main image, EOI, and the like.
Here, SOI (Start Of Image) represents the beginning of JPEG data, and EOI (End Of Image) represents the end of JPEG data. In addition, App1 (Application marker segment 1) stores Exif information.

In this embodiment, as the Exif information, when a player is photographed, the player name is also stored in the image file as the Exif information. Furthermore, the match card, the match venue, the match time, the elapsed time in the match, and the like may be stored.
It is also possible to determine who the photographed player is and store detailed information specific to the player from the database as metadata. When a player decides a try, for example, the timing of that player's appearance, the time in the current match, and the action record of the player during the match ("~The player participates in the starting team and decides one try in the first half 36 minutes. .)) or the like may be stored as metadata.

It should be noted that the server side determines the general information (date, competition card, match venue, match time, shooting time, elapsed time in the match, when shooting the player, player name, etc.) by the metadata creating means. Create metadata for information and post-match information.
In the present embodiment, it is assumed that the camera side stores the above-described metadata regarding general information and cameraman's voice information as metadata corresponding to a still image (photograph). Further, the server side has a function of storing all information (general information, determination information, match end information, professional cameraman voice information, etc.). As the metadata corresponding to the still image (photograph), all information (general information, determination information, match end information, professional cameraman voice information) may be stored on both the camera side and the server side.

In addition, in the present embodiment, as the metadata corresponding to the photograph, the camera side stores only the latest metadata for each photograph, and the server side stores not only the latest metadata for each photograph but also the past. The change history of may be included and stored.
Further, instead of adding and storing the metadata as described above only to the photos of the players of interest, the metadata may be added and stored to the photos of all the players.
The server can further add judgment information by the referee or the like, or information after the end of the game to the metadata such as general information added to the photograph and voice information of the cameraman. In this way, when the metadata is added on the server side, the fact may be notified to the camera one by one.

Further, the server may transfer the photo and the metadata to a predetermined photo agency (other than the image processing device) as a destination when the photo (image) is received from the camera (image processing device). In that case, when the metadata of the photo is changed, the photo and the metadata may be resent together to the photo agency.
In this embodiment, when a photograph of a particular player of interest is taken, the information of the match and the information about this particular player are sent as metadata from the server to the device terminal (camera) as one of the destinations. Further, it is sent to a photo agency or the like registered in advance as a destination other than the camera (image processing device). Then, the cameraman additionally inputs various kinds of metadata such as match information before the start of the match, and further adds voice information of the cameraman as metadata to the taken image (photograph) at the time of shooting. Further, when the photograph with the metadata is transmitted to the server, the determination result by the referee or the like is input as the metadata on the server side.

Such metadata enhances the added value of the photograph. Moreover, metadata can be added in a short time, and high-value-added photos can be quickly sent to the news media.
Further, information such as comments by the cameraman's voice can be corrected or combined with the information recognized by the camera and the information held by the server.

Next, an example of the control flow on the camera side in FIG. 7 will be described using the flowcharts in FIGS. 9A, 9B, and 10. 9A, 9B, and 10 show a focused player display tracking control flow on the camera side.
In FIG. 9A, S101 represents initialization. In S102, it is determined whether or not photo shooting (still image shooting) is selected. If photo shooting is selected, the process proceeds to S103, and if photo shooting is not selected, the process proceeds to S101. In S103, camera setting information is obtained. In S104, it is determined whether or not shooting (designation) of the player of interest is selected. If shooting of the player of interest is selected, the process proceeds to step S105. If shooting of the player of interest is not selected, the process proceeds to step S110, and other processing is performed. To do. In step S105, if there is a player information of interest (ID information of the player of interest) and a password, the password is sent from the camera to the server. As a result, the server side detects the position information of the player of interest and sends it to the camera. In S106, the position information of the player of interest is received from the server.

In S107, the camera itself tracks the player of interest while referring to the position information sent from the server. Here, for example, image recognition is performed by the camera itself to track the player of interest. In that case, the player is tracked based on the recognition result such as the player's uniform number, the player's face information, the physique of the player, or a combination thereof. That is, a part or the whole shape of the player of interest is image-recognized and tracked. However, if the user's shooting position is poor, the field of view of the camera is narrow, or it may be hidden behind other subjects depending on the shooting angle, you may lose sight of it, and if you lose sight of it, you may see it again on the server. You will be sending a request for location information.

S107-2 has shown the example of the mark display as additional information with respect to an attention player. That is, as the additional information, a cursor indicating the player of interest is displayed, a frame is displayed at the position of the player of interest, the color or brightness of the position of the player of interest is conspicuously changed, or a combination thereof is displayed. You may display in characters other than the mark. Then, while the live view image from the image sensor is being displayed on the image display unit, the additional information indicating the position is superimposed on the player of interest.

FIG. 9B shows a specific example of the mark display flow in S107-2. In S120, the relative position of the noted player on the display unit is calculated and calculated based on the position information received from the server. In S121, a mark or the like indicating the position of the player of interest is superimposed on the image while the live view image from the image sensor is displayed on the image display unit.
The tracking operation in S107 as described above may be skipped, and the user may select it with the selection switch so as not to execute it. Alternatively, a mode may be provided in which the tracking operation is executed when the player of interest is in the screen, but the tracking operation is not performed when the player is out of the screen, and the mode can be selected. Furthermore, it automatically determines the competition situation, and automatically stops the tracking operation (display additional information such as arrows) of the target player outside the screen when, for example, the target player enters the bench. You may control. Alternatively, regardless of whether it is on-screen or off-screen, when the server knows that the target player has entered the bench, the display of the position of the target player on the screen, autofocus for the target player, and automatic exposure adjustment for the target player are stopped. It may be controlled to do so.

In S108, it is determined whether or not the tracking continuation of the attention player is OK (successful). If the tracking continuation of the attention player is successful, the process returns to S107, the position of the attention player is recognized again, and attention is paid. Continue shooting athletes. On the other hand, if the tracking fails, the result of S108 is No, and the process proceeds to S109.
In S109, it is determined whether or not the shooting of the noted player is completed. When the shooting of the noted player is completed, the process returns to S101, and if the shooting of the noted player is continued, the process proceeds to S111 of FIG.

In S111 of FIG. 10, it is determined whether or not the image of the selected player of interest has been actually taken. If the image of the player of interest has been taken, the process proceeds to S112, in which audio information of the cameraman at the time of image capturing is not shown. Input from the microphone. If the player of interest has not been photographed, the process proceeds to S107. In S113, the voice information input in S112 is converted into character information by voice recognition and stored. In S114, the metadata (general information) of the player of interest is input from the server. It should be noted that the information (for example, contents of general information) regarding the metadata acquired from the server can be displayed when the user selects it by the menu operation on the display device on the camera side. Alternatively, if the user has made a preset operation through a menu operation, it may be automatically displayed on the screen of the display device. At that time, other information such as the position information of the player of interest may be displayed at the same time, or only the information regarding the metadata acquired from the server may be selectively displayed. Alternatively, in S115, the voice information converted into text information is used as metadata for the photographed picture. In S116, the metadata acquired in S114, S116, etc. is added to the captured image (photograph) as Exif information and recorded on the recording medium. In step S117, the photograph to which the above metadata is added is sent to the server, and then the process proceeds to step S107 of FIG.

Next, a second embodiment which is a modified example of FIG. 10 will be described with reference to the flowchart of FIG. 11 showing an example of the focused player display tracking control flow on the camera side. In FIG. 11, the same reference numerals as those in FIG. 10 indicate the same steps, and the description thereof will be omitted.
In the second embodiment, the metadata added at the time of shooting (voice information converted into character information, etc.) is replaced with metadata such as judgment information of the referee.
That is, in S118 of FIG. 11, it is determined whether or not the metadata of the taken photograph has been updated on the server side. Then, when the metadata of the photographed photograph is updated, the process proceeds to S119, and the metadata at the time of photographing (such as voice information at the time of photographing converted into character information) is replaced with the judgment information of the referee. That is, it updates. In addition, the information about the metadata acquired from the server (for example, the content of the updated metadata) can be displayed when the user selects it by the menu operation on the display device on the camera side. If the metadata of the photographed photograph has not been updated, the process proceeds to S107 in FIG. If the photographed player is not photographed in S111, the process proceeds to S118.

Although the metadata is rewritten according to the determination information in the second embodiment, the determination information may be added as metadata to the photograph and recorded. In this way, the judgment information such as the referee can be additionally saved with the photograph along with the voice memo of the photographer, etc., so that various additional information for the photograph remains, and the analysis and selection of the photograph (image) will be performed later. To help you.
When the judgment information of the referee or the like is stored in the photograph as metadata, the judgment information may be displayed as character information on the screen. This allows the cameraman to confirm on the screen what kind of determination has been made for the photographed image. For example, specifically, it is possible to display on a screen such as "-the player started from the 5th minute of the second half, and the try succeeded in the 35th minute of the second half, but the judgment result was no try."

Similarly, the metadata of the voice information of the cameraman at the time of shooting, the determination information of the referee, and the metadata of the result information after the match may be added to the photograph as metadata and stored, and may be displayed on the screen.
In addition, for example, information such as the time when a player participated in a match as a starting member and retired due to a player change, and the time when a player participated in a match due to a player change was added to the photograph as metadata that focuses on the player and stored or displayed. You may do it.

As explained above, if you upload a photo with metadata from the camera to the server and then change or add metadata (judgment information, match result information, etc.) to the same photo on the server side, The side may only change the metadata.
It should be noted that the server may store not only a general player information about registered players and the like as a database but also a metadata-converted database.

In the embodiment, the server 110 can read the video of the entire competition field and acquire the coordinates, so that the image captured by the professional photographer or the spectator can be analyzed. Thereby, it is possible to know where the shooting is performed from the competition field. That is, the server grasps the entire image of the competition field in advance from a plurality of cameras (fixed camera and moving camera) for the server. As a result, it becomes possible to map the absolute position information of the player of interest in the field and the image viewed by the professional photographer or the audience with the terminal or the digital camera.

Also, when a terminal such as a professional photographer or a camera of a spectator receives the absolute position information of the player from the server, it becomes possible to map the absolute position information with the image currently being shot or monitored. Note that, for example, the absolute position information of the player of interest in the field from the server is (X, Y). It is necessary to convert this absolute position information into relative position information (X′, Y′) when viewed from the camera according to the position information of each camera. The conversion from the absolute position information to the relative position information may be performed on the camera side as in S120, or the relative position information may be sent to each terminal (camera or the like) after conversion on the server side. ..

When the above conversion is executed by a terminal such as a camera, the relative position information (X′, Y) is calculated from the absolute position information (X, Y) sent from the server according to the position information using the GPS of each camera. ´). Based on this relative position information, position information on the display screen on the camera side is set.
On the other hand, when the server performs the above conversion, the server follows the relative position information (X′, Y′) of each camera from the absolute position information (X, Y) according to the position information using the GPS of each camera. Y'). The server sends this relative position information to each camera, and the camera that receives this relative position information sends this relative position information to the position information on the display screen of each camera. And

As described above, it is possible to take a good picture of the player of interest without missing the timing, since the player of interest can be less likely to be lost on the terminal such as the camera of the professional photographer or the camera of the spectator.
With the above-described configuration, the cameraman or the camera owned by the general public sends information about the player of interest to the server, and tracks the player of interest based on the information acquired from the server. In addition to this, the position of the player of interest can be detected by the cameraman or the camera itself held by the general audience.

For example, in rugby competition, as metadata, for example, information 48 hours before the start of the game, change information several hours before the start of the game, information such as change of players during the game, and voice comment information of the cameraman at the time of shooting And so on. Further, information about the judgment of the referee, information about the game result at the end of the game, information obtained by the server from the rugby association, etc. can be considered. Further, there is information obtained as a result of obtaining the device information of the camera and the like and analyzing the device information of the camera by the server, and information obtained from a photographer such as a cameraman.

These metadata are integrated and stored in the server, and necessary information is sent to the camera. The camera adds the information acquired from the server as the metadata of the photo.
The server can also determine who the photographer is and store player-specific information as metadata. For example, the participation timing of the player, the time in the current match, and the action of the player (“AAA player participates in the starting team and decides one try in the first half 36 minutes.”) may be stored as metadata. ..

Next, a control flow for detecting a player of interest on the server side, which corresponds to the first embodiment shown in FIG. 10, will be described with reference to FIGS. 12 and 13. Here, the case where the audio information of the cameraman is included as metadata at the time of shooting will be described.
Based on the ID information of the player of interest sent from a terminal such as a camera, the server recognizes the image of the player of interest and searches the screen for the player of interest. The server detects the player's position information based on images from multiple cameras for the server (fixed camera, moving camera, etc.) and sends the player's position information to professional cameramen and camera terminals of spectators. .. In particular, when a professional photographer or an audience shoots, if the position information of the player of interest is from the server, the player of interest can be surely shot without making a mistake. Furthermore, the information from the server is also important when the player loses sight of a player who becomes a blind spot while the camera is tracking the player of interest. It should be noted that the server side continues to detect the player's position information based on the images from the multiple cameras for the server.

FIG. 12 shows a main flow of detection control of a player of interest on the server side, which corresponds to the first embodiment.
In FIG. 12, initialization is first performed in S201. Next, in step S202, it is determined whether or not photography is selected by the camera. When photography is selected, the process proceeds to step S203, and camera setting information is obtained. At this time, if there is a password in the setting information of the camera, it is also obtained. If photography is not selected, the process proceeds to S201. In S204, it is determined whether or not shooting (designation) of the player of interest is selected. If shooting of the player of interest is selected, the process proceeds to S205, in which the server uses the camera to obtain ID information (for example, player name, profile) of the player of interest. Numbers). If shooting of the player of interest is not selected in S204, the process proceeds to S210 and other processing is performed.

In S206, the server finds the player of interest in the screen by image recognition based on the images from a plurality of cameras (fixed camera, moving camera, etc.) based on the ID information of the player of interest. In S207, the server tracks the player of interest based on the images from the plurality of cameras. In S208, it is determined from the information from the camera whether or not there is a photo shoot (still image shooting) of the noted player, and if there is an actual shoot of the noted player, the process proceeds to S208-2, where an actual photo of the noted player is taken. If there is no shooting, the process returns to S205.
In step S208-2, the photo with the metadata at the time of shooting is received from the camera. Here, the metadata includes metadata such as voice information of the cameraman when the shutter is released by the cameraman and other general information.

In S208-3, the photograph of S208-2 is sent to the photo agency, which is a predetermined destination registered in advance.
As a result, the photo agency can receive the photograph with the taken metadata at an early timing, and can utilize the photograph information.
In S209, it is determined whether or not the shooting of the noted player is completed. When the shooting of the noted player is completed, the process returns to S201, and when the imaging of the noted player is continued, the process returns to S205. Then, the server again recognizes the information from the cameraman or the camera held by the general audience. Then, the information from the cameras for the plurality of servers (fixed camera or moving camera) is searched for to find the player of interest, and in S207, the server continues, based on the images from the plurality of cameras, of the player of interest. Carry out tracking.

If photography is not selected in S202, it is determined in S211 whether metadata input has been selected. If metadata input has been selected, the process proceeds to S212, metadata input is performed, and the metadata is input. If the data input is not selected, the process returns to S201.
The metadata input in S212 is input before the start of the rugby match, and the metadata at that time is, for example, information before the match start (48 hours ago), which was performed prior to the start of the match. For example, there is a change in participating players (information several hours ago).

As a result, the metadata information before the start of the match is temporarily stored on the server side in advance. Then, when the cameraman actually takes a picture, that is, when the shutter is released, it becomes possible to add the audio information of the cameraman as metadata together with the photograph to the photograph on the server side.
This makes it possible to leave the voice memo of the photographer as metadata of the photograph, and, for example, information about the game before the start of the game is automatically entered on the server without the cameraman entering it. Become. Therefore, the photographer can concentrate on shooting and voice memos, and can send a photograph with audio metadata immediately after shooting to a news agency.

Next, an example of a method for finding a target player in S206 and tracking the player in S207 will be described with reference to FIG.
FIG. 13 shows an attention player detection control flow using the uniform number information. In FIG. 13, in S401, the server obtains the uniform number from the data memory 213 based on the ID information of the player of interest, searches for the uniform number from the video information of the plurality of cameras for the server, and recognizes the uniform number. Get the position information of the player with the number. In S402, the absolute position information of the player of interest is acquired by further integrating the position information acquired from the images of the plurality of cameras for the server. In this way, by integrating the information of the plurality of cameras for the server, the accuracy of the absolute position information of the player with a certain uniform number is improved. In S403, the absolute position of the player of interest detected in S402 is transmitted to a terminal such as a camera owned by a professional photographer or a spectator. In S404, it is determined whether or not the tracking of the noted player is continued. If the tracking of the noted player is continued, the process returns to S401, and if the tracking of the noted player is not continued, the flow of FIG. 13 is ended.

Next, an example in which the voice information of the figure photographer is replaced with the determination information of the referee to be the metadata will be described with reference to the flowchart of FIG. FIG. 14 is a flowchart on the server side corresponding to the second embodiment on the camera side shown in FIG.
Since S201 to S207, S208-2, S208-3, and S209 to 212 are the same as those in FIG. 12, description thereof will be omitted.
In S208, it is determined from the information from the camera whether or not the player of interest is actually photographed, that is, whether or not the shutter is released. Then, if there is actual shooting of the player of interest, that is, if the shutter is released, the process proceeds to S208-2, and if there is no actual shooting of the player of interest, that is, if the shutter is not released, the process proceeds to S213.
In S213, it is determined whether or not the determination information input from the voice information of the cameraman exists in the server. If Yes, the process proceeds to S214, and if No, the process proceeds to S205.

In S214, the metadata of the voice information of the cameraman is changed to the judgment information of the referee. Also, since the metadata is changed, the name of the photo is updated.
In this way, the metadata acquired before the start of the match is stored on the server side, and when the cameraman actually shoots, the cameraman's voice information is added to the photo as metadata, Send to. On the server, it is possible to automatically rewrite the metadata when the judgment of the referee is different from the voice memo of the photographer.
In S214-2, the metadata in S214 is sent to the camera. As a result, the camera side can display the information (for example, the contents of the determination information) about the metadata sent from the server when the user selects it by the menu operation on the display device on the camera side.

This enables the camera to synchronize with the metadata on the server side. The camera does not change the photo (still image), only the metadata of the photo.
Next, in S214-3, the photo whose metadata and name have been changed in S214 is sent to the photo agency, and then the process proceeds to S205.
This allows photo agencies to receive photos whose name numbers have been changed when the metadata has changed, which makes the use of photo information more efficient and simplifies the work of the photographer, and speeds up the work. It becomes possible to send photos to the news agency.

Next, FIG. 15 is a flowchart on the server side corresponding to the third embodiment in which the voice information of the cameraman and the determination information of the referee are both metadata.
Since S201 to S207, S208-2, S208-3, and S209 to 212 are the same as those in FIG. 12, description thereof will be omitted.
In S208, it is determined from the information from the camera whether or not the player of interest is actually photographed, that is, whether or not the shutter is released. Then, if there is actual shooting of the player of interest, that is, if the shutter is released, the process proceeds to S208-2, and if there is no actual shooting of the player of interest, that is, if the shutter is not released, the process proceeds to S215.
In S215, it is determined whether or not the server has the determination information when the voice information of the cameraman is input. If Yes, the process proceeds to S216, and if No, the process proceeds to S205.
In S216, the metadata of the determination information is also stored in the metadata of the voice information of the cameraman. Well, the metadata has changed, so I'll change the name of the photo as well.

As a result, the metadata information before the start of the game is stored on the server side, the voice information of the cameraman when the cameraman actually took the picture is added as metadata to the photograph, and the referee It is possible to store the determination of as metadata.
In S216-2, the metadata of S216 is sent to the camera. As a result, the camera side can display the information (for example, the contents of the determination information) about the metadata sent from the server when the user selects it by the menu operation on the display device on the camera side.
This allows the camera to update the metadata to the latest version. The camera does not change the photo, only the photo metadata.
In S216-3, the photograph with the metadata changed in S216 is sent to the photo agency.

As a result, the photo agency can receive the photo whose name has been changed when the metadata is changed, and can quickly publish the photo by utilizing the photo and the metadata. Moreover, the work of the photographer is simplified, and the photograph can be immediately sent to the news agency.

Next, with reference to the flowcharts of FIGS. 16 and 17, an example will be described in which, as the information at the time of shooting, information in which voice information of a professional cameraman, determination information of a referee, and result information of a match are described as metadata.
16 is the same as FIG. 12 except for S209, and the description thereof will be omitted.

When the shooting of the player of interest is finished in S209, the process proceeds to S219 of FIG. In S219, it is determined whether or not the match is over. If the match is over, the process proceeds to S220, and if the game is not over, the process proceeds to S205.
In S220, it is determined whether or not the post-match metadata is present. If the post-match metadata is present, the process proceeds to S221. If the post-match metadata is not present, the process proceeds to S201.
In S221, metadata of audio information input by the cameraman, metadata of determination information, and metadata information after the end of the match are also added. The name of the photo will be changed as the metadata changes.
As a result, the voice information of the cameraman, the judgment result of the referee, and the information after the match can be added to the photograph as metadata.

In S221-2, the metadata of S221 is sent to the camera. As a result, the camera side can display the information (for example, the determination information and the match end information) about the metadata transmitted from the server when the user selects it by the menu operation on the display device on the camera side.
This allows the camera to update the metadata to the latest version. The camera does not change the photo, only the metadata.
In S221-3, the photo with the metadata of S221 is sent to the photo agency.
In the above embodiment, when uploading a photo having metadata from the camera to the server, the camera updates only the metadata for each photo. On the other hand, on the server side, the file of the photograph (captured image) is updated when the metadata is updated, so that the number of photographs becomes plural and the name of the photograph is updated.

Thereby, the voice information of the cameraman, the judgment result of the referee, and the match result information after the match can be stored in the photograph as metadata. The added value of photos can be increased.
For professional photographers and the general public, making decisions in the game is an issue. The referee is a three-member system consisting of one referee and two touch judges. In particular, it may not be possible to make a judgment only by the human eyes, such as a judgment in a scene that seems to have decided a try or a judgment as to whether a foul play has occurred in the game. For this reason, video judgment (TMO (television match official)) is performed to assist the judgment of the referee when it is difficult to judge with the naked eye.

When a tri-scene is shot with a terminal such as a camera owned by a professional photographer or a general spectator, the professional photographer or the general public wants to know immediately whether the captured image was certified as a trial or not. .. Therefore, the judgment is accurately recognized by following the judgment from the images of a plurality of cameras or by analyzing the information on the server based on the information displayed on the electronic bulletin board. Then, by transmitting the judgment result of the referee to a terminal such as a camera of a professional cameraman or a general audience, the user can correctly recognize the judgment result in a timely manner.

There are several methods for detecting whether or not a try has been performed. A try determination control flow (server side) which will be specifically described below is shown in FIG.
In FIG. 18, S1101 represents initialization. Here, the TRY judge flag is cleared. In S1102, it is determined whether or not photography is selected. If photography is selected, the process proceeds to S1103. If photography is not selected, the process proceeds to S1101. In S1103, camera setting information is obtained. In S1104, the balls used in the match are followed from the images of the plurality of cameras.

In S1105, it is determined whether or not the TRY judge flag is 0. If the TRY judge flag is 0, the process proceeds to S1106. If the TRY judge flag is not 0, the process proceeds to S1107. In S1106, it is determined whether there is a try. If there is a try, the process proceeds to S1107, and if there is no try, the process proceeds to S1112. The presence of a try here means that the player is bringing the ball to the try area. For example, a player may be knocking on the ball just before the try, or the defending player may put his hand or body into the ball he is trying to try to prevent grounding. is there. That is, the try is not in a fixed state. In step S1107, the TRY judge flag is set to 1. In S1108, it is determined whether or not there is a try.
FIGS. 19, 20, 21, and 22 show specific examples of determining a try, which will be described later.

In S1109, in the control of S1108, it is determined whether or not the determination result of the presence/absence of a try has been obtained. Proceed to. In step S1110, the TRY judge flag is set to 0. In S1111, the server sends information on whether or not there was a try to a terminal such as a camera owned by a professional cameraman or a general audience. In S1112, it is determined whether or not the game is over. If the game is over, the process proceeds to S1101, and if the game is not over, the process proceeds to S1104.
The above is the control for determining the presence/absence of a try. However, the CPU may perform not only this control but also the detection control of the player of interest shown in FIG.

It should be noted that the control executed by the server in parallel is not limited to this, and a plurality of other controls may be executed simultaneously. On the other hand, the same applies to terminals such as cameras held by professional photographers and general spectators, and a plurality of other controls may be performed simultaneously.
When shooting a tri-scene, you must correctly recognize whether the try succeeded or failed. The server checks to see if the try was successful or the conversion was successful. Here, the example of the trial has been described. However, the same control can be executed for other score scenes as well as the try.

If there is a play that seems to have been tried by analyzing the images of multiple cameras, the movement of the ball was analyzed with multiple cameras to see if the ball was properly grounded in the predetermined area. Recognize whether or not. The server sends information about whether or not it was a try from the recognized movement of the ball to the terminal such as a camera owned by a professional cameraman or a general audience together with player information.
FIG. 19A shows a flow of determining whether or not there is a try from the movement of the ball on the server side.
In FIG. 19(A), in S1201, the server detects a place where the ball exists from the images of the plurality of cameras. In S1202, it is recognized from the movement of the ball whether or not there is a try in a scene considered to be a try from the images of the plurality of cameras.

Next, the trial presence/absence determination flow from the referee operation is shown in FIG. 19(B).
When there is a play that seems to have been performed by analyzing the images of multiple cameras, the action of the referee around the attention player after that is analyzed from the images of the multiple cameras and the image is analyzed. By recognizing, it may be recognized whether or not it was a try from the action of the referee. The server sends the information (motion recognition result) regarding the analysis result of whether or not the motion of this recognized referee was a try, together with the player information to a terminal such as a camera owned by a professional cameraman or a general audience.

In FIG. 19B, in S1301, the server detects the referee's motion near the motion that makes a try determination from the images of the plurality of cameras. In S1302, whether or not there is a try in a scene considered to be a try from the images of the plurality of cameras is recognized from the referee's action.
FIG. 20 illustrates the referee operation for determining whether or not there is a try. In FIG. 20, FIG. 20(A) shows the referee's operation when the try is established. FIG. 20B shows the referee's operation when the try is not established.

Furthermore, a flow of recognizing the presence or absence of a try from the information displayed on the large screen of the stadium will be described with reference to FIG.
When there is a play that seems to have been attempted by analyzing the images of multiple cameras, enter the information displayed on the large screen of the stadium with multiple cameras and try from this screen information Recognize whether or not The server sends information indicating whether or not it was a try from the recognized information on this screen to a terminal such as a camera owned by a professional cameraman or a general audience together with player information.

FIG. 21A shows a flow on the server side for judging the presence/absence of a try from the judgment result of screen display. In FIG. 21A, in S1401, the server detects the information of the determination result displayed on the screen after the movement such as the try from the images from the plurality of cameras. In step S1402, it is recognized from the determination result displayed on the screen whether or not there is a try in a scene considered to be a try from the images of the plurality of cameras.

Next, a flow of recognizing the presence or absence of a try from the score information displayed on the screen will be described with reference to FIG.
When there is a play that seems to have been tried by analyzing the images of multiple cameras, input the score information displayed on the screen based on the images of the multiple cameras, and from the score information of this screen Recognize whether or not it was a try. From the recognized score information on this screen, the server sends information about whether or not it was a try to a terminal such as a camera owned by a professional photographer or a general audience together with player information. If you try, you get 5 points, and if the conversion kick succeeds, you get 2 points.

Also, if you succeed in a penalty kick or a drop goal, you get 3 points. It is possible to recognize whether or not the try is successful by comparing the score before the try is considered to be performed and the score after the try is performed.
FIG. 21B shows a flow in which the server determines whether or not there is a try from the score information on the screen.

In FIG. 21B, in S1501, the server detects the score information displayed on the screen after the movement such as the try from the images of the plurality of cameras. In S1502, it is recognized from the difference in the score information displayed on the screen whether or not there is a try in a scene considered to be a try from the images of the plurality of cameras. In S1503, a try, a conversion kick, a penalty kick, or a drop goal is recognized from the difference in the score information displayed on the screen as to whether or not there is a try in a scene considered to be a try from the images of the plurality of cameras.

Next, a flow of recognizing the presence or absence of a try from the voice information announced in the field will be described with reference to FIG.
Analyze the voice information input from the microphones attached to multiple cameras (fixed camera or moving camera), and if there is a play that seems to have been performed, determine whether it was a try from the voice information from the above microphones. recognize. Based on the recognized voice information, the server sends information indicating whether or not it was a try to a terminal such as a camera owned by a professional cameraman or a general audience together with player information.
FIG. 16 shows a specific flow of determining the presence/absence of a try from the voice information by the server. In FIG. 22, in S1601, the server detects audio information after a try-like movement from the microphones of a plurality of cameras. In S1602, the microphones of the plurality of cameras recognize whether or not there is a try in a scene considered to be a try based on the voice information.

Although the determination of the score of the try is described in the above embodiment, in addition to the score of the try, the score by the conversion after the try and the score by the penalty kick may be considered.
The detection flow on the server side as to whether or not a try has been performed is shown in FIG. 23, but the control on the terminal side such as the camera regarding whether or not a try has been performed will be described.
FIG. 23 shows a try determination control flow on the side of a terminal such as a camera. In FIG. 23, S101 to S107-2 and S110 are the same as the steps of FIG.

In S1620, it is determined whether or not the tracking of the noted player is OK (successful). If the tracking of the noted player has succeeded, the process proceeds to S1621. In S1621, it is determined whether or not the try determination result is sent from the server. When the try determination result is sent from the server, the process proceeds to S1622. When the try determination result is not sent from the server, the process returns to S107 and the camera itself pays attention. Continue tracking players. In S1622, the try determination result is displayed on the display unit of the camera.

If it is determined in S1620 that the player of interest has not been continuously tracked, the process proceeds to step S109 to determine whether or not the shooting of the player of interest has finished. If not, the process returns to step S105. If it is finished, the process proceeds to S1623, it is determined whether or not the try determination result is sent from the server, and if the try determination result is sent from the server, the process proceeds to S1624 and the try determination result is displayed on the display unit of the camera terminal. To do. If the try determination result is not sent from the server, the process returns to S101.
As described above, when it becomes a try-like state, it becomes possible to display on the camera terminal side whether or not the try is successful. As a result, the general audience or the cameraman can correctly recognize the evaluation of the photographed picture, for example. Then, the cameraman can recognize the determination just by looking at the display section of the camera, so that the photographer can appropriately select the photograph to be sent to the news media and the like, and can quickly prepare for the next photographing.

When an offense occurs, multiple cameras (fixed camera or moving camera) recognize the offense situation, but this information is also sent to terminals such as cameras owned by professional photographers and general spectators and displayed on the above terminals. To be done.
Here, FIG. 24 shows a foul judgment control flow on the camera side. FIG. 24 is based on the try determination control flow on the camera side, and is for displaying the foul situation on the camera side.
24, only S7421 to S7424 are different from FIG. That is, S1621, S1622, S1623, and S1624 in FIG. 23 are replaced with S7421, S7422, S7423, and S7424, respectively, and the only difference is that the "trie determination" of each is "foul determination".

As described above, since the foul situation can be displayed on the terminal such as the camera owned by the professional photographer or the general audience, the professional photographer or the general audience can correctly recognize the situation of the photographed picture. In particular, professional photographers need to send photos to the press during the match, which is useful information for this photo selection.
As described above, according to the present embodiment, the server analyzes the surrounding information other than the player of interest (other than the specific target) and transmits the analysis result to the image processing apparatus such as the camera. Then you will be able to understand the real-time situation of the competition such as tries, goals and fouls. Therefore, in particular, a professional photographer or the like can obtain very useful information when selecting a photo and sending it to the news media in a timely manner during a match.

The player's movements may be stored in the server as big data, and AI's movement prediction may be performed based on this big data. In addition, although the example in which only one attention player is designated has been described, a plurality of attention players may be designated. In addition, the attention player can be switched on the way. All players who have participated in the match as players of interest may be used. Also, video and images include not only moving images but also still images. In addition, I have mainly explained the pursuit and tracking of attention players.

However, the information about the player who has the ball or receives the ball may be transmitted to the professional cameraman or the spectator and displayed without chasing only the player who has the attention. Further, although the embodiment has been described by using the example of tracking the player, it is needless to say that the present invention can be applied to a system for tracking a person such as a criminal using a plurality of surveillance cameras. Alternatively, it can be applied not only to a person but also to a system for tracking a specific car in car racing or the like, or a system for tracking a horse in horse racing or the like. In the embodiment, the example has been described in which the player of interest is specified by the camera terminal or the like, but the server may be allowed to specify the player of interest.

In the above embodiments, rugby has been described as an example, but it goes without saying that it can be applied to other sports and field work.
On the server side, the picture sent from the terminal device may be sent to, for example, a broadcasting station or the like via a network (not shown).
As described in the above embodiments, the cameraman wants to add various data as metadata at the time of taking a picture, but there is a risk that he cannot concentrate on the picture taking because of this. However, according to this embodiment, by sending the photograph to the server, various metadata for the photograph can be added or rewritten on the server side, so that the load of the photographer at the time of photographing is significantly reduced. This allows the photographer to focus on the shoot. In addition, the server side can add information that cannot be obtained in the past, such as general information related to the photograph and game determination information. Therefore, it is possible to increase the added value of each photograph, and also to improve the efficiency when the photographs are analyzed, selected, or edited later.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the invention.
Further, a program (software) for realizing a part or all of the control in the present invention to realize the functions of the above-described embodiments may be supplied to the image pickup apparatus or the information processing apparatus via a network or various storage media. Then, the computer (or the CPU, MPU, or the like) in the imaging device or the information processing device may read out and execute the program. In that case, the program and the storage medium storing the program constitute the present invention.

101, 102, 103... Cameras 401, 402, 403... Terminal 110... Server 371... Tracking unit 380... Image display unit

Claims (16)

Display means for displaying images,
Selecting means for selecting a specific object from the image displayed on the display means,
Designation information generation means for generating designation information relating to the specific target selected by the selection means,
Transmitting means for transmitting the designated information generated by the designated information generating means to a server;
Acquisition means for acquiring the position information of the specific target based on the designation information and the predetermined metadata created in the server from the server,
Control means for displaying on the display means additional information based on the position information of the specific target acquired by the acquisition means or information on the metadata;
An image processing apparatus comprising: The image processing apparatus according to claim 1, wherein the image processing apparatus transmits the photographed image to the server by the transmitting unit. The image processing apparatus according to claim 2, wherein the server adds the predetermined metadata to the image transmitted by the transmitting unit. The image processing apparatus according to claim 3, wherein the server transmits the image to which the predetermined metadata is added to a predetermined destination. The image processing apparatus according to claim 4, wherein the predetermined destination includes the image processing apparatus. The image processing apparatus according to claim 4, wherein the predetermined destination includes a destination other than the image processing apparatus. 6. The image processing apparatus according to claim 1, further comprising an input unit for inputting metadata, wherein the transmitting unit transmits the metadata input by the input unit to the server. The image processing device according to one item. The image processing apparatus according to claim 7, wherein the server has a metadata creating unit that creates the predetermined metadata based on the metadata transmitted from the image processing apparatus by the transmitting unit. The image processing apparatus according to claim 8, wherein the server replaces the predetermined metadata created by the metadata creating unit with the metadata sent from the image processing apparatus. The image processing apparatus according to claim 7, wherein the metadata input by the input unit includes audio metadata. 11. The image processing apparatus according to claim 10, wherein the input unit converts the voice into characters to make metadata. Receiving means for receiving designated information about a specific object sent from the image processing device,
Generating means for searching the image for the specific target based on the designation information received by the receiving means and generating data relating to the position of the specific target;
Metadata creating means for creating predetermined metadata,
An image processing server comprising: a transmission unit that transmits the position information of the specific target and the predetermined metadata generated by the generation unit to the image processing apparatus. A computer program for causing a computer to function as each unit of the image processing apparatus according to claim 1. A computer-readable storage medium storing the computer program according to claim 13. A computer program for causing a computer to function as each unit of the image processing server according to claim 12. A computer-readable storage medium that stores the computer program according to claim 15.

Images