JP2003179930A - Moving object extraction method and extraction device - Google Patents

Abstract

(57) [Problem] To provide a moving object extracting method and an extracting device in which a photographing area can cover a desired background and a moving object can have a certain size on a video. The purpose is to do. A moving object extraction method for extracting a moving object from video signals of a plurality of fixed background video cameras, mapping the moving objects to a video captured by a broadcast video camera, and extracting the moving objects, comprising: An identification step of identifying a moving object obtained from an area of a video signal overlapping between cameras; and, when the same moving object is extracted from a video signal of a different fixed background video camera, a center of the screen of the background video camera. Selecting a moving object extracted from a background image having a minimum distance between the moving image and the center of gravity of the moving object, and mapping an area of the moving object selected in the selecting step to a captured image of the broadcast video camera. And an extraction step for extracting a moving object.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving object extracting method and extracting apparatus, and more particularly, to a moving object for extracting a moving object in a video to realize object image coding and an object-linked data broadcasting system. The present invention relates to an extraction method and an extraction device.

[0002]

2. Description of the Related Art In order to realize an object-based image coding and a linked data broadcasting system for broadcasting a video linked to an object, it is indispensable to extract a meaningful or moving object from the video. Has become.

Some conventional moving object extraction methods require a special device such as a background in chroma keys, and some require a special sensor for a range finder.

On the other hand, as a technique for cutting out an object from a natural image acquired by a normal camera or the like, a method of dividing an image into regions based on luminance / color information or a motion vector field, or a method in which a human approximates a position / contour Given that, there is a method of cutting out an object based on the edge information, but among them, the background difference object extraction method that extracts the object by separating the background and the foreground using the statistical information of the luminance value in the time direction is the most It is stable and suitable for this purpose.

Further, in the practice of the conventional moving object extraction method, the camera must be fixed, and pan, tilt, roll, zoom, and
Since various camera operations such as focus and dolly are performed, there are restrictions on the use. However, for example, Japanese Patent Application No. 2001-19607 relating to the invention of the present inventor
As described in the No. 7 application, a fixed background video camera is installed separately from the broadcast video camera at the time of shooting, a moving object is extracted from the background video, and the broadcast video camera of the broadcast video camera is extracted for each frame. Repetition parameters (pan angle,
There is a method of measuring a tilt angle, a roll angle, a zoom ratio, a focus amount, a dolly amount), and extracting a moving object in a broadcast video by using the measured parameters and the extraction result from the background video. Accordingly, by extracting the moving object from the broadcast video, by extracting the moving object from the background video shot by the fixed camera, and projecting the object area between the two cameras of the fixed camera and the background video camera. , A high-precision moving object can be extracted from the background image, and the projective transformation process of the moving object area between the two camera images is a simple coordinate transformation process (mapping). It can be performed with high accuracy.

[0006]

However, in order to provide the contour of the moving object in the broadcast video with a certain degree of accuracy, it is necessary that the moving object in the background video has a certain size. Also, with one background video camera, there are restrictions in specifications or operation of the actual field of view and the distance of the background video camera, and it is possible to handle those that require a wide field of view. Can not. For these reasons, a single background video camera cannot cover a desired background. In addition, sufficient accuracy may not be ensured for the contour of the moving object.

The present invention has been made in view of the above-mentioned points, and a plurality of background image cameras are installed at the time of shooting, and the images from the plurality of background image cameras are connected to cover a desired background. And a moving object extracting method capable of extracting a moving object with high accuracy by allowing the moving object to have a certain size on the image while the shooting area of the background video camera covers a desired background. An object is to provide an extraction device.

[0008]

In order to solve the above problems, the present invention employs means for solving the problems having the following features.

The invention described in claim 1 is a moving object extraction method for extracting a moving object by extracting a moving object from video signals of a plurality of fixed background video cameras and mapping it to a video image captured by a broadcast video camera. There is an identification step of identifying a moving object obtained from a region of video signals that overlap between the plurality of fixed background video cameras, and the same moving object is extracted from video signals of different fixed background video cameras. A selection step of selecting a moving object extracted from the background image in which the distance between the center of the screen of the background video camera and the center of gravity of the moving object is minimum; and a region of the moving object selected in the selection step, It has a step of extracting a moving object by mapping it to a video image captured by a broadcast video camera. To.

According to the first aspect of the present invention, a desired background can be covered by using a plurality of fixed background image cameras. Also, by selecting a background image that minimizes the distance between the center of gravity of the moving object and the center of the screen, and mapping the moving object in the selected background image to the broadcast image, the moving object may have a certain size. Therefore, the extraction accuracy can be improved.

According to a second aspect of the present invention, in the identifying step, between background images in which the distance between the centers of gravity of the moving objects extracted from the image signals captured by the plurality of fixed background image cameras in the real space is different. Is smaller than or equal to a predetermined threshold value and is the smallest compared with the distance between the centers of gravity with other moving objects, and is set as the same moving object.

According to the second aspect of the present invention, it is possible to easily recognize whether or not the moving objects existing in the overlapping area of the background image are the same moving object.

A third aspect of the present invention is a moving object extracting apparatus for extracting a moving object from video signals of a plurality of fixed background video cameras, and mapping the video object to a video captured by a broadcast video camera to extract the moving object. In the case where the same moving object is extracted from the video signals of different fixed background video cameras, the identification unit that identifies the moving objects obtained from the regions of the video signals that overlap between the plurality of fixed background video cameras, A selection unit for selecting a moving object extracted from a background image having a minimum distance between the center of the screen of the background video camera and the center of gravity of the moving object; and a region of the moving object selected by the selection unit, And a extracting unit for extracting a moving object by mapping the image on the image captured by the broadcast image camera.

According to a fourth aspect of the present invention, the identifying section is arranged between background images having different distances between the centers of gravity of the moving objects extracted from the image signals captured by the plurality of fixed background image cameras in the real space. Is smaller than or equal to a predetermined threshold value and is the smallest compared with the distance between the centers of gravity with other moving objects, and is set as the same moving object.

According to the invention described in claims 3 to 4, it is possible to provide an extraction apparatus suitable for the moving object extraction method according to claim 1 or 2.

[0016]

BEST MODE FOR CARRYING OUT THE INVENTION Next, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram of an embodiment of a moving object extracting apparatus according to the present invention. The moving object extracting apparatus of FIG. 1 is configured to include a camera calibration circuit 1, a moving object extracting / tracking / identifying circuit 2, an automatic switching circuit 3, and a broadcast video moving object extracting circuit 4.

In FIG. 1, the camera calibration circuit 1 uses a target point whose coordinate position is clear in a three-dimensional space and the coordinate of the point corresponding to the target point to determine the focal length, the optical axis and the image plane. The camera internal parameters such as the image center position, which is the intersection of, and the lens distortion, are obtained.
It also estimates the position and orientation of the camera.

Moving object extraction / tracking / identification circuit 2
Extracts and tracks a moving object from each background image, and assigns the same identification number (ID) to those corresponding to the same moving object in the scene.

The automatic switching circuit 3 selects a background image so that an optimum moving object can be extracted from the same moving object obtained from a plurality of background images when the extracted moving object area is mapped to a broadcast image. .

Further, the broadcast video moving object extraction circuit 4
Maps the moving object area supplied to the automatic switching circuit 3 to the broadcast video.

With the above contents, it is possible to secure the shooting distance of the fixed camera installed, and it is also possible to extract a moving object with high accuracy from the background image.

Next, the operation will be described in detail.

FIG. 2 is a diagram showing an example of the relationship between the shooting ranges of a plurality of background video cameras.

In the present embodiment, as shown in FIG. 2, the images taken by the plurality of background image cameras are connected to each other, so that an appropriate position, direction, and image are displayed so as to cover the imageable range of the broadcast image camera. Install and fix the background video camera at the corner. In FIG. 2, the number N of background video cameras is N = 3 as an example, but the number of cameras is not limited to this. Also, the shooting is performed by synchronizing the operations of all installed cameras, and a video signal is output.

The camera calibration circuit 1 estimates internal parameters, position, and orientation of the camera by using characteristic points such as intersections of white lines drawn in a sports venue and their image coordinates.

In this embodiment, it is assumed that the moving object is on one plane in the three-dimensional space, and the feature points used in the calibration are those on the plane. In this case, the points in the three-dimensional space and the points on the image have a one-to-one projective relationship. Note that calibration is performed after installing the camera. In addition, the internal parameters of the broadcast video camera are obtained in advance for each lens state and stored in a table.

The specific contents of the above-mentioned estimation method are described in, for example, Japanese Patent Application Laid-Open No. 2001-
As described in Japanese Patent No. 86392, each sensor for measuring the position, orientation, and lens state of the camera is attached to the camera, and these camera operation parameters are acquired in synchronization with a video signal when taking a picture. At the same time, an estimation method or the like is used that estimates the motion of the moving image corresponding to the motion of the camera by receiving the internal parameter of the camera that is obtained in advance by the calibration of the camera.

The extraction / tracking / identification circuit 2 first extracts a moving object from each background image by using the background difference method. In the present embodiment, the method of Sakaida et al. (“Study of moving object extraction technique by background difference”, 1999 IEICE General Conference) is used as the background difference object extraction method.

Here, the outline of the moving object extraction method based on the background difference will be described.

First, the background image is obtained as the mode value of the time-direction luminance value histogram for each pixel of the image. next,
The difference between the image of each frame and the background image is calculated, and the image is classified into pixels of three types of areas according to the following conditions. (1) A pixel whose difference exceeds a certain threshold Th is set as a moving object. (2) Pixels whose difference is less than or equal to a certain threshold Tl (Tl <Th) are set as the background area. (3) Pixels that do not belong to either the background area or the moving object are undetermined areas.

The threshold value Th and the threshold value Tl are determined based on the statistical information of the difference. Finally, in order to make the undetermined area belong to the moving object or the background area, watershed with a marker is applied. The edge image of each frame is used for watershed.

Next, with respect to the moving object extracted in each frame, the moving object is tracked from the relationship between the preceding and succeeding frames, and is associated in the time direction. As a tracking method, for example, Japanese Patent Application No.
By using the method described in the application No. 01-166525, the detection and tracking of an object can be performed with respect to a video input in which noise in an image, deformation of an object, obscuration between objects, appearance and disappearance of an object can occur. You can

Here, the above-mentioned detection / tracking method will be described with reference to a flowchart.

FIG. 3 is an example of a flowchart for explaining a method of detecting and tracking a moving object.

A method of detecting and tracking a moving (video) object described in Japanese Patent Application No. 2001-166525 is as follows.
The video signal is divided into a foreground area and a background area, and a foreground shape signal is generated (S1). Next, an image feature amount signal including at least information characterizing a video object and coordinate information is extracted from one or more of the video signal and the foreground shape signal (S2).

Next, the feature matching area is narrowed down from the video information and the estimated position / shape information (S3), and the feature matching area, the image feature amount signal, and the position / shape information of each video object in the object feature database are obtained. The object existence area information and the reliability h indicating the degree of trust of the object existence area information are generated (S4), and based on the object existence area information and the reliability h,
Estimate the current position and shape of the video object,
Estimated position / shape information is generated (S5).

After that, the estimated position / shape information is monitored (S6), and when the frame-out or disappearance of the video object is detected, the video object which is frame-out or disappeared is identified from the image feature amount in the object feature database. Delete the record corresponding to the number (S
7) Do.

When the reliability h of the collation for the video object having the identification number is monitored (S8) and it is detected that the reliability h reaches a predetermined numerical range, the image feature amount in the object feature database is detected. The image and area shape of the video object having the identification number, which is an element of, the average and covariance of the color vector inside the video object, and the average and covariance of the color vector outside the video object are recalculated and updated (S9). To do.

The above-described operations from S3 to S9 are repeatedly executed (S10) for the number of video objects stored in the object feature database.

Next, the presence area of each video object indicated by the estimated position / shape information is compared with the foreground shape signal (S11), and if a new video object is detected, the image feature in the object feature database is detected. A new record with a new unique identification number is added to the quantity (S1
2) Do.

Since the operation from S1 to S12 is a one-frame video object detecting operation, when tracking the movement of the object, the operation from S1 to S12 is repeatedly executed for the number of consecutive frames (S13). By doing, tracking can be performed.

Here, in the present invention, since a plurality of background image cameras are used for image capturing, there is an area (gray area in FIG. 2) where the image capturing portions of the background image cameras overlap. In such a region, the same moving object (moving object 21 in FIG. 2) in the scene is extracted and tracked in a plurality of background images.

In order to deal with the above-mentioned problem, moving objects are identified between background images and the same identification number is given to moving objects determined to be the same in a scene.

Here, k extracted from the n-th background image
OBJ the second object_nkAnd First, each
OBJ_nkThe Japanese Patent Application No. 20 according to the invention of the present inventor described above.
As described in the 01-0196077 application,
Map each object in a plane in space
OBJ_nkAnd Next, the two objects OBJ
_n1k1And OBJ_n2k2If the following conditions are met
It is determined that they are the same object in the scene.

D (C (OBJ _n1k1 ), C (OBJ _n2k2 )) ≦ D _t ... (1) and D (C (OBJ _{n1 k1} ), C (OBJ _n2k2 )) <D (C (OBJ _{n1k 1} ). , C (OBJ _n2k )), n ₁ ≠ n ₂ , k ≠ k ₂ (2) where C (X) is the center of gravity of the object X and D (X,
Y) represents the distance between points X and Y.

Conditional expression (1) is OBJ _n1k1 and OBJ
_{This means} that the distance between the _centers of gravity of _n2k2 is less than or equal to a certain threshold value D _t . In the present embodiment, as an example, the threshold value D _{t is set} to ½ of the horizontal length of the physical size of the object.
However, the threshold value is not limited to this.

[0048] Further, conditional expression (2), when the object satisfying the equation (1) to the _{n 2} th background image there are _multiple, close to the center of gravity of the _{OBJ N1k1} the center of gravity is another object of the _{OBJ N2k2} Means that.

[0049] After the above-described processing for all the OBJ _nk, imparts a unique identification number to respective dynamic objects in the scene. By giving an identification number, the same moving object photographed by a plurality of background video cameras can be recognized.

The automatic switching circuit 3 is a circuit that determines which background image is used when mapping the same moving object in a scene from a plurality of background images when mapping it to a broadcast image. In the present embodiment, the distance between the center of gravity of the moving object and the center of the screen of the plurality of background images having the moving object is calculated for the moving object corresponding to the same object in the scene extracted from the plurality of background images. The moving object is calculated and mapped to the broadcast image using the background image having the minimum distance. Therefore, a plurality of moving objects in the same screen of the broadcast video may be generated by mapping from different background videos.

The broadcast video moving object extraction circuit 4 is supplied by the broadcast video camera with the result of moving object extraction of the background video selected by the automatic switching circuit 3, the camera internal parameters supplied by the camera calibration circuit 1. This is a circuit part that extracts a moving object of a broadcast video using camera operation parameters.

The processing here is performed by using the camera operation parameter and the camera internal parameter as described in the above-mentioned Japanese Patent Application No. 2001-196077 filed by the present inventor. Is mapped to the broadcast video, the moving object of the broadcast video can be extracted with a unique ID.

As described above, in the moving object extracting method according to the present invention, a broadcast video camera and a plurality of background video cameras are installed at the time of shooting to extract a moving object in a scene from each background video, By identifying moving objects and switching the background image appropriately,
It is possible to map moving objects to broadcast video and extract moving objects with high accuracy.

Further, by using a plurality of background image cameras, the photographing area can cover a desired background without being affected by frame-in and frame-out, and the moving object has a certain size on the image. It becomes possible to have. Further, it is possible to secure sufficient accuracy for the contour of the moving object.

Further, it is possible to realize an object-based image coding and a linked data broadcasting system for broadcasting a video linked with an object.

[0056]

As described above, according to the present invention, a moving object can be extracted so that the shooting area can cover a desired background and the moving object can have a certain size on the image. A method and extraction device can be provided.

[Brief description of drawings]

FIG. 1 is a configuration diagram of an embodiment of a moving object extraction device according to the present invention.

FIG. 2 is a diagram illustrating an example of a relationship between shooting ranges of a plurality of background video cameras.

FIG. 3 is an example of a flowchart for explaining a method of detecting and tracking a moving object.

[Explanation of symbols]

1 Camera calibration circuit 2 Moving object extraction / tracking / identification circuit 3 Automatic switching circuit 4 Broadcast video moving object extraction circuit 21 moving objects

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Toshihiko Misu             1-10-11 Kinuta, Setagaya-ku, Tokyo, Japan             Broadcasting Association Broadcast Technology Institute F-term (reference) 5C022 AB61 AB62 AB63 AB68                 5C059 MB04 MB12 MB21 PP04 PP28                       PP29 SS14 UA02 UA05 UA38                 5L096 FA02 GA08 HA01 JA11

Claims (4)

[Claims] 1. A moving object extraction method for extracting a moving object by extracting moving objects from video signals of a plurality of fixed background video cameras, and mapping to a video image shot by a broadcast video camera to extract the moving objects. The identification step of identifying a moving object obtained from the area of the video signal overlapped between the video cameras, and the same moving object extracted from the video signals of different fixed background video cameras A selecting step of selecting a moving object extracted from a background image in which a distance between a center and a center of gravity of the moving object is minimum; and a region of the moving object selected in the selecting step is mapped to an image captured by the broadcast image camera. Object extraction by performing extraction of moving objects by Method. 2. The identifying step is performed when the distance between the centers of gravity of the moving objects extracted from the video signals captured by the plurality of fixed background video cameras in the real space is less than or equal to a predetermined threshold between different background videos. ,
The moving object extraction method according to claim 1, wherein the smallest moving object is compared with the distance between the centers of gravity of the moving objects and another moving object. 3. A moving object extracting device for extracting moving objects from video signals of a plurality of fixed background video cameras and mapping the captured moving images of broadcast video cameras to extract moving objects. When the same moving object is extracted from the video signals of different fixed background video cameras, the identification unit that identifies the moving objects obtained from the areas of video signals that overlap between video cameras, A selection unit for selecting a moving object extracted from a background image having a minimum distance between the center and the center of gravity of the moving object, and a region of the moving object selected by the selection unit is mapped to an image captured by the broadcast video camera. And a moving object extracting device for extracting a moving object. 4. The identifying unit determines that a distance between the centers of gravity of moving objects extracted from video signals captured by the plurality of fixed background video cameras in a real space is equal to or less than a predetermined threshold between different background videos. ,
The moving object extracting apparatus according to claim 3, wherein the smallest moving object is the same moving object as compared with the distance between the centers of gravity of the moving objects and another moving object.