JP5334328B2 - Moving object detection device, moving object detection method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a moving object detection device, a moving object detection method, and a program, suitable for real-time processing. <P>SOLUTION: A moving object detection device comprises: a detection part 11 for detecting a block size of variable block size movement compensation prediction; a calculation part 12 for calculating a first motion vector for a first macro block including a block of a first prescribed size when the detection part 11 detects the block; and a generation part 13 for setting a moving object area based on the first motion vector and adding also a second macro block including a block of a second prescribed size adjacent to the moving object area. The calculation part 12 calculates a second motion vector for a third macro block not including the blocks of the first and second prescribed sizes adjacent to the moving object area. The generation part 13 adds the third macro block to the moving object area based on the second motion vector. <P>COPYRIGHT: (C)2012,JPO&amp;INPIT

Description

  The present invention relates to a moving object detection apparatus, a moving object detection method, and a program, and more particularly, to an apparatus, method, and program for detecting a moving object in moving image data.

  In recent years, terminals having a function of capturing and / or displaying moving images have become widespread, and capturing and / or viewing of moving images has become a daily routine for users of these terminals. Along with this, a lot of development investment has been made for information extraction from moving images and commercial use of moving images. Specifically, in order to extract a moving object (motion object) in a moving image, for example, background difference, optical flow, particle filter, motion vector, DC (Discrete Cosine) image, skip macroblock (Macro Block) : MB), and methods using motion compensation macroblock sizes have been developed. Hereinafter, each method will be described.

  The moving object detection method using background difference is a method that separates and extracts only moving objects by taking the difference between a background image that shows only the background and a frame image that includes moving objects. In particular, this is particularly effective in processing a moving image in which only a moving object moves (for example, Patent Document 1).

  The moving object detection method using the optical flow is a method in which two video frames having different times are associated with each other with respect to the same target, and the movement amount is expressed as vector data (that is, an optical flow). is there. Typically, in obtaining the optical flow, the partial differentiation of the luminance value and color information at each pixel position is calculated (for example, Non-Patent Document 1).

  A moving object detection method using a particle filter is a method of estimating a moving object using an optical flow at points randomly scattered on a screen (for example, Non-Patent Document 2).

  The motion object method using a motion vector is a method in which a place where a motion vector used in predictive coding of a compressed moving image is generated is regarded as a moving region. Since motion vectors are caused not only by the movement of objects, but also by flickering of fluorescent lamps and shaking of the leaves of trees outdoors, it is necessary to distinguish these noises from actual moving objects. . Therefore, a method of using the magnitude and inner product of a motion vector for noise removal has been proposed (Patent Document 2, Non-Patent Documents 3 and 4).

  The motion object detection method using DC images is calculated using the fact that the DC component of the DCT coefficient in the I (Intra-coded) frame of MPEG (Moving Picture Experts Group) 2 corresponds to the average color of the block. This is a method for identifying a moving object by applying a normal template matching method to a DC image obtained (Non-Patent Documents 5 and 6).

  As a motion object detection method using a skip macroblock, a method of easily removing a background region having no motion using a type assigned to a macroblock in which a motion vector has not occurred is known (for example, non-patent document). 7).

  The motion object detection method using the motion compensated macroblock size is based on the premise that MPEG2 is used as the encoding method, and in the boundary region between the motion object and the background, the macroblock size is 16 × 16. In other words, the moving object region is detected by using the fact that 16 × 8 is likely to be used (for example, Non-Patent Document 8).

Japanese Patent Laid-Open No. 08-221577 JP 2001-250118 A

Yamada, Ito, Ueda, “Study of wave false detection suppression method in background subtraction method”, IEICE Technics, PRMU98-109, 1998 Ya-Dong Wang, Jian-Kang Wu, Ashraf A. Kassim, “Particle Filter for Visual Tracking Using Multiple Cameras,” Proc. MVA2005, Tsukuba, May, pp.16-18, 2005. H. Zen, T. Hasegawa, and S. Ozawa, “Moving object detection from MPEG coded picture,” Proc. Of 1999 International Conference on Image Processing, vol.4, pp.25-29, 1999. Iwasaki, Yokoyama, Watanabe, Koga “Detection and tracking of moving objects in the compression domain using motion vectors of MPEG video data” IEICE D, vol.J91-D, no.6, pp.1592-1603, 2008 Dan Schonfeld and Dan Lelescu, “VORTEX: Video retrieval and tracking from compressed multimedia databases-multiple object tracking from MPEG-2 bit stream,” Journal of Visual Communication and Image Representation, vol.11, no.2, pp.154-182 , 2000. Francesca Manerba, Jenny Benois-Pineau, Riccardo Leonardi, and Boris Mansencal, “Multiple moving object detection for fast video content description in compressed domain,” EURASIP J. Adv. Signal Process, vol.2008, no.1, pp.1- 13, 2008. Wonsang You, M.S.Houari Sabirin, Munchurl Kim, “Real-time detection and tracking of multiple objects with partial decoding in H.264 / A VC bitstream domain,” Proceedings of SPIE (2009), vol.7244, 2009. Simple, “Multiple Moving Object Detection and Tracking System Using MPEG2 Motion Vectors”, The University of Electro-Communications, Master's Thesis, January 2008.

  However, in the moving object detection methods according to Patent Document 1 and Non-Patent Document 1, since calculation in units of pixels is necessary, the total amount of calculation necessary for moving object detection is enormous, and real-time processing is required. Is inappropriate. In addition, according to the moving object detection method using the particle filter according to Non-Patent Document 2, at first glance, it seems that the calculation amount is reduced compared to the optical flow according to the method, but in order to increase the estimation accuracy. Since it is necessary to repeat the optical flow and the likelihood calculation together, it is not suitable for real-time processing. Furthermore, the motion object detection method using motion vectors according to Patent Document 2 and Non-Patent Documents 3 and 4 is also inappropriate for real-time processing.

  Furthermore, the moving object detection method using DC images according to Non-Patent Documents 5 and 6 is an H.264 method in which the interval between I frames is not uniform and is significantly longer than that in the case of MPEG2. When applied to H.264 format moving image data, the reliability of detection accuracy may be degraded. In addition, in the main profile, intra-frame predictive coding is also adopted in the I frame even in the I frame. Therefore, in order to extract color information from the DCT coefficients, at least the I frame must be completely restored, and the processing speed is increased. There is concern about being lost.

  Further, in the object detection method using the skip macroblock according to Non-Patent Document 7, the appearance frequency of the skip macroblock depends on the resolution and the frame type. The parameters have to be adjusted for every profile defined in H.264, which is not suitable for real-time processing. In fact, the number of skipped MBs per frame tends to be larger when the resolution is high or when the B frame is larger than the P frame.

  Furthermore, the object detection method using the motion compensation macroblock size according to Non-Patent Document 8 can detect only a small part of the moving region. It may not be applicable to H.264 format moving image data. In the method according to Non-Patent Document 8, detection is performed by estimating that a 16 × 8 size motion compensation block is generated as a boundary region. In the H.264 format moving image, not only the motion object area but also the motion compensation block of 16 × 8 size is frequently used in the entire screen.

  An object of the present invention made in view of such a point is to provide a moving object detection device, a moving object detection method, and a program suitable for real-time processing.

The invention of the moving object detection device according to the first aspect of achieving the above object,
A variable block size detection unit for detecting a block size used for variable block size motion compensation prediction for image data;
A motion vector calculation unit that calculates a first motion vector for a first macroblock including the block when the variable block size detection unit detects a block of a first predetermined size;
Based on the first motion vector, the first macroblock is set as a first motion object region, and a second macroblock including a second predetermined size block adjacent to the set first motion object region, A moving object area generator for adding to the first moving object area,
The motion vector calculation unit calculates a second motion vector for each of the third macro blocks that are adjacent to the first motion object region and do not include the first and second predetermined size blocks. ,
The moving object region generation unit adds the third macroblock to the first moving object region based on the second motion vector.
It is characterized by this.

The invention according to the second aspect is a moving object detection device according to the first aspect,
A predicting unit configured to predict a position of the moving object region after a predetermined time for the moving object region generated by the moving object region generating unit;
Based on the position of the moving object region predicted by the prediction unit and the second moving object region generated for the image data after the predetermined time by the moving object region generation unit, the first moving object region And a movement object area correspondence determination unit for determining a correspondence with the second movement object area,
It is characterized by providing.

An invention according to a third aspect is a moving object detection device according to the second aspect,
The prediction unit predicts the position of each first macroblock at the time of the second frame subsequent to the first frame based on the motion vector of each first macroblock included in the first moving object region. A macroblock predictor,
The inter-object region correspondence determination unit is configured to determine the position of each first macroblock predicted by the macroblock prediction unit and the second moving object region generated for the second frame by the moving object region generation unit. , Count, and based on the result, determine the correspondence between the first moving object area and the second moving object area,
It is characterized by this.

An invention according to a fourth aspect is a moving object detection device according to the third aspect,
The predicting unit further predicts a position of the first moving object region at a time point of a plurality of third frames for a predetermined time after the first frame using a Kalman filter,
The inter-object region correspondence determination unit further includes the position of the first moving object region predicted using the Kalman filter, and one or more third movements generated by the moving object region generation unit during the predetermined time. The correspondence between the first moving object area and the third moving object area is determined based on the position of the object area.

An invention according to a fifth aspect for achieving the above object is a moving object detection program,
On the computer,
A variable block size detecting step for detecting a block size used for variable block size motion compensation prediction for image data;
A motion vector calculation step of calculating a first motion vector for a first macroblock including the block when the variable block size detection unit detects a block of a first predetermined size;
Based on the first motion vector, the first macroblock is set as a first motion object region, and a second macroblock including a second predetermined size block adjacent to the set first motion object region, A moving object region generating step to be added to the first moving object region,
The motion vector calculation step calculates a second motion vector for each of the third macroblocks that are adjacent to the first motion object area and do not include the first and second predetermined size blocks. ,
The moving object region generating step adds the third macroblock to the first moving object region based on the second motion vector.
It is characterized by this.

The invention according to a sixth aspect is a moving object detection program according to the fifth aspect,
A predicting step of predicting a position of the moving object area after a predetermined time for the moving object area generated by the moving object area generating step;
Based on the position of the moving object area predicted in the prediction step and the second moving object area generated for the image data after the predetermined time in the moving object area generation step, the first moving object area And a movement object area correspondence determination step for determining a correspondence with the second movement object area,
Is executed.

An invention according to a seventh aspect for achieving the above object is a moving object detection method,
A variable block size detecting step for detecting a block size used for variable block size motion compensation prediction for image data;
A motion vector calculation step of calculating a first motion vector for a first macroblock including the block when the variable block size detection unit detects a block of a first predetermined size;
Based on the first motion vector, the first macro block is generated as a first motion object region, and a second macro block including a block of a second predetermined size adjacent to the generated first motion object region, A moving object region generating step to add to the first moving object region,
The motion vector calculation step calculates a second motion vector for each of the third macroblocks that are adjacent to the first motion object area and do not include the first and second predetermined size blocks. ,
The moving object region generating step adds the third macroblock to the first moving object region based on the second motion vector.
It is characterized by this.

The invention according to an eighth aspect is a moving object detection method according to the seventh aspect,
A predicting step of predicting a position of the moving object area after a predetermined time for the moving object area generated by the moving object area generating step;
Based on the position of the moving object area predicted in the prediction step and the second moving object area generated for the image data after the predetermined time in the moving object area generation step, the first moving object area And a movement object area correspondence determination step for determining a correspondence with the second movement object area,
It is characterized by including.

  According to the present invention, it is possible to provide a moving object detection apparatus, method, and program suitable for real-time processing.

It is a functional block diagram which shows roughly the principal part structure of the moving image processing apparatus containing the moving object detection apparatus which concerns on one Embodiment of this invention. It is a flowchart which shows operation | movement of the moving object detection apparatus shown in FIG. It is a figure for demonstrating operation | movement of the moving object detection apparatus shown in FIG. It is a flowchart which shows operation | movement of the moving object detection apparatus shown in FIG. It is an example of the system containing the moving object detection apparatus shown in FIG. It is a figure for demonstrating operation | movement of the system shown in FIG.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a functional block diagram schematically showing a main configuration of a moving image processing apparatus including a moving object detection apparatus according to an embodiment of the present invention. As illustrated in FIG. 1, the moving object region specifying unit 10 includes a variable block size detecting unit 11, a macroblock motion vector calculating unit 12, and a moving object region generating unit 13. On the other hand, the moving object tracking unit 20 includes a macroblock prediction unit 21, a macroblock count unit 22, and a moving object region correspondence determination unit 23.

  The encoding unit 30 is an apparatus for encoding a video signal (image data) having a general configuration. The motion compensation prediction unit 31 is, for example, H.264. H.264 implements variable block size motion compensation. The motion compensation prediction unit 31 includes a motion vector acquisition unit (not shown). The motion vector acquisition unit, for example, based on a frame (t) that is a past frame, a motion vector ( That is, a forward motion vector) is acquired. In this case, the motion compensation prediction unit 31 has seven patterns (16 × 16 pixels, 16 × 8, 8 × 16, 8 × 8, 8 × 4, 8 × 4) in order to optimize the coding amount and the motion compensation accuracy. Motion compensated prediction is performed using a block size (partition size) of 4 × 8, 4 × 4).

  The variable block size detection unit 11 monitors a block size used for motion compensation prediction by the motion compensation prediction unit 31 and detects a block size used for motion compensation prediction by the motion compensation prediction unit 31. The inventors have described H.C. It was found that many partitions having a block size of 8 × 8 pixels (hereinafter referred to as “8 × 8 partitions”) are detected at the boundary portion of the moving object of the H.264 format image data. Based on such knowledge, in this embodiment, the variable block size detection unit 11 notifies the macro block motion vector calculation unit 12 of the information of the 8 × 8 partition, particularly when detecting the 8 × 8 partition. To be configured.

  The macroblock motion vector calculation unit 12 obtains a motion vector of each partition included in a macroblock composed of 16 pixels in the vertical direction and 16 pixels in the horizontal direction from a motion vector detection unit (not shown) of the motion vector macroblock in the motion compensation prediction unit 31. Then, the motion vector of the entire macro block is calculated. Further, the macroblock motion vector calculation unit 12 classifies the direction of the motion vector into any of eight direction areas from 0 to 7, as illustrated in the example in FIG.

  The moving object area generation unit 13 generates a moving object area based on the motion vector of each macroblock and the partition size. Further, the moving object area generation unit 13 outputs the generated moving object area information. The operation of the moving object area specifying unit 10 will be described in detail with reference to FIG. The variable block size detection unit 11, the macroblock motion vector calculation unit 12, and the motion object region generation unit 13 described above are implemented by, for example, a CPU (Central Processing Unit).

  For each frame included in the acquired image data, the macroblock prediction unit 21 uses the forward motion vector for subsequent frames for each macroblock in the motion object region generated by the motion object region generation unit 13 described above. The position at the point of time is predicted.

The macroblock count unit 22 includes a counter (not shown), and the position of the macroblock predicted by the macroblock prediction unit 21 and the motion object region generated for the immediately following frame by the motion object region generation unit 13 Based on the above, the count is executed. As for a specific counting method, for example, a moving object region composed of the position of the macroblock predicted by the macroblock prediction unit 21 is set as the target region r, and the moving object generated for the immediately following frame by the moving object region generation unit 13 is used. A case where the region is the candidate region c will be described. The macroblock count unit 22 compares the target area r and the candidate area c, and counts the number of macroblocks in the target area r that fall within the candidate area c. Then, a reference rate is obtained for each target region r. The reference rate (P) is calculated by the following formula.
P = number of counts of target area r / total number of macroblocks in candidate area c When a plurality of moving object areas are generated by the moving object area generating unit 13 for each target frame, the macroblock counting unit 22 There are a plurality of target regions r and candidate regions c, and for each candidate region, the reference rate (P) is calculated for the plurality of target regions r.

  The inter-moving object region correspondence determination unit 23 selects a target region r having the maximum reference rate (P) calculated by the macroblock count unit 22 for one candidate region c, corresponding to the one candidate region c. It is determined that there are two target regions r. The above-described macroblock prediction unit 21, macroblock count unit 22, and moving object region correspondence determination unit 23 included in the moving object tracking unit 20 are implemented by, for example, a CPU.

  FIG. 2 is a flowchart showing the operation of the moving object area specifying unit shown in FIG. The variable block size detection unit 11 monitors the motion compensation prediction unit 31. The variable block size detection unit 11 detects all 8 × 8 size motion compensation blocks (8 × 8 partitions) included in the frame (S01). As a result, the moving object region specifying unit 10 starts processing.

  Next, the macroblock motion vector calculation unit 12 calculates an average value of motion vectors of all partitions for each macroblock including 8 × 8 partitions detected by the variable block size detection unit 11 (S02). At this time, the macroblock motion vector calculation unit 12 acquires the motion vectors of all partitions in the identified macroblock from the motion compensation prediction unit 31, and calculates an average value thereof.

  The moving object region generation unit 13 determines whether the average value of the motion vectors is positive (S03). When the moving object area generation unit 13 determines that the average value of the motion vectors is positive, the moving object area generation unit 13 connects the corresponding macroblocks, and combines them into one moving object area for each connected macroblock. (Yes in S03, S04). On the other hand, if the moving object area generating unit 13 determines that the average value is not positive, the moving object area detecting operation ends (No in S04, S10).

  Then, the moving object region generation unit 13 determines whether the partition of the macroblock adjacent to the macroblock included in the moving object region is a 16 × 8 or 8 × 16 partition in step S04 (S05). When the moving object region generation unit 13 determines that the partition of the adjacent macroblock is a 16 × 8 or 8 × 16 partition, the moving object region generation unit 13 adds the adjacent macroblock that satisfies the condition of step S05 to the moving object region described above. (Yes in S05, S06). Here, the moving object region generation unit 13 performs the operations of steps S06 and S07 on the macroblock further adjacent to the moving object region in step S07. In this way, the moving object region generation unit 13 determines whether the macroblock adjacent to the moving object region at the time of step S06 does not include a 16 × 8 or 8 × 16 partition, until the determination in steps S05 and S06. Repeat the operation.

  Then, the moving object area generation unit 13 determines whether the partition of the macroblock adjacent to the moving object area at the time of step S06 is a 16 × 16 partition (S07). If the moving object area generation unit 13 determines that the partition of the adjacent macroblock is not a 16 × 16 partition, the moving object area detection operation ends (No in S07, S10). On the other hand, if the moving object area generation unit 13 determines in step S05 that the adjacent macroblock partition is not a 16 × 8 or 8 × 16 partition, the moving object area at the time of step S04 in step S07. It is determined whether the partition of the macroblock adjacent to is a 16 × 16 partition.

  Then, the moving object region generation unit 13 determines the similarity between the macroblock determined to be a 16 × 16 partition in step S07 and the motion vector of each macroblock included in the moving object region in step S06. (S08). At this time, the moving object region generation unit 13 performs similarity determination based on the magnitude and direction of the motion vector. For example, in the direction similarity determination, the moving object region generation unit 13 classifies the direction of the average motion vector into any of the eight direction regions from 0 to 7 shown in FIG. It is determined that the motion vectors belonging to the region are similar in direction.

  If it is determined in step S08 that the moving object area is similar, the moving object area generating unit 13 adds an adjacent macroblock that satisfies the condition of step S08 to the moving object area (Yes in S08, S09). On the other hand, if it is determined in step S08 that they are not similar, the moving object region detection operation is terminated (No in S08, S10).

  FIG. 4 is a flowchart showing the operation of the moving object tracking unit shown in FIG. Here, it is assumed that a moving object tracking process is performed between a frame at time t (hereinafter referred to as frame (t)) and a frame at time t + 1 (hereinafter referred to as frame (t + 1)). The macroblock prediction unit 21 acquires the motion object region information of the frame at time t (hereinafter referred to as frame (t)) from the motion object region generation unit 13 and notifies the macroblock count unit 22 to that effect ( S11). The macroblock count unit 22 prepares a counter for each moving object area included in the frame (t), and resets the counter value to zero (S12).

  The macroblock prediction unit 21 acquires a motion vector from the motion compensation prediction unit 31 for each macroblock (hereinafter referred to as MBx) in each motion object region of the frame (t) acquired in step S11 (S13). . Then, the macroblock prediction unit 21 uses the motion vector acquired in step S13, and in the frame (t), the macroblock corresponding to the position of MBx at the time (t + 1) (hereinafter referred to as MBy). Are identified as predicted macroblocks.

  Then, the macroblock prediction unit 21 acquires the motion object region information of the frame (t + 1) from the motion object region generation unit 13 and provides it to the macroblock count unit 22 together with the information of the predicted macroblock specified in step S14. (S15). The macroblock count unit 22 performs counting based on the information acquired from the macroblock prediction unit 21 (S16). Then, the macroblock count unit 22 calculates the reference rate (P) described above with reference to FIG. 1 for each moving object region of the frame (t + 1) (S17).

  Then, the motion object region correspondence determination unit 23 determines the motion object of the frame (t) having the maximum reference rate (P) calculated in step S17 for the motion object region (candidate region c) of one frame (t + 1). A region (target region r) is determined (S18). In step S18, the moving object area correspondence determination unit 23 determines the target area r having the maximum reference rate (P) for one candidate area c as the same moving object as the candidate area c (Yes in S18). , S19). On the other hand, for the target regions r other than the target region r for which the reference rate (P) is determined to be the maximum in step S18, the moving object tracking process is terminated (No in S18, S20).

  As described above, according to the moving object detection device according to the present embodiment, the variable block size detection unit 11 has a block size used for variable block size motion compensation prediction for image data, and is 8 × 8 partitions (hereinafter referred to as “8 × 8 partitions”). When the first predetermined size block is detected, the macroblock motion vector calculation unit 12 calculates a first motion vector for the first macroblock including the block. Then, the moving object region generation unit 13 sets the first macroblock as the first moving object region based on the first motion vector. Furthermore, the moving object area generation unit 13 includes a second macro including a 16 × 8 or 8 × 16 partition (hereinafter referred to as a second predetermined size block) that is directly or indirectly adjacent to the identified first moving object area. A block is added to the first moving object area. Further, the macroblock motion vector calculation unit 12 performs a second operation on each of the third macroblocks that are adjacent to the generated first moving object region and do not include the first and second predetermined size blocks. A motion vector is calculated. Furthermore, the moving object region generation unit 13 adds the third macroblock to the first moving object region based on the second motion vector. As described above, since the moving object detection process is performed on the basis of a macroblock which is a set of pixels, a moving object detection apparatus suitable for real-time processing can be provided.

  Furthermore, each apparatus according to the present embodiment detects and tracks a moving object triggered by detection of a partition of a predetermined size. The present invention can be applied to various moving image data such as H.264. In addition, using the moving object area generated by the moving object area generating unit 13 according to the present embodiment, it is possible to perform background composition relatively easily compared to the normally used chroma key composition.

  Preferably, the macroblock prediction unit 21 can set a Kalman filter for one or a plurality of moving object regions generated by the moving object region generation unit 13 and perform prediction using them. More preferably, as input information of the Kalman filter, an average motion vector corresponding to the velocity vector of the moving object region can be used as well as the center of gravity position of the moving object region. The barycentric position, average motion vector, and speed information are calculated based on the average value of the position vector of the macroblock and the motion vector included in the motion object area generated by the motion object area generation unit 13. By adding not only the center of gravity position of the moving object area but also the average motion vector to the input of the Kalman filter, the prediction accuracy of the Kalman filter is improved, and a more accurate object position vector and velocity vector at the next time are obtained, not added Compared to the case, the tracking performance can be improved. On the other hand, when only the centroid position of the moving object area is used for the input of the Kalman filter as in the conventional case, the centroid position and velocity vector at the next time are predicted only from the centroid position at the current time. If the direction is changed suddenly or the speed is changed, the prediction result is poor and it is difficult to track the object.

  More preferably, the macroblock prediction unit 21 is a phenomenon in which a certain moving object disappears from the frame for a short period of time by being blocked by an obstacle or intersecting with another moving object. It can be configured to reduce deterioration in detection accuracy due to occurrence of so-called occlusion. In this case, for example, by performing prediction using a Kalman filter for a predetermined period set in advance in the macroblock prediction unit 21, it is possible to track a moving object that has disappeared from the screen due to occlusion.

  At this time, the observation data of the Kalman filter corresponding to the motion object disappeared from the screen due to occlusion is set to 0 for both the position vector and the motion vector corresponding to the velocity information, and the prediction by the Kalman filter is continued. In addition, during a predetermined period, the moving object area correspondence determining unit 23 resembles the moving object area specified by the moving object area specifying unit 13 with the moving object area held by the moving object area specifying unit 13 regardless of the presence or absence of occlusion. Continue to determine whether or not Then, when the occlusion is resolved and the moving object that has disappeared appears on the screen again, the moving object region correspondence determination unit 23 sets the object as a new object, its position, and a prediction result during a predetermined period. Based on the above, the distance to the predicted position by the Kalman filter is compared. If the distance is smaller than a predetermined threshold, the moving object area correspondence determination unit 23 determines that the object area that disappeared due to occlusion has appeared again.

  In this way, the moving object tracking unit 20 including the macroblock prediction unit 21 as described above can also track an object region where occlusion has occurred with high accuracy. The moving object tracking unit 20 is particularly suitable for mounting on a surveillance camera system or the like. This is because it is possible to keep track of a suspicious person who deceives a parking lot or a street corner without losing sight even when an obstacle or the like exists.

  In addition, the moving object detection device 2 according to the above embodiment can be implemented as a system as described below, for example. The system according to the first example is a video (video) distribution system with a video advertisement. In addition to the moving object detection device 2, this system includes, for example, cameras 40a and 40b, a video server 41, a speech recognition system 42, a face image recognition system 43, a UTC (Coordinated Universal Time) server 44, a metadata generation device 45, A metadata provider 46 is included. The face image recognition system holds, for example, a database that associates celebrity face image data with metadata such as the celebrity name and career. The metadata generation device 45 is connected to the Internet (not shown), for example, and can acquire various data via the Internet. The user 49 can view the video by accessing the system from a terminal 47 such as a smartphone or a personal computer (PC), for example. The terminal 47 includes a metadata acquisition unit 48 that acquires metadata from the metadata provider 46. The metadata acquisition unit 48 can be implemented by a CPU, for example.

  The system is configured to detect and track a large number of persons included in live video (real-time video) captured by the cameras 40a and 40b by the moving object area device 2. The moving object area specifying unit 10 of the moving object area apparatus 2 generates an area including each person by the method described in detail with reference to FIGS. Provides object area information. The moving object tracking unit 20 tracks each moving object by the method described in detail with reference to FIGS. 1 and 4 and assigns an identification number (ID) to each detected moving object. Provide tracking information. The moving object tracking information is, for example, information on diagonal coordinate values of an approximate rectangle including the tracked moving object area. In addition, live video captured by the cameras 40 a and 40 b is stored in the video server 41 and provided to the user 49.

  In addition, the voice recognition system 42 converts voice data included in the video data into text, and provides the text to the moving object tracking unit 20 as voice text data. The face image recognition system 43 cooperates to add information such as time, position information of each moving object, and text data as metadata to each moving object. Further, the face image recognition system 43 collates an image file including a moving object (person) with a database stored in advance, and associates an ID of each moving object (person) with metadata such as a name. .

  And the metadata production | generation apparatus 45 acquires near real time text data from a Twitter, a real time search engine, etc. via the internet. Further, the metadata generation unit 45 acquires information on the video acquisition time (current time) based on the UTC time from the UTC server 44 and adds it to the metadata. In addition, the time information may include video time information indicating a relative time within the video.

  Further, the metadata generation device 45 can include the IDs of one or more other objects acting positively or passively on one object in the above-described metadata. Here, “an object that acts positively on one object” means, for example, another person talking to a person (referred to as person A) as one object, or playing tennis with person A Person. On the other hand, “an object that acts passively on one object” means, for example, a wristwatch worn by person A, a sofa on which person A sits, or a wine bottle that person A holds in one hand Etc.

  FIG. 6 shows an example of metadata held in the metadata provider 46. For each object ID, information such as UTC time, video time, position information, audio text data, text data acquired from Twitter, etc., IDs of related objects, and their operations are held as metadata. “Position information” is a diagonal coordinate value of an approximate rectangle including each object region in each of the cameras 1 to N. In the UTC time (16:37:08), the object 1 (ID: 12897654), the object 2: against (ID 18999012), if you say "Hello!" Is, "Hello!" As the voice text data Is held. Then, the ID of the object 2 is held as the ID of the “related object”, and the information “listen to the story” is held as the “motion”. Similarly, various information about each object at each time point is held as metadata.

  The system can also be configured so that the user can interactively specify an object of interest via a screen on which live video is displayed. At this time, the user defines a rectangular region including an object of interest by drawing a diagonal line on the touch panel screen with a finger or a stylus pen, for example. At this time, if the movement of the object in the rectangular area is slow and does not change much even if several consecutive video frames are compared, the position of the object can be determined by capturing the movement of the object in a range larger than the specified boundary. The system can also be configured to obtain information.

  Further, the metadata acquisition unit 48 generates a filter for acquiring (fetching) metadata having contents that match the user's preference and request (that is, metadata dedicated to the user), and the metadata provider 46 receives the filter from the metadata provider 46. The metadata can also be obtained via There is a high possibility that metadata acquired by such information filtering using a filter is attractive and meaningful to the user. Parameters used when the metadata acquisition unit 48 generates a filter include the position, the gender and age of the user, the type of application that the user is running, the field of content included in the metadata, and the surroundings of the user Environment (for example, concerts, parties, schools, workplaces, etc.). In addition, it is also conceivable to use the user's mood and expectation that cannot be measured objectively as the parameters.

  According to this system, it is possible to distribute video advertisements according to user preferences, metadata information, and the like together with video. In addition, when using a plurality of cameras, a plurality of cameras can be integrated into the same tracking by using metadata such as positional information of a moving object to be tracked that is captured at different angles by a plurality of cameras. It is possible to track the subject.

  Furthermore, as one aspect of the present invention, the moving object detection device 2 can be configured as a computer. A program for causing a computer to function as this device is stored in a storage unit provided in the computer. Such a storage unit can be realized by an external storage device such as an external hard disk or an internal storage device such as ROM or RAM. A computer that functions as the above-described device can be realized by control of a CPU or the like. In other words, the CPU can appropriately read from the storage unit a program in which the processing content for realizing the function of each component is described, and realize the function of each component on the computer. Here, the function of each component may be realized by a part of hardware.

  In addition, the program describing the processing contents can be distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM, and such a program can be distributed on a server on a network, for example. Can be distributed by transferring the program from the server to another computer via the network.

  In addition, a computer that executes such a program can temporarily store, for example, a program recorded on a portable recording medium or a program transferred from a server in its own storage unit. As another embodiment of the program, the computer may directly read the program from a portable recording medium and execute processing according to the program, and each time the program is transferred from the server to the computer. In addition, the processing according to the received program may be executed sequentially.

DESCRIPTION OF SYMBOLS 1 Moving image processing apparatus 2 Moving object detection apparatus 10 Moving object area | region specific | specification part 11 Variable block size detection part 12 Macroblock motion vector calculation part 13 Moving object area | region production | generation part 20 Moving object tracking part 21 Macroblock prediction part 22 Macroblock count part 23 motion object region correspondence determination unit 30 encoding unit 31 motion compensation prediction unit 40 camera 41 video server 42 speech recognition system 43 face image recognition system 44 UTC server 45 metadata generation device 46 metadata provider 47 terminal

Claims (8)

A variable block size detection unit for detecting a block size used for variable block size motion compensation prediction for image data;
A motion vector calculation unit that calculates a first motion vector for a first macroblock including the block when the variable block size detection unit detects a block of a first predetermined size;
Based on the first motion vector, the first macroblock is set as a first motion object region, and a second macroblock including a second predetermined size block adjacent to the set first motion object region, A moving object area generator for adding to the first moving object area,
The motion vector calculation unit calculates a second motion vector for each of the third macro blocks that are adjacent to the first motion object region and do not include the first and second predetermined size blocks. ,
The moving object region generation unit adds the third macroblock to the first moving object region based on the second motion vector.
A moving object detection device characterized by the above. The moving object detection device according to claim 1,
A predicting unit configured to predict a position of the moving object region after a predetermined time for the moving object region generated by the moving object region generating unit;
Based on the position of the moving object region predicted by the prediction unit and the second moving object region generated for the image data after the predetermined time by the moving object region generation unit, the first moving object region And a movement object area correspondence determination unit for determining a correspondence with the second movement object area,
A moving object detection device comprising: The prediction unit predicts the position of each first macroblock at the time of the second frame subsequent to the first frame based on the motion vector of each first macroblock included in the first moving object region. A macroblock predictor,
The inter-object region correspondence determination unit is configured to determine the position of each first macroblock predicted by the macroblock prediction unit and the second moving object region generated for the second frame by the moving object region generation unit. , Count, and based on the result, determine the correspondence between the first moving object area and the second moving object area,
The moving object detection device according to claim 2, wherein The predicting unit further predicts a position of the first moving object region at a time point of a plurality of third frames for a predetermined time after the first frame using a Kalman filter,
The inter-object region correspondence determination unit further includes the position of the first moving object region predicted using the Kalman filter, and one or more third movements generated by the moving object region generation unit during the predetermined time. The moving object detection device according to claim 3, wherein a correspondence between the first moving object area and the third moving object area is determined based on a position of the object area. On the computer,
A variable block size detecting step for detecting a block size used for variable block size motion compensation prediction for image data;
A motion vector calculation step of calculating a first motion vector for a first macroblock including the block when the variable block size detection unit detects a block of a first predetermined size;
Based on the first motion vector, the first macroblock is set as a first motion object region, and a second macroblock including a second predetermined size block adjacent to the set first motion object region, A moving object region generating step to be added to the first moving object region,
The motion vector calculation step calculates a second motion vector for each of the third macroblocks that are adjacent to the first motion object area and do not include the first and second predetermined size blocks. ,
The moving object region generating step adds the third macroblock to the first moving object region based on the second motion vector.
A moving object detection program characterized by the above. The moving object detection program according to claim 5,
A predicting step of predicting a position of the moving object area after a predetermined time for the moving object area generated by the moving object area generating step;
Based on the position of the moving object area predicted in the prediction step and the second moving object area generated for the image data after the predetermined time in the moving object area generation step, the first moving object area And a movement object area correspondence determination step for determining a correspondence with the second movement object area,
A motion object detection program characterized by causing A variable block size detecting step for detecting a block size used for variable block size motion compensation prediction for image data;
A motion vector calculation step of calculating a first motion vector for a first macroblock including the block when the variable block size detection unit detects a block of a first predetermined size;
Based on the first motion vector, the first macroblock is set as a first motion object region, and a second macroblock including a second predetermined size block adjacent to the set first motion object region, A moving object region generating step to add to the first moving object region,
The motion vector calculation step calculates a second motion vector for each of the third macroblocks that are adjacent to the first motion object area and do not include the first and second predetermined size blocks. ,
The moving object region generating step adds the third macroblock to the first moving object region based on the second motion vector.
A moving object detection method characterized by the above. The moving object detection method according to claim 7,
A predicting step of predicting a position of the moving object area after a predetermined time for the moving object area generated by the moving object area generating step;
Based on the position of the moving object area predicted in the prediction step and the second moving object area generated for the image data after the predetermined time in the moving object area generation step, the first moving object area And a movement object area correspondence determination step for determining a correspondence with the second movement object area,
A moving object detection method comprising:

Images