JP4555690B2 - Trajectory-added video generation apparatus and trajectory-added video generation program - Google Patents

Description

The present invention relates to a video production technique, and more particularly, to a trajectory-added video generation apparatus and a trajectory-added video generation program for generating a video showing a trajectory of an object.

  Conventionally, a difference is taken between images composing a video taken in time series (a frame image in the case of the non-interlace method, a frame image or a field image (hereinafter simply referred to as a frame image) in the case of the interlace method). There is a device that extracts a video object that is an image of a moving object by a method and expresses the trajectory of the video object on a video by overwriting the video object. However, with this technology, for example, in a video such as a baseball game, when the trajectory of a ball, which is a video object to be extracted, is represented on the video, video objects other than the ball such as a pitcher's arm and bat are extracted. It will be.

  Therefore, a method for stably extracting only a specific video object has been developed (see Non-Patent Document 1). In this method, for each frame image input in time series, a difference image between the frame image input immediately before and a difference image between the frame image input immediately thereafter are obtained. Then, an area in which the sign of the pixel value is different between the two difference images is set as a video object candidate. Further, an area in the frame image where a video object that is likely to be erroneously detected is set as a mask area in advance, and after the video object in the mask area is excluded from the extracted video object candidates, the target becomes the target. Select a video object and display the trajectory.

Therefore, in this method, when displaying the trajectory of a video object that moves at a high speed, such as a baseball ball, the video object that moves at a low speed can be removed and a target video object can be extracted, and it is easy to be erroneously detected. Since the video object can be extracted except for the mask area where the video object exists, the target video object can be tracked stably.
Masaki Takahashi, 3 others, “Baseball pitching trajectory display method”, Technical Report of the Institute of Image Information and Television Engineers, October 21, 2004, VOL. 28, NO. 61, p. 25-28

  However, in the method of Non-Patent Document 1, the extracted video object is synthesized with a frame image, and only the trajectory of this video object is displayed. For example, in an example in which this method is applied to a baseball broadcast, by synthesizing a ball image or a white circle image at a position where the ball is extracted from each frame image in a trajectory image that looks back at the ball muscle after the pitch, The trajectory of this ball is displayed. However, from such a trajectory image, the viewer was unable to determine the ball's power.

  On the other hand, in the conventional baseball broadcast, the speed of the ball is measured with a speed gun, and the result is displayed in a character super. However, viewers who are familiar with baseball can understand the speed of the ball from the displayed numbers, but it may not be understood by viewers who are not, and speed guns have a certain point. However, there is a problem that it is impossible to measure the speed that changes during a pitch.

  Furthermore, in a conventional baseball broadcast pitch trajectory image, for example, even if a trajectory of a changing ball with a large drop is displayed, if the difference between the characteristics of the movement and other pitches is not indicated, It is difficult to tell if it is a changing ball. As a result, there is a problem that even if the trajectory is shown by the trajectory image, the ball type and the characteristics of the pitching are not easily transmitted to the viewer. For this reason, it has been desired to display the ball's sphere and ball type in an easy-to-understand manner.

  In addition, in order to show the difference between a certain pitch and another pitch, when the trajectory of another pitch is also displayed in this pitch trajectory image, if the camera angle changes due to a camera operation between pitches, two trajectories Has been displayed at different locations in the frame image.

The present invention has been made to solve the above-described problems of the prior art. When displaying the trajectory of a video object, the trajectory-added video generation apparatus can display the moving speed and motion characteristics of the video object in an easy-to-understand manner. It is another object of the present invention to provide a trajectory-added video generation program.

In order to solve the above-described problem, the trajectory-added video generation apparatus according to claim 1 extracts and tracks a video object from a frame image that constitutes a video that is captured and input by a camera installed at a predetermined position. A trajectory-added video generation device that generates a trajectory image indicating a trajectory of the video object and adds the generated trajectory image to a frame image that is input after that and is captured by a camera installed at the same position . A video object detecting device; a video delaying unit; and a video object trajectory adding unit. The video object detecting device includes an object candidate image generating unit and an object selecting unit. , a video object position information storage means, and path image information storage means, trace image mode setting means And configured to include an image synthesizing means.

With this configuration, the trajectory-added video generation device detects the position of the video object for each frame image input in time series by the video object detection device. Then, the trajectory-added video generation device generates a current frame image that is a frame image obtained by delaying the input frame image by a predetermined number of frames by the video delay means. The trajectory-added video generation device receives position information indicating the position of the video object from the video object detection device by the video object trajectory adding means, and continues to the plurality of frame images input in time series. A trajectory image indicating the trajectory of the video object included is added to the frame image. The video object detection device generates an object candidate image obtained by extracting a partial area of the frame image input in time series as the video object candidate by the object candidate image generation unit, and the object selection unit From the generated object candidate images, a video object to be extracted is selected based on a predetermined extraction condition, and a position of the selected video object and a feature amount characterizing the video object are extracted. . The video object trajectory adding apparatus stores the position information of the video object input from the video object detection apparatus that detects the position of the video object for each frame image in the video object position information storage unit. Then, the video object trajectory adding device stores in the trajectory image information storage means the color of the trajectory image used to indicate that the video object is high speed when the length of the motion vector of the video object exceeds a threshold value. And image data of a high-speed image of a predetermined color and shape indicating the shape, and a condition that defines the size and direction of the high-speed image used as a trajectory image when the length of the motion vector exceeds a threshold, and , Trajectory image information including a condition for defining that the image of the feature amount extracted by the object selecting means is the trajectory image when the length of the motion vector is equal to or less than a threshold value is stored. The video object locus additional devices, by the trace image mode setting means reads out the position information of the video object from the video object position information storage means, based on the position indicated by the position information, of the plurality of frame images Analyzes the amount of change in the position of the video object. Here, the video object is continuously included in a plurality of frame images input in time series from the outside, and the position of the video object corresponding to each frame image is obtained as position information from the video object detection device. Input and stored in the video object position information storage means. Therefore, for each frame image, the video object trajectory adding device analyzes the amount of change between the position of the video object in the frame image and the position of the video object in the frame image input before the frame image.

The video object trajectory adding device sets the color or shape of the trajectory image based on the amount of change by the trajectory image form setting means, and the trajectory image set by the trajectory image form setting means by the image composition means. Then, one frame image among a plurality of frame images including the video object is continuously synthesized at a position indicated by the position information of the video object. Here, the amount of change in the position indicates the amount of movement of the video object per predetermined time and the direction of movement. Therefore, the video object trajectory adding apparatus sets the color or shape of the trajectory image based on the amount of change, and synthesizes the video object corresponding to each trajectory image in one frame image at the position where the video object is detected. Thus, it is possible to generate a frame image indicating the trajectory of the video object and indicating the speed and moving direction of the video object at the time when the video object corresponding to each of the trajectory images is captured. The trajectory image form setting means calculates the motion vector of the video object and the length of the motion vector from the position of the video object on the frame image and the position of the video object on the frame image immediately before the video object. And determining whether the length of the motion vector exceeds a predetermined threshold based on the amount of change and the trajectory image information stored in the trajectory image information storage unit, and the color of the trajectory image Set the shape, size, and direction.

  Note that when the input video signal is captured by the interlace method, the video object trajectory adding apparatus may perform an operation on each frame image on the field image. Further, the video object trajectory adding device may set the color, size, etc. of the trajectory image by calculating by a predetermined method from the change amount of the position of the video object by the trajectory image form setting means. The color, shape, and the like of the trajectory image may be set by selecting an image, texture, shape, or the like stored in advance in the storage unit based on the amount of change in the position of the object.

The trajectory-added video generation apparatus according to claim 2 extracts and tracks a video object from a frame image that constitutes a video that is captured and input by a camera installed at a predetermined position, and tracks the video object. A trajectory-added video generation device that generates a trajectory image indicating the following, and adds the generated trajectory image to a frame image captured by a camera installed at the same position that is input thereafter, and a video object detection device Video object trajectory adding means, and the video object detecting device comprises object candidate image generating means and object selecting means, and the video object trajectory adding means includes video object position information. storage means, the trace image information storage means, and path image position setting means, trajectory image configuration settings And means, and configured to include an image synthesizing means.

With this configuration, the trajectory-added video generation device detects the position of the video object for each frame image input in time series by the video object detection device. The trajectory-added video generation device generates a current frame image that is a frame image obtained by delaying the input frame image by a predetermined number of frames by the video delay means. The trajectory-added video generation device receives position information indicating the position of the video object from the video object detection device by the video object trajectory adding means, and continues to the plurality of frame images input in time series. The video object that is continuously included in a past trajectory image that is a trajectory image indicating a past trajectory that is a trajectory of the video object included and a plurality of other frame images captured from a position where the frame image is captured. The other frame trajectory image indicating the trajectory is added to one frame image of the other plurality of frame images. The video object detection device generates an object candidate image obtained by extracting a partial area of the frame image input in time series as the video object candidate by the object candidate image generation unit, and the object selection unit From the generated object candidate images, a video object to be extracted is selected based on a predetermined extraction condition, and a position of the selected video object and a feature amount characterizing the video object are extracted. . The video object trajectory adding means stores the video object position information input from the video object detection device in the video object position information storage means in association with the trajectory information for specifying the trajectory of the video object, A predetermined color and shape indicating the color and shape of the trajectory image used to indicate that the video object is high-speed when the length of the motion vector of the video object exceeds a threshold value in the trajectory image information storage means Image data of the high-speed image, a condition that defines the size and direction of the high-speed image used as a trajectory image when the length of the motion vector exceeds the threshold, and the length of the motion vector is the threshold It is specified that the feature image extracted by the object selection means is the trajectory image when Storing the path image information including that condition. Then, the trace image position setting means, from said video object position information storage means, reads out the past trajectory position information is position information corresponding to the locus information indicating a past trajectory, the position indicated by the past trajectory position information Past The locus position.

Then, based on the past trajectory position by the trajectory image form setting means, as the amount of change in the position of the video object, the position of the video object on the frame image and the video on the frame image immediately before the video object and a position of the object, the motion vector of the image object, and calculates the length of the motion vector, based on a the amount of change and the path image information stored in the storage unit path image information, the motion vector It is determined whether the length exceeds a predetermined threshold, and the color , shape , size, and direction of the trajectory image are set, and are continuously included in a plurality of other frame images from the video object position information storage unit. Read the position information of the video object, and the position of the other frame trajectory that is the position indicated by this position information Based on the change in position of the video object by analyzing similarly, the amount of change and the path image information color other frame path image based on the stored path image information in the storage unit, shape, size Set the direction . Further, the image synthesis means synthesizes the past trajectory image set by the trajectory image form setting means to the past trajectory position of one of the other frame images, and the other frame trajectory of the frame image. The other frame trajectory image set by the trajectory image form setting means is synthesized at the position.

  As a result, the video object trajectory adding apparatus can add the trajectory of the video object previously detected together with the trajectory of the video object in the frame image to the frame image, and other frame trajectory images indicating these trajectories. In addition, the past trajectory image can be combined with the frame image by setting the color or shape indicating the speed or moving direction of the video object when the video object corresponding to each of the past trajectory images is captured.

Furthermore, the trajectory-added video generation apparatus according to claim 3 extracts and tracks a video object from a frame image that constitutes a video that is captured and input by a camera installed at a predetermined position, and tracks the video object. A trajectory-added video generation device that generates a trajectory image indicating the following, and adds the generated trajectory image to a frame image captured by a camera installed at the same position that is input thereafter, and a video object detection device Video object trajectory adding means, and the video object detecting device comprises object candidate image generating means and object selecting means, and the video object trajectory adding means includes video object position information. storage means, the trace image information storage means, and path image position setting means, trajectory image configuration settings And means, and configured to include an image synthesizing means.

With this configuration, the trajectory-added video generation device detects the position of the video object for each frame image input in time series by the video object detection device. Then, the trajectory-added video generation device generates a current frame image that is a frame image obtained by delaying the input frame image by a predetermined number of frames by the video delay means. The trajectory-added video generation device receives position information indicating the position of the video object from the video object detection device by the video object trajectory adding means, and continues to the plurality of frame images input in time series. The video object that is continuously included in a past trajectory image that is a trajectory image indicating a past trajectory that is a trajectory of the video object included and a plurality of other frame images captured from a position where the frame image is captured. The other frame trajectory image indicating the trajectory is added to one frame image of the other plurality of frame images. The video object detection device generates an object candidate image obtained by extracting a partial area of the frame image input in time series as the video object candidate by the object candidate image generation unit, and the object selection unit From the generated object candidate images, a video object to be extracted is selected based on a predetermined extraction condition, and a position of the selected video object and a feature amount characterizing the video object are extracted. . The video object trajectory adding means stores in the video object position information storage means the position information of the video object input from the video object detection device in association with the trajectory information for specifying the trajectory of the video object, A predetermined color and shape indicating the color and shape of the trajectory image used to indicate that the video object is high speed when the length of the motion vector of the video object exceeds a threshold value in the trajectory image information storage means Image data of the high-speed image, a condition that defines the size and direction of the high-speed image used as a trajectory image when the length of the motion vector exceeds a threshold, and the length of the motion vector is a threshold It is specified that the feature image extracted by the object selection means is the trajectory image when Storing the path image information including that condition. Then, the trace image position setting means, wherein the video object position information storage means, and the past trajectory position information is position information corresponding to the locus information indicating a past trajectory of the at least one video object other plurality of frame images The position information is read, a shift amount for shifting the position indicated by the past locus position information is set based on the position information, and the position indicated by the past locus position information is shifted by this shift amount. Set the position.

Then, based on the past trajectory position by the trajectory image form setting means, as the amount of change in the position of the video object, the position of the video object on the frame image and the video on the frame image immediately before the video object and a position of the object, the motion vector of the image object, and calculates the length of the motion vector, based on a the amount of change and the path image information stored in the storage unit path image information, the motion vector It is determined whether the length exceeds a predetermined threshold, and the color , shape , size, and direction of the past trajectory image are set, and are continuously included in a plurality of other frame images from the video object position information storage means. The position information of the video object to be read is read, and the other frame locus that is the position indicated by this position information Based on the location, the analyzes in the same manner the amount of change in position of the video object, the other frame path image based on the the variation between the trajectory image information track image information stored in the storage means color, shape, Set the size and direction . Further, the image synthesis means synthesizes the past trajectory image set by the trajectory image form setting means to the past trajectory position of one of the other frame images, and the other frame trajectory of the frame image. The other frame trajectory image set by the trajectory image form setting means is synthesized at the position.

  Thus, the video object trajectory adding device shifts the position of the trajectory of the previously detected video object in accordance with the position of the video object in the frame image or the trajectory of the video object, and converts the trajectory into the frame image. The other frame trajectory image and the past trajectory image indicating these trajectories can be combined, and the color or speed indicating the speed or moving direction of the video object at the time when the video object corresponding to each of the trajectory images is captured. It can be set to a shape and combined with a frame image.

  It should be noted that the plurality of frame images including the video object constituting the past trajectory and the other plurality of frame images to which the trajectory image is synthesized are taken by a camera installed at the same position. The camera angle may be different. In the case where a continuous frame image including the video objects constituting these two loci is captured, the position of one locus does not correspond to the position of the other locus due to a change in camera angle. In addition, the video object trajectory adding apparatus can shift the position of the past trajectory and add the past trajectory image to a position corresponding to the position of the trajectory of the video object of another plurality of frame images.

According to a fourth aspect of the present invention, a trail-added video generation program extracts and tracks a video object from a frame image that constitutes a video that is captured and input by a camera installed at a predetermined position, and tracks the video object. The video input from the video object detection device to add the generated trajectory image to a frame image captured by a camera installed at the same position, which is input after that Video object position information storage means for storing the position information of the object, and the color of the trajectory image used to indicate that the video object is high-speed when the length of the motion vector of the video object exceeds a threshold; Image data of a high-speed image having a predetermined color and shape indicating the shape, and the length of the motion vector A condition that defines the size and direction of the high-speed image used as a trajectory image when the threshold value is exceeded, and the feature amount extracted by the object selection means when the length of the motion vector is equal to or less than the threshold value. A trajectory image information storage means for storing trajectory image information including a condition for defining the image as the trajectory image , an object candidate image generation means, an object selection means, a video delay means, a trajectory image; It was made to function as a form setting means and an image composition means.

According to such a configuration, the trajectory-added video generation program generates an object candidate image obtained by extracting a partial region of the frame image input in time series as the video object candidate by the object candidate image generation unit. Then, the object selection means selects a video object to be extracted from the generated object candidate images based on a predetermined extraction condition, and determines the position of the selected video object and the video object. Extract feature quantities to be characterized. Then, a current frame image which is a frame image obtained by delaying the input frame image by a predetermined number of frames is generated by the video delay means. The trajectory-added video generation program reads the position information of the video object from the video object position information storage device by the trajectory image form setting means, and includes video objects continuously based on the position indicated by the position information. As a change amount of the position of the video object in the plurality of frame images, the position of the video object is calculated from the position of the video object on the frame image and the position of the video object on the frame image immediately before the video object. a motion vector, and calculates the length of the motion vector, based on a the amount of change and the path image information stored in the storage unit path image information, or the length of the motion vector exceeds a predetermined threshold value And the color , shape , size, and direction of the trajectory image are set. Further, the trajectory-added video generation program synthesizes the trajectory image set by the trajectory image form setting unit with the position indicated by the position information of one of the plurality of frame images by the image compositing unit. .

As a result, the trajectory-added video generation program generates a frame image indicating the trajectory of the video object and indicating the speed and moving direction of the video object at the time when the video object corresponding to each trajectory image is captured. be able to.

The trajectory-added video generation apparatus and the trajectory-added video generation program according to the present invention have the following excellent effects.

  According to the first or fourth aspect of the present invention, it is possible to provide a trajectory image of a video object that can easily display the moving speed and motion characteristics of the video object.

  According to the second aspect of the present invention, the trajectory of the video object extracted before the video object trajectory can be displayed together with the trajectory of the video object in the video. Can be displayed. Therefore, the trajectory of the previously extracted video object can be displayed as a comparison target, and the difference in speed and motion characteristics between this video object and the video object in the video can be clearly indicated. It is possible to provide a trajectory image that can easily display the characteristics of the speed and movement of the viewer to the viewer.

  According to the third aspect of the present invention, the trajectory of the video object extracted before can be displayed at a position that can be easily compared with the trajectory of the video object, along with the trajectory of the video object in the video. By using the trajectory as a comparison target, it is possible to clearly show the difference in speed and motion characteristics between the previously extracted video object and the video object in the video. Accordingly, it is possible to provide a trajectory image that can easily display the characteristics of the speed and movement of the image object in the image to the viewer.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Here, the video object trajectory adding means (video object trajectory adding device) in the present invention extracts and tracks a video object with a movement to be tracked, and the trajectory (current trajectory) of the video object in the video is recorded. A case will be described in which the present invention is applied to a trajectory-added video generation apparatus that generates a video to which a trajectory (past trajectory) of a video object in a previously captured video is added.

[Configuration of locus added video generation device]
First, the configuration of the trajectory-added video generation device will be described with reference to FIG. FIG. 1 is a block diagram showing a configuration of a trajectory-added video generation apparatus including video object trajectory adding means of the present invention. The trajectory-added video generation device 1 extracts and tracks a video object from a frame image constituting an input video, and the trajectory image indicating the trajectory of the video object is displayed in a color corresponding to the amount of change in the position of the video object. Are added to the frame image. The trajectory-added video generation apparatus 1 includes a video object extraction unit 10, a video delay unit 20, and a video object trajectory addition unit 30.

  The video object extracting means (video object detection device) 10 extracts and tracks a video object to be tracked for each frame image from the input video. The video object extraction unit 10 includes an object candidate image generation unit 11, an object selection unit 12, an extraction condition storage unit 13, a position prediction unit 14, a search area setting unit 15, and an ID number addition unit 16.

  The object candidate image generating means 11 generates an object candidate image obtained by extracting video object candidates to be tracked for each frame image from the input video. Here, the object candidate image generation unit 11 includes a difference image (difference image 1) between a frame image from which a candidate for a video object is extracted and a frame image input after one frame from the frame image, and an extraction target. A difference image (difference image 2) between the frame image to be input and a frame image input one frame before this frame image, and based on these difference images (difference image 1, difference image 2), A candidate image is generated. The object candidate image generation unit 11 includes image delay units 111 and 112, difference image generation units 113 and 114, and a candidate image generation unit 115.

  The image delay units 111 and 112 delay the input video signal by one frame. For example, the image delay units 111 and 112 may include a general delay circuit, or the input video signal may be converted into digital data in units of one frame. It may be composed of a memory for storing. Here, the image delay unit 111 receives a video signal from the outside, and the image delay unit 112, the difference image generation units 113 and 114, and the object selection unit 12 as a video signal obtained by delaying the video signal by one frame. The result is output to the feature amount analysis unit 122. Further, the image delay unit 112 receives a video signal delayed by one frame from the image delay unit 111, and further delays the video signal by one frame to obtain a video signal delayed by two frames. It outputs to the difference image generation unit 114.

  The difference image generation units 113 and 114 calculate a difference between the signal levels of the luminance signals of the two input video signals and generate a difference image for each frame image, and are configured by a general subtraction circuit or the like. Can do. Here, the frame image of the video signal delayed by one frame by the image delay unit 111 is set as a video object candidate extraction target. Then, the difference image generation unit 113 subtracts the signal level of the luminance signal of the video signal input from the outside from the signal level of the luminance signal of the video signal delayed by one frame input from the image delay unit 111. Thus, the difference image 1 is generated. Further, the difference image generation unit 114 receives the video signal delayed by one frame input from the image delay unit 111 from the signal level of the luminance signal of the video signal delayed by two frames input from the image delay unit 112. The difference image 2 is generated by subtracting the signal level of the luminance signal. The difference images generated here (difference image 1 and difference image 2) are output to the candidate image generation unit 115.

  The candidate image generation unit 115 uses the pixel values (luminance signal) of the difference images (the difference image 1 and the difference image 2) input from the difference image generation units 113 and 114 for the search area set by the search area setting unit 15 described later. The object candidate image is generated by determining for each pixel whether the signal level of each pixel satisfies a predetermined condition. Here, for each pixel, the pixel value of the difference image 1 and the pixel value of the difference image 2 in the pixel are different from each other, and the pixel value is set to “1 ( White) ”, and in other cases“ 0 (black) ”, an object candidate image is generated. As a result, an area where the pixel value is “1 (white)” can be extracted as a candidate video object. The object candidate image generated here is output to the object selection means 12.

  The object selection unit 12 extracts (tracks) from the object candidate images generated by the object candidate image generation unit 11 based on the extraction condition indicated by the extraction condition information 131 stored in the extraction condition storage unit 13. The target video object is selected, and the position of the video object and the feature quantity characterizing the video object are extracted. Here, the object selection unit 12 includes a labeling unit 121, a feature amount analysis unit 122, a filter processing unit 123, and an object selection unit 124.

  The labeling unit 121 assigns numbers (labels) to regions that are candidates for video objects in the object candidate images generated by the object candidate image generating unit 11. That is, the labeling unit 121 assigns one number to a connected area (connected area) having a pixel value of “1 (white)” that is an area of the video object. As a result, the video object candidates in the object candidate image are numbered.

  The feature amount analysis unit 122 analyzes video object candidates corresponding to the labels labeled by the labeling unit 121, and analyzes the position of the video object candidate and the feature amount characterizing the video object. Here, the feature amount analysis unit 122 corresponds to the video object corresponding to the label labeled by the labeling unit 121 in the frame image delayed by one frame input from the image delay unit 111 of the object candidate image generation unit 11. The feature amount is analyzed for each area to be processed. The analyzed position and feature amount of the video object are output to the filter processing unit 123.

  Here, as the position of the video object, the position of the center of gravity of the video object, the vertex coordinates of polygon approximation, the control point coordinates of the spline curve, and the like can be used. For example, the area, brightness, color, and circularity of the video object can be used as the feature amount of the video object.

The area indicates the number of pixels of the video object, for example. Further, the luminance indicates an average value of the luminance of each pixel in the video object. The color indicates an average value of colors (for example, RGB values) of each pixel in the video object. For this color, the background color of the background image may be set as an initial value, and the amount of change from the background color may be set as a threshold value. The circularity is an index indicating the complexity of the shape, and has a larger value as it becomes closer to a circle. The circularity e is expressed by the following equation (1), where S is the area of the video object and L is the perimeter.
e = 4πS / L ² (1)

  The filter processing unit 123 extracts the extraction condition information 131 stored in the extraction condition storage unit 13 based on the feature amount input from the feature amount analysis unit 122 for each video object candidate numbered by the labeling unit 121. The video objects to be extracted (tracked) are narrowed down by determining whether or not the video objects satisfy the extraction condition indicated by. The label and position of the selected video object are output to the object selection unit 124. If there is no video object that matches the extraction condition, the filter processing unit 123 notifies the ID number adding unit 16 to that effect.

  That is, for each video object candidate, the filter processing unit 123 performs extraction conditions (for example, area, luminance, color, and circularity) indicated by the extraction condition information 131, and filter processing (area filter, luminance filter, color filter). And a circularity filter), a video object that satisfies the extraction condition is selected.

  Based on the position of the video object with the label input from the filter processing unit 123, the object selection unit 124 calculates the position calculated by the position prediction unit 14 described later based on the position of the video object of the frame image one frame before. The video object closest to the predicted position of the video object is selected as the video object to be tracked. The position of the selected video object is output to the position predicting unit 14 and the ID number adding unit 16 as position information indicating the trajectory of the video object in the frame image. Further, the feature amount of the video object selected here is output to the drawing unit 351 of the image composition means 35.

  The extraction condition storage unit 13 stores conditions for selecting a video object to be extracted (tracked) by the object selection unit 12 and includes a general storage unit such as a hard disk. The extraction condition storage unit 13 stores extraction condition information 131 indicating various extraction conditions.

  The extraction condition information 131 is information describing the extraction condition of the video object to be extracted, and describes, for example, at least one extraction condition of area, luminance, color, and circularity. This extraction condition information 131 is a filter (area filter, luminance filter) for the filter processing unit 123 of the object selection unit 12 to select a video object to be extracted from the object candidate images generated by the object candidate image generation unit 11. , Color filter and circularity filter).

  The extraction condition information 131 stores a predetermined initial value and a threshold value indicating an allowable range as conditions for the area filter, the luminance filter, the color filter, and the circularity filter. As a result, video objects having features outside the threshold can be excluded from video object candidates to be extracted.

  The position prediction unit 14 analyzes the predicted position of the video object in the next frame image based on the position information of the video object input from the object selection unit 124 of the object selection unit 12. Here, the position prediction unit 14 includes a linear prediction unit 141, a curve prediction unit 142, and a switching unit 143.

  The linear prediction unit 141 uses the linear prediction to determine the predicted position of the video object in the next frame image based on the position information (centroid coordinates, etc.) of the video object input from the object selection unit 124 of the object selection unit 12. Is to be calculated. Here, the linear prediction unit 141 can calculate the predicted position of the video object in the next frame image, for example, by applying a Kalman filter or the like to the barycentric coordinates. The predicted position calculated here is output to the switching unit 143.

  The curve prediction unit 142 approximates the trajectory of the video object based on the position information (center of gravity coordinates, etc.) of the video object input from the object selection unit 124 of the object selection unit 12 to obtain the next frame image. The predicted position of the video object in is calculated. The predicted position calculated here is output to the switching unit 143.

  The switching unit 143 outputs one of the predicted position input from the linear prediction unit 141 and the predicted position input from the curve prediction unit 142 as the predicted position of the video object of the next frame image. . Here, the switching unit 143 outputs the predicted position input from the linear prediction unit 141, for example, when the position information of the already extracted video object is less than a predetermined number necessary for curve prediction, When the number is greater than or equal to the predetermined number, the predicted position input from the curve prediction unit 142 may be output. Further, when the value of the coefficient in the quadratic curve approximation of the trajectory in the curve prediction unit 142 is largely changed due to erroneous detection or the like, or when the coefficient exceeds a predetermined value, the prediction from the curve prediction unit 142 to the linear prediction unit 141 is performed. It is good also as switching to a position and outputting. The predicted position selected here is input to the object selection unit 124 of the object selection unit 12 and the search area setting unit 15.

  The search area setting unit 15 sets the search area of the video object in the next frame image based on the predicted position of the video object in the next frame image calculated by the position prediction unit 14. The search area is an area where an object candidate image is generated from the next frame image by the object candidate image generation means 11. The position and size of the search area set here are output to the candidate image generation unit 115 of the object candidate image generation unit 11. Here, the search area setting means 15 may set an area having a size input from an input means (not shown).

  The ID number adding unit 16 uses the position information of the video object continuously tracked from the frame image input in time series by the object selecting unit 12 as the position information constituting the trajectory of a series of movements of the same video object. Assuming that there is a trajectory number (trajectory information) indicating this trajectory, a sequential number (ID number) is added in time series to the position information for each trajectory number. Here, the ID number adding unit 16 determines that the extraction of a series of motion video objects by the object selection unit 12 is completed based on the notification input from the filter processing unit 123 that there is no video object that satisfies the extraction condition. Judgment can be made to set the trajectory number. The position information to which the ID number is added is stored in the video object position information storage unit 31.

  The video delay means 20 delays the frame image of the video signal input from the outside by a predetermined number of frames in order to synchronize with the trajectory image synthesized by the image synthesis means 35 described later. The video signal delayed here is output to the image composition means 35 of the video object locus adding means 30.

  The video object trajectory adding means (video object trajectory adding apparatus) 30 is a current frame which is a frame image of the video signal input from the video delay means 20 based on the position information of the video object input from the video object extracting means 10. The current trajectory image (other frame trajectory image), which is a trajectory image indicating the trajectory (current trajectory) of the video object included in the frame image, is set in the image in a color and shape according to the amount of change in the position of the video object. Thus, a trajectory image composed of a frame image obtained by compositing the trajectory image is generated. The video object trajectory adding means 30 includes video object position information storage means 31, trajectory image position setting means 32, trajectory image form setting means 33, trajectory image information storage means 34, and image composition means 35.

  Here, the video object trajectory adding means 30 receives the trajectory of the video object photographed before the current frame image together with the current trajectory image when a command for synthesizing the past trajectory image is input from an input means (not shown). The past trajectory image, which is a trajectory image indicating (past trajectory), is also synthesized with the current frame image. Then, the amount of change in the position of the video object in the past trajectory is shifted by shifting the display position of the past trajectory as a whole so that the position of the first video object in the past trajectory matches the position of the first video object in the current trajectory. We decided to synthesize past trajectory images with colors and shapes according to the conditions. Further, the video object trajectory adding means 30 synthesizes only the current trajectory image with the current frame image when the command for synthesizing the past trajectory image is not input from the input means (not shown).

  The video object position information storage means (video object position information storage device) 31 stores the position information of the video object input from the ID number adding means 16 and is a general storage medium such as a hard disk. The position information stored here is information indicating the position of the video object extracted by the object selection means 12, and is, for example, the barycentric position of the video object. This position information is referred to and used by the trajectory image position setting means 32 when setting the position where the past trajectory image is to be synthesized, and the trajectory image form setting means 33 sets the color and shape of the current trajectory image. It is used when referred to.

  Here, the video object position information storage means 31 stores the position information of the video object and the trajectory number (trajectory information) in association with each other, and further, in time series for each position information and trajectory number. The assigned ID numbers are stored in correspondence with each other. Here, the position information stored in the video object position information storage means 31 will be described with reference to FIG. FIG. 2 is an explanatory diagram for explaining an example of the position information stored in the video object position information storage means, and (a) is a position of the video object when the position information of the video object of one trajectory is input. An example of the position information stored in the information storage means, (b) is an example of the position information stored in the video object position information storage means when the position information of the video object of two trajectories is input.

First, the position information to which the ID number is added is input from the ID number adding means 16 to the video object position information storage means 31. Here, it is assumed that the position information (x ₁ , y ₁ ) to which the ID number “1” is added is input from the ID number adding unit 16 to the video object position information storage unit 31. Here, for example, it is assumed that the lower left of the frame image is the origin, the x axis is set in the horizontal direction, and the y axis is set in the vertical direction. Then, as shown in FIG. 2A, the video object position information storage unit 31 uses the input position information C1 (x ₁ , y ₁ ) as the trajectory number “1” (T1) and the ID number “1” ( And stored in correspondence with I1).

When the next video object is extracted from the continuous frame image by the object selection unit 12, the ID number “2” is added to the video object position information storage unit 31 from the ID number addition unit 16. Position information (x ₂ , y ₂ ) is input. Then, as shown in FIG. 2A, the video object position information storage unit 31 uses the position information C2 (x ₂ , y ₂ ) as the trajectory number “1” (T1) and the ID number “2” (I2). It memorize | stores corresponding to.

Then, it is assumed that n ₁ −2 video objects are sequentially extracted from the continuous frame images by the object selection unit 12. Then, as shown in FIG. 2A, the video object position information storage unit 31 stores the position information C3,..., Cn ((x ₃ , y ₃ ),..., (X _n , y _n )). The trajectory number “1” (T1) and the ID numbers “3”,..., “N” (I3,..., In) are stored in association with each other.

  Here, it is assumed that the video object is not extracted from the continuous frame images by the video object extraction means 10. Then, this notification is input to the ID number adding means 16, and based on this notification, the ID number adding means 16 adds 1 to each trajectory number stored in the video object position information storage means 31 and rewrites it. In FIG. 2A, the trajectory number “1” (T1) stored in the video object position information storage unit 31 is rewritten to “2” (T2 in FIG. 2B). As a result, the trajectory number indicating the trajectory of the video object of the current frame image is always “1”.

Thereafter, when the video object extraction unit 10 extracts the first video object among video objects that draw different trajectories, the ID number “1” is stored in the video object position information storage unit 31 from the ID number adding unit 16. The position information (x _{n + 1} , y _{n + 1} ) to which “is added is input. Then, as shown in FIG. 2B, the video object position information storage unit 31 uses the position information Cn + 1 (x _{n + 1} , y _{n + 1} ) as the trajectory number “1” (T1 ′) and the ID number “ 1 "(In + 1) and stored. As described above, the video object position information storage unit 31 can store the trajectory number and the ID number in association with the position information.

  Returning to FIG. 1, the description will be continued. The trajectory image position setting means 32 sets a position (past trajectory position) where the past trajectory image is drawn and synthesized by the image compositing means 35 described later. Here, the trajectory image position setting unit 32 includes a position information reading unit 321 and an additional position setting unit 322.

  The position information reading unit 321 includes the position information of the trajectory of the video object captured before the current frame image (past trajectory position information) and the same trajectory number as the video object of the current frame image (here, the trajectory number “1”). ) Position information is read out from the video object position information storage means 31. Here, the position information reading unit 321 reads position information corresponding to the trajectory number input from the outside together with a command for synthesizing the past trajectory image from an input unit (not shown) from the video object position information storage unit 31. Here, in the case where the trajectory number is not input together with the past trajectory image compositing command, the position information reading unit 321 has a trajectory number indicating the trajectory immediately before the video object of the current frame image (here, The position information of the trajectory number “2”) is read from the video object position information storage unit 31.

  Also, here, in order to match the position of the first video object in the past trajectory with the position of the first video object in the current trajectory, the position information reading unit 321 has position information of the same trajectory number as the video object of the current frame image. Among them, the position information whose ID number is 1 is read. The position information read here is output to the additional position setting unit 322.

  The additional position setting unit 322 sets the position of the past trajectory image (past trajectory position) based on the position information input from the position information reading unit 321. Here, the additional position setting unit 322 shifts the position indicated by the position information of the video object of the past trajectory input from the position information reading unit 321 according to the position of the first video object of the current trajectory. Set the trajectory position. The past trajectory position set here is output to the position change amount calculation unit 331 of the trajectory image form setting unit 33 and the composition unit 352 of the image composition unit 35.

  Here, the additional position setting unit 322 sets the past locus position based on the position information indicating the past locus and the position information of the first video object in the current locus. Here, referring to FIG. 3 (refer to FIG. 1 as appropriate), a method of setting the past locus position will be described. FIG. 3 is an explanatory diagram for explaining an example of a method for setting the past trajectory position by the additional position setting unit. FIG. 3A shows the position information of the video object of the past trajectory read by the position information reading unit. The schematic diagram schematically showing the position indicated by the position information of the first video object in the current trajectory, and (b) is the schematic diagram schematically showing the past trajectory position set by the additional position setting unit. . Here, the x-axis is set in the horizontal direction and the y-axis is set in the vertical direction of the frame image.

The position information reading unit 321 reads n _k pieces of position information (here, centroid coordinates) of the trajectory number “k” and the position information of the first video object (ID number “1”) of the current trajectory. 3 (a), the location information Ck1 with the trajectory number “k”, the ID number “1” is (xk1, yk1), and the location information Ck2 with the ID number “2”, as shown in FIG. the (xk2, yk2), ID number is the position information Ckn _k of "n _k" and (xkn _k, ykn _k), currently the location information C11 of the first video object trajectory and (x11, y11).

Then, the additional position setting unit 322 matches the position information Ck1 (xk1, yk1) of the first (ID number “1”) of the past locus with the position information C11 (x11, y11) of the first video object of the current locus. Therefore, it is moved in the x and y axis directions by (x11−xk1, y11−yk1), and set to the past locus position Dk1 (x11, y11). The other location information Ck2 past trajectory, ..., a position indicated by CKN _k also, x, is moved in the y-axis direction by (x11-xk1, y11-yk1 ), past trajectory position Dk2 (xk2 + x11-xk1, yk2 + y11 −yk1),..., Dkn _k (xkn _k + x11−xk1, ykn _k + y11−yk1).

  In this way, the additional position setting unit 322 synthesizes the past trajectory image at a position shifted as a whole so that the position of the first video object in the past trajectory matches the position of the first video object in the current trajectory. Can be set.

  Here, the additional position setting unit 322 sets the position of the first video object in the past trajectory to the position of the first video object in the current trajectory. Is not limited to this. For example, the position of the first video object in the past trajectory may be set to be a predetermined distance above the position of the first video object in the current trajectory, or the video object and ID number of the current frame image May be set so that the position of the video object of the same past locus becomes the position of the video object of the current frame image.

  Returning to FIG. 1, the description will be continued. The trajectory image form setting means 33 sets the color and shape of the current trajectory image based on the position indicated by the position information of the video object in the current frame image read from the video object position information storage means 31, and Based on the past locus position set by the additional position setting unit 322 of the image position setting means 32, the color and shape of the past locus image are set. The trajectory image form setting unit 33 includes a position change amount calculation unit 331 and a form setting unit 332.

  The position change amount calculation unit 331 reads position information of the same trajectory number (here, trajectory number “1”) as the video object of the current frame image, and a current trajectory position (another frame trajectory) that is a position indicated by this positional information. The amount of change in the position of the video object is calculated based on the position), and the position of the video object in the past trajectory is determined based on the past trajectory position set by the additional position setting unit 322 of the trajectory image position setting unit 32. The amount of change is calculated. The amount of change calculated here is output to the form setting unit 332. Further, the position information of the same trajectory number (here, trajectory number “1”) as the video object of the current frame image read out here is output to the drawing unit 351 of the image compositing means 35.

Here, it is assumed that the change amount of the position of the video object on the frame image is proportional to the moving distance (speed) of the video object per predetermined time, and the position change amount calculation unit 331 performs the current locus or the past locus. For each of the video objects constituting the video object, from the position of the video object on the frame image and the position of the video object on the frame image immediately before the video object, the motion vector B of the video object and the motion The length | B | of the vector B is calculated as the amount of change in position. Then, the x-axis in the horizontal direction at the lower left of the screen as an origin, sets the y-axis in the vertical direction, the position of the video object (x _t, y _t), the position of the previous frame of the video object (x _{t- 1} , y _t-1 ), the motion vector B and the length | B | of the motion vector B are expressed by the following equations (2) and (3).

  The form setting unit 332 generates a trajectory image (current trajectory image and past trajectory) based on the position change amount calculated by the position change amount calculation unit 331 and the trajectory image information 341 stored in the trajectory image information storage unit 34. Image) color and shape. The color and shape information set here is output to the drawing unit 351 of the image composition means 35.

  Here, with reference to FIG. 4 and FIG. 5, a method for setting the color and shape of the trajectory image by the form setting unit 332 will be described by taking as an example the case where the trajectory of the pitch is displayed on the video in a baseball broadcast or the like. To do. FIG. 4 is a schematic diagram schematically illustrating an example of a current frame image obtained by synthesizing the current trajectory image set by the form setting unit. FIG. 5 is a schematic diagram schematically showing an example of a current frame image obtained by synthesizing the current trajectory image and the past trajectory image set by the form setting unit.

  Here, it is assumed that an image of a pitch such as a baseball game is input from the outside, and the change amount of the ball position is calculated by the position change amount calculation unit 331. The length | B | of the motion vector B calculated by the position change amount calculation unit 331 is proportional to the velocity of the ball. Further, the x component of the motion vector B increases when the ball moves linearly, and the y component increases when it moves like a mountain. Therefore, here, the form setting unit 332 determines whether each of the length | B | of the motion vector B, the x component, and the y component exceeds a predetermined threshold, and stores it in the trajectory image information storage unit 34 described later. Based on the stored trajectory image information 341, the color and shape (shape, size, direction) of the trajectory image are set.

  Here, for example, when the image data of the flame image and the length | B | of the motion vector B exceed the threshold value, the trajectory image information storage unit 34 sets the color and shape of the trajectory image to the flame image. If the size is c · | B | (c is a constant), the direction is the direction of the motion vector B, and the length | B | The image of the feature amount analyzed by the analysis unit 122 is stored as the trajectory image information 341.

  For example, when the length | B | of the motion vector B calculated by the position change amount calculation unit 331 exceeds the threshold based on the current locus position, the form setting unit 332 shows the configuration shown in FIG. In this manner, the direction of the motion vector B is directed, and flame images (current trajectory images) g, g,... Having a size (c · | B |) proportional to the length | B | By setting in this way, the ball is a fast ball, the direction in which the ball moves, and the change in the ball speed during throwing (for example, the current trajectory image g increases as the ball moves to the right in FIG. 4). It can be displayed in a manner that is easy for the viewer to understand.

  Further, for example, when the length | B | of the motion vector B calculated by the position change amount calculation unit 331 based on the past trajectory position is equal to or smaller than the threshold value, the form setting unit 332 displays the configuration as shown in FIG. In addition to the current trajectory images (flame images g, g,...) Shown in FIG. 4, the feature amount images (past trajectory images) g ′, g ′,. . By setting in this way, the difference between the ball trajectory (current trajectory) and the past trajectory in the current frame image, the difference in ball power, and the like can be easily displayed to the viewer.

  Further, when the value of the y component of the motion vector of the video object exceeds the threshold value, the trajectory image information storage unit 34 sets the image to a predetermined shape, and information such as the shape, color, and size of the image May be stored as the trajectory image information 341, and the form setting unit 332 may set the trajectory image of the video object having a mountain-like movement as an image having a predetermined shape.

  The trajectory image information storage unit 34 stores information for setting the color and shape of the trajectory image by the form setting unit 332, and includes a storage unit such as a general hard disk. Here, the trajectory image information storage unit 34 stores data such as images and textures of various shapes, the color and shape of the trajectory image, the length | B |, x component of the motion vector B of the video object for each frame image. And data corresponding to the value of the y component are stored in advance as trajectory image information 341.

  Returning to FIG. 1, the description will be continued. The image synthesizing unit 35 draws and combines the trajectory image set by the trajectory image form setting unit 33 with the frame image input from the video delay unit 20. Here, the image composition unit 35 includes a drawing unit 351 and a composition unit 352.

  The drawing unit 351 receives the current trajectory image set by the trajectory image form setting unit 33 at the position indicated by the position information of the same trajectory number as the video object of the current frame image input from the position change amount calculation unit 331. In addition to drawing, the past trajectory image set by the trajectory image form setting means 33 is drawn at the position input from the additional position setting unit 322. The image drawn here is output to the synthesis unit 352.

  The combining unit 352 combines the image input from the drawing unit 351 with the frame image of the video signal input from the video delay unit 20. The video composed of the frame images synthesized here is output to the outside as a trajectory video.

  The configuration of the trajectory-added video generation device 1 according to the present invention has been described above, but the present invention is not limited to this. For example, here, a case where a pitching trajectory is displayed in a baseball broadcast video has been described as an example. However, the trajectory-added video generation device according to the present invention is not limited to a baseball pitching trajectory, and various movements are involved. When applied to a video of a simple video object, the trajectory can be indicated by a trajectory image corresponding to the characteristics of speed and motion. Further, here, the video object extraction means 10 generates two difference images of a frame image from which the video object is to be extracted and a frame image input one frame before and after the frame image, thereby generating a video object. However, the video object detection method of the video object extraction means 10 is not limited to this, and it is sufficient that the position of the video object can be detected for each frame image. For example, the video object may be detected from the frame image depending on the shape of the video object to be extracted, or may be detected based on the color of the video object.

  Here, the video object position information storage unit 31 stores the position information of the video object, the trajectory number, and the ID number in association with each other, and the position information reading unit 321 includes the past trajectory position information. The position information having the ID number 1 is read out from the position information of the same trajectory number as that of the video object of the current frame image. For example, the video object position information storage unit 31 only includes the position information and the trajectory number. May be stored in association with each other. At this time, for example, when the video object moves from left to right in the video, the position information reading unit 321 includes the position information of the past trajectory and the position information of the same trajectory number as the video object of the current frame image. The position information having the smallest x coordinate value may be read out.

  Further, here, the trajectory number is set by the ID number adding means 16, and the video object position information storage means 31 stores the trajectory number and the position information of the video object in association with each other. Any information may be used as long as the locus can be specified. For example, the trajectory-added video generation apparatus 1 does not include the ID number adding unit 16, and the video object position information storage unit 31 uses the time when the frame image from which the video object is extracted as trajectory information, and the trajectory. Information and position information may be stored in association with each other. At this time, for example, the position information reading unit 321 assumes that the position information in which the time of the locus information is continuous is the position information indicating a series of movement locus, and the position of the locus designated based on the command input from the outside. Information can be read out. Further, the video object position information storage unit 31 may store the position information as text data or the like, for example, and record it by changing the line for each locus. At this time, the position information reading unit 321 can read the position information of an arbitrary locus using the row number as the locus information.

  Further, here, the trajectory image position setting unit 32 reads the past trajectory position information and the position information of the same trajectory number as the video object of the current frame image from the video object position information storage unit 31 by the position information reading unit 321. The past position is set from the position information by the additional position setting unit 322. However, the trajectory image position setting unit 32 reads the past locus position information by the position information reading unit 321, and adds the additional position setting unit. By 322, the position indicated by the past locus position information may be set as the past locus position.

  In addition, here, one video to which a trajectory is added is input from the outside, and the video object position information storage unit 31 stores the two-dimensional coordinates of the video object in the frame image of the video as the position information. The additional position setting unit 322 sets the past trajectory position based on the two-dimensional coordinates, and the trajectory image form setting means 33 analyzes the change amount of the position of the video object based on the two-dimensional coordinates, and the color of the trajectory image. However, for example, it is also possible to further input video images taken from different places and analyze the three-dimensional position (three-dimensional coordinates) of the video object.

  At this time, the video object extracting means 10 analyzes the three-dimensional coordinates of the video object based on the two-dimensional coordinates of the video object extracted from two or more videos and the camera parameters of the photographed camera. The dimensional coordinates may be stored in the video object position information storage unit 31 as position information. In addition, the two-dimensional coordinates of the video object extracted from two or more videos by the video object extraction unit 10 are stored in the video object position information storage unit 31 as position information, and the position information reading unit 321 and the trajectory image form setting are stored. The position change amount calculation unit 331 of the means 33 may analyze the three-dimensional coordinates based on the camera parameters of the photographed camera and the read position information. Then, the additional position setting unit 322 shifts the three-dimensional coordinates of the past trajectory by a certain shift amount based on the past trajectory and the position of the current trajectory, and converts the shifted three-dimensional coordinates in the frame image by perspective projection conversion or the like. By converting to two-dimensional coordinates, a more accurate past trajectory can be added to the video. In addition, the form setting unit 332 sets the color and shape of the trajectory image based on the amount of change in the position of the three-dimensional coordinates, thereby adding an image showing more accurate speed and motion characteristics to the video as the trajectory image. can do.

  Further, here, the trajectory image form setting unit 33 calculates the change amount of the position of the video object, and the color and shape of the trajectory image corresponding to the change amount are stored in the trajectory image information storage unit 34. The video object adding apparatus according to the present invention sets only one of color and shape based on the change amount of the position of the video object by the trajectory image form setting unit. It is good as well. For example, the trajectory image form setting means may set the trajectory image as a circular image of a predetermined color, changing only the magnitude according to the magnitude | B | of the motion vector B, or An image having a predetermined size and a predetermined shape may be set by changing only the color according to the magnitude | B | of the motion vector B. Furthermore, the video object adding device of the present invention does not necessarily include the trajectory image information storage unit 34, and calculates and sets the color and shape of the trajectory image directly from the amount of change in the position of the video object. Also good.

  Note that the video object trajectory adding means 30 of the trajectory-added video generation device 1 is realized by causing a general computer to execute a program (video object trajectory adding program) and operating a computing device or a storage device in the computer. Can do.

[Operation of locus added video generation device]
Next, with reference to FIG. 6 and FIG. 7 (refer to FIG. 1 as appropriate), the operation of the trajectory-added video generation device 1 according to the present invention will be described. FIG. 6 is a diagram illustrating an image generated by adding a trajectory position of a subject from a frame image and adding a current trajectory image indicating a current trajectory and a past trajectory image indicating a past trajectory to the frame image. It is the flowchart which showed the operation | movement which produces | generates the locus | trajectory image | video performed. FIG. 7 is a flowchart showing an operation (video object extraction operation) in which the video object extraction means extracts the position of the video object from the frame image.

  First, the operation of the trajectory-added video generation device 1 will be described with reference to FIG. The trajectory-added video generation device 1 uses the video object extraction means 10 to extract the position and feature amount of the video object from the frame image input from the outside by the video object extraction operation described later (step S10). Then, the trajectory-added video generation device 1 determines whether the video object is extracted by the video object extraction operation by the video object extraction means 10 (step S11). If not extracted (No in step S11), the process directly proceeds to step S25. If it is extracted (Yes in step S11), the trajectory-added video generation device 1 stores the video object position information extracted in step S10 in the video object position information storage unit 31 of the video object trajectory adding unit 30. Is stored (step S12).

  Subsequently, the trajectory-added video generation apparatus 1 determines whether a command for synthesizing the past trajectory image to the frame image is input from the outside by the position information reading unit 321 of the trajectory image position setting unit 32 (step S13). Then, when the command for synthesizing the past trajectory image to the frame image is not input from the outside (No in step S13), the trajectory-added video generation device 1 includes the position change amount calculation unit of the trajectory image form setting unit 33. By 331, the position information of the current locus is read from the video object position information storage means 31 (step S14). Then, the trajectory-added video generation device 1 calculates the change amount of the position of the video object based on the position information read in step S14 by the position change amount calculation unit 331, and stores the change amount and the trajectory image information. Based on the trajectory image information 341 stored in the means 34, the color and shape of the current trajectory image are set (step S15). Further, the trajectory-added video generation device 1 draws the current trajectory image set in step S15 at the position indicated by the position information read in step S14 by the drawing unit 351 of the image compositing means 35 (step S15). S16), the process proceeds to step S24.

  On the other hand, when a command for combining past trajectory images is input (Yes in step S13), the position information reading unit 321 determines whether the trajectory number of the trajectory to be combined with the command is input (step S17).

  If no trajectory number has been input (No in step S17), the trajectory-added video generation device 1 uses the position information reading unit 321 to store 1 of the video object of the current frame image from the video object position information storage unit 31. The position information of the track number indicating the previous track and the position information of the first video object of the current track are read (step S18), and the process proceeds to step S20. When the trajectory number is input (Yes in step S17), the trajectory-added video generation device 1 uses the position information reading unit 321 to detect the position corresponding to the input trajectory number from the video object position information storage unit 31. Information and position information of the first video object in the current locus are read (step S19). The trajectory-added video generation device 1 sets a position where the past trajectory image is synthesized in the current frame image based on the position information read out in step S18 or step S19 by the additional position setting unit 322 (step S18). S20).

  Further, the trajectory-added video generation device 1 reads the position information of the current trajectory from the video object position information storage unit 31 by the position change amount calculation unit 331 of the trajectory image form setting unit 33 (step S21). Then, the trajectory-added video generation device 1 uses the position change amount calculation unit 331 to calculate the change amount of the position of the video object based on the position indicated by the position information read in step S21. The color and shape of the current trajectory image are set based on the trajectory image information 341 stored in the trajectory image information storage unit 34, and the position of the video object is determined based on the position read in step S18 or step S19. Is calculated, and the color and shape of the past trajectory image are set based on the amount of change and the trajectory image information 341 stored in the trajectory image information storage unit 34 (step S22).

  Subsequently, the trajectory-added video generation device 1 draws the current trajectory image set in step S22 at the position indicated by the position information read in step S21 by the drawing unit 351, and is set in step S20. The past trajectory image set in step S22 is drawn at the position (step S23). Furthermore, the trajectory-added video generation device 1 combines the image drawn in step S16 or step S23 with the current frame image by the combining unit 352 (step S24).

  Then, the trajectory-added video generation device 1 determines whether a new frame image has been input from the outside by the video object extraction means 10 (step S25). When a new image is input (Yes in step S25), the process returns to step S10, and operations after the video object extraction operation for extracting a video object from the input frame image are performed. If no new frame image is input (No in step S25), the operation ends.

[Video object extraction operation]
Next, referring to FIG. 7 (refer to FIG. 1 as appropriate), the trajectory-added video generation device 1 extracts a video object from a frame image input from the outside by the video object extraction means 10 (video object detection device). The object extraction operation (step S10 in FIG. 6) will be described.

  First, the trajectory-added video generation device 1 uses the search region setting unit 15 of the video object extraction unit 10 to generate a frame image (frame image to be extracted) immediately before the frame image input from the outside by the position prediction unit 14. Based on the predicted position calculated from the position of the video object in the previous frame image), a search area for the video object is set (step S40). When the predicted position is not calculated by the position predicting unit 14 for the video object of the frame image immediately before the frame image to be extracted, the trajectory-added video generation device 1 displays the entire area of the frame image, Alternatively, the specified area is set. Then, the trajectory-added video generation device 1 generates the difference image 1 and the difference image 2 by the object candidate image generation unit 11 and the frame image immediately before the frame image input from the outside (the frame image to be extracted). ) To generate an object candidate image for the search region set in step S40 (step S41).

  Further, the locus-added video generation device 1 attaches a label to a region that is a candidate video object by the labeling unit 121 of the object selection unit 12 (step S42). Then, the trajectory-added video generation apparatus 1 analyzes the position and the feature amount of the candidate video object corresponding to the label labeled in step S42 by the feature amount analysis unit 122 (step S43). Subsequently, the trajectory-added video generation device 1 uses the filter processing unit 123 to extract a video object to be extracted from the video object candidates corresponding to the label labeled in step S42 based on the feature amount analyzed in step S43. Filter processing for narrowing down is performed (step S44). Here, the filter processing unit 123 determines whether the feature amount of each video object candidate matches the extraction condition indicated by the extraction condition information 131 stored in the extraction condition storage unit 13. Process.

  Furthermore, the trajectory-added video generation apparatus 1 determines whether there is a video object that satisfies the extraction condition in the filter processing performed in step S44 by the filter processing unit 123 (step S45). If there is no video object that satisfies the extraction condition (Yes in step S45), the trajectory-added video generation device 1 uses the filter processing unit 123 to indicate that there is no video object to be extracted in this frame image. Judgment is made and the operation is finished as it is. If there is a video object that matches the extraction condition (No in step S45), the trajectory-added video generation device 1 uses the object selection unit 124 to perform the filtering process in step S44 and that matches the extraction condition. A video object to be extracted is selected from the list (step S46). Here, the object selection unit 124 selects the highest predicted position calculated from the position of the video object of the frame image immediately before the frame image to be extracted by the position prediction unit 14 among the video objects that meet the extraction condition. A near video object is selected as a video object to be extracted.

  The trajectory-added video generation apparatus 1 adds an ID number to the position information indicating the position of the video object selected in step S46 by the ID number adding unit 16 (step S47). Further, the trajectory-added video generation apparatus 1 calculates the predicted position of the video object in the next frame image based on the position of the video object selected in step 45 by the position prediction unit 14 (step S48), and ends the operation. To do. The predicted position calculated here is determined by the search area setting means 15 when the search area is set in the next frame image to be extracted (step S40) and by the object selection section 124 of the object selection means 12. This is used when a video object is selected in the next frame image to be extracted (step S46).

It is the block diagram which showed the structure of the locus | trajectory addition image | video production | generation apparatus provided with the image | video object locus | trajectory addition means in this invention. Explanatory drawing for demonstrating the example of the positional information memorize | stored in the video object positional information storage means of the locus | trajectory addition image generation apparatus in this invention, (a) is the case where the positional information on the image object of one locus | trajectory is input FIG. 5B shows an example of position information stored in the video object position information storage means. FIG. 5B shows position information stored in the video object position information storage means when the position information of the video object of two trajectories is input. It is an example. Explanatory drawing for demonstrating the example of the method of setting a past locus position by the additional position setting part of the locus | trajectory addition image generation apparatus in this invention, (a) is the image | video of the past locus read by the position information reading part. Schematic diagram schematically showing the position indicated by the position information of the object and the position information of the first video object of the current trajectory, (b) schematically shows the past trajectory position set by the additional position setting unit. It is a schematic diagram. It is a schematic diagram which shows typically the example of the present frame image with which the present locus image set by the form setting part of the locus | trajectory addition image generation apparatus in this invention was synthesize | combined. It is a schematic diagram which shows typically the example of the present frame image by which the present locus image set by the form setting part of the locus | trajectory addition image generation apparatus in this invention and the past locus image were synthesize | combined. The trajectory-added video generation apparatus according to the present invention extracts the position of the subject from the frame image, and generates a trajectory video in which the current trajectory image indicating the current trajectory and the past trajectory image indicating the past trajectory are added to the frame image. It is the shown flowchart. It is the flowchart which showed the operation | movement (video object extraction operation | movement) in which the video object extraction means of the locus | trajectory addition video generation apparatus in this invention extracts the position of a video object from a frame image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Trajectory addition video generation apparatus 10 Video object extraction means (video object detection apparatus)
20 video delay means 30 video object trajectory adding means (video object trajectory adding apparatus)
31 Video object position information storage means (video object position information storage device)
32 Trajectory image position setting means 33 Trajectory image form setting means 35 Image composition means

Claims (4)

A video object is extracted and tracked from a frame image that constitutes an image captured and input by a camera installed at a predetermined position, and a trajectory image indicating the trajectory of the video object is generated, and then input the same A trajectory-added video generation device that adds the generated trajectory image to a frame image photographed by a camera installed at a position,
A video object detection device for detecting the position of a video object for each frame image input in time series ;
Video delay means for generating a current frame image that is a frame image obtained by delaying the input frame image by a predetermined number of frames;
Position information indicating the position of the video object is input from the video object detection device , and a trajectory image indicating a trajectory of the video object included in succession in the plurality of frame images input in time series is obtained from the frame. Video object trajectory adding means for adding to the image ,
The video object detection device includes:
Object candidate image generation means for generating an object candidate image obtained by extracting a partial area of the frame image input in time series as a candidate for the video object;
An object that selects a video object to be extracted from the generated object candidate images based on a predetermined extraction condition, and extracts a position of the selected video object and a feature amount that characterizes the video object A selection means,
The video object trajectory adding means is
Video object position information storage means for storing position information of the video object input from the video object detection device;
Image data of a high-speed image of a predetermined color and shape indicating the color and shape of a trajectory image used to indicate that the video object is high-speed when the length of the motion vector of the video object exceeds a threshold value And a condition that defines the size and direction of the high-speed image used as a trajectory image when the length of the motion vector exceeds a threshold, and the object when the length of the motion vector is equal to or less than the threshold. Trajectory image information storage means for storing trajectory image information including conditions for defining the image of the feature amount extracted by the selection means as the trajectory image;
The position information of the image object is read from the image object position information storage means, and the image object on the frame image is obtained as a change amount of the position of the image object of the plurality of frame images based on the position indicated by the position information. And the motion vector of the video object and the length of the motion vector are calculated from the position of the video object and the position of the video object on the previous frame image of the video object, and the amount of change and the trajectory image information storage are calculated. Based on the trajectory image information stored in the means, it is determined whether the length of the motion vector exceeds a predetermined threshold, and the trajectory image form setting for setting the color , shape , size, and direction of the trajectory image Means,
A trajectory characterized by comprising: a trajectory image set by the trajectory image form setting means; and an image compositing means for synthesizing the trajectory image at a position indicated by the position information of one of the plurality of frame images. Additional video generation device. A video object is extracted and tracked from a frame image that constitutes an image captured and input by a camera installed at a predetermined position, and a trajectory image indicating the trajectory of the video object is generated, and then input the same A trajectory-added video generation device that adds the generated trajectory image to a frame image photographed by a camera installed at a position,
A video object detection device for detecting the position of a video object for each frame image input in time series ;
Video delay means for generating a current frame image that is a frame image obtained by delaying the input frame image by a predetermined number of frames;
A trajectory image indicating a past trajectory that is a trajectory of the video object that is continuously included in the plurality of frame images input in time series, by inputting position information indicating the position of the video object from the video object detection device. A past trajectory image, and another frame trajectory image indicating the trajectory of the video object continuously included in the other plurality of frame images captured from the position where the frame image was captured. Video object locus adding means for adding to one of the frame images ,
The video object detection device includes:
Object candidate image generation means for generating an object candidate image obtained by extracting a partial area of the frame image input in time series as a candidate for the video object;
An object that selects a video object to be extracted from the generated object candidate images based on a predetermined extraction condition, and extracts a position of the selected video object and a feature amount that characterizes the video object A selection means,
The video object trajectory adding means is
Video object position information storage means for storing position information of the video object input from the video object detection device in association with trajectory information for specifying the trajectory of the video object;
Image data of a high-speed image of a predetermined color and shape indicating the color and shape of a trajectory image used to indicate that the video object is high-speed when the length of the motion vector of the video object exceeds a threshold value And a condition that defines the size and direction of the high-speed image used as a trajectory image when the length of the motion vector exceeds a threshold, and the object when the length of the motion vector is equal to or less than the threshold. Trajectory image information storage means for storing trajectory image information including conditions for defining the image of the feature amount extracted by the selection means as the trajectory image;
The past trajectory position information, which is position information corresponding to the trajectory information indicating the past trajectory, is read from the video object position information storage means, and the trajectory image position setting with the position indicated by the past trajectory position information as the past trajectory position is set. Means,
Based on the past trajectory position, as the amount of change in the position of the video object, from the position of the video object on the frame image and the position of the video object on the frame image immediately before the video object, the video The motion vector of the object and the length of the motion vector are calculated, and the length of the motion vector exceeds a predetermined threshold based on the amount of change and the trajectory image information stored in the trajectory image information storage unit. And setting the color , shape , size, and direction of the trajectory image, and the position information of the video object continuously included in the plurality of other frame images from the video object position information storage means. Based on the position of the other frame trajectory that is the position indicated by this position information, And similarly analyzed the amount of change in position of the defect, the other frame color locus image, shape, size, direction set based on the the variation between the trajectory image information storage means the stored path image information Trajectory image form setting means to perform,
The past trajectory image set by the trajectory image form setting means is combined with the past trajectory position of one of the other frame images, and the other frame trajectory position of the frame image is combined. A trajectory-added video generation apparatus comprising: an image compositing unit that synthesizes a trajectory image of another frame set by the trajectory image form setting unit . A video object is extracted and tracked from a frame image that constitutes an image captured and input by a camera installed at a predetermined position, and a trajectory image indicating the trajectory of the video object is generated, and then input the same A trajectory-added video generation device that adds the generated trajectory image to a frame image photographed by a camera installed at a position,
A video object detection device for detecting the position of a video object for each frame image input in time series ;
Video delay means for generating a current frame image that is a frame image obtained by delaying the input frame image by a predetermined number of frames;
A trajectory image indicating a past trajectory that is a trajectory of the video object that is continuously included in the plurality of frame images input in time series, by inputting position information indicating the position of the video object from the video object detection device. A past trajectory image, and another frame trajectory image indicating the trajectory of the video object continuously included in the other plurality of frame images captured from the position where the frame image was captured. Video object locus adding means for adding to one of the frame images ,
The video object detection device includes:
Object candidate image generation means for generating an object candidate image obtained by extracting a partial area of the frame image input in time series as a candidate for the video object;
An object that selects a video object to be extracted from the generated object candidate images based on a predetermined extraction condition, and extracts a position of the selected video object and a feature amount that characterizes the video object A selection means,
The video object trajectory adding means is
Video object position information storage means for storing position information of the video object input from the video object detection device in association with trajectory information for specifying the trajectory of the video object;
Image data of a high-speed image of a predetermined color and shape indicating the color and shape of a trajectory image used to indicate that the video object is high-speed when the length of the motion vector of the video object exceeds a threshold value And a condition that defines the size and direction of the high-speed image used as a trajectory image when the length of the motion vector exceeds a threshold, and the object when the length of the motion vector is equal to or less than the threshold. Trajectory image information storage means for storing trajectory image information including conditions for defining the image of the feature amount extracted by the selection means as the trajectory image;
From the video object position information storage means, read the past trajectory position information that is position information corresponding to the trajectory information indicating the past trajectory, and the position information of at least one video object of the other plurality of frame images, A trajectory that sets a shift amount for shifting the position indicated by the past trajectory position information based on the position information and sets a past trajectory position by shifting the position indicated by the past trajectory position information by the shift amount. Image position setting means;
Based on the past trajectory position, as the amount of change in the position of the video object, from the position of the video object on the frame image and the position of the video object on the frame image immediately before the video object, the video The motion vector of the object and the length of the motion vector are calculated, and the length of the motion vector exceeds a predetermined threshold based on the amount of change and the trajectory image information stored in the trajectory image information storage unit. And setting the color , shape , size, and direction of the past trajectory image, and the position information of the video object continuously included in the other frame images from the video object position information storage means Is read out, and based on the other frame trajectory position, which is the position indicated by this position information, The amount of change in the position of the object by analyzing the same manner, the other frame color locus image, shape, size, direction set based on the the variation between the trajectory image information storage means the stored path image information Trajectory image form setting means to perform,
The past trajectory image set by the trajectory image form setting means is combined with the past trajectory position of one of the other frame images, and the other frame trajectory position of the frame image is combined. A trajectory-added video generation apparatus comprising: an image compositing unit that synthesizes a trajectory image of another frame set by the trajectory image form setting unit . A video object is extracted and tracked from a frame image that constitutes an image captured and input by a camera installed at a predetermined position, and a trajectory image indicating the trajectory of the video object is generated, and then input the same In order to add the generated trajectory image to the frame image taken by the camera installed at the position ,
Video object position information storage means for storing position information of the video object input from the video object detection device;
Image data of a high-speed image of a predetermined color and shape indicating the color and shape of a trajectory image used to indicate that the video object is high-speed when the length of the motion vector of the video object exceeds a threshold value And a condition that defines the size and direction of the high-speed image used as a trajectory image when the length of the motion vector exceeds a threshold, and the object when the length of the motion vector is equal to or less than the threshold. A trajectory image information storage unit that stores trajectory image information including a condition for defining that the image of the feature amount extracted by the selection unit is the trajectory image ;
Object candidate image generation means for generating an object candidate image obtained by extracting a partial region of the frame image input in time series as a candidate for the video object;
An object that selects a video object to be extracted from the generated object candidate images based on a predetermined extraction condition, and extracts a position of the selected video object and a feature amount that characterizes the video object Selection means,
Video delay means for generating a current frame image that is a frame image obtained by delaying the input frame image by a predetermined number of frames;
The position information of the image object is read from the image object position information storage means, and the image object on the frame image is obtained as a change amount of the position of the image object of the plurality of frame images based on the position indicated by the position information. And the motion vector of the video object and the length of the motion vector are calculated from the position of the video object and the position of the video object on the previous frame image of the video object, and the amount of change and the trajectory image information storage are calculated. Based on the trajectory image information stored in the means, it is determined whether the length of the motion vector exceeds a predetermined threshold, and the trajectory image form setting for setting the color , shape , size, and direction of the trajectory image means,
A trajectory characterized by causing the trajectory image set by the trajectory image form setting means to function as an image synthesizing means for synthesizing the trajectory image at a position indicated by the position information of one of the plurality of frame images. Additional video generation program.

Images