JPH04352076A - Method for offering moving image and method for machine moving image data - Google Patents

Abstract

PURPOSE:To obtain a data managing method for expressing the movement of a moving object as a function from time to a position without requiring much storage capacity in time space description and to retrieve a moving image by a visual method based upon movement retrieval by automating the processing of image working or the like based upon movement by using the time spatial description of the moving object. CONSTITUTION:A current frame time is applied to the previously formed function from time to a position, a position (x, y) on a screen is calculated, processing required by a user is added by using the calculated result and a picture is displayed. Thus the position of the moving object can be specified within a frame switching time, and when the moving object is synthesized with a CG e.g. a moving image capable of moving the CG in accordance with the movement of the moving object can be offered.

Description

[Detailed description of the invention]

0001

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for presenting and retrieving moving images based on the movement of moving objects in an information representation system using moving images.

0002

2. Description of the Related Art Conventionally, moving images are presented by cutting out a certain section (called a scene) on the time axis and editing it. Furthermore, in interactive applications using moving images, the user's operations are limited to the entire presented scene or stationary icons.

[0003] Scene retrieval has been performed interactively by presenting an image (called a frame) at a certain point on the time axis, which can serve as a criterion for use, using various user interfaces. Alternatively, this was done based on keyword information that was added interactively to the scene in advance.

0004

[Problems to be Solved by the Invention] However, the method of using information only on the time axis as described above has the following problems.

For example, consider a landscape scene in which several people are moving around. If you want to compose an arrow that moves according to the movement of person A in order to emphasize and present a certain person A, the video itself does not have position information of person A, so it is necessary to manually draw the arrow for each frame. There must be. Furthermore, when changing the shape of an arrow, it is necessary to redraw the arrow for each frame, which is inefficient.

[0006]Furthermore, even if an attempt is made to search this scene using keywords based on the movement of person A, appropriate query processing cannot be realized because the keywords representing movement vary from person to person.

An object of the present invention is to present not only a conventional method for presenting and retrieving moving images depending on the time axis, but also a method for managing data on movement of a moving body that includes information on position and shape on an image as well as time. , and to provide a method for presenting and searching moving images based on this.

[0008]

[Means for Solving the Problems] The moving image presentation method of the first invention records the position of a moving object at each time in advance, thereby making the position of the moving object a function of time, and using this. It is characterized by the presentation of moving objects based on movement by performing processing tailored to the position of the moving object at each time.

[0009] The moving image data management method of the second invention is based on the method of managing motion image data at a time (referred to as reference time) that well represents the characteristics of motion in a space-time consisting of three dimensions, two dimensions representing the screen and one dimension representing time. A straight line or curve that approximates the position of the moving object, obtained by giving multiple positions of the moving object.
Alternatively, the trajectory of the movement of the moving object expressed by a combination of these is used as the spatiotemporal information of the moving object, and when time is given, the position of the object is returned.

The moving image presentation method of the third invention is based on the first invention, by recording the shape of the moving object in addition to the position of the moving object at each time, thereby making the position and shape of the moving object a function of time. , is used to perform processing tailored to the position and shape of the moving object at each time, thereby presenting the moving object based on changes in movement and shape.

[0011] In the moving image data management method of the fourth invention, in the second invention, by also providing a rectangle that approximates the shape of the moving object at the reference time, the trajectory of the movement of the moving object and the shape at the reference time are determined. An approximate rectangle is used as the spatio-temporal information of the moving object, and when time is given, the position of the object is returned, and in addition, an approximate rectangle calculated using the relative time to the reference time is returned based on the approximate rectangle at the reference time. It is characterized by

[0012] The moving image retrieval method of the fifth invention uses the motion vector information of the moving object obtained from the moving image data management method of the second or fourth invention as a search key, and performs scene retrieval based on the movement. It is characterized by

[0013]

[Operation] A moving image is a periodic (NT) of still images (frames).
Since the SC method is realized by continuous display at 30 frames per second, in order to present a moving image based on the movement of a moving object, it is necessary to specify the position of the moving object in each frame. Furthermore, when the movement of an object is accompanied by a change in shape, it is necessary to specify the shape in addition to specifying the position of the moving object. This work can be done either visually by humans or by using image processing algorithms.
In the former, it is impossible to complete processing within the frame switching time, and in the latter, there are still many technical problems that have not been solved, making it inappropriate for general use. Therefore, a method can be considered in which the position is recorded in advance as a spatiotemporal representation as a function of time, and moving images are presented based on this.

[0014] The invention according to claim 1 provides a method of presenting a moving image using the spatio-temporal information of the moving object. As a result, the position of the moving object can be specified within the frame switching time, and by compositing with CG, for example, it becomes possible to present a moving image in which the CG moves in accordance with the movement of the moving object. Furthermore, if a certain area including the position of a moving object is treated as a button in hypertext, it becomes possible to use the moving object as a button.

[0015] However, recording the position of a moving object in detail every time requires an enormous storage capacity, which is not practical. The invention according to claim 2 provides spatiotemporal information of a moving object, and has the effect of reducing storage capacity.

The effect brought about by the invention described in claim 1 cannot be applied to presentations based on changes in the size of an object due to operations such as zooming or panning in camera work, or changes in the shape of the object itself. Therefore, the above-mentioned effect can be obtained by expressing the shape in addition to the position as spatiotemporal information as a function of time.

The third aspect of the invention provides spatiotemporal information including the position and shape of the moving object. This allows the position and shape of the moving object to be specified within the frame switching time, making it possible to synthesize the moving object and CG in accordance with changes in camera work and shape, and to use the moving object as a hypertext button.

However, if the position and shape of a moving object were to be recorded in detail every time, the storage capacity would be enormous, which would be impractical. The invention according to claim 4 provides spatiotemporal information of a moving object, and has the effect of reducing storage capacity.

Furthermore, since the data management method according to claim 2 or 4 can be mathematically regarded as a trajectory in a three-dimensional space, it can be expressed as motion vector information on a two-dimensional plane by simple mathematical processing. It is possible. Therefore, it becomes easy to realize a graphic user interface on the screen.

The invention as set forth in claim 5 provides a method for performing scene retrieval using motion vector information expressed on this two-dimensional plane as a retrieval key. This makes it possible to search for a scene by inputting the movement of a moving object using a pointing device or the like.

[0021]

DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows a flowchart of the invention according to claim 1. Here, step 1 in the diagram
1 to 17 indicate each step of the flowchart.

The moving image presentation process of this embodiment is executed, for example, once every frame or once every n frames. While playing a moving image such as a video, step 1 is synchronized with this.
The process is started in step 1, and the current frame time Tf is read out from the moving image reproduction device in step 12. Next, step 1
Function F from time to position created in advance in 3.
(t) is given a frame time Tf, and the position P(x
, y). Using this result, in step 14, the user adds the desired processing based on P(x,y),
In step 15, this result is output on the screen. step16
In this step, determine whether processing is requested for the next frame as well, and if there is a request, repeat step 12 and later,
If not, the process ends in step 17.

[0023] Here, the processing in step 14 is performed using CG
If it is a drawing, the CG will be linked to the animal body as a result.
You can get the effect of moving. Also, if the mouse input result performed in advance is P(x, y) or P(x, y
), the moving object itself can be used as a hypertext button, and a link can be created from the moving object.

[0024] Although other processes can be considered in step 14, as long as the main moving image presentation method is used, step 1
It is within the scope of the present invention regardless of the process applied in step 4.

FIG. 2 is a diagram for explaining an embodiment of the invention according to claim 2. As shown in the figure, a space-time 1 is constituted by an X-Y plane representing a screen and a time axis representing a frame time T, and a locus 2 of the movement of a moving object is drawn within this space. The time and space including the movement of the moving body are defined by the start frame 3 at the start time T1 and the end time T4.
end frame 4 at and an appropriate reference time T2,
It is divided by reference frame 5 at T3, and the function of each section defined thereby is expressed as a function with respect to time, which is approximated by a straight line or a curve.

In this embodiment, the movement locus of the moving object is expressed as shown in FIG. 3, and the frame time Tf
is given, f(t) or g(t
) to obtain the position P(x,y) of the moving object on the screen.

FIG. 4 shows a flowchart in the invention according to claim 3. Here, step in the figure
21 to 27 indicate each step of the flowchart.

The moving image presentation process of this embodiment is executed, for example, once every frame or once every n frames. While playing a moving image such as a video, synchronize with this and step 2.
The process is started in step 1, and the current frame time Tf is read out from the moving image reproduction device in step 22. Next, step 2
The frame time Tf is given to the function G(t) from time to position and shape created in advance in step 3, and the position P(x, y) on the screen and the shape Fig are calculated. Using this result, in step 24, processing desired by the user is added based on P and Fig, and in step 25, this result is output on the screen. In step 26, it is determined whether processing is requested for the next frame as well, and if there is a request, repeat step 22 and subsequent steps; otherwise, proceed to step 27.
It ends with.

[0029] Here, the processing in step 24 is CG
If it is a drawing, the CG will be linked to the animal body as a result.
You can get the effect of moving. Also, if the mouse input result performed in advance is P(x, y) or P(x, y
), the moving object itself can be used as a hypertext button, and a link can be created from the moving object.

FIG. 5 is a diagram for explaining the invention according to claim 4. A space-time 6 similar to that shown in FIG. 2 is constructed, and a locus 7 of the position and shape of the moving object with respect to time is drawn within this space. The time and space including the movement of the moving body is divided into a start frame 8 with a start time T1, an end frame 10 with an end time T3, and a reference frame 9 with an appropriate reference time T2, and the time of each section defined by this is divided into Each position function is approximated by a straight line or a curve, and expressed as a locus of position relative to time. Furthermore, information on a rectangle 11 that approximates the shape of a moving object is provided for each of the start, end, and reference frames. Thereby, it is possible to calculate the approximate rectangle of the moving object in any frame by performing simple calculations.

FIG. 6 is a diagram for explaining an embodiment of the invention set forth in claim 5. This method uses the trajectory on the XY plane obtained by erasing time from the equation expressing the trajectory of the moving object in space and time as search keys 13, 14, 15, and 16 for each scene that indicate the movement of the moving object. Used every time. On the other hand, since the movement of the moving object appears as a trajectory on the screen (X-Y plane) to the eyes of the user viewing the moving image, the user uses the pointing device to trace the movement trajectory 12 of the moving object that the user has memorized. It is possible to search for a scene based on movement by inputting it and matching it with the search key that indicates movement.

In this embodiment, the search key 14 searches for a scene that matches the input motion locus and includes the motion description 17.

Furthermore, it is obvious that this embodiment can be used in combination with a scene search method using conventional techniques such as keywords, and this is also within the scope of the present invention.

[0034]

Effect of the invention As explained above, claims 1 and 3
According to the described invention, processing such as image processing based on movement can be automated by using the spatiotemporal description of a moving object. This makes it possible to reduce the time required to prepare a presentation when, for example, a moving object is to be displayed with emphasis during a presentation. Furthermore, since it becomes possible to use a moving object as a button in hypertext, it becomes possible to expand the range of applications that use moving images.

The invention according to claims 2 and 4 provides a data management method in which the movement of a moving object is a function from time to position. It is possible to significantly reduce storage capacity compared to .

Furthermore, according to the invention as set forth in claim 5, it is possible to perform moving image retrieval using a visual method called motion retrieval, whereas conventionally the only retrieval method was by inputting keywords from a keyboard. This is an effective method for retrieving moving images in which the user's memory is likely to be visual.

[Brief explanation of drawings]

FIG. 1 is a flowchart in the invention according to claim 1.

FIG. 2 is a diagram showing an embodiment of the data management method according to the invention as claimed in claim 2;

FIG. 3 is a diagram for explaining the embodiment of FIG. 2;

FIG. 4 is a flowchart in the invention according to claim 3.

FIG. 5 is a diagram for explaining an embodiment of the data management method according to the invention as set forth in claim 4;

FIG. 6 is a diagram for explaining an embodiment of a motion-based moving image search according to the invention;

[Explanation of symbols]

1 Space-time 2 Trajectory 3 Start frame 4 End frame 5 Reference frame 6 Space-time 7 Trajectory 8 Start frame 9 Reference frame 10 End frame 11 Approximate rectangle 12 Motion trajectory input

Claims (5)

[Claims] Claim 1: By recording the position of the moving object at each time in advance, the position of the moving object is made a function of time, and this is used to perform processing tailored to the position of the moving object at each time. A moving image presentation method characterized by presentation of moving objects based on movement. [Claim 2] By providing a plurality of positions of the moving object at times (referred to as reference times) that well represent the characteristics of the movement on a three-dimensional space-time consisting of two dimensions representing the screen and one dimension representing time. A moving image characterized in that the trajectory of the movement of the moving object expressed by a straight line or curved line that approximates this, or a combination thereof, is used as spatiotemporal information of the moving object, and the position of the object is returned when time is given. Data management methods. Claim 3: By recording the shape as well as the position of the moving object at each time, the position and shape of the moving object are made a function of time, and this is used to record the position and shape of the moving object at each time. 2. The moving image presentation method according to claim 1, wherein the moving image presentation method is characterized in that the moving object is presented based on changes in movement and shape by performing processing tailored to the moving image. Claim 4: By also providing a rectangle that approximates the shape of the moving body at the reference time, the trajectory of the movement of the moving body and the approximate rectangle at the reference time are used as spatiotemporal information of the moving body, and the time is given. 3. The moving image data management method according to claim 2, further comprising returning an approximate rectangle calculated based on an approximate rectangle at a reference time using a time relative to the reference time. 5. A moving image search method, characterized in that motion vector information of a moving object obtained from the moving image data management method according to claim 2 or 4 is treated as a search key, and a scene search is performed based on movement.