KR101737087B1 - Method and apparatus for video stabilization by compensating sigth direction of camera - Google Patents

Abstract

Frame camera movement is measured based on a relative angle of a camera sight line direction relative to a reference camera eye line direction for each frame of a video frame sequence, a camera movement path of a frame sequence is generated using inter- Disclosed is a video stabilization method for determining a camera sight line direction adjustment angle with respect to a user's gaze direction using a camera movement path and compensating a camera gaze direction using a camera gaze direction adjustment angle for each frame.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video stabilization method and a video stabilization method,

The present invention relates to a video stabilization technique for correcting motion and shake of a video camera.

Conventional video stabilization techniques are classified into two-dimensional analysis-based techniques and three-dimensional analysis-based techniques.

The technique based on the two-dimensional analysis includes a step of estimating motion, a step of setting a camera path through estimated motion, and a step of generating an image according to a camera movement path. In the motion estimation step, motion is measured by interpreting the photographed image as a two-dimensional motion model. In an actual camera, an image in which a three-dimensional space is projected as a two-dimensional image is obtained, but a two-dimensional motion model can not accurately model an actual three-dimensional motion. Therefore, although the technique based on the two-dimensional analysis can not completely perform the video stabilization operation, the efficient result can be obtained with a relatively small amount of calculation.

Video stabilization techniques based on 3D analysis reconstruct 3D spatial information and camera position information from 2D images using motion structure analysis techniques in general. A new camera position is defined using the reconstructed information, and image information at a new position is synthesized. The video stabilization technique based on 3D analysis requires a considerable amount of computation compared to the technique based on 2D analysis, but accurate video stabilization can be derived. However, the video stabilization process based on 3D analysis may cause image distortion in some areas.

The present invention discloses a video stabilization method and a video stabilization apparatus for providing a scene similar to an actual motion perceived by human vision while correcting camera movement and shake accurately and rapidly.

The video stabilization method according to the first embodiment of the present invention includes the steps of measuring camera-to-frame camera movement based on a relative angle of view in the direction of a camera's eye line relative to a reference camera's eye direction for each frame of a frame sequence of video; Generating a camera movement path of the frame sequence using the inter-frame camera movement, and determining a camera parameter for each frame using the camera movement path; And compensating the camera motion using the camera parameters for each frame.

A video stabilization method according to a second embodiment of the present invention includes the steps of measuring camera motion for each frame of a frame sequence of a video; Generating a camera movement path of the frame sequence using the camera movement for each frame and determining an angle of the camera's gaze direction between the user's gaze direction and the camera's gaze direction using the camera movement path; And compensating the camera line direction using the camera line direction adjusting angle for each frame.

According to a third aspect of the present invention, there is provided a video stabilization method comprising: measuring camera motion for each frame of a frame sequence of a video; Determining a target area in the line-of-sight direction of the frame sequence; Generating a camera movement path of the frame sequence using the camera movement for each frame and determining a camera's gaze direction adjustment angle with respect to a gaze direction toward the target area using the camera movement path; And compensating the camera line direction using the camera line direction adjusting angle for each frame.

The video stabilization method according to the fourth embodiment of the present invention includes the steps of: measuring inter-frame camera movement based on a difference angle of a camera sight line direction relative to a reference camera eye line direction for each frame of a frame sequence of video; Generating a camera movement path of the frame sequence using the inter-frame camera movement, and determining an angle of view direction adjustment of the camera relative to the user's gaze direction using the camera movement path; And compensating the camera line direction using the camera line direction adjusting angle for each frame.

A video stabilization method according to a fifth embodiment of the present invention includes the steps of: receiving a selection request of a target area in a line direction of the frame sequence from a user through a user interface; Measuring camera-to-frame camera motion based on a difference angle of a camera line-of-sight direction relative to a reference camera line-of-sight direction for each frame of a frame sequence of video; Generating a camera movement path of the frame sequence using the inter-frame camera movement, and determining a camera eye direction adjustment angle with respect to a visual line direction toward the target area using the camera movement path; And compensating the camera line direction using the camera line direction adjusting angle for each frame.

A video stabilization method according to a sixth embodiment of the present invention includes: receiving at least one selection request of a size and a stabilization level of a margin region for video stabilization from a user through a user interface; Frame camera movement relative to the reference camera eye line direction of the frame sequence of the video based on at least one of the size of the selected margin area and the stabilization level, Determining a gaze direction adjustment angle, and stabilizing the video camera by compensating the camera gaze direction using the camera gaze direction adjustment angle for each frame.

A video stabilization method according to a seventh embodiment of the present invention measures a relative frame-to-frame camera movement relative to a reference camera eye line direction of a frame sequence of a video and adjusts a camera eye direction adjustment angle And stabilizing the camera of the video by compensating the direction of the camera line-of-sight using the camera's gaze direction adjustment angle for each frame; Measuring a stabilization related indicator while performing the video camera stabilization step; And warning in real time if the stabilization related indicator exceeds a predetermined threshold.

The video stabilization apparatus according to the first embodiment of the present invention includes a camera motion measurement unit for determining a difference angle of a camera sight line direction relative to a reference camera sight line direction for each frame of a video frame sequence, ; A camera parameter determination unit for generating a camera movement path of the frame sequence using the inter-frame camera movement and determining camera parameters for each frame using the camera movement path; And a camera motion compensation unit for compensating the camera motion using the camera parameters for each frame.

A video stabilization apparatus according to a second embodiment of the present invention includes: a camera motion measurement unit for measuring camera motion for each frame of a frame sequence of a video; Generating a camera movement path of the frame sequence using the camera movement for each frame and determining a camera direction of view direction adjustment angle relative to the user's gaze direction using the camera movement path; And a camera view direction direction compensation unit for compensating the camera sight direction using the camera sight direction adjustment angle for each frame.

A video stabilization apparatus according to a third embodiment of the present invention comprises: a camera motion measurement unit for measuring camera motion for each frame of a frame sequence of a video; A line-of-sight-direction target area determining unit for determining a line-of-sight direction target area in the frame sequence; Generating a camera movement path of the frame sequence using the camera movement for each frame and determining a camera's gaze direction adjustment angle for determining a camera's gaze direction adjustment angle relative to the gaze direction toward the target area using the camera movement path part; And a camera direction of sight direction adjusting angle compensation unit for compensating the camera sight direction using an angle of view direction of the camera relative to the direction of the eye toward the target area for each frame.

The video stabilization apparatus according to the fourth embodiment of the present invention includes a camera motion measurement unit for determining a difference angle of a camera sight line direction relative to a reference camera sight line direction for each frame of a video frame sequence, ; Generating a camera movement path of the frame sequence using the inter-frame camera movement, and determining a camera eye direction adjustment angle in relation to the user's gaze direction using the camera movement path; And a camera view direction direction compensation unit for compensating the camera sight direction using the camera sight direction adjustment angle for each frame.

The video stabilization apparatus according to the fifth embodiment of the present invention includes a gaze direction target region user input unit for receiving a request for selecting a target region in the line direction of the frame sequence from a user through a user interface; A camera motion measurement unit for measuring a camera movement between frames by determining a difference angle of a camera sight line direction relative to a reference camera sight line direction for each frame of a video frame sequence; Generating a camera movement path of the frame sequence by using the inter-frame camera movement, and determining a camera eye direction adjustment angle corresponding to a direction of a gaze toward the target area using the camera movement path part; And a camera view direction direction compensation unit for compensating the camera sight direction using the camera sight direction adjustment angle for each frame.

A video stabilization apparatus according to a sixth embodiment of the present invention includes a margin area user input unit receiving a request for selection of at least one of a size of a margin area and a stabilization level from a user through a user interface; Frame camera movement relative to a reference camera sight line direction based on at least one of a magnitude and a stabilization level of the selected margin area and determines a camera eye line direction adjustment angle in relation to a visual line direction toward a target area And a video camera stabilizing unit for stabilizing the camera of the video by compensating the direction of the camera line of sight using the camera line direction adjusting angle for each frame.

The video stabilization apparatus according to the seventh embodiment of the present invention measures a relative frame-to-frame camera movement relative to a reference camera sight line direction of a frame sequence of a video, And stabilizing the camera of the video by compensating the direction of the camera line of sight using the angle of view direction of the camera for each frame; A stabilization related indicator measuring unit for measuring a stabilization related indicator while performing the video camera stabilization step; And a video stabilization warning unit that warns in real time when the stabilization related index exceeds a predetermined threshold value.

The present invention discloses a computer-readable recording medium on which a program for implementing a video stabilization method of the present invention is recorded.

Fig. 1 shows a block diagram of a video stabilization apparatus according to the first embodiment.
Fig. 2 shows a block diagram of a video stabilization apparatus according to a second embodiment.
3 shows a block diagram of a video stabilization apparatus according to the third embodiment.
4 shows a block diagram of a video stabilization apparatus according to the fourth embodiment.
5 shows a block diagram of a video stabilization apparatus according to a fifth embodiment.
Fig. 6 shows a block diagram of a video stabilization apparatus according to the sixth embodiment.
7 shows a block diagram of a video stabilization apparatus according to the seventh embodiment.
FIG. 8 illustrates a camera motion measurement scheme according to an embodiment.
FIG. 9 illustrates a camera rotation angle measurement method in the Z-axis direction according to an embodiment.
FIG. 10 shows the camera movement path, and FIG. 11 shows the existing video stabilization method.
FIG. 12 illustrates a camera movement path in which camera movement is adjusted based on the global camera line-of-sight direction of the camera movement path according to an embodiment.
13 illustrates a camera movement path in which camera motion is adjusted based on the direction of the camera's gaze toward the target area in accordance with one embodiment.
Figure 14 illustrates user interfaces for adjusting the size and video stabilization degree of the margin region according to the video stabilization operation according to one embodiment.
15 illustrates a user interface for a user to set a target area in the camera eye direction for a video stabilization operation in accordance with one embodiment.
16 illustrates a user interface for alerting an excessive video stabilization operation while performing a video stabilization operation according to one embodiment.
17 shows a flowchart of a video stabilization method according to the first embodiment.
18 shows a flowchart of a video stabilization method according to the second embodiment.
19 shows a flowchart of a video stabilization method according to the third embodiment.
20 shows a flowchart of a video stabilization method according to the fourth embodiment.
Fig. 21 shows a flowchart of a video stabilization method according to the fifth embodiment.
22 shows a flowchart of a video stabilization method according to the sixth embodiment.
23 shows a flowchart of a video stabilization method according to the seventh embodiment.

Hereinafter, video stabilization apparatuses according to various embodiments of the present invention are described with reference to Figs. The detailed stepwise operation of the video stabilization devices according to various embodiments is described above with reference to Figs. 8-13. 14 to 16, user interfaces for video stabilization devices according to various embodiments are also illustrated. 17-23 disclose video stabilization methods in accordance with various embodiments of the present invention.

The video stabilization apparatuses 100 and 400 according to the first and fourth embodiments of FIGS. 1 and 4 can be applied to a video stabilization system that corrects camera movement using only the relative camera rotation angles measured as inter- Technique.

The video stabilization apparatuses 200, 300, and 400 according to the second, third, and fourth embodiments of FIGS. 2, 3, and 4 are configured to determine whether the camera- Discloses a video stabilization technique for correcting camera shake using only the difference between the gaze directions.

The video stabilization devices 500, 600 and 700 according to the fifth to seventh embodiments of FIGS. 5, 6 and 7 are the same as the video stabilization devices 100, 200, 300, 400 according to the first to fourth embodiments ), And discloses a video stabilization technique including a user interface.

Fig. 1 shows a block diagram of a video stabilization apparatus according to the first embodiment.

The video stabilization apparatus 100 according to the first embodiment includes an inter-frame camera motion measurement unit 110, a camera parameter determination unit 120, and a camera motion compensation unit 130. The video stabilization apparatus 100 according to the first embodiment can output stabilized video by compensating camera movement using only the relative motion between the camera's eye directions of two frames as camera motion information.

The inter-frame camera motion measuring unit 110 according to the first embodiment acquires a frame sequence of video and measures inter-frame camera motion for each frame of the frame sequence. The camera parameter determination unit 120 according to the first exemplary embodiment generates a camera movement path of a frame sequence using the inter-frame camera movement measured by the inter-frame camera motion measurement unit 110, Camera parameters are determined for each frame. The camera motion compensation unit 130 according to the first embodiment compensates the camera motion for each frame by using the camera parameters determined by the camera parameter determination unit 120. [

The inter-frame camera motion measuring unit 110 can determine relative camera line direction information for each frame as compared to the reference camera eye direction as inter-frame camera motion information. For example, the inter-frame camera motion measuring unit 110 can determine a difference angle between the camera line direction of the current frame and the reference camera line direction as inter-frame camera motion information.

The reference camera view direction for determining the relative camera movement in the camera view direction of the current frame may be the camera view direction of the previous frame. For example, the inter-frame camera motion measuring unit 110 may determine a relative angle difference between a previous frame and a current frame in a direction of the camera line-of-sight for each pair of consecutive frames in the frame sequence.

Further, the reference camera sight line direction may be commonly determined with respect to the entire frame sequence. For example, the inter-frame camera motion measuring unit 110 measures the average of the difference angles of the camera line-of-sight direction between a pair of consecutive frames in the frame sequence, It may be determined in the direction of the camera line of sight. Accordingly, the inter-frame camera motion measuring unit 110 measures the average difference angle (A) between the camera line-of-sight direction, i.e., the camera line-of-sight direction, per frame, based on the average difference angle of the camera line- The difference angle AB between the camera's sight directions B may be measured as camera motion information.

The inter-frame camera motion measuring unit 110 can determine the difference angle of the inter-frame camera view direction as the camera motion information by the rotation angles in the X, Y, and Z axis directions. The inter-frame camera motion measuring unit 110 can measure the relative angle changes in the X and Y axis directions between the camera line direction of the current frame and the reference camera line direction to determine the X and Y axis components of the camera motion.

The inter-frame camera motion measuring unit 110 can determine the camera rotation angle in the Z-direction by using the camera movement in the linear direction generated in predetermined areas on the Z-axis plane by the camera rotation in the Z-axis direction. One or more sample regions for analyzing the camera movement in the linear direction generated in accordance with the camera rotation in the Z-axis direction may be selected.

The inter-frame camera motion measuring unit 110 determines errors occurring between the reference frames in the reference camera view direction and the sample areas of the current frame at the rotation angles within the predetermined range in the Z-axis camera rotation angles, And the total error is determined for each camera rotation angle. The inter-frame camera motion measuring unit 110 may detect a camera rotation angle at which the total error of the sample areas becomes the smallest among the Z-axis camera rotation angles within a predetermined range, and determine the relative camera rotation angle in the Z-axis direction.

The inter-frame camera motion measuring unit 110 can determine the difference angle of the relative eye direction of the inter-frame camera using the frames of the video sequence, that is, images. The inter-frame camera motion measuring unit 110 according to another embodiment may predict a camera motion using a sensor. For example, the inter-frame camera motion measuring unit 110 determines the difference angle of the relative direction of the inter-frame camera using the camera motion information obtained using the gyro sensor, the G sensor, the acceleration sensor, .

 The camera parameter determination unit 120 may accumulate the measured inter-frame camera motion information in the inter-frame camera motion measurement unit 110 to generate a camera motion path of the frame sequence. In addition, the camera parameter determination unit 120 can generate a global camera movement path of the camera movement path. For example, a global camera movement path can be created through low-pass filtering on the camera motion path. The degree of stabilization of the global camera movement path can be adjusted according to the size and intensity of the window for low-pass filtering.

Since the inter-frame camera movement measured by the inter-frame camera motion measuring unit 110 uses a relative camera movement such as a difference angle in the direction of the camera's gaze, the camera gaze of the camera movement path generated by the camera parameter determination unit 120 Direction and the global eye direction can also be defined as the relative camera movement, for example, a difference angle with respect to a predetermined reference camera eye direction.

The camera parameter determination unit 120 may determine the camera adjustment value in consideration of the global camera movement path for each frame. For example, when the camera movement path is changed to the global camera movement path through the video stabilization, the camera parameter determination unit 120 determines the camera movement path based on the actual camera movement path generated through the accumulation of the inter- The difference between the movement paths can be determined as the camera adjustment value.

The camera movement path can be defined as a line of sight vector of the camera on the movement path, that is, the camera position and the camera line direction. Therefore, the camera parameter determination unit 120 can determine, as the camera adjustment value, the displacement of the camera position between the actual camera movement path and the global camera movement path and the relative directional difference in the camera's gaze direction.

The camera parameter determination unit 120 determines a camera movement path based on the difference between the current camera line direction and the reference camera line direction on the camera movement path received from the inter-frame camera motion measurement unit 110, It is possible to determine a relative difference angle, that is, a rotation angle, between the camera movement paths. For example, the camera parameter determination unit 120 determines the angle of difference between the current camera line-of-sight direction and the reference camera line-of-sight direction on the camera movement path and the rotation angle between the global camera- , And can be determined by the angle of view direction of the camera.

The camera adjustment value determined for each frame by the camera parameter determination unit 120 may be output to the camera motion compensation unit 130 as a camera parameter for each frame. In particular, the inter-frame camera motion measuring unit 110 extracts only the amount of change in the camera's eye direction, that is, the rotation angle in the direction of the camera's eyes, as the camera parameter, and transmits the extracted rotation angle to the camera motion compensating unit 130.

The camera motion compensation unit 130 may compensate camera movement for each frame using camera parameters for each frame. For example, the camera motion compensating unit 130 may rotate the camera line direction on the actual camera moving path for each frame by the eye direction adjustment value using the camera parameters for each frame.

The video stabilization apparatus 100 according to an exemplary embodiment measures a camera movement relative to a frame of a video frame sequence, accumulates camera movement relative to each other to generate a camera movement path, Can be performed. In this case, relative camera movement between frames temporally preceding in the frame sequence of the single view video may be measured, and the camera movement path may be predicted.

Also, the video stabilization apparatus 100 according to the first embodiment may apply the video stabilization technique to the stereoscopic video. Specifically, the inter-frame camera motion measurement unit 110 of the video stabilization apparatus 100 according to an exemplary embodiment can measure inter-view camera motion with respect to the left and right viewpoint frames corresponding to the stereoscopic video . A relative directional difference, that is, a difference angle, in the direction of the camera line of sight of the remaining viewpoint can be determined with reference to the reference point-of-view camera direction among the left viewpoint and the right viewpoint. Accordingly, the inter-frame camera motion measuring unit 110 may measure the camera motion between the two viewpoints for each frame in time order, so that the video stabilizer 100 may perform the video stabilization operation on the stereoscopic video.

Fig. 2 shows a block diagram of a video stabilization apparatus according to a second embodiment.

The video stabilization apparatus 200 according to the second embodiment includes a camera motion measurement unit 210, a camera's gaze direction adjustment angle determination unit 220, and a camera's gaze direction compensation unit 230. The video stabilization apparatus 200 according to the second embodiment generates a camera movement path using camera movement and compensates for camera movement by adjusting only the direction of the camera's sight line except for the camera position in the camera movement path, Can be output.

The camera motion measurement unit 210 according to the second embodiment receives a frame sequence of a video and measures camera motion for each frame of the frame sequence. The camera-view-direction-direction-adjusting-angle determining unit 220 according to the second embodiment generates a camera movement path of the frame sequence using the camera movement for each frame measured by the camera-motion measuring unit 210, And determines the angle of view direction of the camera relative to the user's gaze direction. The camera's gaze direction compensation unit 230 according to the second embodiment compensates the camera's gaze direction for each frame using the camera's gaze direction adjustment angle determined by the camera's gaze direction adjustment angle determination unit 220, The video stabilization apparatus 200 according to the embodiment can complete the video stabilization operation.

The camera motion measuring unit 210 can determine the camera line-of-interest vector for each frame to measure camera motion for each frame. The camera direction of sight direction adjustment unit 220 can generate a camera movement path of a frame sequence by accumulating camera motion for each frame determined by the camera motion measurement unit 210. [ Therefore, the camera movement path generated by the camera's gaze direction adjustment angle determination unit 220 may include information on the gaze vector of the camera for each frame.

The camera gaze direction adjusting angle determining unit 220 can predict the user's gaze direction using the camera moving path of the frame sequence. For example, the camera's gaze direction adjustment angle determination unit 220 can separate the global camera gaze direction and the regional camera gaze direction from the generated camera movement path, and determine the global camera gaze direction as the user's gaze direction.

There are various methods for separating the global camera line-of-sight direction from the camera movement path of the frame sequence. For example, the global camera movement path of the frame sequence is determined through low-pass filtering on the camera movement path, and the camera's gaze direction on the global movement path can be determined as the global camera gaze direction. However, the method of determining the global movement route is not limited to the low-pass filtering.

The camera direction of sight direction adjusting section 220 can determine the camera parameters using only the camera direction of the camera except for the camera position in the camera moving path. That is, the camera's gaze direction adjusting angle determining unit 220 can determine the angle of difference between the user's camera's gaze direction and the current camera's gaze direction on the camera moving path by the camera's gaze direction adjusting angle.

The camera sight line direction of the camera movement path can be adjusted to face the center of the frame. That is, the camera's gaze direction adjustment angle determination unit 220 may detect the camera's gaze direction adjustment angle for moving the current camera's gaze direction to the center of the frame.

The camera's gaze direction compensation unit 230 can compensate camera movement by rotating the current camera's gaze direction by the camera's gaze direction adjustment angle determined by the camera's gaze direction adjustment angle determination unit 220. [

Since the video stabilization apparatus 200 according to the second embodiment adjusts only the direction of the camera line of sight except for the camera position in the camera movement path, the camera position is maintained in the actual camera movement path, The video stabilization of the effect can be performed.

In addition, the camera motion measuring unit 210 measures the camera motion between the both viewpoints for each of the frames in time sequence with respect to the left view frame and the right view frame corresponding to each other of the stereoscopic video, The stabilization device 200 may also perform video stabilization operations on the stereoscopic video.

3 shows a block diagram of a video stabilization apparatus according to the third embodiment.

The video stabilization apparatus 300 according to the third embodiment includes a camera motion measurement unit 310, a visual direction target area determination unit 315, a camera's gaze direction adjustment angle determination unit 320, and a camera's gaze direction compensation unit 330 ). The video stabilization apparatus 300 according to the third embodiment adjusts only the camera line direction of the camera movement path to compensate for the camera movement as in the video stabilization apparatus 200 according to the second embodiment, Can be arbitrarily selected.

The camera motion measuring unit 310 according to the third embodiment receives the frame sequence of the video and measures the camera motion every frame of the frame sequence. The gaze direction target area determining unit 315 according to the third embodiment determines a target area of the frame sequence facing the camera line direction. The camera view direction adjustment angle determiner 320 according to the third exemplary embodiment generates a camera movement path of the frame sequence using the camera movement of each frame measured by the camera motion measurement unit 310, The angle of view direction of the camera relative to the gaze direction toward the target area is determined for each point. The camera's gaze direction compensation unit 330 compensates the camera's gaze direction every frame using the camera's gaze direction adjustment angle determined by the camera's gaze direction adjustment angle determination unit 320 according to the third embodiment.

The camera motion measuring unit 310 may determine a camera eye direction vector for each frame to measure camera motion for each frame.

The gaze direction target area determining unit 315 can analyze the camera line-of-sight direction of each frame of the frame sequence to determine the target area facing the camera's gaze direction. The gaze direction target area determining unit 315 can determine the target area in the camera's gaze direction by analyzing the camera line-of-sight direction of each frame of the frame sequence. For example, the gaze direction target region determining unit 315 may analyze the camera line-of-sight directions of the frame sequence to detect a representative target area toward the camera line-of-sight direction of the frame sequence, or to detect a target area in the global camera- have.

Or the gaze direction target area determining unit 315 can determine the target area in the camera's gaze direction based on a request to arbitrarily select a target area in the direction of the camera's gaze direction or to select a target area in the camera's sight direction direction received from the outside have. The gaze direction target area determining unit 315 can transmit information on the target area in the gaze direction to the camera gaze direction adjusting angle determiner 320. [

The camera view direction adjustment angle determiner 320 generates a camera movement path of the frame sequence using the camera movement for each frame measured by the camera motion measurement unit 310, Determines the angle of view direction of the camera relative to the line of sight.

The camera view direction adjustment angle determiner 320 can generate a camera movement path of a frame sequence by accumulating camera motion for each frame determined by the camera motion measurement unit 310. [ The camera movement path generated by the camera direction of sight direction adjustment unit 320 may include information about the camera's sight line vector for each frame.

The camera gaze direction adjusting angle determining unit 320 can predict the user's gaze direction using the camera moving path. When the camera view direction adjusting angle determining unit 320 receives information on the target area in the view direction from the view direction target area determining unit 315, The adjustment angle can be determined.

The camera's gaze direction adjusting angle determiner 320 can determine the camera gaze direction adjusting angle with respect to the gaze direction toward the target area using the user's gaze direction or the global camera's gaze direction.

First, the camera's gaze direction adjustment angle determination unit 320 can separate the global camera gaze direction and the regional camera gaze direction from the current camera gaze direction of the camera movement path. The camera's gaze direction adjustment angle determiner 320 can determine a first adjustment angle in the direction of the camera's gaze relative to the global camera's gaze direction for each frame.

Based on the information on the target area in the camera view direction received from the sight direction target area determination unit 315, the camera view direction direction adjustment angle determination unit 320 determines, based on information about a point at which the global camera eye direction is oriented on the frame, The displacement or coordinate difference between the regions can be analyzed. The camera's gaze direction adjustment angle determination unit 320 detects the difference angle between the global camera gaze direction and the camera's gaze direction toward the target area using the displacement or coordinate difference between the point where the global camera gaze direction is directed and the target area , And can be determined by the second adjustment angle in the camera view direction.

The camera's gaze direction adjustment angle determination unit 320 may adjust the camera's gaze direction by integrating the first adjustment angle and the second adjustment angle in the camera's eye direction every frame. The camera motion compensation unit 330 can compensate for the camera shake by compensating the direction of the camera's gaze based on the information on the camera's gaze direction adjustment angle outputted from the camera's gaze direction adjustment angle determiner 320. [

In addition, the camera motion measuring unit 310 measures the camera motion between the both viewpoints in frames according to the time order with respect to the left view frame and the right view frame corresponding to each other of the stereoscopic video, Apparatus 300 may also perform video stabilization operations on stereoscopic video.

4 shows a block diagram of a video stabilization apparatus according to the fourth embodiment.

The video stabilization apparatus 400 according to the fourth embodiment includes an inter-frame camera motion measurement unit 410, a camera's gaze direction adjustment angle determination unit 420, and a camera's gaze direction compensation unit 430. In the video stabilization apparatus 400 according to the fourth embodiment, the camera movement path is generated by measuring the camera movement only by relative movement between the camera's eye directions of the two frames, and only the camera view direction So that stabilized video can be output by compensating the camera movement.

The inter-frame camera motion measuring unit 410 according to the fourth embodiment receives the frame sequence of the video and measures inter-frame camera motion based on the relative difference in the direction of the camera's eye line relative to the direction of the reference camera line per frame . The camera view direction adjustment angle determiner 420 according to the fourth exemplary embodiment generates a camera movement path of a frame sequence using the inter-frame camera motion information obtained by the inter-frame camera motion measurement unit 410, And determines the angle of view direction of the camera relative to the direction of the target camera using the movement path. The camera's gaze direction compensation unit 430 according to the fourth embodiment compensates the camera's gaze direction for each frame using the camera's gaze direction adjustment angle determined by the camera's gaze direction adjustment angle determination unit 420. [

The inter-frame camera motion measuring unit 410 according to the fourth embodiment performs the same operation as the inter-frame camera motion measuring unit 210 according to the second embodiment. That is, the inter-frame camera motion measuring unit 410 can determine the relative angle difference between the pair of consecutive frames in the time sequence of the frame sequence as the camera motion information. The inter-frame camera motion measuring unit 410 may transmit the inter-frame camera motion information to the camera's visual-axis direction adjusting angle determining unit 420. [

The inter-frame camera motion measuring unit 410 may measure relative camera rotation angles relative to the direction of the reference camera eye line for each of the X, Y, and Z axis directions to determine the camera motion information. The inter-frame camera motion measuring unit 410 can determine the difference angle of relative direction of the inter-frame camera using the frames of the video sequence, that is, images. Also, the inter-frame camera motion measuring unit 410 may determine the difference angle of the relative line-of-sight direction of the inter-frame camera using the predicted camera motion using the sensor.

The camera view direction adjustment angle determiner 420 according to the fourth embodiment accumulates inter-frame camera movement based on the inter-frame camera motion information received from the inter-frame camera motion measurement unit 210, A movement path can be generated. The camera movement path generated by the camera's gaze direction adjustment angle determination unit 420 can be defined only by the relative difference value in the camera's gaze direction except for the camera position information.

The current camera gaze direction information can be extracted from the camera movement path. The camera gaze direction adjustment angle determination unit 420 can determine the camera gaze direction adjustment angle for changing the current camera gaze direction to a desired camera gaze direction.

First, the camera's gaze direction adjusting angle determining unit 420 can predict the user's gaze direction from the current camera's gaze direction on the camera moving path. The camera gaze direction adjusting angle determining unit 420 can determine the global moving path using the camera moving path and determine the gaze direction on the global moving path as the user's gaze direction. The camera gaze direction adjusting angle determining unit 420 can determine the first adjusting angle in the camera view direction with respect to the user's gaze direction (global camera gaze direction) for each frame.

The video stabilization apparatus 400 according to the fourth embodiment may further include a gaze direction target area determination unit 415. [ When the target area information is received from the gaze direction target area determining unit 415, the camera gaze direction adjusting angle determining unit 420 determines, for each point on the camera moving path of the frame sequence, The angle of view direction adjustment can be determined.

In this case, the camera-view-direction-direction-adjusting-angle determining unit 420 determines whether or not the camera's gaze direction (the direction of the global camera line direction) in comparison with the user's gaze direction And the second adjustment angle in the direction of the camera line of sight between the user's gaze direction (the global camera's gaze direction) and the camera's gaze direction toward the target area may be integrated to determine the camera's gaze direction adjustment angle.

The camera's gaze direction adjustment angle determination unit 420 determines the camera's gaze direction using the rotation angle information between the current camera's gaze direction and the reference camera's gaze direction on the camera movement path received from the interframe camera motion measurement unit 410, The direction adjustment angle can be determined.

For example, the camera's gaze direction adjustment angle determination unit 420 determines the rotation angle between the current camera line direction and the reference camera line direction on the camera movement path, and the rotation angle between the reference camera line direction and the camera line direction toward the target area And can be determined by the angle of view direction of the camera. Since the current camera line-of-sight direction is defined as a relative rotation angle with respect to the reference camera line-of-sight direction, the angle of camera line-of-sight direction adjustment can be easily determined using the relative rotation angle between the reference camera line-

The camera gaze direction compensating unit 430 can correct the camera movement by rotating the camera gaze direction by the camera gaze direction adjusting angle for each frame.

The gaze direction target area determining unit 415 may set the global camera gaze direction of the camera moving path to the initial value of the gaze direction toward the target area.

In the video stabilization apparatus 400 according to the fourth embodiment, in a manner similar to the video stabilization apparatuses 100, 200, and 300 according to other embodiments, the camera motion measurement unit 410 may measure the correspondence of the stereoscopic video The video stabilization operation may be performed on the stereoscopic video by measuring the camera motion between the both viewpoints for each of the frames in the time sequence with respect to the left viewpoint frame and the right viewpoint frame.

5 shows a block diagram of a video stabilization apparatus according to a fifth embodiment.

The video stabilization apparatus 500 according to the fifth embodiment includes a gaze direction target region user input section 520 and a video stabilization apparatus 510. [ The video stabilization apparatus 510 according to the fifth embodiment may be the video stabilization apparatuses 300 and 400 according to the third and fourth embodiments of the present invention described above.

The gaze direction target area user input unit 520 according to the fifth embodiment receives a selection request of the target area in the gaze direction of the frame sequence from the user through the user interface. The video stabilization apparatus 510 according to the fifth embodiment can receive information on the target area in the direction of the camera line of sight transmitted from the gaze direction target area user input unit 520. [ The video stabilizer 510 can adjust the camera's gaze direction in relation to the camera gaze direction toward the target area in the same manner as the video stabilization devices 300 and 400. [ Thus, the video stabilization apparatus 500 according to the fifth embodiment can perform video stabilization that generates a result similar to a human visual perception by adjusting the direction of the camera's visual line of the video to a target area desired by the user .

6 shows a block diagram of a video stabilization system according to a sixth embodiment.

The video stabilization apparatus 600 according to the sixth embodiment includes a margin region user input unit 620 and a video stabilization apparatus 610. [ The video stabilization apparatus 610 according to the sixth embodiment may be the video stabilization apparatuses 100, 200, 300 and 400 according to the first, second, third and fourth embodiments of the present invention described above.

The margin area user input unit 620 according to the sixth embodiment receives a selection request of at least one of the size of the margin area and the stabilization level for video stabilization from the user through the user interface. The margin area user input unit 620 may determine the stabilization level based on the size of the margin area selected by the user or may determine the size of the margin area based on the stabilization level selected by the user. Also, the margin area user input unit 620 may receive a selection request of a size and a stabilization level of a margin region of a predetermined combination from the user.

The video stabilization apparatus 610 according to the sixth embodiment can receive information on the target area in the direction of the camera's gaze transmitted from the margin area user input unit 620. [ The video stabilizer 610 may compensate camera movement by adjusting the direction of the camera line of sight in relation to the direction of the predetermined camera line direction in the same manner as the video stabilization devices 100, 200, 300, and 400. However, the video stabilizing device 610 may restrict the adjustment amount of the camera's gaze direction based on at least one of the size of the selected margin area and the stabilization level. For example, based on the size or stabilization level of the margin region, the size of the measured camera movement, the size of the camera's gaze direction adjustment angle, and the amount of change due to the camera's gaze direction compensation may be limited.

Thus, the video stabilization apparatus 600 according to the sixth embodiment can prevent unreasonable video stabilization while performing video stabilization similar to human visual perception by adjusting the camera's eye direction of the video.

7 shows a block diagram of a video stabilization apparatus according to the seventh embodiment.

The video stabilization apparatus 700 according to the seventh embodiment includes a video stabilization apparatus 710, a stabilization index measurement section 720 and a video stabilization warning section 730. [

The video stabilization apparatus 710 according to the seventh embodiment includes the video stabilization apparatuses 100, 200, 300, 400, 500 according to the first, second, third, fourth, fifth, , 600). That is, the video stabilization device 710 measures camera movement for each frame of video, generates a camera movement path using the camera movement, adjusts the direction of the camera's eye direction The camera movement of the video can be stabilized by determining the angle and compensating the direction of the camera line of sight.

The stabilization index measurement unit 720 according to the seventh embodiment can detect the stabilization related index by sensing the video camera stabilization operation of the video stabilization apparatus 710 in real time. The video stabilization warning unit 730 according to the seventh embodiment can warn in real time when the measured stabilization related index exceeds a predetermined threshold value.

The stabilization related index according to an exemplary embodiment may include at least one of a size of a margin region generated through a video stabilization operation, a motion size of a video camera sensed through a sensor, and a size of a camera movement between frames.

The stabilization index measurement unit 720 according to the seventh embodiment detects the video camera stabilization operation of the video stabilization apparatus 710 in real time and performs the inter-frame camera motion measurement operation, the camera eye direction adjustment angle determination operation, For at least one of the compensation operations, a stabilization related indicator can be measured.

Accordingly, the video stabilization apparatus 710 according to the seventh embodiment can detect the video stabilization operation in real time, and alert the user that excessive video stabilization is performed when the video stabilization index exceeds the danger level.

The video stabilization method for correcting the camera shake using the camera rotation angle relative to each other as the camera motion information is described above with reference to the video stabilization apparatuses 100 and 400 of Figs. 1 and 4, A video stabilization method has been described with reference to the video stabilization devices 200, 300, and 400 to adjust only the camera line-of-sight direction of the camera movement path to be directed toward a predetermined target area. By the video stabilization method according to various embodiments of the present invention, camera motion correction and video stabilization similar to human visual perception can be implemented at high speed.

The user B who watches the video photographed by the shaking camera may feel severe dizziness because the camera shakes or moves when the user A who shoots the video with the camera shoots while riding the camera. On the other hand, the user A does not feel dizzy due to the image visually perceived by the user A even though the user A is shaking or moving.

The reason is that although the position of the eye of the user A changes abruptly according to the motion of the user A, the line of sight of the user A is directed to the object which is continuously focused. That is, since the gaze direction continues to be fixed toward the focus target and the focus target is continuously positioned at the center of the image of the eye of the user A, even though the position of the user eye continues to move, I do not feel dizziness.

As described above, the motion of the human eye in which the eye direction of the user's eyes continuously changes as the position of the eyes continues to move is also applied to the camera. Accordingly, the video stabilization devices 100, 200, 300, 400, 500, 600, and 700 according to various embodiments of the present invention adjust the camera's gaze direction toward a predetermined focus object according to camera movement, Video stabilization can be implemented.

To this end, the present invention discloses a video stabilization technique for correcting camera movement by compensating for a camera line-of-sight direction by a difference between a camera line direction of a current frame and a predicted user line direction using only the user's gaze direction in the camera movement path. In addition, since the present invention adjusts only the direction of the camera line-of-sight, only a rotation angle of the camera line relative to the visual line direction, i.e., the line-of-sight direction is measured, and only the camera line direction on the camera movement path is predicted.

First, an embodiment of a method for predicting relative camera motion between temporally adjacent frames will be described below. (K + 1) th image and the (K + 1) th image in order to match the gaze direction of the K th and (K + 1) th images when the gaze directions between the K th image and the You do not need to know the exact line of sight. Instead of knowing only the difference in the direction of the gaze vector between the K-th image and the (K + 1) -th gaze, that is, the direction of the inter-frame gaze direction, Can be rotated so as to coincide with the viewing direction of the K-th image. Accordingly, the video stabilization apparatuses 100, 400, 500, 600, and 700 according to an embodiment can predict only the relative gaze vector difference between two frames.

In addition, the user's gaze direction can be predicted using relative camera motion, and the camera's gaze direction can be adjusted by measuring the difference between the camera gaze direction of the current frame and the predicted user gaze direction. In order to align the gaze directions of the N frames with one gaze direction, the gaze directions of the N frames relative to the reference gaze direction may be changed. Camera motion during the frame sequence of N frames can be predicted using only relative directional differences between the N frames and the reference line-of-sight direction, even though the reference line-of-sight direction is not precisely defined in the three-dimensional space.

For example, for N frames, the average of the (N-1) gaze direction differences may be determined by obtaining the differences of the relative gaze directions, i.e., (N-1) gaze directions, between adjacent frames. Thus, the average difference information of (N-1) eye directions can be defined as the final reference line direction of N frames. The final reference viewing direction may also be defined as the rotation angle relative to the initial viewing direction, such as the gaze vectors of the N frames. At this time, the final reference line of sight may be the user's line of sight direction, and the final reference line of sight may be changed to another predetermined target line direction.

In addition, the camera shake can be corrected similarly to the human vision by adjusting only the direction of the camera line of sight by the difference angle between the final reference line direction and the current camera line direction without changing the camera position.

8 and 9, a camera motion measurement method according to various embodiments of the present invention in which a relative camera rotation angle between frames is measured will be described in detail.

FIG. 8 illustrates a camera motion measurement scheme according to an embodiment.

In order to accurately determine the correlation between two images taken at other general positions, information about the distance between the center points of the two cameras and the direction of the camera gaze vectors is needed. However, by using the information about the distance between the center points of two cameras and the direction of the camera's eye vectors, it is possible to measure the motion between two cameras by obtaining the exact position and coordinates of the objects formed on the two images and then determining the displacement or coordinate difference Requires a considerable amount of computation.

On the other hand, according to the camera motion measurement method according to the present invention, by measuring the relative angle between the two camera's gaze directions, i.e., the rotation angle of the remaining camera's gaze direction relative to the reference camera's gaze direction, . In addition, the difference in the gaze direction vectors of the two cameras can be determined in the X, Y, and Z-axis directions, respectively.

For example, the camera view direction 820 of the original image 800 of the K-th image toward the current position 810 is changed to the reference camera view direction 830 toward the image center 860, so that the current position 810 Can be formed at the center 860 of the image.

In fact, since the camera movement between consecutive images is not large compared to the distance between the camera and the object being photographed, the distance between the two cameras can be approximated to zero. In addition, a trigonometric function can be used to measure relative angles between specific points through the positional relationship between the center of the image and specific points on the image. However, the trigonometric function can be approximated to a general polynomial because the rotation component in the camera motion between consecutive images is not really large. Assuming the characteristics between these two cameras, the rotation angle of the camera can be simply described by a general polynomial using the coordinates of specific points on the image, and a relative angle change value in the X-axis and Y-axis directions is derived as a result .

The rotation angle? Between the current camera view direction 820 and the reference camera view direction 830 can be predicted using the center coordinate m of the current position 810 and the center coordinate m 'of the image center 860. If the displacement between m and m 'has components in the X-axis and Y-axis directions, the rotation angle in the X-axis direction and the rotation in the Y-axis direction in accordance with the displacement in the X- Angles may be predicted.

FIG. 9 illustrates a camera rotation angle measurement method in the Z-axis direction according to an embodiment.

The rotation angle in the Z-axis direction can be measured using the linear movement of a plurality of sample points in the image. For example, rotation of the image 900 in the Z-axis direction may cause movement in each linear direction on the four sample points, A, B, C, and D points.

By the same rotation, point A and point C have linear motion in opposite directions, and point B and point D have a linear motion in the opposite direction. For example, when a clockwise rotation occurs in the Z-axis direction, the point A is right, the point B is down, the point C is left, and the point D is a straight line.

If the horizontal length and the vertical length of the image are different, the linear motion size of point A and point C and the linear motion size of point B and point D are different by the same rotation. For example, when the horizontal and vertical lengths of the image are W and H and the image rotates by the angle of rotation θ, the linear momentum of the point B and D is about W / 2 * θ, and the points A and C are about H / 2 * [theta]. Therefore, it is preferable that the magnitude of the linear motion due to the rotational motion of each sample point is determined based on the horizontal and vertical lengths of the image.

Thus, by the same rotation, a linear motion of a predetermined magnitude and direction occurs at the sample points, and the magnitude and direction of the linear motion vectors occurring at the sample points corresponding to the rotation angle are determined from the origin of the Z- Can be determined based on the distance and direction to the sample point.

After determining a cost value, such as a sum of absolute differences (SAD) at each matching point on the current frame and the reference frame, for each sample point, the magnitude and direction of movement in which the final cost value based on the course value of all sample points is minimized The vector may be determined as a motion vector between the current frame and the reference frame. That is, the cost values at the points A, B, C, and D are respectively determined and then integrated to determine one final cost value. The motion amount of the motion vector in which the final cost value is minimized is determined as the final linear motion amount . When the linear motion amount is changed to the rotation motion, the rotation angle in the Z-axis direction can be determined.

As with sample points A, B, C, and D, you can increase or decrease the number of sample points for momentum measurement. A method of measuring a motion vector of a sample region by selecting a sample region around a sample point in order to measure an accurate motion at each point may be used.

In order to measure the camera movement in the Z-axis direction between frames, the inter-frame camera motion measurement units (100, 400) measure the camera movement in the direction of the straight line generated in the sample areas on the Z- The size and direction of the motion vectors can be determined. The cost based on the error (c = ab) between the image (a) shifted by each linear motion vector and the corresponding area (b) of the frame in the reference camera eye direction is measured and integrated for each sample area, Can be determined.

The inter-frame camera movement measuring unit 100, 400 compares the total cost of the sample areas determined for each rotation angle, detects the rotation angle that generates the minimum total cost, and stores the rotation angle in the Z- It may be determined by the rotation angle.

Accordingly, the inter-frame camera motion measuring unit 100, 400 estimates only the camera rotation angle without predicting the actual camera position and displacement among the inter-frame camera movement, and transmits the information on the inter-frame camera rotation angle to the camera parameter determination unit 120, or the camera's gaze direction adjustment angle determination unit 220, 320, 420. Therefore, it is possible to reduce the calculation amount for measuring the accurate displacement of the camera movement among the calculation amount for measuring the camera movement and reduce the amount of information to be transmitted to the camera parameter determination unit 120 or the camera view direction adjustment angle determination unit 220, 320, Can be reduced.

FIG. 10 shows the camera movement path, and FIG. 11 shows the existing video stabilization method.

The camera movement path 1000 can be obtained by using the acquired video as a result of the user taking the camera while moving. Information on the camera moving along the camera movement path 1000 can be defined in terms of the camera position and the camera view direction for each frame.

By analyzing the frame sequence of the camera movement path 1000, a global movement path 1030 occurring along the camera movement path 1000 can be obtained. The actual camera view direction 1002, 1004, 1006, 1008, 1010, 1012, 1014, 1016, and 1018 generated in each frame of the camera movement path 1000 is determined based on the global sight line based on the global motion path 1030 Direction and the direction of the local line of sight that occurs in each frame.

The actual camera line-of-sight directions 1002 and 1004 correspond to the global camera line-of-sight directions 1032 and 1034 along the global movement path 1030, And can be separated in the gaze direction.

According to the conventional video stabilization method, the actual camera position and the gaze direction on the camera movement path 1000 are predicted, and a method in which the global camera movement path 1030 is matched is used.

That is, after the actual camera line-of-sight directions 1002 and 1004 and the global camera line-of-sight directions 1032 and 1034 are respectively measured with reference to the origin of the coordinate system, the actual camera line-of-sight directions 1002 and 1004 and the global camera- , 1034 are determined. The actual camera line-of-sight directions 1002 and 1004 can be rotated by? 2 and? 4 and matched with the global camera view direction 1032 and 1034.

In addition, the actual camera position of the camera view direction 1002, 1004 on the camera movement path 1000 and the position on the global camera movement path 1030 corresponding thereto are predicted. The difference D2, D4 between the actual camera position of the camera view direction 1002, 1004 and the corresponding position on the global camera movement path 1030 is determined. The positions of the actual camera line-of-sight directions 1002 and 1004 can be shifted by the position differences D2 and D4, and can be matched to the positions on the global movement path 1030. [

Therefore, the video stabilization method according to the conventional method accurately measures both the position and the gaze direction of the killer camera on the camera movement path, the camera position and the gaze direction on the global movement path, and determines that the actual camera movement path is a global movement path Should be adjusted to match.

The video stabilization apparatuses 200, 300, and 400 according to the second, third, and fourth embodiments generate a camera movement path to predict a user's gaze direction, The camera shake can be corrected using only the difference between the user's gaze directions. 12 and 13, a camera shake correction method according to various embodiments of the present invention for adjusting the camera's gaze direction toward a predetermined viewing direction will be described in detail.

FIG. 12 illustrates a camera movement path in which camera movement is adjusted based on the global camera line-of-sight direction of the camera movement path according to an embodiment. 13 illustrates a camera movement path in which camera motion is adjusted based on the direction of the camera's gaze toward the target area in accordance with one embodiment.

The camera view direction adjustment angle determination units 220, 320, and 420 of the video stabilization apparatuses 200, 300, and 400 according to the second, third, and fourth embodiments include a camera motion measurement unit 220 The camera movement path 1000 can be generated using the camera rotation angle information obtained from the inter-frame camera motion measurement unit 420 or the inter-frame camera motion measurement unit 420. The current camera line directions 1002, 1004, 1006, 1008, 1010, 1012, 1014, 1016, and 1018 can be defined for each frame according to the camera movement path 1000.

The video stabilization devices 200, 300, and 400 according to the second, third, and fourth embodiments of the present invention do not move the actual position of the camera, It is not necessary to estimate the exact camera position on the camera movement path, but only a relative rotation angle of the camera in the direction of the camera line direction in the predetermined direction is required.

That is, the camera direction of sight direction determining unit 220, 320, and 420 determines the global motion path 1030 generated along the camera motion path 1000 by analyzing the frame sequence of the camera motion path 1000 The camera line directions 1002, 1004, 1006, 1008, 1010, 1012, 1014, 1016, and 1018 generated in each frame of the camera movement path 1000 are the global sight line based on the global motion path 1030, Direction 1032, 1034, 1036, 1038, 1040, 1042, 1044, 1046, 1048 and the local eye direction occurring in each frame.

The camera gaze direction adjustment angle determination units 220, 320, and 420 determine whether or not the global gaze direction based on the camera gaze directions 1002, 1004, 1006, 1008, 1010, 1012, 1014, 1016, and 1018 and the global movement path 1030 The camera's gaze direction adjustment angle can be determined based on the rotation angle or the angle difference between the gaze directions 1032, 1034, 1036, 1038, 1040, 1042, 1044, 1046, The camera direction of sight direction adjusting units 220, 320, and 420 may transmit information on the camera direction of sight direction adjustment determined for each frame to the camera direction direction compensating units 230, 330, and 430.

In this case, since the camera movement path generated according to the fourth embodiment is generated based only on the inter-frame camera rotation angle information, the accurate camera position and the accurate camera photographing angle are not determined on the camera movement path, The continuous rotation angle in the camera's gaze direction along the movement path can be determined. Accordingly, the camera viewing directions 1002, 1004, 1006, 1008, 1010, 1012, 1014, 1016, and 1018 determined by the camera viewing direction adjusting angle determining unit 420 may be a rotation angle with respect to a predetermined reference viewing direction. Similarly, the direction of the camera's gaze direction, which is determined by the camera's gaze direction adjustment angle determination unit 420, is defined as a global gaze direction, a regional gaze direction, a camera gaze direction adjustment angle toward a predetermined gaze direction, and a direction difference relative to the reference gaze direction, .

The camera gaze direction compensation units 230, 330, and 430 according to the second, third, and fourth embodiments detect the direction of the camera eye, which is received from the camera gaze direction adjustment angle determination unit 220, 320, Based on the adjustment angle information, it is possible to compensate the camera movement by adjusting the current camera line-of-sight direction by the angle of the camera's gaze direction.

For example, the camera line-of-sight direction compensation units 230, 330, and 430 may detect camera direction of the camera 1002, 1004, 1006, 1008, 1010, 1012, 1014, 1016, It can be rotated by the angle of view direction adjustment.

According to FIG. 12, the camera's gaze direction adjustment angle determination units 220, 320, and 420 can determine the camera's gaze direction adjustment angle information based on the global gaze direction. The camera view direction adjustment angle determination units 220, 320, and 420 determine the camera view direction adjustment angle determination unit 220, 320, and 420 in accordance with the current camera direction (1002, 1004, 1006, 1008, 1010, 1012, 1014, 1016, 1044, 1046, and 1048) may be measured to determine the angle of view direction of the camera. That is, the camera gaze direction compensation units 230, 330, and 430 rotate the camera view direction 1002, 1004, 1006, 1008, 1010, 1012, 1014, 1016, Can be matched with the global eye direction (1032, 1034, 1036, 1038, 1040, 1042, 1044, 1046, 1048).

Therefore, only the camera direction of the camera can be determined by the camera direction of sight direction compensation units 230, 330, and 430 without changing the position of the current frame on the camera moving path 1000, , 1064, 1066, 1068). In other words, the adjusted camera view directions 1052, 1054, 1056, 1058, 1060, 1062, 1064, 1066, 1046, 1048).

According to Fig. 13, the camera gaze direction adjustment angle information can be determined so that the camera gaze direction is directed to a predetermined target point. The camera's gaze direction adjusting angle determining units 220, 320 and 420 measure the difference angle between the current camera gaze direction on the camera moving path 1000 and the target gaze direction toward the target point 1330, You can decide. That is, the camera view direction direction compensation units 230, 330, and 430 detect the current camera view direction 1002, 1004, 1006, 1008, 1010, 1012, 1014, 1016, 1334, 1336, 1338, 1340, 1342, 1344, 1346, and 1348 toward the target point 1330. In this case,

Therefore, the camera direction of sight 1332, 1334, 1336, 1338, 1340, 1342, 1344, 1346, and 1348 in which the camera movement is compensated by the camera direction direction compensation units 230, And may be adjusted to focus on the target point 1330. [

Accordingly, the video compensation apparatuses 200, 300, and 400 according to the second, third, and fourth embodiments of the present invention can detect not only the global eye direction along the global camera movement path, , The camera view direction can be adjusted so as to be directed to a predetermined target point or fixed to the target sight direction. The predetermined target point or the target line of sight direction of the camera line-of-sight direction can be arbitrarily set according to the intention of the user who shoots the video.

In addition, the video compensation method according to the embodiment is implemented not only in the video photographing apparatus but also in the video reproducing apparatus, so that the video stabilization operation can be performed during the video reproduction, so that the predetermined target point or the target sight direction , And can be arbitrarily set according to the intention of the user who watches the video.

The video compensating device 200, 300, 400 according to the second, third, and fourth embodiments according to the embodiment is configured such that the camera's eye direction is continuously fixed to a predetermined target point without changing the actual camera moving path and position So that it is possible to acquire a video similar to a motion that a human perceives visually. In addition, since the camera movement can be compensated by adjusting the direction of the camera line of sight using only the direction of the camera line of sight without using the accurate displacement on the camera movement path and using only a relative change amount or rotation angle of the camera line direction, The amount of computation can be reduced as compared with the existing methods of restoring the camera accurately and adjusting the camera displacement and the direction of the camera line of sight.

14 to 16, user interfaces for effectively utilizing the video stabilization scheme of the video stabilization apparatus 100, 200, 300, 400 according to the first, second, third, and fourth embodiments of the present invention are disclosed do.

Figure 14 illustrates user interfaces for adjusting the size and video stabilization degree of the margin region according to the video stabilization operation according to one embodiment.

The camera motion compensation units 130, 230, 330, and 430 of the video stabilization apparatuses 100, 200, 300, and 400 according to the first, second, third, . The direction of the camera line of sight in the Z axis direction can be changed by rotating the image in parallel with the Z axis plane by the rotation angle in the Z axis direction among the rotation angle components in the X, Y, and Z axis directions of the camera view direction adjustment angle .

When the video stabilization is performed according to the rotation of the camera's gaze direction, since the image portion corresponding to the rotation of each frame is lost or not completely reconstructed, the angle of view that the user can actually see becomes narrower than that of the original image. Finally, the user of the video playback apparatus can only view the image within the angle of view remaining in the middle portion, or only the reconstructed effective image can be stored in the video photographing apparatus or the storage device. The stronger the video stabilization degree, the larger the area (margin area) lost due to the video stabilization. If the video stabilization is performed excessively without considering the margin area, the effective image may be excessively small. That is, there may be a tradeoff between the video stabilization effect and the size of the margin region.

Therefore, as the video stabilization apparatus 610 of the video stabilization apparatus 600 according to the sixth embodiment, the video stabilization apparatuses 100, 200, 300, and 700 according to the first, second, third, fourth, fifth, 400, 500, 700) can be mounted. The marginal region user input 620 of the video stabilization apparatus 600 according to the sixth embodiment provides a user interface 1400, 1420, 1440, 1460 that enables the user to coordinate the marginal region with the degree of video stabilization .

The user interface 1400 displays a slice bar in which the user can freely select the ratio of the size of the margin area or the width of the margin area to the entire image area. The marginal region user input unit 620 of the first, second, third, and fourth embodiments may be configured to determine whether the marginal region input through the user interface 1400 based on the width of the marginal region input through the user interface 1400, The degree of stabilization can be determined.

The user interface 1420 displays a slice bar where the user can freely select a video stabilization level that indicates the degree of video stabilization. The marginal region user input unit 620 of the first, second, third, and fourth embodiments may be configured to compare the marginal region width or the entire image width corresponding thereto based on the video stabilization level input through the user interface 1420 The ratio can be determined.

The user interface 1440 displays a slice bar where the user can freely select a trade-off of the video stabilization level relative to the margin area. The user interface 1460 displays a lookup table of the video stabilization level relative to the margin area so that the tradeoff between the margin area and the video stabilization level can be selected in the optimized combination. A user may select a margin size, a video stabilization level, or margin area and a tradeoff of video stabilization using a user controller associated with the user interface 1400, 1420, 1440, 1460.

The margin area user input unit 620 of the first, second, third, and fourth embodiments determines the size or the degree of video stabilization of the margin area based on the user input through the user interface 1400, The adjustment angle determination units 120, 220, 320, and 420 can determine the adjustment angle of the camera view direction.

15 illustrates a user interface for a user to set a target area in the camera eye direction for a video stabilization operation in accordance with one embodiment.

The video stabilizing apparatuses 300, 400, 600 and 700 according to the third, fourth, sixth and seventh embodiments can be mounted as the video stabilizing apparatus 510 of the video stabilizing apparatus 500 according to the fifth embodiment have. The gaze direction target region user input section 520 of the video stabilization apparatus 500 according to the fifth embodiment may provide a user interface 1500 for the user to set a target region or a target gaze direction in the camera's gaze direction . The user interface 1500 displays a selection window 1510 that can freely move up, down, left, and right on the current video shooting scene or the photographed video image. The user can move the selection window 1510 to a target point at which the camera's gaze direction is desired to be fixed using the movement controller of the selection window 1510 associated with the user interface 1500. [

The gaze direction target area determination units 315 and 415 of the third and fourth embodiments determine the target area and the target gaze direction based on the user input through the user interface 1500 and thereby determine the camera gaze direction adjustment angle The units 320 and 420 can determine the angle of view of the camera.

16 illustrates a user interface for alerting an excessive video stabilization operation while performing a video stabilization operation according to one embodiment.

200, 300, 400, 400 according to the first, second, third, fourth, fifth, and sixth embodiments as the video stabilization apparatus 710 of the video stabilization apparatus 700 according to the seventh embodiment. 500, and 600 may be mounted. The stabilization index measurement unit 720 of the video stabilization apparatus 700 according to the seventh embodiment can measure the degree of video stabilization stepwise in the video stabilization operation. The stabilization index measurement unit 720 can measure the stabilization related index by sensing the video stabilization degree in real time during the video stabilization operation.

The video stabilization apparatus 700 according to the seventh embodiment realizes a stabilization related index such as a size of a margin area generated through a video stabilization operation, a motion size of a video camera sensed through a sensor, . For example, the greater the difference between the camera movement path generated by the video stabilization apparatus 700 according to the seventh embodiment and the accumulated inter-frame camera view direction, the larger the camera's gaze direction adjustment angle, , Excessive video stabilization may be expected. Therefore, the camera adjustment angle can be used as a stabilization index.

The stabilization index measurement unit 720 according to the seventh embodiment can detect the stabilization related index in the video stabilization operation of the video stabilization apparatus 710 in real time. For example, the stabilization-related indicator can be measured during the video stabilization operation, the inter-frame camera motion measurement step, the camera's gaze direction adjustment angle determination step, and the camera's gaze direction compensation step. The video stabilization warning unit 730 according to the seventh embodiment is configured such that, for each of the inter-frame camera motion measurement step, the camera visual axis direction adjustment angle determination step, and the camera visual direction direction compensation step during the video stabilization operation, The user can be warned in real time or inform the user about the current stabilization related state. The real-time alert may be implemented with an alert window 1610 or an audio alert sound 1620 displayed on the user interface 1600 screen.

For example, if the camera movement detected by the stabilization index measurement unit 720 exceeds a threshold value during the inter-frame camera movement measurement operation, the video stabilization warning unit 730 may warn in real time. In addition, if the angle of the camera gaze directional angle detected by the stabilization index measurement unit 720 exceeds the threshold value during the camera gaze direction adjustment angle determination operation, the video stabilization warning unit 730 can warn in real time. If the marginal area detected by the stabilization index measuring unit 720 exceeds the threshold during the camera gaze direction compensation operation, the video stabilization warning unit 730 can warn in real time.

If the video stabilization warning unit 730 can predict the margin area based on the camera motion information and the camera eye direction adjustment angle information in the inter-frame camera motion measurement step and the camera eye direction direction adjustment angle determination step during the video stabilization process, The video stabilization warning unit 730 can warn in real time when the margin area exceeds the threshold value throughout the video stabilization process.

The user can predict the degree of motion of the user according to the guidance or warning according to the video stabilization state of the video stabilization apparatus 700. [ That is, if there is a warning or a warning that the video stabilization state is excessive, the user is prevented from excessively moving the camera because the movement of the user camera requiring stabilization of the video is analyzed to be large.

The threshold of the stabilization related indicator may also be determined by the video stabilization apparatus within the amount of calculation that can stably process the current system and video. The threshold value of the stabilization related index may be set based on a tradeoff between the margin area set by the user and the degree of video stabilization.

17 shows a flowchart of a video stabilization method according to the first embodiment.

In step 1710, for each frame of the frame sequence of the video, the difference angle in the current camera eye direction relative to the reference camera eye direction is determined, and based on the relative difference angle in the current camera eye direction with respect to the reference camera eye direction, Motion is measured.

In step 1720, the camera movement path of the frame sequence is generated using the inter-frame camera movement, and the camera parameters are determined for each frame using the camera movement path. The camera parameters may include compensation values for camera displacement and camera orientation on the camera movement path for camera motion compensation.

In step 1730, camera motion is compensated for using a camera parameter for each frame.

The video stabilization method according to the first embodiment corresponds to a video stabilization method in which the video stabilization apparatus 100 according to the first embodiment extracts only relative rotation angle information according to inter-frame camera movement to compensate camera movement.

18 shows a flowchart of a video stabilization method according to the second embodiment.

In step 1810, camera motion is measured for each frame of the frame sequence of the video. In step 1820, the camera movement path of the frame sequence is generated using the camera movement for each frame, and the camera angle of view direction adjustment angle between the user's gaze direction and the camera's gaze direction is determined using the camera movement path. In step 1830, the camera's gaze direction is adjusted using the camera's gaze direction adjustment angle for each frame.

The video stabilization method according to the second embodiment corresponds to the video stabilization method in which the video stabilization apparatus 200 according to the second embodiment adjusts only the camera line direction on the camera movement path to the user's gaze direction to compensate camera movement.

19 shows a flowchart of a video stabilization method according to the third embodiment.

In step 1910, camera motion is measured for each frame of the frame sequence of video. In step 1920, a target area in the eye direction in the frame sequence is determined. In step 1930, a camera movement path of the frame sequence is generated using the camera movement for each frame, and the camera's gaze direction adjustment angle with respect to the gaze direction toward the target area is determined using the camera movement path. In step 1940, the camera's gaze direction is compensated using the camera's gaze direction adjustment angle for each frame.

The video stabilization method according to the third embodiment is a video stabilization method in which the video stabilization apparatus 300 according to the third embodiment compensates camera movement by adjusting only the direction of the camera sight line toward the target area in the camera movement path, Corresponding.

20 shows a flowchart of a video stabilization method according to the fourth embodiment.

In step 2010, for each frame of the frame sequence of the video, the difference angle of the current camera line-of-sight direction relative to the reference camera line-of-sight direction is determined, and based on the relative difference angle of the current camera- Motion is measured.

In step 2020, the camera movement path of the frame sequence is generated using the inter-frame camera movement, and the camera view direction adjustment angle in relation to the user's gaze direction is determined using the camera movement path. At this time, the user's gaze direction can be set to the direction of the camera's gaze toward the target area according to user input.

In step 2030, the camera's gaze direction is compensated using the camera's gaze direction adjustment angle for each frame.

In the video stabilization method according to the fourth embodiment, the video stabilization apparatus 400 according to the fourth embodiment extracts only the relative rotation angle information according to the inter-frame camera movement, Corresponding to a video stabilization scheme that compensates for camera motion by adjusting only the eye direction toward the target area.

Fig. 21 shows a flowchart of a video stabilization method according to the fifth embodiment.

In step 2110, a selection request of the target area in the eye direction of the frame sequence is received from the user via the user interface. In step 2120, for each frame of the frame sequence of video, the difference angle of the current camera eye direction relative to the reference camera eye direction is determined, and based on the relative difference angle of the current camera eye direction with respect to the reference camera eye direction, Motion is measured.

In step 2130, the camera movement path of the frame sequence is generated using the inter-frame camera movement, and the camera's gaze direction adjustment angle in relation to the sight line direction toward the target area is determined using the camera movement path. In step 2140, the camera's gaze direction is compensated using the camera's gaze direction adjustment angle for each frame.

The video stabilization method according to the fifth embodiment corresponds to the video stabilization method in which the video stabilization apparatus 500 according to the fifth embodiment compensates for camera motion in consideration of a target area in the visual direction according to user input. Therefore, the video stabilization method according to the fifth embodiment can follow the video stabilization methods according to the first, second, third, and fourth embodiments in consideration of the target area in the eye direction according to the user input.

22 shows a flowchart of a video stabilization method according to the sixth embodiment.

At step 2210, a request for selection of at least one of the size of the margin region and the stabilization level for video stabilization from the user is received via the user interface.

In step 2220, relative frame-to-frame camera movement relative to the reference camera eye direction is measured, based on at least one of the magnitude and the stabilization level of the selected margin area, A video stabilization operation is performed in which the camera's gaze direction adjustment angle is determined and the camera's gaze direction is stabilized by compensating the camera's gaze direction using the camera's gaze direction adjustment angle for each frame. The video stabilization operation of step 2220 may follow video stabilization methods according to the first, second, third, and fourth embodiments.

The video stabilization method according to the sixth embodiment corresponds to the video stabilization scheme in which the video stabilization apparatus 600 according to the sixth embodiment compensates for camera motion in consideration of a margin area according to user input.

23 shows a flowchart of a video stabilization method according to the seventh embodiment.

In step 2310, the relative inter-frame camera movement relative to the reference camera eye direction is measured in the frame sequence of the video, and the camera eye direction adjustment angle in relation to the visual line direction toward the target area is determined, The angle of view of the camera is compensated by using the angle, and the camera of the video is stabilized.

In step 2320, during the video camera stabilization step of step 2310, the stabilization related indicator is measured. In step 2330, a warning is given in real time if the stabilization related indicator exceeds a predetermined threshold.

The video stabilization method according to the seventh embodiment corresponds to the video stabilization method of the video stabilization apparatus 700 according to the seventh embodiment.

Conventional video stabilization techniques based on 2D analysis compensate the original image using a simple parallel movement technique or a simple warping technique, resulting in image distortion. In addition, existing techniques based on three-dimensional analysis can reconstruct a new image by reconstructing a three-dimensional image, so that a more accurate image can be produced. However, a tremendous amount of computation is required to reconstruct a three-dimensional image and a complete reconstruction is also difficult Still image distortion occurs.

On the other hand, the video stabilization apparatuses 100, 200, 300, and 400 and the video stabilization methods of FIGS. 17 to 22 according to various embodiments of the present invention are based on the difference of the gaze vector direction And the 3D direction is implemented by adjusting the view direction of the camera for each frame according to the camera movement path instead of the simple parallel movement of the camera position, so that the video having little distortion to human vision can be outputted.

The above-described embodiments of the present invention can be embodied in a general-purpose digital computer that can be embodied as a program that can be executed by a computer and operates the program using a computer-readable recording medium. The computer-readable recording medium includes a storage medium such as a magnetic storage medium (e.g., ROM, floppy disk, hard disk, etc.), optical reading medium (e.g., CD ROM,

The present invention has been described with reference to the preferred embodiments. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

Claims (61)

Measuring camera-to-frame camera motion based on a relative angle of view of a camera relative to a reference camera sight line direction for each frame of a frame sequence of video;
Generating a camera movement path of the frame sequence using the inter-frame camera movement, and determining a camera parameter for each frame using the camera movement path; And
And compensating the camera motion using the camera parameters for each of the frames. 2. The method of claim 1, wherein the inter-
And determining a relative angle difference between the pair of consecutive frames of the camera sequence direction in the time sequence of the frame sequence. 3. The method of claim 2, wherein the inter-
Further comprising the step of determining an average of the relative angles of the camera line-of-sight differences between the consecutive frames of the frame sequence in the direction of the reference camera line of sight. 2. The method of claim 1, wherein the inter-
And measures a relative camera rotation angle in the X, Y, and Z axis directions. 5. The method of claim 4,
And determining the camera rotation angle in the Z-direction using a linear motion generated in predetermined areas on the Z-axis plane by rotation in the Z-axis direction. 6. The method according to claim 5, wherein the step of determining the camera rotation angle in the Z-
Determining a magnitude and direction of linear motion vectors occurring in predetermined regions on a Z axis plane corresponding to respective rotation angles in a predetermined range of rotation angles;
Determining a cost based on an error between an image shifted by a linear motion vector for each of the predetermined areas and a corresponding area of a frame in the direction of the reference camera view line and summing the cost for each predetermined area; And
Determining a total sum cost for each of the predetermined angles for each of the rotation angles to detect a rotation angle at which a minimum total cost is generated and determining the detected rotation angle as a camera rotation angle in the Z axis direction The video stabilization method comprising the steps of: The method according to claim 6,
Wherein the magnitude and direction of the linear motion vectors generated in the predetermined areas corresponding to the rotation angles are determined based on a distance and a direction from the origin of the Z axis plane to the predetermined area, . 2. The method according to claim 1,
Accumulating the inter-frame camera movement to generate a camera movement path of the frame sequence;
Generating a global camera movement path of the camera movement path;
And determining a camera adjustment value for each frame in relation to the global camera movement path. 2. The method of claim 1,
Rotating the camera view direction by the eye direction adjustment value among the camera parameters for each frame; And
And moving the camera position by a displacement adjustment value of the camera parameters for each frame. 2. The method of claim 1, wherein the inter-
Between the left and right viewpoint frames corresponding to each other of the stereoscopic video, relative to the reference-point camera sight line direction, Wherein the video stabilization method comprises the steps of: Measuring camera motion for each frame of a frame sequence of video;
Generating a camera movement path of the frame sequence using the camera movement for each frame and determining an angle of the camera's gaze direction between the user's gaze direction and the camera's gaze direction using the camera movement path; And
And compensating the camera line direction using the camera line direction adjusting angle for each frame. 12. The method according to claim 11, wherein the camera-
Separating a global camera view direction and a regional camera view direction from the camera movement path; And
And determining a difference angle between the global camera view direction and the current camera view direction as the camera view direction adjustment angle. 12. The method according to claim 11, wherein the camera-
Wherein the camera rotation angle for moving the current camera line direction of the camera movement path to the center of the frame is determined by the camera angle of view direction adjustment angle. The method as claimed in claim 11,
And rotating the camera eye direction of the frame by the camera eye direction adjusting angle. 12. The method of claim 11,
Between the left and right viewpoint frames corresponding to each other of the stereoscopic video, relative to the reference-point camera sight line direction, Wherein the video stabilization method comprises the steps of: Measuring camera motion for each frame of a frame sequence of video;
Determining a target area in the line-of-sight direction of the frame sequence;
Generating a camera movement path of the frame sequence using the camera movement for each frame and determining a camera's gaze direction adjustment angle with respect to a gaze direction toward the target area using the camera movement path; And
And compensating the camera line direction using the camera line direction adjusting angle for each frame. The method as claimed in claim 16, wherein the camera-
And predicting a user's gaze direction using the camera movement path. 18. The method as claimed in claim 17, wherein the camera-
Separating the global camera view direction and the regional camera view direction from the camera movement path and determining the global camera view direction as the user view direction;
Determining a difference angle between the global camera line-of-sight direction and the current camera line-of-sight direction as a first adjustment angle in the camera line direction;
A difference angle between the global camera sight line direction and the camera sight line direction toward the target area on the basis of the displacement or coordinate difference between the point where the global camera eye line direction is on the frame and the target area, Determining a second adjustment angle; And
And determining the sum of the first adjustment angle and the second adjustment angle in the camera sight line direction as the camera's gaze direction adjustment angle. 17. The method as claimed in claim 16,
And rotating the camera view direction by the camera view direction adjusting angle for each frame. 17. The method of claim 16, wherein the inter-
Between the left and right viewpoint frames corresponding to each other of the stereoscopic video, relative to the reference-point camera sight line direction, Wherein the video stabilization method comprises the steps of: Measuring camera-to-frame camera motion based on a relative angle of view of a camera relative to a reference camera sight line direction for each frame of a frame sequence of video;
Generating a camera movement path of the frame sequence using the inter-frame camera movement, and determining an angle of view direction adjustment of the camera relative to the user's gaze direction using the camera movement path; And
And compensating the camera line direction using the camera line direction adjusting angle for each frame. 22. The method of claim 21, wherein the inter-
And determining a relative angle difference between the pair of consecutive frames of the camera sequence direction in the time sequence of the frame sequence. 23. The method of claim 22, wherein the inter-
Further comprising the step of determining an average of the relative angles of the camera line-of-sight differences between the consecutive frames of the frame sequence in the direction of the reference camera line of sight. 22. The method of claim 21, wherein the inter-
And measures a relative camera rotation angle in the X, Y, and Z axis directions. 25. The method of claim 24, wherein the inter-
And determining the camera rotation angle in the Z-direction using a linear motion generated in predetermined areas on the Z-axis plane by rotation in the Z-axis direction. 26. The method according to claim 25, wherein the step of determining the camera rotation angle in the Z-
Determining a magnitude and direction of linear motion vectors occurring in predetermined regions on a Z axis plane corresponding to respective rotation angles in a predetermined range of rotation angles;
Determining a cost based on an error between an image shifted by a linear motion vector for each of the predetermined areas and a corresponding area of a frame in the direction of the reference camera view line and summing the cost for each predetermined area; And
Determining a total sum cost for each of the predetermined angles for each of the rotation angles to detect a rotation angle at which a minimum total cost is generated and determining the detected rotation angle as a camera rotation angle in the Z axis direction The video stabilization method comprising the steps of: 27. The method of claim 26,
Wherein the magnitude and direction of the linear motion vectors generated in the predetermined regions corresponding to the rotation angles are determined based on a distance and a direction from the origin of the Z axis plane to the predetermined region, . 22. The method of claim 21, wherein the camera viewing direction adjusting angle determining step comprises:
And accumulating the inter-frame camera movement to generate a camera movement path of the frame sequence. 29. The method of claim 28, wherein the camera-
And predicting the user's gaze direction using the camera movement path. 30. The method as claimed in claim 29, wherein the camera-
Separating the global camera view direction and the regional camera view direction from the camera movement path and determining the global camera view direction as the user view direction; And
Further comprising the step of determining a camera viewing direction adjustment angle in relation to the global camera viewing direction for each point on the camera movement path. ≪ Desc / Clms Page number 19 > 22. The method of claim 21, wherein the camera viewing direction adjusting angle determining step comprises:
Determining a target area in the line-of-sight direction of the frame sequence; And
Generating a camera movement path of the frame sequence using the inter-frame camera movement and determining an angle of camera sight direction adjustment with respect to a visual line direction toward the target area. 32. The method of claim 31, wherein the camera-
And rotating the camera view direction by the camera view direction adjusting angle for each frame. 32. The method according to claim 31,
Wherein the global camera line-of-sight direction of the camera movement path is set to the gaze direction toward the target area. 22. The method of claim 21, wherein the inter-
Between the left and right viewpoint frames corresponding to each other of the stereoscopic video, relative to the reference-point camera sight line direction, Wherein the video stabilization method comprises the steps of: Receiving a selection request of a target area in a line-of-sight direction from a frame sequence of video from a user through a user interface;
Measuring camera-to-frame camera motion based on a relative angle of view of the camera's eye relative to a reference camera's eye direction for each frame of the frame sequence;
Generating a camera movement path of the frame sequence using the inter-frame camera movement, and determining a camera eye direction adjustment angle with respect to a visual line direction toward the target area using the camera movement path; And
And compensating the camera line direction using the camera line direction adjusting angle for each frame. Receiving a selection request of at least one of a size of a margin region and a stabilization level for video stabilization from a user through a user interface; And
Frame camera movement relative to the reference camera eye line direction in the frame sequence of the video based on at least one of the size of the selected margin area and the stabilization level, Determining a direction adjustment angle, and stabilizing the video camera by compensating the direction of the camera line-of-sight using the camera line-of-sight direction adjustment angle for each frame. 37. The method of claim 36, wherein the step of stabilizing the video camera comprises:
Measuring camera motion for each frame of a frame sequence of video;
Generating a camera movement path of the frame sequence using the camera movement for each frame to determine a user's gaze direction of the camera movement path and determining a camera's gaze direction adjustment angle with respect to the user's gaze direction for each frame; And
And compensating the camera line direction using the camera line direction adjusting angle for each frame. 37. The method of claim 36, wherein the step of stabilizing the video camera comprises:
Measuring camera motion for each frame of a frame sequence of video;
Determining a target area in the line-of-sight direction of the frame sequence;
Generating a camera movement path of the frame sequence using the camera movement for each frame and determining a camera's gaze direction adjustment angle with respect to a gaze direction toward the target area using the camera movement path; And
And compensating the camera line direction using the camera line direction adjusting angle for each frame. 37. The method of claim 36, wherein the step of stabilizing the video camera comprises:
Measuring camera-to-frame camera motion based on a relative angle of view of a camera relative to a reference camera sight line direction for each frame of a frame sequence of video;
Determining a target area in the line-of-sight direction of the frame sequence;
Generating a camera movement path of the frame sequence using the inter-frame camera movement, and determining a camera eye direction adjustment angle with respect to a visual line direction toward the target area using the camera movement path; And
And compensating the camera line direction using the camera line direction adjusting angle for each frame. 37. The method of claim 36, wherein receiving a selection request from the user further comprises:
And determining a stabilization level based on a size of the margin area selected by the user. 37. The method of claim 36, wherein receiving a selection request from the user further comprises:
And determining a size of the margin region based on the stabilization level selected by the user. 37. The method of claim 36, wherein receiving a selection request from the user further comprises:
And receiving a selection request of a size and a stabilization level of a margin region of a predetermined combination from the user. 37. The method of claim 36, wherein the step of stabilizing the video camera comprises:
Limiting at least one of a size of the measured camera movement, a size of the camera's gaze direction adjustment angle, and a variation amount according to the camera's gaze direction compensation based on at least one of the size of the selected margin area and the stabilization level The video stabilization method comprising the steps of: Frame camera movement relative to the direction of the reference camera line of the frame sequence of the video is measured and the angle of view direction of the camera's gaze direction is determined in relation to the gaze direction toward the target area, Stabilizing the camera of the video by compensating the direction of the camera line of sight;
Measuring a stabilization related indicator while performing the video camera stabilization step; And
And alerting in real time if the stabilization related indicator exceeds a predetermined threshold. 45. The method of claim 44, wherein measuring the stabilization-
A size of a margin region generated through the stabilization operation, a size of a motion of a video camera detected through a sensor, and a size of a motion of a camera in a frame. 45. The method of claim 44, wherein the stabilization-
Measuring the stabilization related indicator during at least one of the inter-frame camera movement measurement operation, the camera view direction adjustment angle determination operation, and the camera view direction direction compensation operation. A video stabilization apparatus comprising:
A camera motion measurement unit for measuring a camera movement between frames by determining a difference angle of a camera sight line direction relative to a reference camera sight line direction for each frame of a video frame sequence;
A camera parameter determination unit for generating a camera movement path of the frame sequence using the inter-frame camera movement and determining camera parameters for each frame using the camera movement path; And
And a camera motion compensation unit for compensating the camera motion using the camera parameters for each of the frames. A video stabilization apparatus comprising:
A camera motion measurement unit for measuring camera motion for each frame of a frame sequence of a video;
Generating a camera movement path of the frame sequence using the camera movement for each frame and determining a camera direction of view direction adjustment angle relative to the user's gaze direction using the camera movement path; And
And a camera gaze direction compensation unit for compensating the camera gaze direction using the camera gaze direction adjustment angle for each frame. A video stabilization apparatus comprising:
A camera motion measurement unit for measuring camera motion for each frame of a frame sequence of a video;
A line-of-sight-direction target area determining unit for determining a line-of-sight direction target area in the frame sequence;
Generating a camera movement path of the frame sequence using the camera movement for each frame and determining a camera's gaze direction adjustment angle for determining a camera's gaze direction adjustment angle relative to the gaze direction toward the target area using the camera movement path part; And
And a camera direction of sight direction adjusting angle compensating unit for compensating the direction of the camera sight line by using a camera sight direction adjusting angle corresponding to a viewing direction toward the target area for each frame. A video stabilization apparatus comprising:
A camera motion measuring unit for measuring a camera movement between frames by determining a difference angle of a relative camera line direction relative to a reference camera eye line direction for each frame of a frame sequence of video;
Generating a camera movement path of the frame sequence using the inter-frame camera movement, and determining a camera eye direction adjustment angle in relation to the user's gaze direction using the camera movement path; And
And a camera gaze direction compensation unit for compensating the camera gaze direction using the camera gaze direction adjustment angle for each frame. 51. The video stabilization apparatus of claim 50,
Further comprising a gaze direction target area determining unit for determining a gaze direction target area in the frame sequence,
Wherein the camera view direction adjusting angle determining unit is configured to generate a camera moving path of the frame sequence using the inter-frame camera movement and to adjust a camera view direction adjusting angle corresponding to a viewing direction toward the target area using the camera moving path image Lt; / RTI >
Wherein the camera's gaze direction compensation unit compensates the camera's gaze direction using a camera's gaze direction adjustment angle for each frame with respect to the gaze direction toward the target area. A video stabilization apparatus comprising:
A gaze direction target area user input unit for receiving a selection request of a gaze direction target region in a frame sequence of video from a user through a user interface;
A camera motion measuring unit for measuring a camera movement between frames by determining a relative angle of a camera sight line direction relative to a reference camera eye line direction for each frame of the frame sequence;
Generating a camera movement path of the frame sequence by using the inter-frame camera movement, and determining a camera eye direction adjustment angle corresponding to a direction of a gaze toward the target area using the camera movement path part; And
And a camera gaze direction compensation unit for compensating the camera gaze direction using the camera gaze direction adjustment angle for each frame. A video stabilization apparatus comprising:
A margin area user input unit for receiving a selection request of at least one of a size of a margin area and a stabilization level from a user through a user interface; And
Frame camera movement relative to the reference camera eye line direction based on at least one of the magnitude and the stabilization level of the selected margin area and determines a camera eye direction adjustment angle in relation to the eye direction toward the target area, And a video camera stabilizing unit for stabilizing the camera of the video by compensating the direction of the camera's gaze direction using the camera's gaze direction adjusting angle for each frame. A video stabilization apparatus comprising:
Frame camera movement relative to the direction of the reference camera line of the frame sequence of the video is measured and the angle of view direction of the camera's gaze direction is determined in relation to the gaze direction toward the target area, A video camera stabilizing unit for stabilizing a camera of the video by compensating the camera visual direction;
A stabilization related indicator measuring unit for measuring a stabilization related indicator while performing the video camera stabilization step; And
And a video stabilization warning unit for warning in real time when the stabilization related index exceeds a predetermined threshold value. 11. A computer-readable recording medium on which a program for implementing the video stabilization method according to any one of claims 1 to 10 is recorded. A computer-readable recording medium having recorded thereon a program for implementing the video stabilization method according to any one of claims 11 to 15. A computer-readable recording medium on which a program for implementing the video stabilization method according to any one of claims 16 to 20 is recorded. 34. A computer-readable recording medium on which a program for implementing the video stabilization method of any one of claims 21 to 34 is recorded. 34. A computer-readable recording medium on which a program for implementing the video stabilization method of claim 35 is recorded. A computer-readable recording medium on which a program for implementing the video stabilization method according to any one of claims 36 to 43 is recorded. 46. A computer-readable recording medium on which a program for implementing the video stabilization method of any one of claims 44 to 46 is recorded.

Images