JP2006310936A - System for generating video image viewed at optional viewpoint - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology of tracing a particular object at all times, generating its video image and providing a video image resulting from viewing the object from an easy to see position by automatically and optionally controlling a viewpoint position and a sight line direction. <P>SOLUTION: Video information items from a multiple-lens imaging section 1 comprising many cameras located at different viewpoint positions and different sight line directions are composed into a video image in an optional viewpoint position and an optional sight line direction. An object position and direction detection section 2 detects the position and the direction of the object and a virtual camera position and direction calculation section 3 calculates the viewpoint position and the sight line direction of a virtual camera with respect to the object. Further, a video generating section 4 uses multiple-lens video data from the multiple-lens imaging section 1 to generate an image according to the information about the viewpoint position and the sight line direction of the virtual camera transmitted from the virtual camera position and direction calculation section 3. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an arbitrary viewpoint video generation system, and more specifically, video from a plurality of cameras arranged at different viewpoint positions and line-of-sight directions. The present invention relates to an arbitrary viewpoint video generation system that generates a video at a camera viewpoint position and a line-of-sight direction.

Many research institutes / universities, etc. have an arbitrary viewpoint video generation technology for generating a virtual camera that does not actually exist and generates an image in which a subject is viewed from a desired arbitrary gaze direction and arbitrary viewpoint position using the virtual camera. It has been studied. This is because a scene (subject) is shot using a large number of real cameras, and the video data is processed, assuming that there is actually a virtual camera at a point where the camera does not exist. This is a technology for generating video as if it were taken by a virtual camera. For example, if a technique such as Non-Patent Document 1 or Non-Patent Document 2 is used, the user can manipulate the viewpoint position and line-of-sight direction of the virtual camera to watch the video while tracking the specific subject. Taking sporting sports as an example, you can always track and watch your favorite players.
On the other hand, when a moving player is photographed with a camera, there is an automatic tracking technology that automatically controls the camera direction to track the player (see, for example, Patent Document 1).
JP 2004-80163 A Kitahara, Sato, Ota, “3-D image display capable of reproducing motion parallax using the multi-eye stereo method—generation and evaluation of display image”, Television Society Journal, Vol. 1268-1276 (1996) Primm, Fujii, Tanimoto: "Real-time system for free viewpoint TV", IEICE Technical Report, IE2002-120 (2002) Inamoto, Saito: “Free viewpoint viewing system for 3D soccer video based on viewpoint interpolation”, Journal of the Institute of Image Information and Television Engineers, Vol. 58, no. 4, pp. 529-539 (2004) Kimura, Saito: “Method for generating player viewpoint video from multi-viewpoint images of tennis”, IEICE General Conference, D-12-172 (2004) Otani, Kishino: “Model-based estimation of human postures from multi-view images using genetic algorithms”, Journal of the Institute of Image Information and Television Engineers, Vol. 51, no. 12, pp. 2107-2115 (1997) Kaoru, Saito: “Analysis of player motion using multi-view soccer video”, IEICE General Conference, D-12-123 (2004)

  If the arbitrary viewpoint video generation technology as described above is used, it is possible to always track and watch a specific player, but the user always adjusts the viewpoint position and line-of-sight direction of the virtual camera according to the movement of the player. There is a problem that it is necessary and the burden on the user is large. In order to reduce the burden, it is desirable to have a function that can automatically track players.

  However, in the automatic tracking technology described in Patent Document 1, the camera position is fixed, and depending on the relationship between the camera position and the position of the player, for example, when the player moves away from the camera, the video of the player becomes smaller, or the player There is a problem that a desired image cannot be obtained for the player, such as being able to shoot only the back of the player.

In order to solve the above problems, the present invention realizes the following two points when generating an arbitrary viewpoint video.
(1) A specific player is always tracked, and an image in the viewpoint position and the line-of-sight direction of the virtual camera is generated.
(2) Automatically control the viewpoint position and line-of-sight direction of the virtual camera in accordance with the position and orientation of the player, and provide an image viewed from a position where the player can easily see.
For example, it is possible to automatically move the virtual camera to a position and a direction that are always a fixed distance in front of the player, and to generate an image in the viewpoint position and the line-of-sight direction of the virtual camera.

  According to a first aspect of the present invention, there is provided a video system that synthesizes or selects a video in a viewpoint position and a gaze direction of an arbitrary virtual camera from a video obtained by photographing a certain subject from one or more viewpoint positions and a gaze direction. And a calculation unit that calculates the viewpoint position and the line-of-sight direction of the virtual camera that satisfies the above.

  A second means of the present invention comprises means for detecting the position and orientation of one or more subjects in the first means, and information on the position and orientation of one or more subjects obtained by the detecting means. The viewpoint position and the line-of-sight direction of the virtual camera satisfying the predetermined condition are calculated using.

  According to a third means of the present invention, in the second means, the predetermined condition is that the virtual camera is positioned in a specific direction with respect to the subject and the line-of-sight direction of the virtual camera faces the direction of the subject. It is characterized by being.

  According to a fourth means of the present invention, in the second means, the predetermined condition is that the visual point direction of the virtual camera is fixed and the viewpoint position is changed, and the subject can always be photographed from the viewpoint position and the visual line direction of the virtual camera. It is characterized by being.

  According to a fifth means of the present invention, in the second to fourth means, the predetermined condition is that a plurality of subjects are included in an image in a viewpoint position and a line-of-sight direction of the virtual camera. Is.

  According to a sixth means of the present invention, in the second means, the predetermined condition is that the viewpoint position and the line-of-sight direction of the subject coincide with the viewpoint position and the line-of-sight direction of the virtual camera. It is.

  The seventh means of the present invention is characterized in that in the second to sixth means, processing is performed in which a dead band is provided for information on the position and direction of the subject.

  The eighth means of the present invention is characterized in that in the second to seventh means, low-pass filter processing is applied to information on the position and direction of the subject.

  According to a ninth means of the present invention, in the second to eighth means, the detecting means for detecting the position and orientation of one or more subjects is one or more subjects for one or more viewpoint positions and line-of-sight directions. It is a detection means to detect using the multi-view video image | photographed from this, It is characterized by the above-mentioned.

  According to a tenth means of the present invention, in the second to eighth means, the detecting means for detecting the position and orientation of the one or more subjects uses an image obtained by photographing one or more subjects from above. It is the detection means to detect, It is characterized by the above-mentioned.

  According to an eleventh means of the present invention, in the second to eighth means, the detecting means for detecting the position and orientation of the one or more subjects is a sensor capable of detecting the location and orientation of the subject on the subject itself. And detecting means for detecting using the information on the position and orientation of the subject acquired by the sensor.

  A twelfth means of the present invention is characterized in that, in the second to eighth means, a detection means in which any detection means of the means of claims 9 to 11 is combined.

  According to a thirteenth means of the present invention, in the first to twelfth means, based on an image obtained by photographing the certain subject from one or more viewpoint positions and line-of-sight directions, using the calculation result by the calculating means. An arbitrary viewpoint video generation technique for synthesizing videos in the viewpoint position and the line-of-sight direction of the virtual camera is used.

  According to a fourteenth aspect of the present invention, in any one of the first to twelfth means, an arbitrary virtual camera is used from a video obtained by photographing the certain subject from one or more viewpoint positions and line-of-sight directions, using the calculation result by the calculation means. A video that is most similar to the video at the viewpoint position and the line-of-sight direction is selected, and the selected video is used as an image at the viewpoint position and the line-of-sight direction of an arbitrary virtual camera.

  According to a fifteenth means of the present invention, in the first to fourteenth means, an image obtained by photographing a certain subject from one or more viewpoint positions and line-of-sight directions is delayed for a predetermined time, and an arbitrary image is obtained based on the delayed image. The virtual camera's viewpoint position and line-of-sight direction video are synthesized, or the video image most similar to the virtual camera's viewpoint position and line-of-sight direction image is selected from the delayed video, and the virtual camera's viewpoint position is selected. And an image in the line-of-sight direction.

  According to a sixteenth means of the present invention, in the fifteenth means, a video obtained by photographing a subject from one or more viewpoint positions and line-of-sight directions is delayed for a predetermined time, and calculation of the viewpoint position and line-of-sight direction of the virtual camera is calculated. It is characterized by being performed relatively earlier than the video.

  The seventeenth means of the present invention is characterized in that, in the first to sixteenth means, means for outputting a video generated in the viewpoint position and the line-of-sight direction of the virtual camera is provided.

  According to an eighteenth means of the present invention, in the seventeenth means, a two-dimensional display is used as the means for outputting the generated video.

  According to a nineteenth means of the present invention, in the seventeenth means, a stereoscopic display is used as the means for outputting the generated video.

  According to a twentieth means of the present invention, in the first to nineteenth means, there is provided means for detecting a condition in which the subject in the generated video is backlit, and the subject in the generated video is not backlit. Is defined as the predetermined condition.

  The twenty-first means of the present invention is characterized in that, in the twentieth means, backlight detection is performed using an image obtained by photographing a certain subject from one or more viewpoint positions and line-of-sight directions.

According to the video system of the present invention, even if the subject moves, the viewpoint position and the line-of-sight direction of the virtual camera are automatically changed in accordance with the movement of the subject, and the subject is always observed from a certain easy-to-see relative position. Is possible.
In addition, when there are a plurality of subjects, it is possible to automatically change the viewpoint position and the line-of-sight direction of the virtual camera so that they always enter the screen.
In addition, by automatically changing the subject position and orientation to match the viewpoint position and line-of-sight direction of the virtual camera, it is possible to synthesize a scene viewed by the subject.
In addition, by applying dead band processing to the subject position and orientation information, even if the subject position and orientation information changes finely, fine changes below a certain level α can be ignored, and the position of the virtual camera Alternatively, it is possible to prevent the direction from being changed too much and generate a stable image. Therefore, it is possible to prevent the user from getting drunk with the video.
Moreover, it is possible to apply low-pass filter processing to information on the subject position and orientation. By this processing, the high frequency component of the information on the subject position and direction is removed, the virtual camera position is prevented from changing too finely, and a stable image can be generated. Therefore, it is possible to prevent the user from getting drunk on the video.
Also, by delaying the video and relatively pre-reading the processing of the subject position and orientation information, it is possible to control the position and direction of the virtual camera in advance of the future movement of the subject.
Further, by determining whether or not the generated screen is in the backlight state, the viewpoint position and the line-of-sight direction of the virtual camera can be controlled so as to avoid the viewpoint position and the line-of-sight direction of the virtual camera that is in the backlight state.

First embodiment:
FIG. 1 is a block diagram showing a schematic configuration of a main part for explaining a first embodiment of an arbitrary viewpoint video generation system according to the present invention. In FIG. 1, 10 indicates an arbitrary viewpoint video generation system according to the present invention. The arbitrary viewpoint video generation system 10 includes a multi-view video shooting unit 1, a subject position and orientation detection unit 2, a virtual camera position and direction calculation unit 3, a video generation unit 4, and a video output unit 5.

  The multi-view video shooting unit 1 arranges a large number of cameras around a subject, and obtains video data groups obtained by shooting the subject from various viewpoint positions and line-of-sight directions.

  The subject position and orientation detection unit 2 performs processing for detecting the position and orientation of one or more subjects, and sends the detected information to the virtual camera position and direction calculation unit 3.

The means for detecting the position and orientation of the subject will be described below.
Regarding the positions of the subjects, for example, Non-Patent Document 3 to Non-Patent Document 4 describe a method for obtaining the positions of a plurality of subjects from multi-view video data. Further, for example, Non-Patent Document 5 to Non-Patent Document 6 describe a method for estimating the direction of a subject from multi-view video data. By using these methods, it is possible to detect the positions and orientations of a plurality of subjects from multi-view video data obtained from the multi-view video shooting unit 101.
As another method, the subject is provided with a sensor such as a GPS (Global Positioning System) or a gyro sensor that can detect the position and orientation of the subject in advance, and the information obtained from the sensor is used to obtain the position and orientation of the subject. It is also possible to use a means of detecting.
Of course, other means may be used, or these means may be used in combination.

  The virtual camera position and direction calculation unit 3 uses the information on the position and orientation of the subject obtained from the subject position and orientation detection unit 2 to calculate information on the viewpoint position and line-of-sight direction of the virtual camera with respect to the subject. This calculation method will be described in detail later.

  The video generation unit 4 uses the multi-view video data from the multi-view video shooting unit 1 and from the virtual camera according to the viewpoint position and line-of-sight direction information sent from the virtual camera position and direction calculation unit 3. A virtual image in which the subject is viewed is generated. As a method for generating this virtual image, for example, the following two methods A and B are conceivable.

  A. This is a method of synthesizing videos in the viewpoint position and line-of-sight direction of a virtual camera from multi-view video data using the arbitrary viewpoint video generation technology described in Non-Patent Document 1 and Non-Patent Document 2. In this case, it is possible to synthesize and display an image at the viewpoint position and the line-of-sight direction where the camera originally does not exist. The video changes smoothly according to changes in the viewpoint position and line-of-sight direction of the virtual camera. In addition, if an arbitrary viewpoint video generation technique is used, it is also possible to generate a stereoscopically displayable video by synthesizing videos from two viewpoints separated by the distance between the left and right eyes.

B. The video closest to the viewpoint position and the line-of-sight direction of the virtual camera is selected from the multi-view video data and displayed. The advantage is that it is a simple method and does not require complicated image composition processing. Instead, only the video from where the camera originally resides can be seen, and the video sometimes switches to the camera closest to the virtual camera's position and direction as the virtual camera's position and direction change. Become a shape.
Of course, other methods may be used.

  The video output unit 5 displays the video generated by the video generation unit 4. A case of a general two-dimensional display or a case of a three-dimensional display capable of stereoscopic viewing can be considered.

  For the processing in the virtual camera position and direction calculation unit 3, several processing methods can be considered depending on the subject. Hereinafter, each will be described in detail.

  FIG. 2 is a diagram for explaining a first example of calculating the viewpoint position and line-of-sight direction of a virtual camera when the present invention is applied to a sports broadcast. In the figure, reference numeral 21 denotes a soccer player assumed as a subject. , 22 indicates a virtual camera, arrow A indicates the front direction of the soccer player 21, arrow B indicates the moving direction of the soccer player 21, and arrow C indicates the line-of-sight direction of the virtual camera 22. In FIG. 2, a bird's-eye view of a soccer player 21 moving sequentially in the direction of arrow B while changing the body direction, that is, the front direction (arrow A) in order to look around.

  Here, if information on the position and orientation of the specific subject obtained from the subject position and orientation detection unit 2 is used, the virtual camera 22 is always located at a certain distance and direction with respect to the subject, and the virtual camera 22 is always the subject. It is possible to control so as to face the direction (arrow C).

  In FIG. 2, even if the subject player 21 changes position and direction, control is performed so that the virtual camera 22 is always located at a fixed distance in the front direction (arrow A) of the player 21 and faces the subject (arrow C). It shows how it is. With this control, it is possible to always generate an image when the player 21 is viewed at a constant distance from the front regardless of the position and direction of the player 21. Of course, the same processing can be performed not only on the front direction of the subject but also on an arbitrary direction such as the back side, side surface, and top surface of the subject. By controlling the virtual camera position in this way, it is possible to always observe the subject from a certain easy-to-see relative position.

  FIG. 3 is a diagram for explaining a second example of calculating the viewpoint position of the virtual camera when the present invention is applied to a sports broadcast. In the figure, 21 is a short-distance running player assumed as a subject, 22 indicates a virtual camera, arrow A indicates the front direction of the sprinter 21, arrow B indicates the direction of movement of the sprinter 21, and arrow C indicates the direction of the virtual camera 22. FIG. 3 shows a state in which a short-distance running player 21 is running while looking around to see the other players.

  At this time, if the information on the position of the specific subject obtained from the subject position and orientation detection unit 2 is used, the viewpoint position of the virtual camera 22 always keeps photographing the subject while the absolute direction of the virtual camera 22 is fixed. Can be controlled. In the second example shown in FIG. 3, the player 21 is displayed on the screen while keeping the line-of-sight direction (arrow C) of the virtual camera 22 always perpendicular to the course (movement direction B of the sprinter 21). The viewpoint position of the virtual camera 22 is moved so as to be captured at the center. In this way, by controlling the viewpoint position of the virtual camera 22, for example, it is possible to always see from the position directly beside the player in 100 m running, and it is possible to generate an image that clearly shows which player is currently at the top it can. Of course, it is possible to perform the same processing by fixing the orientation of the virtual camera in an arbitrary direction as well as a direction perpendicular to the course.

  4 and 5 are diagrams for explaining a third example of calculating the position of the virtual camera when the present invention is applied to a sports broadcast. In FIGS. 4 and 5, 21 is assumed to be a subject. A soccer player, 22 is a virtual camera, 23 is a ball, arrow A is the front direction of the soccer player 21, and arrow D indicates that the range indicated by the arrow D is the shooting range of the virtual camera 22. 4 shows a situation where the ball 23 is present near the player 21, and FIG. 5 shows a situation where the ball 23 is being separated from the player 21 because the player 21 kicks the ball 23. At this time, if the information on the position of the specific subject obtained from the subject position and orientation detection unit 2 is used, it is assumed that both the player 21 and the ball 23 fall within the shooting range in both cases of FIGS. The position of the camera 22 can be controlled.

  In FIG. 5, by moving the position of the virtual camera 22 backward from the position of the virtual camera 22 in FIG. 4, both the player 21 and the ball 23 are within the shooting range. The third example may be used in combination with the first and second examples for calculating the position and direction of the virtual camera.

  FIG. 6 is a diagram for explaining a fourth example of calculating the viewpoint position and line-of-sight direction of the virtual camera when the present invention is applied to a sports broadcast, where 21 is a soccer player assumed as a subject, 22 is The virtual camera, arrow B indicates the moving direction of the soccer player 21, and arrow E indicates the front direction of the soccer player 21 and the line of sight of the virtual camera 22. FIG. 6 shows a situation in which a soccer player 21 moves in the direction of arrow B while changing the body direction, that is, the front direction (arrow E) in order to look around.

  Here, if the information on the position and orientation of the specific subject obtained from the subject position and orientation detection unit 2 is used, the player 21 can match the position and orientation of the virtual camera 22 with the viewpoint position and orientation of the player 21. It is possible to synthesize an image when the player 21 is seen from the viewpoint position, that is, a scene that the player is viewing. FIG. 6 shows a state in which the position and direction of the virtual camera 22 are following the viewpoint position and line-of-sight direction of the player 21.

Second embodiment:
FIG. 7 is a block diagram showing a schematic configuration of a main part for explaining a second embodiment of the arbitrary viewpoint video generation system according to the present invention. In FIG. 7, 10 is an arbitrary viewpoint video generation system according to the present invention. The viewpoint video generation system 10 includes a multi-view video shooting unit 1, a subject position / orientation detection unit 2, a virtual camera position / direction calculation unit 3, a video generation unit 4, a video output unit 5, and an information recording unit 6. That is, the information recording unit 6 is added to the first embodiment shown in FIG. 1, and the information recording unit 6 is the position and orientation of the specific subject obtained from the subject position and orientation detection unit 2. This is a part for recording the dead band information necessary for processing the information and the past information of the position and orientation of the specific subject.

FIG. 8 is a flowchart illustrating an example of calculating subject direction information by providing a dead band for subject direction information (subject direction information).
First, information on the direction of the subject (angle F) is obtained from the subject position and orientation detection unit 2 (step S1).
Next, it is determined whether or not the angle F is within the dead band range recorded in the information recording unit 6 (step S2).
If it is within the dead band range (YES in step S2), the orientation information of the subject before the temporary point is read from the information recording unit 6 and output as data for calculating the viewpoint position and the line-of-sight direction of the virtual camera. (Step S3).
If it is out of the dead band range (NO in step S2), first, the information recording unit 6 is overwritten with the angle F-α as the lower limit value of the new dead band and the angle F + α as the upper limit value (step S4). The angle F is recorded as a new value of the subject orientation information in the information recording unit 6 (step S5), and the angle F is output as data for calculating the viewpoint position and line-of-sight direction of the virtual camera (step S6).
Finally, the viewpoint position and line-of-sight direction of the virtual camera are calculated (step S7).
The above processing is performed for each frame of the video. Α is the level of the dead band (reference value).

When the processing with the dead band is not performed, the virtual camera position and direction always change slightly in order to follow the change in the direction of the subject, and the user who sees the generated image may get drunk on the video.
On the other hand, by adding the dead band processing described above, even if the information on the direction of the subject (angle F) changes finely, a fine change below a certain level α can be ignored, and the position of the virtual camera or / And it is possible to prevent the direction from changing too finely and to generate a stable image. Therefore, it is possible to prevent the user from getting drunk with the video.

  Although the example in the case of applying to the orientation information of the subject has been described with reference to FIG. 8, it is of course possible to add the same processing to the location information of the subject.

  It is also possible to record past subject position and direction information in the information recording unit 6 and apply low-pass filter processing to the subject position and direction information arranged in time series. By this processing, the high frequency component of the information on the subject position and direction is removed, and the position of the virtual camera is prevented from changing too finely, and a stable image can be generated. Therefore, it is possible to prevent the user from getting drunk on the video.

Third embodiment:
FIG. 9 is a block diagram showing a schematic configuration of a main part for explaining a third embodiment of the arbitrary viewpoint video generation system according to the present invention. In FIG. 9, 10 is an arbitrary viewpoint video generation system according to the present invention. The viewpoint video generation system 10 includes a multi-view video shooting unit 1, a subject position and orientation detection unit 2, a virtual camera position and direction calculation unit 3, a video generation unit 4, a video output unit 5, an information recording unit 6, and a video delay unit 7. Consists of That is, the video delay unit 7 is added to the second embodiment shown in FIG.

  The video delay unit 7 executes a function of temporarily storing the multi-view video data from the multi-view video shooting unit 1 and sending it to the video generation unit 4 after being delayed. By delaying the video in this way, the subject position and orientation information obtained from the subject position and orientation detection unit 2 becomes information on the previous time relative to the video. Therefore, it is possible to perform processing equivalent to prefetching the position and orientation information of the subject (hereinafter referred to as virtual prefetching processing).

  A specific example of the third embodiment will be described with reference to FIGS. 10 and 11, reference numeral 21 is a soccer player assumed as a subject, 22 is a virtual camera, 23 is a soccer ball, arrow A is the front direction of the soccer player 21, and arrow D is the range indicated by the arrow D. This indicates that the shooting range is the virtual camera 22. 10 shows a situation where the ball is present near the player 21, and FIG. 11 shows a situation where the ball 23 is being separated from the player because the player 21 kicks the ball 23.

  In the first embodiment described with reference to FIGS. 4 and 5, it is possible to control the position of the virtual camera so that both the player 21 and the ball 23 enter the screen in both cases of FIGS. 4 and 5. Indicated. If virtual prefetch processing is further applied to the first embodiment, the player 21 kicks the ball 23 and knows that the ball 23 will leave first, so the viewpoint position and line-of-sight direction of the virtual camera are moved early, It becomes possible to control the viewpoint position and the line-of-sight direction of the virtual camera so that the traveling direction of the ball 23 falls within the screen (inside the arrow D indicating the shooting range).

  When the position of the virtual camera 22 shown in FIG. 10 is compared with the position of the virtual camera 22 shown in FIG. 4, in FIG. 10, the player 21 kicks the ball 23 and the virtual camera 22 is positioned and oriented. For the purpose of control, the position and orientation of the virtual camera is such that the traveling direction of the ball 23 enters the screen. Similarly, when the position of the virtual camera 22 shown in FIG. 11 is compared with the position of the virtual camera 22 shown in FIG. 5, the position of the virtual camera 22 shown in FIG. Position and orientation.

  By performing such virtual prefetching processing, it becomes easier for the viewer to follow the ball. Also, it becomes easier to see what is ahead of the moving direction of the ball. Alternatively, since the processing is delayed, it is possible to prevent the ball from coming out of the screen.

  Information other than the subject position and orientation information can also be used as information used for controlling the viewpoint position and line-of-sight direction of the virtual camera. For example, it is possible to control the viewpoint position and the line-of-sight direction of the virtual camera so as to avoid the viewpoint position and the line-of-sight direction of the virtual camera that is in the backlight state by using means for determining whether the generated screen is in the backlight state. . Alternatively, a high-brightness subject is determined from the video of the multi-view video photographing unit 1, and the viewpoint position and line-of-sight direction of the virtual camera are controlled so that the subject does not enter the generation screen. It is also possible to avoid the viewpoint position and the line-of-sight direction.

  As described above, it is also possible to set a predetermined condition for calculating the viewpoint position and the line-of-sight direction of the virtual camera that the generation screen does not enter the backlight state. In addition, various predetermined conditions can be used to calculate the viewpoint position and the line-of-sight direction of the virtual camera.

  In addition, a process of controlling the viewpoint position and the line-of-sight direction of the virtual camera so as to avoid the viewpoint position and the line-of-sight direction of the virtual camera that is in the backlight state, It is also possible to use a process for calculating the line-of-sight direction.

In this patent, as shown in FIG. 1 to FIG. 7 to FIG. 9, the data obtained by the multi-view video photographing unit 1 is directly input to the video generation unit 4 to the video delay unit 7. The embodiment is not limited to this. For example, the same mechanism can be applied to a system that applies to broadcasting, that is, a system in which multi-view video data captured by the multi-view video capturing unit 1 is transmitted via radio waves or a network, and the viewer receives and processes it. It is. A similar mechanism is also applied to a system applied to package media, that is, a system in which multi-view video data captured by the multi-view video capturing unit 1 is temporarily recorded on some recording medium, and is read and processed on the viewer side. Applicable.

It is a block diagram which shows the principal part schematic structure for demonstrating the 1st Example of the video system by this invention. It is a figure for demonstrating the 1st example which calculates the position and direction of a virtual camera. It is a figure for demonstrating the 2nd example which calculates the position of a virtual camera. It is a figure for demonstrating the 3rd example which calculates the position of a virtual camera. It is a figure for demonstrating the 3rd example which calculates the position of a virtual camera. It is a figure for demonstrating the 4th example which calculates the position and direction of a virtual camera. It is a block diagram which shows the principal part schematic structure for demonstrating the 2nd Example of the video system by this invention. FIG. 6 is a flowchart illustrating a process of providing a dead band for information on the direction of a subject (subject direction information) and calculating the direction information of the subject. It is a block diagram which shows the principal part schematic structure for demonstrating the 3rd Example of the video system by this invention. It is a figure for demonstrating the 3rd Example of this invention. It is a figure for demonstrating the 3rd Example of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Multi-eye image | video imaging | photography part, 2 ... Subject position and direction detection part, 3 ... Virtual camera position and direction calculation part, 4 ... Image | video production | generation part, 5 ... Image | video output part, 6 ... Information recording part, 7 ... Image | video delay part , 21 ... Player, 22 ... Virtual camera, 23 ... Ball.

Claims (21)

  The viewpoint position and line of sight of a virtual camera that satisfies a predetermined condition in a video system that synthesizes or selects an image in the viewpoint position and line of sight of an arbitrary virtual camera from an image of a subject taken from one or more viewpoint positions and line of sight An arbitrary viewpoint video generation system comprising a calculation means for calculating a direction.   Means for detecting the position and orientation of one or more subjects, and using the information on the position and orientation of one or more subjects obtained by the detection means, the viewpoint position and line of sight of a virtual camera that satisfies the predetermined condition The arbitrary viewpoint video generation system according to claim 1, wherein a direction is calculated.   3. The arbitrary viewpoint video generation according to claim 2, wherein the predetermined condition is that the virtual camera is positioned in a specific direction with respect to the subject, and the line-of-sight direction of the virtual camera faces the direction of the subject. system.   3. The arbitrary viewpoint video according to claim 2, wherein the predetermined condition is that the visual point direction of the virtual camera is fixed and the viewpoint position is changed, and the subject can be always photographed from the viewpoint position and the visual line direction of the virtual camera. Generation system.   5. The arbitrary viewpoint video generation system according to claim 2, wherein the predetermined condition is that a plurality of subjects are included in a video in a viewpoint position and a line-of-sight direction of a virtual camera.   3. The arbitrary viewpoint video generation system according to claim 2, wherein the predetermined condition is that a viewpoint position and a line-of-sight direction of a subject coincide with a viewpoint position and a line-of-sight direction of a virtual camera.   The arbitrary viewpoint video generation system according to any one of claims 2 to 6, wherein a process of providing a dead band is performed on information on a position and a direction of a subject.   The arbitrary viewpoint video generation system according to any one of claims 2 to 7, wherein a low-pass filter process is applied to information on a position and a direction of a subject.   The detection means for detecting the position and orientation of one or more subjects is a detection means for detecting one or more subjects using a multi-view image captured from one or more viewpoint positions and line-of-sight directions. The arbitrary viewpoint video generation system according to any one of claims 2 to 8.   The detection means for detecting the position and orientation of the one or more subjects is detection means for detecting using one or more subjects photographed from above. The arbitrary viewpoint video generation system according to 1.   The detection means for detecting the position and orientation of the one or more subjects has a sensor capable of detecting the location and orientation of the subject in the subject itself, and uses the information on the location and orientation of the subject acquired by the sensor. The arbitrary viewpoint video generation system according to any one of claims 2 to 8, wherein the arbitrary viewpoint video generation system is detection means for detecting.   The arbitrary viewpoint video generation system according to any one of claims 2 to 8, further comprising a detection unit that is a combination of the detection units of the detection units according to claims 9 to 11.   Arbitrary viewpoint video generation technology for synthesizing video in the viewpoint position and line-of-sight direction of the virtual camera based on video obtained by photographing the certain subject from one or more viewpoint positions and line-of-sight directions using the calculation result by the calculation means 13. The arbitrary viewpoint video generation system according to claim 1, wherein the arbitrary viewpoint video generation system is used.   Using the calculation result by the calculation means, select an image most similar to the image in the viewpoint position and line-of-sight direction of an arbitrary virtual camera from the images obtained by photographing the certain subject from one or more viewpoint positions and line-of-sight directions, and select The arbitrary viewpoint video generation system according to any one of claims 1 to 12, wherein the generated video is used as a video in a viewpoint position and a line-of-sight direction of an arbitrary virtual camera.   An image obtained by photographing a subject from one or more viewpoint positions and line-of-sight directions is delayed for a predetermined time, and an image in the viewpoint position and line-of-sight direction of any virtual camera is synthesized or delayed based on the delayed image. 15. The method according to claim 1, further comprising: selecting an image most similar to an image in a viewpoint position and a line-of-sight direction of an arbitrary virtual camera from the captured image, and setting the image in the viewpoint position and the line-of-sight direction of an arbitrary virtual camera. Or the arbitrary viewpoint video generation system according to claim 1.   16. The video obtained by photographing a subject from one or more viewpoint positions and line-of-sight directions is delayed for a predetermined time, and the viewpoint position and line-of-sight direction of the virtual camera are calculated relatively earlier than the video. The arbitrary viewpoint video generation system described in 1.   17. The arbitrary viewpoint video generation system according to claim 1, further comprising means for outputting a video generated at a viewpoint position and a line-of-sight direction of the virtual camera.   The arbitrary viewpoint video generation system according to claim 17, wherein a display of a two-dimensional display is used as the means for outputting the generated video.   The arbitrary viewpoint video generation system according to claim 17, wherein a stereoscopic display is used as the means for outputting the generated video.   20. The apparatus according to claim 1, further comprising means for detecting a condition in which the subject in the generated video is backlit, wherein the predetermined condition is that the subject in the generated video is not backlit. Or the arbitrary viewpoint video generation system according to claim 1.   21. The arbitrary viewpoint video generation system according to claim 20, wherein backlight detection is performed using video obtained by photographing a certain subject from one or more viewpoint positions and line-of-sight directions.

Images