JP2017156880A - Image processing apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform natural display even in the case where a virtual viewpoint moves in an arbitrary direction, in a free viewpoint video.SOLUTION: An image processing device includes: an acquisition part for acquiring an image which is acquired by imaging a subject coming into contact with a reference surface defined in a real space from a plurality of viewpoints; a calculation part for calculating the position on the reference surface in which the reference surface comes into contact with the subject, from the image acquired by the acquisition part; and a deformation part for deforming an image of the subject included in the image acquired by the acquisition part, based on the position calculated by the calculation part, when composing a composite image corresponding to the viewpoint which is not included in the plurality of viewpoints from the image acquired by the acquisition part.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing apparatus and an image processing method.

  2. Description of the Related Art Conventionally, a technique for generating a video from a free viewpoint other than the camera viewpoint (hereinafter referred to as a free viewpoint video) has been proposed for a sports scene or the like. This technology synthesizes videos from virtual viewpoints that are not arranged based on videos taken by multiple cameras, and displays the results on the screen for viewing videos from various viewpoints. It is possible.

  Here, among techniques for synthesizing free viewpoint videos, there is a technique for synthesizing free viewpoint videos at high speed using a simple model called a billboard (see Non-Patent Document 1). In the technology using this billboard, the texture of the object to be modeled is accurately cut out from the video, and it is made to stand on the ground in a virtual space as a thin billboard model, thereby generating a free viewpoint video.

  Here, in general, in the billboard system, the billboard is arranged such that the lowest point (for example, a person's foot) of a certain billboard is in contact with the ground of the virtual space. When the virtual viewpoint moves in the horizontal direction, the billboard is rotated in accordance with the movement of the virtual viewpoint, and when the virtual viewpoint moves in the vertical direction, the direction of the billboard is not changed.

Hayashi, K .; Saito, H., "Synthesizing Free-Viewpoing Images from Multiple View Videos in Soccer Stadium ADIUM," in Computer Graphics, Imaging and Visualisation, 2006 International Conference on, vol., No., Pp.220-225, 26-28 July 2006

  The method described in Non-Patent Document 1 is superior in that a high-quality free viewpoint video can be synthesized at high speed. However, the posture of the target person cannot be appropriately expressed by using a billboard (a model with only a texture having no thickness).

  The present invention has been made in view of these problems, and an object thereof is to provide a technique capable of performing natural display even when a virtual viewpoint moves in an arbitrary direction in a free viewpoint video.

  One embodiment of the present invention relates to an image processing apparatus. This image processing apparatus includes an acquisition unit that acquires an image obtained by imaging a subject that is in contact with a reference plane defined in real space from a plurality of viewpoints, and the reference plane and the subject from the image acquired by the acquisition unit. A calculation unit that calculates a position on a reference plane that is in contact with and a position calculated by the calculation unit when combining a composite image corresponding to a viewpoint that is not included in a plurality of viewpoints from the image acquired by the acquisition unit. And a deforming unit that deforms an image of the subject included in the image acquired by the acquiring unit.

  It should be noted that any combination of the above-described constituent elements, or those obtained by replacing the constituent elements and expressions of the present invention with each other between apparatuses, methods, systems, computer programs, recording media storing computer programs, and the like are also included in the present invention. It is effective as an embodiment of

  According to the present invention, natural display can be performed even when a virtual viewpoint moves in an arbitrary direction in a free viewpoint video.

It is a schematic diagram which shows a free viewpoint image delivery system provided with the image processing apparatus which concerns on embodiment. It is a block diagram which shows the function and structure of the image processing apparatus which concerns on embodiment. It is explanatory drawing which shows the correspondence of the coordinate on the image plane of a camera, and a field coordinate. 4A to 4C are explanatory diagrams illustrating an example of processing in the background difference unit. It is explanatory drawing which shows the extraction process of the skeleton in a billboard control point determination part. It is explanatory drawing which shows the specific process of the control point in a billboard control point determination part. FIGS. 7A to 7F are explanatory diagrams illustrating a grounding position estimation process in the billboard posture estimation unit. It is an image figure which shows the estimation result of the attitude | position with respect to an original image. It is explanatory drawing which shows the variable regarding a billboard reference | standard figure. It is explanatory drawing which shows the reconstruction process of the billboard in a billboard reconstruction part. 2 is a flowchart showing a flow of a series of processes in the image processing apparatus of FIG. 1. It is a figure which shows the image in a camera viewpoint, and the image in the virtual viewpoint moved to the perpendicular direction from the camera viewpoint.

  Hereinafter, the same or equivalent components, members, and processes shown in the drawings are denoted by the same reference numerals, and repeated description is appropriately omitted. In addition, in the drawings, some of the members that are not important for explanation are omitted.

  In the method described in Non-Patent Document 1, for example, when a virtual viewpoint is moved in the vertical direction when a pose such as a push-up is taken, the posture of the person cannot be expressed appropriately, as if the hand (the bottom of the billboard) It may be composed as if it stands on the ground at a point) and the feet are floating. FIG. 12 is a diagram showing an image 102 at the camera viewpoint and an image 104 at the virtual viewpoint moved in the vertical direction from the camera viewpoint. In the image 102 from the camera viewpoint, the person appears to touch the ground with both hands and feet. However, in the image 104 at the virtual viewpoint, the display of the billboard itself is not changed, and it is interpreted that the person is standing on the ground with his hand, so that the person's feet appear to be floating (reference numeral 103).

  The reason for this is that the conventional method targets images taken from a distance (for example, images from soccer stadium seats), and does not consider the difference in depth caused by the posture of the person. Is assumed to be standing on the ground with one point (foot).

  In contrast, in the embodiment, the posture of a person's billboard is simply estimated using calibration data of a plurality of cameras measured in advance, and the billboard is artificially deformed using the information. Thus, even when the virtual viewpoint moves in the free viewpoint video, adaptive display is performed.

  FIG. 1 is a schematic diagram illustrating a free viewpoint image distribution system 110 including an image processing device 200 according to an embodiment. The free-viewpoint image distribution system 110 includes a plurality of cameras 116, 118, and 120, an image processing device 200 connected to the cameras, and a mobile terminal 114 such as a mobile phone, a tablet, a smartphone, or an HMD (Head Mounted Display). . The image processing apparatus 200 and the portable terminal 114 are connected via a network 112 such as the Internet. In the free viewpoint image distribution system 110, for example, a plurality of cameras 116, 118, and 120 arranged in a studio image the dancer 124 standing on the ground 126. The plurality of cameras 116, 118, and 120 transmit captured images to the image processing apparatus 200, and the image processing apparatus 200 processes these images. The user of the portable terminal 114 designates a desired viewpoint with respect to the image processing apparatus 200, and the image processing apparatus 200 synthesizes an image when the dancer 124 is viewed from the designated viewpoint (virtual viewpoint), via the network 112. To the mobile terminal 114.

  In addition, although FIG. 1 demonstrated the case where the dancer in a studio was imaged, it is not restricted to this, For example, when imaging a fitness instructor, when imaging a tennis game, when imaging a soccer game, The technical idea of the present embodiment can be applied when imaging a subject that is in contact with a reference plane in real space. Further, a stationary terminal such as a desktop PC, a laptop PC, or a TV receiver may be used instead of the portable terminal 114.

  FIG. 2 is a block diagram illustrating functions and configurations of the image processing apparatus 200 according to the embodiment. Each block shown here can be realized by hardware such as a computer (CPU) (Central Processing Unit) and other elements and mechanical devices, and software can be realized by a computer program or the like. The functional block realized by those cooperation is drawn. Therefore, it is understood by those skilled in the art who have touched this specification that these functional blocks can be realized in various forms by a combination of hardware and software.

  The image processing apparatus 200 synthesizes an image of an arbitrary virtual viewpoint from images taken by a plurality of cameras. The image processing apparatus 200 includes an image reading unit 202, 204, 206, a background difference unit 208, 210, 212, a camera calibration unit 214, 216, 218 (may be collectively referred to as a camera calibration unit 220), A billboard control point determination unit 222, 224, 226, a billboard posture estimation unit 228, a billboard reconstruction unit 230, 232, 234, and a composite video output unit 236 are provided. For each of the N cameras (N is a natural number of 2 or more), an image reading unit, a background difference unit, a camera calibration unit, a billboard control point determination unit, and a billboard reconstruction unit are provided.

  The image processing apparatus 200 generates a free viewpoint video in accordance with a billboard system based on a plurality of camera videos obtained by photographing the same subject. In the conventional billboard method, the billboard is not deformed (including rotation) with respect to the movement of the virtual viewpoint in the vertical direction, but the image processing apparatus 200 according to the present embodiment does not change the virtual viewpoint in the vertical direction. For movement, a billboard is generated in accordance with the virtual viewpoint.

  In the following, it is assumed that time synchronization between a plurality of cameras is performed in advance. In the following, modeling of a person is mainly assumed, but the technical idea of the present embodiment can be applied to modeling of other subjects. The virtual viewpoint is a virtual viewpoint that can be arbitrarily designated by the user, for example, and is not included in the actual viewpoints in which the cameras 1 to N are arranged.

[Outline of Image Processing Device 200]
In the image processing in the image processing apparatus 200, a surface in real space such as the ground surface (ground), a floor surface, or a wall surface is defined as a reference surface. For example, the ground is modeled as a two-dimensional surface. The image reading units 202, 204, and 206 function as an acquisition unit that acquires an image obtained by imaging a person in contact with the ground with the cameras 1 to N arranged at a plurality of different viewpoints.

  The camera calibration unit 220 receives images taken by a plurality of cameras as inputs, associates the ground in real space (sometimes referred to as a field) with a camera image for each camera, and outputs it. This calibration operation need only be performed for one camera when a fixed camera is assumed, and can be calculated by using a known positional relationship between cameras for the other cameras. The background difference units 208, 210, and 212 separate the background and foreground in the image using the background difference method, and output a binarized image.

  The billboard control point determination units 222, 224, and 226 determine pixels in the billboard to be controlled to obtain a final synthesis result based on the person billboards obtained in the background difference units 208, 210, and 212. To do. The billboard posture estimation unit 228 estimates the contact position of the person using the association information with the real space obtained by the camera calibration unit 220, and what kind of posture the person actually has in the three-dimensional real space. Estimate what you are taking. Further, the billboard posture estimation unit 228 calculates a polygon (referred to as a billboard reference figure) representing the polygon. The billboard reconstruction units 230, 232, and 234 are the billboard obtained by the background difference units 208, 210, and 212, the control points obtained by the billboard control point determination units 222, 224, and 226, and the billboard posture estimation unit. Based on the billboard reference graphic obtained in 228 and information on the virtual viewpoint selected by the user, a billboard optimal for the virtual viewpoint is constructed.

[Camera calibration unit 220]
The camera calibration unit 220 calculates a characteristic point of a field image in an image taken at a certain time (for example, an intersection of white lines of a tennis court or a soccer ground) and a corresponding point on the field in an actual real space. Perform the association. Here, when the subject is included in a general sports video, since the size of the court etc. is standardized, the point on the image plane corresponds to which coordinate on the field in the real space (world coordinate system) It is possible to calculate what to do.

FIG. 3 is an explanatory diagram showing a correspondence relationship between coordinates on the image plane of the camera 402 and field coordinates. When the coordinates on the two-dimensional image plane of the camera 402 are (u, v) and the coordinates (field coordinates) on the field plane of the world coordinate system are (x ′, y ′), the correspondence between the two is homography. matrix
And a scalar value s can be expressed as follows.
... (Formula 1)

S and H can be obtained by inputting the set of corresponding points in Equation 1, and mutual conversion between the coordinates of an arbitrary pixel on the image plane and the field coordinates becomes possible. Here, the homography matrix representing the correspondence between the coordinates on the image plane of the i-th camera and the field coordinates is
And

  The camera calibration method for the field is not limited to the above.

Next, an image captured at time t
The following processing is independently performed for each camera with respect to ( _mi and _ni are the width and height of the image of the i-th camera, respectively).

[Background difference unit 208, 210, 212]
Background subtraction unit _{208, 210,} 212 to classify each pixel of _I ^{i t} into two between the background and the _foreground, to separate _I ^{i t} in the background and foreground. In the present embodiment, this separation may be realized using a known background subtraction method, for example. The background difference units 208, 210, and 212 assign 0 and 1 to the pixel values set as the background and the foreground, respectively. By separating the background and the foreground, a rough region including a person can be extracted. The background difference units 208, 210, and 212 generate a texture by extracting a connected pixel set from the pixel set extracted as the foreground from the image. The background difference units 208, 210, and 212 output the generated texture as a billboard. The output billboard corresponds to an image of a person included in an image captured by the camera.

4A to 4C are explanatory diagrams illustrating examples of processing in the background difference units 208, 210, and 212. FIG. 4 (a) shows the image I _i ^t of the captured time t by i-th camera. Background subtraction unit 208, 210, 212 processes the image _I ^{i t} as an original image. FIG. 4B shows the result of applying the background subtraction method to the original image of FIG. The black part is determined to be the background and 0 is assigned. The white part is determined to be the foreground, and 1 is assigned. FIG. 4C shows a billboard that the background difference units 208, 210, and 212 process and output the original image of FIG.

[Billboard Control Point Determination Unit 222, 224, 226]
The billboard control point determination units 222, 224, and 226 set or determine pixels (control points) inside the billboard to be used for billboard reconstruction according to the virtual viewpoint from the billboards of the cameras. First, the billboard control point determination units 222, 224, and 226 generate an abstract image obtained by abstracting the billboard. Specifically, the billboard control point determination units 222, 224, and 226 apply a known reduction process to the texture pixel set constituting the billboard, and the human skeleton in the billboard is connected. This is expressed as a pixel set (skeleton).

  FIG. 5 is an explanatory diagram showing a skeleton extraction process in the billboard control point determination units 222, 224, and 226. Billboard 502 is “reduced” and eventually becomes a connected set of lines or skeleton 504. Reference numeral 506 indicates a state in which the obtained skeleton 504 is superimposed on the original billboard 502.

  The billboard control point determination units 222, 224, and 226 specify, as control points, pixels that satisfy a predetermined condition from a set of pixels included in the obtained skeleton. The predetermined condition is, for example, that the number of connections is not two. The number of connections is the number of pixels belonging to the pixel set among the eight pixels around the pixel in a pixel set included in the obtained skeleton. The pixel selected in this way becomes an end portion of the skeleton or a branch portion of the skeleton. That is, the end portion of the skeleton and the branch portion of the skeleton are used as control points.

  FIG. 6 is an explanatory diagram showing control point identification processing in the billboard control point determination units 222, 224, and 226. As for the control points, it is desirable to indicate the same part of the person even when the billboard moves in the moving image. Therefore, the feature point in the billboard that is closest to the control point can be used (a slight error in the coordinates of the control point does not greatly affect the result). It is also possible to manually give the initial value of the control point.

[Billboard Posture Estimation Unit 228]
The billboard posture estimation unit 228 uses the binary mask (for example, the image shown in FIG. 4B) of the person area obtained by the background difference units 208, 210, and 212 on the field where the field and the person are in contact with each other. It functions as a calculation unit for calculating the position. In other words, the billboard posture estimation unit 228 estimates the posture of a person using a binary mask.

First, the billboard posture estimation unit 228 extracts a place where the person is actually in contact with the field by the following processing. A binary mask corresponding to the image I _i ^t at time t captured by the i-th camera is denoted as J _i ^t . The correspondence between the coordinates on the image plane of the i-th camera and the field coordinates is obtained by using the homography matrix H _i calculated by the camera calibration unit 220 according to Equation 1,
... (Formula 2)
It is expressed.
The billboard posture estimation unit 228 performs projection and integration on the field coordinates as follows with respect to the binary masks at the time t of all the cameras using Expression 2.
... (Formula 3)
Here, F ^t is a matrix that holds values in each of the field coordinates, and F ^t (x ″, y ″) is a value in the field coordinates (x ″, y ″).

  In the above projection operation, among the pixels on the image plane of the camera, those corresponding to the place where the person is in contact with the field plane in the real space are projected to the same field coordinates regardless of the camera. Therefore, a rough position where the person is in contact with the ground can be calculated by integrating all the projected values.

FIGS. 7A to 7F are explanatory diagrams illustrating the grounding position estimation processing in the billboard posture estimation unit 228. FIG. FIG. 7A shows a binary mask J ₁ ^t corresponding to the image I ₁ ^t at time t captured by the first camera. FIG. 7B shows a binary mask J ₂ ^t corresponding to the image I ₂ ^t at time t captured by the second camera. FIG. 7C shows a binary mask J ₃ ^t corresponding to the image I ₃ ^t at time t captured by the third camera. FIG. 7D shows a mask obtained as a result of projecting _three binary masks J ₁ ^t , J ₂ ^t , and J ₃ ^t onto the field and integrating them. The F ^t places 710 the person is in contact with the field is calculated as a larger value (high value enough color in FIG. 7 (d) is close to black F ^t).

Billboard posture estimation unit 228, by performing the following threshold processing on the F ^t, estimates the candidate position on the field the person is in contact with the field.
... (Formula 4)

Figure 7 (e) shows what the f ^t obtained by equation 4, is projected to the image plane of a camera. A set 712 of black pixels corresponds to a person's ground contact position. FIG. 7 (f) shows a ground contact point and a circumscribed polygon. Through the above operation, a set of pixels 712 on the image plane determined to be the person's ground contact position is obtained. The billboard posture estimation unit 228 determines the centroid of the connected pixel set as locations (grounding points) 702, 704, 706, and 708 where the person is actually in contact with the pixel set 712. A circumscribed polygon α obtained by constructing a complete graph with respect to the contact points 702, 704, 706, and 708 indicates a general posture of the person. FIG. 8A is an image diagram showing a posture estimation result with respect to the original image of FIG. FIG. 8B is an image diagram showing a result of posture estimation for the original image in FIG. In FIG. 8B, four ground points 802, 804, 806, 808 are obtained.

FIG. 9 is an explanatory diagram showing variables related to the billboard reference graphic. The billboard posture estimation unit 228 generates a figure Θ that roughly includes a person by translating the obtained circumscribed polygon α in the y direction on the image plane. Of this figure Θ, a parallelogram that includes two line segments that are the locus of the vertex of the circumscribed polygon α in translation (looks like a side on the image)
And M is a natural number. When the parallelogram obtained as a result of performing the expansion process on the parallelogram θ _j is expressed as ζ _j , the billboard posture estimation unit 228 sets the union of ζ _j
Ζ _j is determined so as to circumscribe the person's billboard. Ζ _j determined in this way is referred to as a billboard reference graphic. The billboard reference figure ζ _j is a reference when the billboard reconstructing units 230, 232, and 234 deform the billboard.

[Billboard reconstruction unit 230, 232, 234]
The billboard reconstructing unit 230 selects the billboard obtained by the background difference unit 208, the control points obtained by the billboard control point determination unit 222, the billboard reference figure obtained by the billboard posture estimation unit 228, and the user selection. Based on the information on the virtual viewpoint, the optimal billboard for the virtual viewpoint is constructed. When the billboard reconstruction unit 230 synthesizes a composite image corresponding to the virtual viewpoint from the image acquired by the image reading unit 202, the billboard reconstruction unit 230 generates a billboard based on the ground contact position calculated by the billboard posture estimation unit 228. It functions as a deforming part that deforms. The same applies to the billboard reconstruction units 232 and 234. In this specification, the case where the virtual viewpoint is moved in the vertical direction from the actual viewpoint will be described, and the horizontal movement of the virtual viewpoint is handled by the rotation of the billboard as in the conventional method.

  FIG. 10 is an explanatory diagram showing billboard reconfiguration processing in the billboard reconfiguration units 230, 232, and 234. An image indicated by reference numeral 154 is an image at time t captured by a certain camera. The image indicated by reference numeral 156 is a synthesized image synthesized as an image at time t captured from the virtual viewpoint.

The coordinates on the image plane of the control points included in each billboard reference figure ζ _j obtained by the billboard posture estimation unit 228
And

ζ _j clearly has two ground points from its definition. The field coordinates corresponding to the contact point are calculated in the billboard posture estimation unit 228, and the field coordinates do not change due to the movement of the virtual viewpoint. Using this property, the billboard reconstruction unit 230 projects the field coordinates on the image plane of the virtual viewpoint based on the positional relationship between the virtual viewpoint moved vertically and the field. Thereby, the billboard reconstruction unit 230 calculates the position (for example, the position of the point of reference numeral 152) on which the ground point corresponding to the field coordinates should be displayed on the virtual viewpoint image. That is, the billboard reconstruction unit 230 calculates information about how the billboard reference figure ζ _j obtained from the image captured by the camera should appear in the virtual viewpoint.

Let η _j be a new billboard reference figure obtained from the current virtual viewpoint. Billboard reconstruction unit 230 maps
Projected onto the virtual viewpoint screen
The billboard is deformed using (points to the coordinates on the image plane of the virtual viewpoint). Note that the mapping X _j is not applied except for the coordinates of the control points. This is to suppress unnaturalness at the boundary of the billboard reference figure.

  The billboard reconstruction unit 230 finally deforms the billboard by moving control points on the billboard including the grounding point. In this modification, the two-dimensional image is treated as a triangle mesh, and the original structure is not destroyed while being deformed according to the movement of the control point. The billboard deformed by this operation is employed as a billboard for combining with a virtual viewpoint image. Note that some positions of the control points after being moved include a ground contact position on the virtual viewpoint image (for example, the position of the point 152).

  The combined video output unit 236 functions as a combining unit that combines the deformed billboard obtained as a result of the deformation of the billboard by the billboard reconstructing units 230, 232, and 234 into a virtual viewpoint composite image. The synthesized video output unit 236 generates a synthesized image obtained by synthesizing the deformed billboard as an image from the virtual viewpoint, and transmits the generated image to the mobile terminal 114 via the network 112.

The operation of the image processing apparatus 200 having the above configuration will be described.
FIG. 11 is a flowchart showing a flow of a series of processes in the image processing apparatus 200. The image processing apparatus 200 acquires an image including an image of a person from an imaging apparatus such as a camera arranged at each of a plurality of viewpoints set around a person in contact with the ground (S12). The image processing apparatus 200 extracts a binary mask of the person area from each of the acquired images (S14). The image processing apparatus 200 projects each of the extracted binary masks onto a field in real space (S16). The image processing apparatus 200 estimates the position of the portion where the projected images overlap more largely as the ground contact position (S18). The image processing apparatus 200 cuts out a billboard from the image acquired from the camera, and sets a control point on the billboard (S20). The image processing apparatus 200 determines how to move the control point based on the estimated contact position (S22). The image processing apparatus 200 deforms the billboard by moving the control points in the determined manner of movement (S24). The image processing apparatus 200 combines the deformed billboard with the image and outputs it as an image from the virtual viewpoint (S26).

  In the embodiment described above, examples of the holding unit are a hard disk and a semiconductor memory. Further, based on the description of the present specification, each unit is configured by a CPU (not shown), a module of an installed application program, a module of a system program, a semiconductor memory that temporarily stores the content of data read from the hard disk, or the like. It will be appreciated by those skilled in the art who have touched this specification that this can be achieved.

  According to the image processing apparatus 200 according to the present embodiment, it is possible to eliminate an unnatural display that becomes a problem when the virtual viewpoint is moved in the vertical direction by the conventional billboard method. In this embodiment, the posture of a person is estimated to deform the billboard. However, since an actual accurate three-dimensional model (thick model) is not required, such a three-dimensional model is constructed. The calculation cost is lower than the method of performing the above.

  The configuration and operation of the image processing apparatus 200 according to the embodiment have been described above. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to each component and combination of processes, and such modifications are within the scope of the present invention.

In the embodiment, the description has been given of the case to deform the billboard using control points is not limited thereto and may be applied to mapping X _j for example, all pixels included in the billboard reference graphic zeta _j.

  110 free viewpoint image distribution system, 112 network, 114 mobile terminal, 200 image processing apparatus.

Claims (10)

An acquisition unit for acquiring an image obtained by capturing an image of a subject in contact with a reference plane defined in real space from a plurality of viewpoints;
A calculation unit that calculates a position on the reference surface where the reference surface and the subject are in contact with each other, from the image acquired by the acquisition unit;
When synthesizing a composite image corresponding to a viewpoint that is not included in the plurality of viewpoints from the image acquired by the acquisition unit, the image acquired by the acquisition unit is based on the position calculated by the calculation unit. An image processing apparatus comprising: a deformation unit configured to deform an image of the included subject.   The image processing apparatus according to claim 1, further comprising a combining unit that combines an image obtained as a result of deformation by the deforming unit with the combined image.   The deformation unit calculates and calculates a corresponding position on the composite image corresponding to the position calculated by the calculation unit based on a positional relationship between viewpoints not included in the plurality of viewpoints and the reference plane. The image processing apparatus according to claim 1, wherein the image of the subject is deformed based on the corresponding position. A setting unit for setting at least one control point on the subject image;
The deformation unit deforms the image of the subject by moving at least one control point set by the setting unit,
The image processing apparatus according to claim 3, wherein the position of at least one control point after being moved includes a calculated corresponding position.   The image processing apparatus according to claim 1, wherein the reference plane is a surface in the real space. An abstraction unit that generates an abstract image obtained by abstracting the image of the subject;
The image processing apparatus according to claim 4, wherein the setting unit sets a control point for the abstract image generated by the abstraction unit.   The image processing apparatus according to claim 6, wherein the setting unit sets a branch portion and / or an end portion of the abstract image generated by the abstraction unit as a control point.   The calculation unit determines a position on the reference plane where the reference plane and the subject are in contact with each other based on a correspondence relationship between an image obtained by imaging the subject from each of the plurality of viewpoints and the reference plane. The image processing apparatus according to claim 1, wherein the image processing apparatus calculates the image processing apparatus. Obtaining an image obtained by imaging a subject in contact with a reference plane defined in real space from a plurality of viewpoints;
Calculating a position on the reference plane where the reference plane and the subject are in contact with each other from the acquired image;
When combining a synthesized image corresponding to a viewpoint that is not included in the plurality of viewpoints from the acquired image, the image of the subject included in the acquired image is deformed based on the calculated position. And an image processing method. A function for acquiring an image obtained by imaging a subject in contact with a reference plane defined in real space from a plurality of viewpoints;
A function of calculating a position on the reference plane where the reference plane and the subject are in contact with each other from the acquired image;
When combining a synthesized image corresponding to a viewpoint that is not included in the plurality of viewpoints from the acquired image, the image of the subject included in the acquired image is deformed based on the calculated position. A computer program for causing a computer to realize functions.