KR20180083245A - Apparatus and method for processing information of multi camera - Google Patents

Abstract

An apparatus and method for processing the information of a multi camera are disclosed. The apparatus for processing the information of a multi camera according to an embodiment of the present disclosure may include a plurality of cameras for capturing an image including at least one object, a positional information verifying part provided in each of the plurality of cameras and verifying positional information between the at least one object and the camera; and an image processing part for processing image information obtained from the plurality of cameras using the positional information received from the positional information verifying part. It is possible to effectively process creation or display of realistic media.

Description

TECHNICAL FIELD [0001] The present invention relates to an information processing apparatus and method for a multi-

The present disclosure relates to an image processing method and apparatus, and more particularly, to a method and apparatus for processing an image obtained through a multi-camera system.

The use of high definition (HD) content is increasing, and realistic media contents such as three-dimensional images and ultra high definition (UHD) images are being developed to meet the needs of viewers who want more realistic images.

2. Description of the Related Art [0002] In recent years, there has been a demand for providing a high-resolution panoramic image content capable of providing a wide field of view (FOV) Interest in content is growing.

In order to construct such high-resolution panoramic image contents and virtual reality (VR) contents, techniques for acquiring multi-view images using a plurality of cameras, techniques for generating panoramic images using multi-view images, And multi-projection display technology for rendering.

The technical object of the present disclosure is to provide a method and apparatus that can effectively process creation or display of real-sensible media contents.

It is another object of the present invention to provide a method and apparatus for efficiently processing a multi-view image constituting real-sensible media contents.

The technical objects to be achieved by the present disclosure are not limited to the above-mentioned technical subjects, and other technical subjects which are not mentioned are to be clearly understood from the following description to those skilled in the art It will be possible.

According to one aspect of the present disclosure, an information processing apparatus of a multi-camera can be provided. The apparatus comprising: a plurality of cameras for capturing an image including at least one object; a positional information verifying unit provided for each of the plurality of cameras, for verifying positional information between the at least one object and the camera; And an image processing unit for processing image information obtained from the plurality of cameras using the positional information received from the positional information verifying unit.

According to another aspect of the present disclosure, a composite image providing method using a multi-camera can be provided. The method includes the steps of receiving a plurality of images photographed by each of a plurality of cameras, receiving a plurality of pieces of positional information provided from a plurality of position determination units respectively provided at positions corresponding to the plurality of cameras, Matching original pixels included in each of the plurality of images with synthetic pixels included in the composite image in consideration of the relationship between the position information, the plurality of images, and the composite image corresponding to the camera of the composite image, Determining a value of a source pixel in each of the image units and applying the value to the value of the composite pixel; and generating a composite image by combining the values of the composite pixels.

The features briefly summarized above for this disclosure are only exemplary aspects of the detailed description of the disclosure which follow, and are not intended to limit the scope of the disclosure.

According to the present disclosure, a method and apparatus capable of effectively processing generation or display of sensible media contents can be provided.

In addition, according to the present disclosure, a method and apparatus for efficiently processing a multi-view image constituting real-sensible media contents can be provided.

The effects obtainable from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below will be.

1 is a block diagram showing a configuration of an information processing apparatus of a multi-camera according to an embodiment of the present disclosure.
2 is a diagram illustrating a relationship between an image processing apparatus and an image of a multi-camera according to an embodiment of the present disclosure.
FIGS. 3A and 3B are diagrams illustrating a geometric relationship of an original image as a basis of a composite image used in an information processing apparatus of a multi-camera according to an embodiment of the present disclosure; FIG.
4 is a diagram illustrating a relationship between an original image and a composite image processed in an information processing apparatus of a multi-camera according to an embodiment of the present disclosure.
5 is a flowchart showing a procedure of an information processing method of a multi-camera according to an embodiment of the present disclosure.
FIG. 6 is a diagram illustrating an example of an operation for converting position information into the information processing method of the multi-camera according to the embodiment of the present disclosure.
7 is a diagram illustrating a procedure of a method of providing a composite image using a multi-camera according to another embodiment of the present disclosure.
8 is a block diagram illustrating a computing system that implements an apparatus and method for processing information of a multi-camera and a method and apparatus for providing a composite image using a multi-camera according to an embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, which will be easily understood by those skilled in the art. However, the present disclosure may be embodied in many different forms and is not limited to the embodiments described herein.

In the following description of the embodiments of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present disclosure rather unclear. Parts not related to the description of the present disclosure in the drawings are omitted, and like parts are denoted by similar reference numerals.

In the present disclosure, when an element is referred to as being "connected", "coupled", or "connected" to another element, it is understood that not only a direct connection relationship but also an indirect connection relationship May also be included. Also, when an element is referred to as " comprising "or" having "another element, it is meant to include not only excluding another element but also another element .

In the present disclosure, the terms first, second, etc. are used only for the purpose of distinguishing one element from another, and do not limit the order or importance of elements, etc. unless specifically stated otherwise. Thus, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly a second component in one embodiment may be referred to as a first component .

In the present disclosure, the components that are distinguished from each other are intended to clearly illustrate each feature and do not necessarily mean that components are separate. That is, a plurality of components may be integrated into one hardware or software unit, or a single component may be distributed into a plurality of hardware or software units. Thus, unless otherwise noted, such integrated or distributed embodiments are also included within the scope of this disclosure.

In the present disclosure, the components described in the various embodiments are not necessarily essential components, and some may be optional components. Thus, embodiments consisting of a subset of the components described in one embodiment are also included within the scope of the present disclosure. Also, embodiments that include other elements in addition to the elements described in the various embodiments are also included in the scope of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings.

FIG. 1 is a block diagram showing a configuration of an information processing apparatus of a multi-camera according to an embodiment of the present disclosure, and FIG. 2 is a diagram illustrating a relationship between an information processing apparatus and an image of a multi- FIG.

Referring to FIGS. 1 and 2, an information processing apparatus of a multi-camera may include a plurality of cameras 11a and 11b, position information confirmation units 13a and 13b, and an image processing unit 15. [

The plurality of cameras 11a and 11b are cameras provided on the basis of the multi-camera system, and may be cameras capable of photographing predetermined images by setting shooting directions, shooting angles, and the like. In addition, the plurality of cameras 11a and 11b can receive or receive a control command for controlling the start and end of shooting, and can control shooting or ending of the video in response to the control command, Can be synchronized with each other.

In the embodiment of the present invention, an image captured through the plurality of cameras 11a and 11b may include a still image and a moving image obtained by combining still images captured in predetermined time units.

The position information confirmation units 13a and 13b may be devices for acquiring correlation position information of the plurality of cameras 11a and 11b. For example, the positional information verifying units 13a and 13b may determine the distance between the subject included in the image captured by each of the plurality of cameras 11a and 11b and the plurality of cameras 11a and 11b Meter.

The location information confirmation units 13a and 13b may be provided inside the plurality of cameras 11a and 11b or may be provided outside the plurality of cameras 11a and 11b and may include a plurality of cameras 11a and 11b Can measure the distance to the subject included in the area corresponding to the image photographed by the photographer.

The positional information verifying units 13a and 13b can output, as positional information, a positional information verifying device identifier for identifying each device and distance data for measuring the distance to the detected subject (or object).

As another example, the location information confirmation units 13a and 13b may include a depth camera, and the area captured by the depth camera includes a plurality of cameras 11a and 11b And may be set as an area corresponding to an area to be photographed. The depth camera can photograph and provide a depth image of the corresponding region.

The image processing unit 15 can confirm information on the photographing direction, photographing angle, and the like of the plurality of cameras 11a and 11b provided on the basis of the multi-camera system. The image processing unit 15 photographs the plurality of cameras 11a and 11b The composite image can be generated by reflecting the geometric information of the image.

For example, referring to FIGS. 3A and 3B, the first image 301 and the second image 302 illustrate images captured by cameras 11a and 11b located at different shooting locations.

In the case of images taken at different shooting locations, it is difficult to find corresponding points due to the geometrical errors between the two images. That is, as illustrated in FIG. 3A, the first image 301 and the second image 302 are different from each other in terms of the difference in position of the camera for photographing the point P, the error caused by the arrangement of the camera, There may be differences in the internal characteristics of the camera and the like. In consideration of this, the image processing unit 15 sets a line (i.e., an epipolar line) 310 corresponding to each of the first image 301 and the second image 302, Image Rectification) processing. In the epipolar line 310, a point of the first image 301 corresponds to a line of the second image 302 in a three-dimensional image (e.g., a stereo image), which corresponds to the corresponding line. In order to match the polar lines 310, parameter information inside and outside the camera (e.g., camera focal length, principal point) of the camera that captured the first image 301 and the second image 302 The position of the image, the distortion of the image, and the relative rotation angle and position between the cameras).

Furthermore, the color difference between the two points of view caused by the color characteristics of the plurality of cameras and the influence of illumination and shadows may also lead to subjective image quality deterioration at the time of image synthesis. The image processing unit 15 may extract the corresponding points existing in the first image 301 and the first image 302 and correct the color difference between the corresponding points.

The image processing unit 15 generates a three-dimensional composite image 305 through a geometric process for converting the image-aligned first image 301 and the second image 302 into a three-dimensional coordinate system. For example, the image processing unit 15 may generate the three-dimensional composite image 305 by projecting the image-aligned first image 301 and the second image 302 onto the cylinder region or the spherical surface.

As described above, the image processing unit 15 can use the position information of the plurality of cameras 11a and 11b in processing the alignment and geometric transformation of images photographed from the plurality of cameras 11a and 11b. The positional information of the plurality of cameras 11a and 11b may be provided from the positional information verifying units 13a and 13b.

Further, when the position information confirmation units 13a and 13b provide the position information including the position information identifying device identifier identifying each device and the distance data measuring the distance to the detected object (or object) The processing unit 15 may convert the location information identifying device identifier into an object identifier and process the object identifier. Accordingly, the image processing unit 15 can check the distance data from the camera to the object using the object identifier, and use it for image alignment and geometric transformation processing.

On the other hand, when the position information confirmation units 13a and 13b provide a depth image, the image processing unit 15 can calculate the distance data from the corresponding camera to the object based on the depth image, It can be used for image alignment and geometric transformation processing.

4 is a diagram illustrating a relationship between an original image and a composite image processed in an information processing apparatus of a multi-camera according to an embodiment of the present disclosure.

The composite image 410 may be formed by combining original images 401 and 402 captured from a plurality of cameras. As described above, the photometric and geometric variations of the original images 401 and 402 .

For example, the composite pixel X _{C included} in the composite image 410 may include a first original pixel X _A provided in the first original image 401, The first weight W _A and the second original pixel X _B may be combined with a second weight W _B considering a photometric relationship and a geometric variation relationship. At this time, the first weight W _A and the second weight W _B can be calculated in advance from the image synthesis algorithm, and the pre-calculated data can be managed in a look-up table format.

Further, the composite image can be configured by combining a plurality of original images. Depending on how many original images are combined to form a composite image, the weights reflected on the original image can be set differently. Accordingly, a weight applied to the original image may be set in the lookup table according to the number of original images composing the composite image.

Based on this, the image processing unit 15 can check the number of the cameras 11a and 11b and check the lookup table to which the weight corresponding to the number of the cameras 11a and 11b is applied. The weight information included in the identified lookup table may be applied to each of the original images provided from the plurality of cameras 11a and 11b to form a composite image.

5 is a flowchart showing a procedure of an information processing method of a multi-camera according to an embodiment of the present disclosure.

The information processing method of the multi-camera can be performed by the above-described information processing apparatus of the multi-camera.

In step S51, the information processing apparatus of the multi-camera can shoot and acquire a plurality of original images shot from a plurality of cameras based on the multi-camera system.

In step S52, the information processing apparatus of the multi-camera can acquire the correlated position information corresponding to each of the plurality of cameras. For example, an information processing apparatus of a multi-camera can acquire distance data confirmed by a subject included in an image photographed by each of a plurality of cameras and a distance measurer capable of measuring a distance between the plurality of cameras.

Particularly, the information processing apparatus of the multi-camera generates and outputs positional information including a positional information identifying apparatus identifier for identifying the positional information confirming apparatus and distance data for measuring the distance to the detected object (or object) .

In step S53, the information processing apparatus of the multi-camera can confirm information about the photographing direction, photographing angle, and the like of a plurality of cameras provided on the basis of the multi-camera system, The composite image can be generated by reflecting the information.

In the process of generating the composite image, the information processing apparatus of the multi-camera may use the position information provided in step S52 as the geometric information. Specifically, the received location information may include a device identifier 601 (see FIG. 6) and distance data 602, which can not specify an object included in the image using the device identifier 601. Thus, , And convert the device identifier 601 detected in the received location information into the object identifier 611.

Since the position information converted based on the object identifier can be matched to each object included in the plurality of original images, it is important to align or calibrate the original image in the process of generating the composite image. Can be used.

On the other hand, as another example, the correlated position information may be obtained and provided with a depth image taken from a depth camera. In this case, the information processing apparatus of the multi-camera can calculate the distance data from the camera to the object on the basis of the depth image, and the calculated distance data can be used for image alignment and geometric transformation processing.

7 is a diagram illustrating a procedure of a method of providing a composite image using a multi-camera according to another embodiment of the present disclosure.

A composite image providing method using a multi-camera can be performed by the information processing apparatus (particularly, the image processing unit) of the above-described multi-camera.

The information processing apparatus of the multi-camera receives an original image provided from a plurality of cameras (S71), and receives position information corresponding to each of the original images (S72).

On the other hand, (see 410, FIG. 4) composite image the composite pixel (X _C) is a first a first original pixel (X _A) a geometric variation between brightness (photometric), and in having a source image (401) included in The first weight W _A and the second original pixel X _B may be configured by combining a second weight W _B considering a photometric and a geometric variation relationship between the first weight W _A and the second original pixel X _B. At this time, the first weight W _A and the second weight W _B can be calculated in advance from the image synthesis algorithm, and the pre-calculated data can be managed in a look-up table format. Further, the composite image can be configured by combining a plurality of original images. Depending on how many original images are combined to form a composite image, the weights reflected on the original image can be set differently. Accordingly, a weight applied to the original image may be set in the lookup table according to the number of original images composing the composite image.

In consideration of the above points, the information processing apparatus of the multi-camera confirms the number of original images constituting the synthesized image (S73). The information processing apparatus of the multi-camera can confirm the weight information applied to each of the original pixels included in the original image by checking the weight table set in consideration of the number of original images constituting the combined image (S74) .

Next, the information processing apparatus of the multi-camera applies the weight information to each of the original pixels included in the original image to set the value of the combined pixel included in one region of the combined image (S75).

The steps S71 through S75 described above are repeatedly performed until the input of all the original images constituting the composite image is completed, and the composite image is outputted as the values of the composite pixels included in the composite image are all set (S76).

8 is a block diagram illustrating a computing system that implements an apparatus and method for processing information of a multi-camera and a method and apparatus for providing a composite image using a multi-camera according to an embodiment of the present disclosure.

8, a computing system 1000 includes at least one processor 1100, a memory 1300, a user interface input device 1400, a user interface output device 1500, (1600), and a network interface (1700).

The processor 1100 may be a central processing unit (CPU) or a memory device 1300 and / or a semiconductor device that performs processing for instructions stored in the storage 1600. Memory 1300 and storage 1600 may include various types of volatile or non-volatile storage media. For example, the memory 1300 may include a ROM (Read Only Memory) and a RAM (Random Access Memory).

Thus, the steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by processor 1100, or in a combination of the two. The software module may reside in a storage medium (i.e., memory 1300 and / or storage 1600) such as a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, You may. An exemplary storage medium is coupled to the processor 1100, which can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral to the processor 1100. [ The processor and the storage medium may reside within an application specific integrated circuit (ASIC). The ASIC may reside within the user terminal. Alternatively, the processor and the storage medium may reside as discrete components in a user terminal.

Although the exemplary methods of this disclosure are represented by a series of acts for clarity of explanation, they are not intended to limit the order in which the steps are performed, and if necessary, each step may be performed simultaneously or in a different order. In order to implement the method according to the present disclosure, the illustrative steps may additionally include other steps, include the remaining steps except for some steps, or may include additional steps other than some steps.

The various embodiments of the disclosure are not intended to be all-inclusive and are intended to illustrate representative aspects of the disclosure, and the features described in the various embodiments may be applied independently or in a combination of two or more.

In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In the case of hardware implementation, one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays A general processor, a controller, a microcontroller, a microprocessor, and the like.

The scope of the present disclosure is to be accorded the broadest interpretation as understanding of the principles of the invention, as well as software or machine-executable instructions (e.g., operating system, applications, firmware, Instructions, and the like are stored and are non-transitory computer-readable medium executable on the device or computer.

Claims (16)

A plurality of cameras for capturing an image including at least one object,
A positional information verifying unit provided in each of the plurality of cameras for verifying positional information between the at least one object and the camera;
And an image processing unit that processes the image information acquired from the plurality of cameras using the positional information received from the positional information verification unit.
The method according to claim 1,
The location information verifying unit,
A device identifier for identifying the location information verifying unit,
And generates position information including positional data between the at least one object and the camera.
3. The method of claim 2,
Wherein the image processing unit comprises:
The device identifier and location information including the location data,
And converts the device identifier into an object identifier that identifies the at least one object.
The method according to claim 1,
The location information verifying unit,
And a depth camera for acquiring a depth image of the image.
5. The method of claim 4,
Wherein the image processing unit comprises:
And provides the depth image of the image as the position information.
The method includes capturing an image including at least one object using a plurality of cameras,
Identifying location information from each of the plurality of cameras to the at least one object;
And processing the image information acquired from the plurality of cameras using the position information.
The method according to claim 6,
The step of confirming the position information comprises:
A device identifier for identifying the location information verifying unit,
And generating position information including positional data between the at least one object and the camera.
8. The method of claim 7,
The process of processing the image information includes:
Identifying the device identifier and location information including the location data;
And converting the device identifier into an object identifier that identifies the at least one object.
The method according to claim 6,
The step of confirming the position information comprises:
And acquiring a depth image of the image from a depth camera provided at a position corresponding to each of the plurality of cameras.
10. The method of claim 9,
The process of processing the image information includes:
And providing a depth image of the image as the position information.
A method of providing a composite image in which a plurality of images are combined,
Receiving a plurality of images respectively photographed from a plurality of cameras;
Receiving a plurality of pieces of positional information provided from a plurality of position determination units provided at positions corresponding to the plurality of cameras;
A step of matching original pixels provided in each of the plurality of images with synthetic pixels included in the composite image in consideration of the relationship between the position information, the plurality of images, and the composite image corresponding to the plurality of cameras, and,
Determining a value of an original pixel in each of the image units and applying the value to the value of the combined pixel;
And generating a composite image in which values of the composite pixels are combined with each other.
12. The method of claim 11,
Wherein the step of matching the original pixel with the composite pixel comprises:
Determining a number of the plurality of images included in the composite image;
And checking the composite pixel to which the original pixel is applied and the importance of the composite pixel from a predetermined importance table in consideration of the number of the images.
13. The method of claim 12,
Wherein the step of applying the value of the original pixel to the value of the composite pixel comprises:
And applying a value of the original pixel reflecting the importance to the value of the original pixel and applying the value of the original pixel reflecting the importance to the value of the composite pixel.
An apparatus for providing a composite image in which a plurality of images are combined,
A plurality of cameras for capturing an image including at least one object,
A positional information verifying unit provided in each of the plurality of cameras for verifying positional information between the at least one object and the camera;
A source pixel included in each of the plurality of images is matched with a composite pixel included in the composite image in consideration of the relationship between the position information, the plurality of images, and the composite image corresponding to the plurality of cameras, And an image processor for generating a composite image by combining the values of the composite pixels with the values of the composite pixels by checking the values of the original pixels in the respective image units. .
15. The method of claim 14,
Wherein the image processing unit comprises:
Checking the number of the plurality of images included in the composite image,
Wherein the synthesized pixel to which the original pixel is applied and the importance level are checked from a predetermined importance table in consideration of the number of the images.
15. The method of claim 14,
Wherein the image processing unit comprises:
Wherein a value of the original pixel reflecting the importance of the original pixel and a value of the original pixel reflecting the importance are applied to the value of the composite pixel.

Images