JP2013029995A - Imaging system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To inconspicuously display image gaps in all connection sections when images taken by a plurality of cameras are connected and displayed in one round manner.SOLUTION: An image processor 16 generates a synthesized image by synthesizing image signals from the plurality of imaging units 1 to 4, and applies gradation processing to a signal level in an image. A signal level detector 14 detects the signal levels of the image signals from the plurality of imaging units in an overlapping area of the image signals. A gradation controller 15 sets a gradation processing throughput to the image processor 16 so that the signal level of each photographed image in the overlapping area becomes a prescribed level difference or lower.

Description

  The present invention relates to an imaging system that synthesizes and displays images from a plurality of imaging units.

  A technique is known in which a plurality of cameras (imaging devices) are installed on the front, rear, left, and right sides of an automobile, and a captured image from each camera is subjected to viewpoint conversion processing to generate and display a vehicle peripheral image. With respect to this technology, Patent Document 1 describes a configuration for generating a vehicle peripheral image that has no discontinuity due to a luminance difference and that is less uncomfortable for the driver with a small amount of calculation. In this document, a photometric priority area is determined from the imaging areas of the imaging apparatus, and the photometry important area is measured more preferentially than other areas to calculate the appropriate exposure of the imaging area. It is stated that the luminance of a plurality of images having overlapping imaging regions is almost the same, and the cuts caused by the luminance difference are hardly visible between the images.

JP 2007-31970 A

  In the conventional techniques including the above-mentioned document 1, among the images of adjacent cameras, by adjusting so that the luminance and white balance of the areas where the shooting ranges overlap are equal, This makes the cuts in the image inconspicuous. In that case, the brightness difference between the cameras is adjusted for an image of a region where the shooting ranges of two adjacent cameras overlap. When three or more cameras are installed and the images of each camera are connected and displayed in a round, the brightness adjustment described above must be performed in each connecting section. Adjustment deviation occurs at the connecting portion. That is, it is difficult to make the image cutout inconspicuous in all the connecting portions.

  An object of the present invention is to provide an imaging system that displays image breaks inconspicuously at all connecting portions even when images from a plurality of cameras are connected and displayed in a round.

  The present invention relates to an imaging system that combines a plurality of captured images to generate a composite image, and combines a plurality of image capturing units that capture a subject while overlapping the capturing regions and image signals from the plurality of image capturing units. An image processing unit that performs gradation processing on the signal level in the image, a signal level detection unit that detects the signal level of the image signal from the plurality of imaging units, and a signal level detected by the signal level detection unit And a gradation control unit that sets a gradation processing amount performed by the image processing unit, the signal level detection unit detects a signal level in an overlapping area among the captured images of each imaging unit, and the gradation control unit The gradation processing amount is set so that the signal level in the overlapped area of the captured images is equal to or less than a predetermined level difference.

  The imaging system of the present invention further includes an exposure control unit that controls the exposure condition of each imaging unit in accordance with the signal level detected by the signal level detection unit, and the exposure control unit determines the signal level in the overlapping region of each captured image. The exposure amount is adjusted so as to be less than a predetermined level difference.

  According to the present invention, even when images from a plurality of cameras are connected and displayed in a round, the cuts in the images can be displayed inconspicuously at all the connecting portions.

1 is a system configuration diagram showing an embodiment of an imaging system according to the present invention. The figure which shows the example of arrangement | positioning of an imaging part, and an imaging | photography range. The figure which shows the example of the synthetic | combination screen which connected the picked-up image from each imaging part of FIG. The figure which shows correction | amendment of the signal level by exposure control and gradation control. The figure which shows the 1st example of the correction method of an image signal (combination of exposure control and gradation control). The figure which shows the modification of FIG. The figure which shows the modification of FIG. The figure which shows the modification of FIG. The figure which shows the 2nd example of the correction method of an image signal (only gradation control). The figure which shows the modification of FIG. The figure which shows the modification of FIG. The system block diagram which shows the other Example of the imaging system by this invention. The figure which shows the internal structure of the duplication area | region determination part 18. FIG. The figure which shows the operation | movement which restrict | limits exposure correction amount and gradation correction amount.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a system configuration diagram showing an embodiment of an imaging system according to the present invention. Here, as an example, four cameras (imaging units) are installed and the captured images of the cameras are connected and displayed. The configuration of the imaging system 10 will be described.

  The imaging units 1 to 4 include a lens (not shown) for collecting incident light from a subject, an image sensor that photoelectrically converts the light collected by the lens and outputs it as an image signal, and an electronic shutter control for each image sensor. Part (not shown) and a gain control part (not shown). The imaging units 1 to 4 receive a control signal from an exposure control unit 12 (to be described later) and change the exposure conditions (such as shutter time).

  The image input unit 11 is an input interface that inputs image signals from the imaging units 1 to 4 and sends the image signals to a subsequent image signal processing unit. Here, gain control and filter processing for signal level adjustment are performed. Furthermore, an AD (Analog to Digital) conversion process, a synchronization adjustment process for adjusting the processing timing between the image sensor and the signal processing unit, and the like may be included.

  The exposure control unit 12 controls exposure conditions (aperture, electronic shutter of the image sensor, and gain control) for the imaging units 1 to 4 and a gain of the image input unit 11. At that time, in accordance with signal level information received from a signal level detection unit 14 to be described later, the exposure conditions are controlled so that the connection between the image capturing units 1 to 4 is not conspicuous.

  The overlapping area storage unit 13 is a memory that holds information (hereinafter referred to as overlapping area information) indicating an area overlapping with an adjacent captured image among the captured images of the imaging units 1 to 4, and is arranged in the arrangement of the imaging units 1 to 4. Pre-stored. Note that the overlap area information may be held on the program.

  The signal level detection unit 14 receives an image signal from the image input unit 11 and detects a signal level in the overlapping image region according to the overlapping region information held in the overlapping region storage unit 13. The signal level detected here is information indicating the luminance distribution state of the signal in the overlapping image region, such as the absolute value (luminance level), histogram distribution, maximum value / minimum value, and average value of the input pixel signal. The signal level detection unit 14 outputs information on the detected signal level to the exposure control unit 12 and a gradation control unit 15 described later.

  The gradation control unit 15 sets the processing amount of the gradation processing so that the cuts are not conspicuous in the connection portion of the images from the imaging units 1 to 4 according to the signal level information received from the signal level detection unit 14.

  The image processing unit 16 performs gradation processing for correcting the change in signal level so as to be moderate with respect to the input image in accordance with the gradation processing amount set by the gradation control unit 15. And the image from each imaging part 1-4 is connected and synthesize | combined. For example, in the case of a vehicle peripheral image, a vehicle peripheral image viewed from a virtual viewpoint above the host vehicle is generated by viewpoint conversion processing. As other image processing, noise removal, gamma correction, contour enhancement, filter processing, zoom processing, camera shake correction, image recognition, and the like are performed.

  The image output unit 17 outputs the image signal synthesized by the image processing unit 16 to an image display device (monitor, TV), an image recording device, or the like. At that time, a process of converting to the signal format of the output device is performed. For example, NTSC, PAL conversion, HDMI conversion, conversion to a network transmission system, and the like are performed. The image output unit 17 may be configured to include gain control for signal level adjustment, filter processing, compression processing by encoding, and the like.

  Although the imaging system 10 of the present example includes the imaging units 1 to 4, the imaging units 1 to 4 may be handled as external devices. Further, although the image display apparatus is handled as an external device, a configuration including the image display apparatus in the imaging system 10 may be possible.

  FIG. 2 is a diagram illustrating an example of the arrangement of the imaging unit and an imaging range. Here, an example in which four imaging units 1 to 4 are arranged to photograph the periphery of the automobile 9 is shown as viewed from above the automobile. The imaging unit 1 is disposed on the rear of the vehicle, the imaging unit 2 is disposed on the left side of the vehicle, the imaging unit 3 is disposed on the front of the vehicle, and the imaging unit 4 is disposed on the right side of the vehicle. The left and right end regions of the imaging range of each imaging unit overlap with the imaging range of the adjacent imaging unit, so that a peripheral image of the automobile is acquired without being lost.

  FIG. 3 is a diagram illustrating an example of a composite screen in which captured images from the imaging units illustrated in FIG. 2 are connected. A front (front) shooting range 93 obtained from the imaging unit 3 is arranged at the center of the screen, and a left side and right side shooting ranges 92 and 94 obtained from the imaging units 2 and 4 are arranged adjacent thereto. . Furthermore, a rear (back) imaging range 91 obtained from the imaging unit 1 is arranged on the outer side. These imaging ranges have overlapping left and right ends between adjacent imaging ranges, and the overlapping areas are indicated by diagonal lines. The position of this overlapping area is stored in the overlapping area storage unit 13 of FIG.

  In the overlapping area, connecting portions 95 to 98 that serve as joints of images are provided, and the image signals from the imaging portions 1 to 4 are switched and displayed. In the following description, the image displayed in the front shooting range 93 is the main image, the image displayed in the left and right shooting ranges 92 and 94 is the sub image, and the image displayed in the rear (back) shooting ranges 91 and 91 ′. It will also be called a rear image. In the case of the vehicle periphery image described below, the main image is information that is more important to the driver than the sub image, so that the image quality is displayed with higher priority. Also, the priority of the rear image is set next to the main image.

Here, the exposure control and gradation control performed in this embodiment will be described.
FIG. 4 is a diagram showing signal level correction by exposure control and gradation control. In exposure control, for example, the exposure time is changed, which corresponds to multiplying the image signal level (luminance level) in the correction section (P1 to P2) by a uniform correction coefficient k. As a result, the corrected signal level is translated in the vertical direction while maintaining the original signal level gradient. The correction section (P1 to P2) covers the entire screen obtained from the imaging unit to be corrected.

  On the other hand, in gradation control, calculation processing is performed to multiply the image signal level (luminance level) in the correction section (P1 to P2) by a correction coefficient k corresponding to the position. For example, when the correction coefficients k1 and k2 at the positions P1 and P2 are given, the correction coefficient k at the intermediate position P is obtained by the linear interpolation method shown in the equation (1) and smoothly changed.

k = (1−α) × k1 + α × k2
Where α = (P−P1) / (P2−P1) (1)
According to this, the gradient of the signal level in the correction section is gently changed. The interpolation formula is not limited to this, and the correction amount in the section may be appropriately weighted.

Next, a correction method to which the above exposure control and gradation control are applied will be described using a specific example.
FIG. 5 is a diagram illustrating a first example of a method for correcting an image signal. In the composite screen in which the captured images obtained from the imaging units 1 to 4 are connected, the average luminance level (average value in the vertical direction of the screen) at each horizontal position is shown. Photographed images (luminance levels) obtained from the respective imaging units 1 to 4 are denoted by reference numerals 101 to 104, and connecting portions are denoted by reference numerals 95 to 98.

  This example assumes a typical luminance distribution obtained by normal exposure and has the following characteristics. The main image 103 from the front imaging unit 3 has a bright subject at the center of the screen, and the luminance decreases in the left and right areas of the screen, but the entire screen is within the optimum exposure range. The rear image 101 from the rear imaging unit 1 is assumed to be illuminated by a headlight from a rear automobile, and has the same distribution as the main image 103 from the front imaging unit 3.

  On the other hand, the sub image 102 from the imaging unit 2 on the left side has a dark subject in the center of the screen and is bright in the left and right areas of the screen. This is because the rear image 101 adjacent to the left side (there is a car behind) becomes brighter than the side of the car. In such a scene, when exposure control is performed with a weight placed at the center of the sub-image 102, the center of the screen is optimally exposed, but the left and right areas of the screen tend to be overexposed. The same applies to the sub image 104 from the imaging unit 4 on the right side surface.

  As a result, in normal exposure, a difference in luminance level occurs in the connection portions 95 to 98 of the respective photographed images, and the cuts between the images are very conspicuous. In this embodiment, therefore, exposure control and gradation control are combined to make the image cutout inconspicuous.

  First, with regard to the connecting portions 96 and 97 of the main image 103 and the sub images 102 and 104, the main image 103 is prioritized and its luminance level is fixed, and the luminance levels of the sub images 102 and 104 adjacent thereto are matched. Correct with exposure control. That is, the level 3L in the vicinity of the connecting portion 96 on the left side of the main image 103 is detected, and the exposure of the imaging unit 2 on the left side surface is set so that the level 2R in the vicinity of the connecting portion 96 of the sub image 102 adjacent thereto is the same level. Implement control. As a result of this exposure control, the level of the sub-image 102 obtained from the imaging unit 2 is corrected (translated) as indicated by 102a. Similarly, exposure control is performed on the sub-image 104 obtained from the imaging unit 4 on the right side surface, and the level is corrected as indicated by 104a.

  By this exposure control, it is possible to make the image breaks between the main image 103 and the sub-images 102a and 104a adjacent to the main image 103 in the connecting portions 96 and 97 inconspicuous. However, the image breaks at the connecting portions 95 and 98 remain, and even if the above exposure control is repeated, the breaks at all the connecting portions cannot be resolved.

  Therefore, with respect to the connecting portions 95 and 98 of the rear image 101 and the corrected sub images 102a and 104a, the rear image 101 is prioritized and its luminance level is fixed, and the luminance levels of the sub images 102a and 104a adjacent thereto are fixed. Correct with gradation control to match. That is, the connecting unit 95 detects the level 1R in the vicinity of the connecting unit 95 of the rear image 101, and multiplies the correction coefficient k so that the level 2aL in the vicinity of the connecting unit 95 of the sub-image 102a adjacent thereto is the same level. In this way, gradation control is implemented. In this example, the section on which gradation processing is performed is a portion from P1 to P2, and is limited to an area of about ½ on the left side of the sub-image 102a. That is, in this correction, the level 2aN of the sub-image 102a at P2 is fixed, and the gradient of the portion reaching P1 is changed to connect to the level 1R of the rear image 101. As a result of this gradation control, the level of the sub-image 102 obtained from the imaging unit 2 is corrected as indicated by 102b. Similarly, gradation control is performed in the sections P5 to P6 for the sub-image 104 obtained from the imaging unit 4, and the level is corrected as indicated by 104b.

  By adding gradation control in this way, level adjustment at each connecting portion can be freely adjusted regardless of the level difference situation at other connecting portions, and all connecting portions 95, 96, 97, At 98, the image break can be made inconspicuous. The size of the section to which the gradation control is applied is not limited to this, and may be changed as appropriate. Hereinafter, various modifications will be described in the case of combining exposure control and gradation control.

  FIG. 6 shows a modification of FIG. 5 in which the gradation control application section is further narrowed. Here, a case is shown in which the present invention is applied to only about one third of the sub-images 102a and 104a (P1 to P2 ', P5' to P6). According to the gradation control, a smooth image is obtained, but the gradation difference (resolution) is lowered. Therefore, the application section may be determined in consideration of the balance between the two.

  FIG. 7 shows a modification of FIG. 5, in which the luminance level of the main image 103 is corrected by exposure control. That is, in the left connecting portion 96, both the level 3L of the main image 103 and the level 2R of the sub-image 102 are corrected by exposure control so that the same level 3aL is obtained. The level of the corrected main image 103 is 103a, and the level of the sub image 102 is 102a. Further, gradation control is applied to the sections P1 to P2 of the sub image 102a so as to be connected to the rear image 101, and the level is corrected as indicated by 102b. The same applies to the right connecting portion 97. In the example of FIG. 7, by correcting the luminance level of the main image 103, it is possible to prevent the luminance levels of the sub images 102 and 104 from deviating from the optimum exposure range and improve the exposure state of the entire composite image.

  FIG. 8 shows a modification of FIG. 7 in which exposure control is performed on the main image and only gradation control is performed on the sub image. That is, after the luminance level of the main image 103 is corrected to 103a by exposure control, the left sub-image 102 is divided into sections P1 to P2 and sections P2 to P3, and gradation control is performed. As a result, while being fixed at the level 2N at the intermediate position P2, the section P1 to P2 is corrected to the level 102b, and the section P2 to P3 is corrected to the level 102c. The same applies to the right sub-image 104. The intermediate position P2 in this case can also be set to an arbitrary position.

  In the first embodiment, the method of combining the exposure control and the gradation control in order to make the discontinuity between adjacent images inconspicuous has been described. On the other hand, in the second embodiment, the image signal (luminance level) is corrected only by gradation control.

  FIG. 9 is a diagram illustrating a second example of the image signal correction method. First, the luminance level is fixed for the priority main image 103. Then, gradation control is performed on the sub-images 102 and 104 adjacent thereto. Specifically, the left sub-image 102 is divided into sections P1 to P2 and sections P2 to P3. Then, while being fixed at the level 2N at the intermediate position P2, the section P1 to P2 is corrected to the level 102b, and the section P2 to P3 is corrected to the level 102c. As a result, the luminance level in the connecting unit 95 can be adjusted to the level 1R of the rear image 101, and the luminance level in the connecting unit 95 can be adjusted to the level 3L of the main image 103. The right sub-image 104 is similarly corrected.

  In this embodiment, the sub-images 102 and 104 are connected to the luminance level of the main image 103 only by gradation control without performing exposure control. Normally, the main image 103 is photographed in the optimum exposure state, so that the sub-image region adjacent to the main image 103 is less likely to deviate from the optimum exposure due to correction. In addition, since gradation control is applied to all sections of the sub-image, the uncomfortable feeling of the image due to the correction does not concentrate on the specific area.

  FIG. 10 shows a modification of FIG. 9, which is a case where the application section of gradation control is limited. Here, a case is shown in which the present invention is applied only to a section close to the connecting portion on both sides except for the central region of the sub-images 102 and 104. Specifically, the sub-image 102 is limited to approximately ３ regions P1 to P2 ′ close to the connecting portion 95 and approximately 略 regions P2 ″ to P3 close to the connecting portion 96. As a result, the luminance level of the sub-image 102 is corrected to 102b and 102c, and remains unchanged in the sections P2 ′ to P2 ″. The same applies to the right sub-image 104.

  FIG. 11 shows a modification of FIG. 9 and shows a case where gradation control is performed without dividing a sub-image section. That is, this is corrected as one section for the ranges P1 to P3 of the sub-image 102. As a result, the luminance level of the sub-image 102 is corrected to 102b, but the tendency of the original luminance distribution is maintained, so that the uncomfortable feeling is reduced. The same applies to the right sub-image 104.

  In the first and second embodiments, the same processing (selection of exposure control and gradation control) is performed on the left and right sub-images 102 and 104 with the main image 103 as the center. Different processing may be selected on the left and right. In addition, the priority image is fixed to the front (front) main image 103. For example, in the case of a vehicle peripheral image, the priority image according to the vehicle running state is set to the left side surface, the right side surface, or the back (rear) image. By switching, the image display is easier to see for the driver.

  FIG. 12 is a system configuration diagram showing another embodiment of the imaging system according to the present invention. In the present embodiment, an overlapping area determination unit 18 is added to the configuration of the first embodiment (FIG. 1). The overlapping area determination unit 18 has a function of recognizing the shooting overlapping area of each imaging unit from the image signal. Then, either the overlapping area information separately stored in the overlapping area storage unit 13 or the overlapping area information obtained by recognition from the image signal is selected and output to the signal level detection unit 14. This configuration is effective when the overlapping area information between the imaging units is not held in advance in the overlapping area storage unit 13. Further, the exposure control unit 12 and the gradation control unit 15 have a function of limiting the correction amount in order to suppress image quality deterioration due to correction.

  FIG. 13 is a diagram illustrating an internal configuration of the overlapping area determination unit 18. The image recognizing unit 21 extracts an overlapping portion of the captured image from the input image signal of each imaging unit. As a recognition method at that time, a correlation such as edge information, color information, and luminance information of an image can be used. Next, the overlapping area information generation unit 22 generates area information of the extracted overlapping image portion. The selection unit 23 selects either the region information stored in the overlapping region storage unit 13 in advance or the region information generated by the overlapping region information generation unit 22 through image recognition. The selection by the selection unit 23 may be performed by the user, or may be automatically switched to the overlap region information generation unit 22 side when there is no overlap region storage unit 13. The selected area information is sent to the signal level detection unit 14 to detect the signal level of the image signal in the area.

  FIG. 14 is a diagram illustrating an operation for limiting the exposure correction amount and the gradation correction amount. The exposure control unit 12 and the gradation control unit 15 determine the maximum value of the correction amount in order to suppress image quality deterioration due to the correction, and perform control so that the correction amount does not exceed the maximum value. For example, when exposure control is performed by the connecting unit 96 and the luminance level of the image 102 is matched with the luminance level of the image 103, it is necessary to shift from the level Q1 to the level Q3, but the correction amount exceeds the maximum value kmax. . In this case, the correction amount is suppressed to the maximum value kmax, the shift to level Q2 is performed, and the corrected image is 102a. Due to this limitation, the exposure is not over-corrected and image quality deterioration is prevented.

  Further, when gradation control is performed by the connecting unit 95 and the luminance level of the image 102a is adjusted to the luminance level of the image 101, it is necessary to shift from the level Q4 to the level Q6, but the correction amount exceeds the maximum value kmax. . In this case, the correction amount is suppressed to the maximum value kmax, the shift to level Q5 is performed, and the corrected image is set to 102b. By this limitation, gradation overcorrection is suppressed and image quality deterioration is prevented.

  As mentioned above, although several Example was shown about the imaging system by this invention, and its operation | movement, each Example is effective even if it combines individually or suitably. According to these embodiments, when images from a plurality of cameras are connected and displayed in a round, there is an effect that the cuts of the images can be displayed inconspicuously at all the connecting portions.

1-4 ... Imaging unit (camera),
9 ... car,
10: Imaging system,
11: Image input unit,
12 ... Exposure control unit,
13: Overlapping area storage unit,
14 ... Signal level detection unit,
15 ... gradation control unit,
16: Image processing unit,
17 ... image output unit,
18 ... Overlapping region determination unit,
21 ... Image recognition unit,
22 ... detection area information generation unit,
23 ... selection part,
91-94 ... Shooting area,
95-98 ... connection part,
101-104: Images (luminance levels) from the imaging unit.

Claims (8)

In an imaging system that generates a composite image by connecting a plurality of captured images,
A plurality of imaging units that shoot subjects while overlapping the shooting areas;
An image processing unit that combines image signals from the plurality of imaging units to generate a composite image, and performs gradation processing on a signal level in the image;
A signal level detector that detects signal levels of image signals from the plurality of imaging units;
A gradation control unit that sets a gradation processing amount performed by the image processing unit in accordance with the signal level detected by the signal level detection unit,
The signal level detection unit detects a signal level in a region overlapping each other among the captured images of the imaging units,
The imaging system, wherein the gradation control unit sets a gradation processing amount so that a signal level in an overlapping region of the captured images is equal to or less than a predetermined level difference. The imaging system according to claim 1,
An exposure control unit that controls an exposure condition of each imaging unit according to the signal level detected by the signal level detection unit;
The exposure control unit adjusts an exposure amount so that a signal level in an overlapping region of the captured images is equal to or less than a predetermined level difference. The imaging system according to claim 1 or 2,
An overlapping area storage unit that holds information of areas where the captured images of the respective imaging units overlap each other;
The imaging system according to claim 1, wherein the signal level detection unit detects a signal level in accordance with overlapping area information held by the overlapping area storage unit. The imaging system according to claim 1 or 2,
An overlapping area determination unit that generates overlapping area information by extracting overlapping areas from the captured images of each imaging unit,
The imaging system according to claim 1, wherein the signal level detection unit detects a signal level in accordance with the overlapping region information generated by the overlapping region determination unit. The imaging system according to claim 1 or 2,
When a priority order is set for the captured image from the imaging unit, an image with a high priority is a main image, and an image with a low priority is a sub-image,
The imaging system according to claim 1, wherein the gradation control unit controls the sub-image to perform gradation processing. The imaging system according to claim 5, wherein
The imaging system according to claim 1, wherein the gradation control unit performs control so that gradation processing is performed on a partial region of the sub-image that is not adjacent to the main image. The imaging system according to claim 1 or 2,
The gradation control unit limits an amount of processing so that a correction amount of a signal level does not exceed a predetermined amount by gradation processing. The imaging system according to claim 2,
The imaging system according to claim 1, wherein the exposure control unit limits the adjustment amount so that the signal level correction amount does not exceed a predetermined amount by adjusting the exposure amount.

Images