JP2023130999A - Imaging system, imaging system control method, and program - Google Patents

Abstract

An object of the present invention is to provide an in-vehicle camera system that can prevent object information from becoming unrecognizable due to overexposure when a flicker reduction function is enabled. SOLUTION: Imaging means has a first imaging function that reduces flicker and a second imaging function that expands a dynamic range, and switches between the first imaging function and the second imaging function; a range calculation means for calculating a brightness distribution range of an image output from the means, and determining which of the first imaging function and the second imaging function to use based on information about the calculated brightness distribution range; An imaging system comprising: imaging system control means for transmitting a control signal for switching between the first imaging function and the second imaging function to the imaging function switching means. [Selection diagram] Figure 1

Description

The present invention relates to an imaging system that appropriately controls a flicker reduction function and a dynamic range extension function.

Some recent vehicle-mounted camera systems generate an image with an expanded dynamic range (HDR: High Dynamic Range) by combining images acquired under a plurality of different exposure conditions. Specifically, one composite image is generated by combining the low-brightness pixels of an image acquired with a long exposure time and the high-brightness pixels of an image acquired with a short exposure time. By performing such processing, it is possible to obtain an image with a high dynamic range that maintains the gradation of dark areas and can express bright subjects without blown out highlights.

Incidentally, in recent years, digital traffic lights, road traffic signs, and car tail lamps that use LEDs have become available.

However, since LEDs periodically turn on and off, even if it appears to the naked eye that the LED is on, the image signal captured by the LED shows that the LED is off, resulting in LED flicker. This phenomenon occurs. The appearance of LED flicker differs depending on the shutter method of the imaging device. In the case of the global shutter method, the brightness of all pixels in the LED area in the image signal changes simultaneously from frame to frame, and depending on the timing of the exposure period, some frames may be completely turned off. When the shutter method is a rolling shutter method, the brightness of the LED area portion in the image signal varies line by line, and depending on the timing of the exposure period, some lines may be completely turned off. This LED flicker phenomenon occurs because the exposure period of the imaging device overlaps with the period when the LED is off, and the exposure is only during the period when the LED is off.The shorter the exposure time, the more the LED is completely turned off. The phenomenon where the light goes out is more likely to occur. That is, when an imaging device is used for vehicle-mounted applications, if the exposure time is too short, a situation will occur in which information such as traffic lights and road traffic signs cannot be acquired. In recent years, with the rapid spread of self-driving cars, if information such as traffic lights and road traffic signs becomes difficult to recognize due to LED flicker during self-driving, there is a risk that self-driving cars will misestimate their own position. .

In Patent Document 1, the amplitude component of flicker is detected from image signals taken with multiple types of exposure times, and the blending ratio during image composition is changed based on the detection result, thereby reducing the influence of flicker while achieving a wide dynamic range. Techniques for generating range images have been proposed.

In addition, Patent Document 2 discloses a technology that reduces flicker by performing cycle photography using long exposure and short exposure, and detecting and correcting flicker by looking at the difference between frames for each image under each exposure condition. Proposed.

By the way, some recent vehicle-mounted sensors are equipped with a flicker reduction (LED flicker mitigation) function. An example of a flicker reduction function is a function that prevents the LED from completely disappearing by limiting the exposure time to a time longer than the blinking cycle of the LED. Generally, the exposure time is limited to 11 ms or more, assuming the longest blinking cycle of the LED, 90 Hz. However, since it is not necessarily necessary to assume 90 Hz, the limiting condition of the exposure time can be changed as appropriate according to the blinking cycle of the LED.

JP2016-146592 JP2014-179924

Depending on the flicker and the timing of the exposure period, the LED lighting state may not be detected, and it may take a long time to detect the flicker, or it may not even be possible to detect the flicker component. Furthermore, if the flicker reduction function limits the exposure time to a predetermined time or more, the effect of expanding the dynamic range will be reduced, and the range of brightness that can express the subject will be limited. As a result, for example, if a self-driving car constantly activates the flicker reduction function while driving autonomously, it would be difficult to use the flicker reduction function in environments where a wide dynamic range is required, such as when switching from a dark place to a bright place, such as at the exit of a tunnel. In some cases, overexposure occurs. This overexposure may make it impossible to recognize not only traffic lights and road traffic signs, but also all subject information.

The present invention has been made in view of the above-mentioned problems, and provides an imaging system that can prevent object information from becoming unrecognizable due to insufficient dynamic range when using a flicker reduction function. The purpose is to provide

In order to achieve the above object, the invention according to claim 1 of the present application:
an imaging means for switching between the first imaging function and the second imaging function, the imaging means having a first imaging function for reducing flicker and a second imaging function for expanding a dynamic range; a range calculation means for calculating a brightness distribution range of the image;
Based on information on the calculated luminance distribution range, it is determined which of the first imaging function and the second imaging function is to be used, and a control signal for switching between the first imaging function and the second imaging function is generated. The imaging system is characterized by comprising: imaging system control means for transmitting data to the imaging function switching means.

According to the present invention, it is possible to prevent object information from becoming unrecognizable due to insufficient dynamic range when flicker reduction is enabled.

1 is a configuration diagram of an in-vehicle camera system 100 according to a first embodiment. FIG. 2 is an explanatory diagram showing an exposure control method according to the first embodiment. FIG. 3 is an explanatory diagram showing the LED lighting state and exposure period of the first embodiment. 5 is a diagram showing how an LED looks when performing exposure control according to the first embodiment. FIG. FIG. 2 is a conceptual diagram for explaining a method of generating an HDR image according to the first embodiment. 7 is a flowchart when switching from the flicker reduction function to the dynamic range extension function of the first embodiment. This is an output image from the HDR image generation unit 109 when photographing the tunnel exit direction in the first embodiment. 7 is a calculation result of the brightness distribution range in the output image of FIG. 7 of the first embodiment. FIG. 6 is an explanatory diagram regarding a method of changing the exposure time when switching from the flicker reduction function to the dynamic range extension function of the first embodiment. 7 is a flowchart when switching from the dynamic range extension function to the flicker reduction function in the first embodiment. 1 is a diagram showing a hardware configuration of an in-vehicle camera system 100 according to a first embodiment.

<First embodiment>
Embodiment 1 of the present invention will be described below.

FIG. 1 is a configuration diagram of an in-vehicle camera system 100 according to a first embodiment. The in-vehicle camera system 100 includes two systems: an imaging system 101 and an in-vehicle system 102.

First, the imaging system 101 will be explained. The imaging system 101 includes an imaging system control section 103, an imaging section 104, and an imaging system image processing section 108. Further, the imaging unit 104 includes a flicker reduction unit 105, a dynamic range expansion unit 106, and an imaging function switching unit 107, and the imaging system image processing unit 108 includes at least an HDR (high dynamic range) image generation unit 109 and a range calculation unit 110.

The imaging system control unit 103 controls the entire imaging system 101. Specifically, it controls the imaging unit 104, the imaging function switching unit 107, the imaging system image processing unit 108, and the HDR image generation unit 109, and acquires information from the range calculation unit 110 and the in-vehicle system 102.

The imaging unit 104 has a function of converting an optical image of a subject formed by a lens (not shown) into an analog electrical signal using a photoelectric conversion element (photodiode), and a function of converting the analog electrical signal into a digital signal. have The imaging unit 104 also has a function of controlling the exposure time according to an exposure control signal input from the imaging system control unit 103.

FIG. 11 is a diagram showing the hardware configuration of the in-vehicle camera system 100. H11 is a CPU that controls various devices connected to the system bus H21. H12 is a ROM that stores BIOS programs and boot programs. H13 is a RAM, which is used as the main storage device of H11, which is the CPU. H14 is an external memory that stores programs processed by the in-vehicle camera system 100. The input unit H15 performs processing related to inputting information such as a keyboard and a mouse. The display unit H16 outputs the calculation results of the vehicle-mounted camera system 100 to the display device according to the instructions from H11. Note that the display device may be of any type, such as a liquid crystal display device, a projector, or an LED indicator. H17 is a communication interface that performs information communication via a network, and the communication interface may be Ethernet, and any type such as USB, serial communication, wireless communication, etc. is not restricted. H18 is an I/O, which inputs an image from the imaging unit 104.

Here, the exposure control method in this embodiment will be explained using FIG. 2.

In FIG. 2, VD (Vertical Driving pulse) is a signal indicating the start timing of one frame. A VD control signal is transmitted from the imaging system control unit 103 to the imaging unit 104, and when the VD signal rises, controls such as reset and readout for acquiring one frame worth of images are started.

The Reset signal is a control signal for setting the potential of the photoelectric conversion element to a reset (reference) potential by connecting the photoelectric conversion element to a reset power supply, and the reset control is executed while the Reset signal is High.

The Read signal is a control signal for reading charges generated in the photoelectric conversion element, and charge readout control is started at the timing when the Read signal rises.

The exposure time is a period from the falling timing of Reset to the rising edge of Read, and indicates the period during which the photoelectric conversion element is irradiated with light and charges are accumulated. The amount of charge generated within the photoelectric conversion element changes depending on the length of the exposure time. In other words, the output brightness level with respect to incident light changes depending on the length of the exposure time. In this embodiment, the first frame is photographed with a short exposure time, and the second frame is photographed with a long exposure time. From the third frame onward, exposure control is performed in which short-second exposure and long-second exposure are alternately repeated every other frame. Although this embodiment will be described using two types of exposure conditions, long-second exposure and short-second exposure, three or more types of exposure conditions may be combined. By increasing the exposure conditions and performing image synthesis, performance can be expected to improve due to dynamic range expansion processing.

The period shown in the sensor output in FIG. 2 indicates a period during which the imaging unit 104 converts the electrical signal read from the photoelectric conversion element into a digital luminance signal according to Read control, and further outputs the signal to the HDR image generation unit 109.

The above is the exposure control method in this embodiment.

The imaging unit 104 includes a flicker reduction unit 105, a dynamic range extension unit 106 having a dynamic range extension function, and an imaging function switching unit 107 for switching these functions. Here, the flicker reduction function and dynamic range extension function will be explained using FIGS. 3 and 4.

First, the flicker reduction function will be explained. FIG. 3 is a graph showing the LED lighting state and exposure period over time. Further, FIG. 4 is a diagram showing the appearance of an LED when the exposure control shown in FIG. 3 is performed, using a traffic light as an example.

In FIG. 3, the LED lighting state indicates whether the LED is on or off. In this embodiment, a High period indicates a lighting state, and a Low period indicates a non-lighting state. Furthermore, the present embodiment will be described on the assumption that the blinking cycle of the LED is 11 ms.

The VD and exposure period are the same as those shown in FIG. 2 described above, so their explanation will be omitted.

In FIG. 3, the LED lighting state and the exposure period are completely asynchronous. In this embodiment, t1 to t2 indicates a period when the LED is on, t2 to t3 indicates a period when the LED is off, and t4 to t5 indicates an exposure period of the imaging unit 104.

In the case of FIG. 3A, since the LED lighting state is "on" during the period from t4 to t5, charge is normally accumulated in the imaging unit 104. In other words, when the imaging system 101 photographs a traffic light at the control timing shown in FIG. 3(a), an image with the LED of the traffic light turned on is output, similar to what a person would see when looking at the traffic light with the naked eye. . At that time, the traffic light appears as if the LED is lit as shown in FIG. 4(a). In FIG. 4, the left LED is lit assuming a green signal, but the same applies to signals of other colors.

In the case of FIG. 3B, since the LED lighting state is "off" during the period from t4 to t5, no charge is accumulated in the imaging unit 104. In other words, when the imaging system 101 photographs a traffic light at the control timing shown in FIG. 3(b), an image with the LED of the traffic light turned off is output, which is different from how a person would see the traffic light with the naked eye. . At that time, the traffic light appears as if the LED of the traffic light is off, as shown in FIG. 4(b). Figure 4(b) assumes a global shutter type in-vehicle sensor, but in the case of a rolling shutter type, the image may be such that a specific line of the LED of a traffic light is partially turned off, as shown in Figure 4(c). be.

LED flicker is a phenomenon in which the LED area of an image acquired by the imaging system 101 is turned off as shown in FIGS. The shorter the time, the higher the probability of occurrence. A flicker reduction function is a function that suppresses the occurrence of LED flicker.

In order to make the flicker reduction function effective, it is necessary to set restrictions on exposure conditions. Specifically, the exposure time when acquiring a short exposure image is limited to at least 11 ms. This is because the blinking period of the LED is long, about 90 Hz, so by setting the minimum exposure time to 11 ms or more, the lighting period of the LED necessarily overlaps with the exposure period of the imaging unit 104. The LED lighting state and exposure period when the flicker reduction function is enabled are as shown in FIG. 3(c). By setting the period from t4 to t5 to be 11 ms or more, it is possible to always perform exposure with the LED turned on, no matter how much the exposure timing shifts. In this embodiment, the blinking cycle of the LED is set to 90 Hz, but if the blinking cycle of the LED is longer, the minimum exposure time may be set even longer. By activating the flicker reduction function, it is possible to prevent information on LED-based traffic lights, road traffic signs, and the like from being completely erased in images acquired from in-vehicle sensors.

Next, the dynamic range extension function will be explained. The dynamic range extension function is a function for extending the dynamic range further than in the enabled state of the flicker reduction function without imposing restrictions on the exposure time due to the flicker reduction function. Since the dynamic range is expanded even when the flicker reduction function is enabled, this embodiment will be described using the expression "dynamic range expansion." When using the dynamic range extension function, the flicker reduction function will be canceled. In this embodiment, the minimum exposure time is not limited by the flicker reduction function, and the exposure time when acquiring a short exposure image is set to be shorter than 11 ms. By doing so, it is possible to express a subject with a wider dynamic range without overexposure than when the imaging unit 104 performs normal dynamic range expansion processing with the flicker reduction function enabled. Become. The states shown in FIGS. 3(a) and 3(b) described above correspond to the exposure control method when the dynamic range extension function is enabled.

The above is the explanation about the flicker reduction function and the dynamic range extension function.

The imaging function switching unit 107 is for switching between the flicker reduction unit 105 and the dynamic range expansion unit 106 based on a control signal from the imaging system control unit 103. The concept of a specific control method for in-vehicle use in this embodiment will be explained. To prevent complete loss of information such as traffic lights and road traffic signs when a self-driving car drives itself, the flicker reduction function is usually enabled. However, if it is determined that the negative effect of the loss of object information due to insufficient dynamic range outweighs the loss of object information due to LED flicker, the flicker reduction function is temporarily canceled and switched to the dynamic range expansion function. Images acquired from the imaging unit 104 using this control method are output to the imaging system image processing unit 108.

The imaging system image processing section 108 includes at least an HDR image generation section 109 and a range calculation section 110. Further, other processing units may be provided as necessary, such as a gradation compression processing unit for reducing the amount of data of an output image.

The HDR image generation unit 109 is for performing processing to generate an HDR (high dynamic range) image using two image signals acquired from the imaging unit 104 under two different exposure conditions. A method for generating an HDR image in this embodiment will be explained using FIG. 5.

FIG. 5(a) shows an outline of the configuration of the HDR image generation unit 109. The HDR image generation section 109 includes a long exposure image memory 501, a short exposure image memory 502, a brightness correction section 503, and an image composition section 504.

Among the image signals output from the imaging unit 104, images acquired by long exposure are stored in the long exposure image memory 501. Further, images acquired by short exposure are stored in short exposure image memory 502. The brightness correction unit 503 is for adjusting the brightness level of an image acquired by short exposure to a brightness level equivalent to an image acquired by long exposure. By using this means to match the luminance values of the long exposure image and the short exposure image, the two images can be combined without any discomfort.

The processing in the brightness correction unit 503 will be specifically explained using FIG. 5(b). In FIG. 5B, the horizontal axis shows the light amount of the subject, and the vertical axis shows the output brightness of the imaging unit 104.

5A is a graph showing the output brightness relative to the light amount of the object during long exposure, 5B is a graph showing the output brightness relative to the light amount of the object during short exposure, and 5B' is the brightness calculated by the brightness correction unit 503 for the output brightness of 5B. A graph after the correction process is performed is shown. Further, Sat written on the vertical axis indicates the saturation level of the photoelectric conversion element.

Since 5A has a long exposure time, it quickly reaches the Sat level when the amount of light on the subject increases. Since 5B has a short exposure time, it reaches the Sat level after the amount of light on the subject becomes greater than in 5A. The brightness correction unit 503 multiplies the brightness output of 5B by Gain to match the brightness outputs of 5A and 5B. Specifically, the gain is calculated by dividing the exposure time during long exposure by the exposure time during short exposure, and the brightness output of the short exposure image is directly multiplied by the Gain. By doing so, the brightness output of the image acquired by long exposure and the brightness output of the image acquired by short exposure become the same brightness value.

The image synthesis unit 504 synthesizes the luminance output of 5A and the luminance output of 5B' to generate one dynamic range enlarged image in which the long exposure image and the short exposure image are synthesized. A method of generating a composite image using the image composition unit 504 in this embodiment will be explained using FIG. 5(c). In FIG. 5(c), the horizontal axis represents the input luminance, and the vertical axis represents the combination ratio of the long exposure image and the short exposure image. In addition, the brightness from α to β on the horizontal axis is the brightness near the switching point between the long exposure image and the short exposure image, and ideally the brightness of β is set to the brightness near Sat in Figure 5(b). It is preferable to do so.

If the input brightness value is equal to or less than α, the brightness value 5A of the long exposure image is used as 100%. When the brightness value is α to β, the brightness value 5A of the long exposure image and the brightness value 5B' of the short exposure image are each multiplied by the synthesis ratio shown in the graph of FIG. Add up the results. If the brightness value is equal to or greater than β, the brightness value 5B' of the short exposure image is used as 100%. By performing this processing, it is possible to combine an image obtained by long exposure and an image obtained by short exposure to generate a single HDR composite image with an expanded dynamic range. The HDR composite image generated by the HDR image generation unit 109 is output to the range calculation unit 110 and the in-vehicle system image processing unit 112 in the in-vehicle system 102 .

The range calculation unit 110 calculates the brightness distribution range of the HDR composite image generated by the HDR image generation unit 109. Specifically, brightness distribution information is generated by counting the frequency of occurrence of each brightness value in the same manner as when generating a histogram. Then, the difference between the maximum brightness value and the minimum brightness value in the generated brightness distribution graph is calculated as the brightness distribution range. A detailed explanation of the brightness distribution range in this embodiment will be given later.

The in-vehicle system 102 includes an in-vehicle system control section 111, an in-vehicle system image processing section 112, and a traveling speed acquisition section 113.

The in-vehicle system control unit 111 controls the in-vehicle system 102 as a whole. Specifically, it acquires image information from the in-vehicle system image processing unit 112, acquires the running speed of the own vehicle from the running speed acquisition unit 113, performs determination processing using the information, and performs image capture as necessary. The determination result is transmitted to the system control unit 103 as information.

The in-vehicle system image processing section 112 has at least an image receiving function. The image receiving function is a function for receiving an image signal output from the imaging system 101. The received image signals are used for recognition purposes for autonomous driving and visual recognition purposes for the driver to understand the situation outside the vehicle. The in-vehicle system image processing unit 112 performs necessary image processing depending on the purpose and application of the image. Furthermore, when the image processing unit 108 of the imaging system performs gradation compression processing, it may also perform decompression processing, or perform other processing such as white balance processing and development processing as necessary.

The traveling speed acquisition unit 113 is for acquiring traveling speed information of the own vehicle. Specifically, the running speed is calculated based on the rotational speed of the tires and is to be displayed on a speed display unit (not shown).

The outline of the configuration of the in-vehicle camera system 100 in this embodiment has been described above. This configuration is only a configuration for realizing the method of switching between the flicker reduction function and the dynamic range expansion function described in this embodiment, so as long as the configuration achieves the purpose, it is limited to this configuration. It's not something you can do.

A control flow when switching from a state in which the flicker reduction function is enabled to a dynamic range extension function in this embodiment will be explained using the flowchart of FIG. 6.

In S601, the imaging function switching unit 107 sets the function to be prioritized between the flicker reduction unit and the dynamic range expansion unit as an initial value. In this embodiment, automatic driving is assumed, and the flicker reduction function is given priority. Specifically, the imaging system control unit 103 enables the flicker reduction unit 105 via the imaging function switching unit 107 in the imaging unit 104. In this embodiment, the flicker reduction unit 105 is enabled, but if it is desired to prioritize the dynamic range extension function over the flicker reduction function, the dynamic range extension unit 106 may be enabled as an initial setting. When the process of S601 is completed, the process advances to S602.

In S602, the travel speed acquisition unit 113 is used to acquire travel speed information of the host vehicle. In this embodiment, as described above, the traveling speed information of the own vehicle is calculated from the information on the rotational speed of the tires. When the process of S602 is completed, the process advances to S603.

In S603, the in-vehicle system control unit 111 determines whether the traveling speed acquired in S602 is a predetermined speed or higher. Here, the reason for performing the determination process in S603 will be briefly explained. When the driving speed is slow, such as in traffic jams or when driving at low speeds, the danger of driving is low in the first place, and even if there is an environment that requires a wide dynamic range, the moving speed of the car is slow, so it is difficult to follow changes in brightness. will arrive in plenty of time. More than that, in environments such as urban areas where traffic jams and low-speed driving are likely to occur, the inability to obtain information on traffic lights, road traffic signs, etc. is more likely to impede driving. In view of the above-mentioned situation, in this embodiment, processing is performed to prioritize the flicker reduction function over the dynamic range expansion function. In S603, if the traveling speed is equal to or higher than the predetermined speed, the in-vehicle system control unit 111 transmits information that the traveling speed is equal to or higher than the predetermined value to the imaging system control unit 103, and the process advances to S604. If the traveling speed is less than the predetermined speed, the process returns to S602.

In S604, the range calculation unit 110 calculates a brightness distribution range for the image signal output from the HDR image generation unit 109. Specifically, first, the frequency of occurrence of each brightness value is counted in the same manner as when generating a general histogram, and a brightness distribution graph is generated from the count results.

Here, an example of the results of generating a brightness distribution graph will be described using FIGS. 7 and 8. In this embodiment, a description will be given assuming a tunnel exit as a specific location where a dynamic range of a predetermined value or more is required.

FIG. 7 shows an output image from the HDR image generation unit 109 when photographing the tunnel exit direction from inside the tunnel using the imaging system 101. In FIG. 7, the area 7A indicates the tunnel exit, and the area 7B indicates the inner wall of the tunnel and the road located on the near side of the tunnel exit, that is, inside the tunnel. In the image of FIG. 7, the area 7A has saturated brightness due to insufficient dynamic range, resulting in blown-out highlights, and the area 7B has low brightness and is dark.

FIG. 8 shows a brightness distribution graph in the image shown in FIG.

In FIG. 8, the horizontal axis indicates the brightness value, and the saturation brightness level when the LED Flicker Mitigation function is enabled is defined as Sat (LFM), which is taken as 100% (reference). . Further, the resolution of the brightness value is classified into brightness ranges in 10% increments, such as 0 to 9%, 10 to 19%, and 20 to 29%. In this embodiment, the luminance ranges are classified into 10 to reduce the image processing load, but the number of luminance range classifications may be larger, or the output luminance values may be used as they are to generate a graph. Also good.

The vertical axis indicates how many times pixels in each luminance range occur within one frame of an image, that is, the frequency of occurrence. Specifically, it shows the result of counting the frequency of occurrence for each brightness range based on the brightness value of the image signal output from the HDR image generation unit 109. The gray graph shows the frequency of occurrence of unsaturated pixels, and the shaded graph shows the frequency of occurrence of saturated pixels. Further, the original brightness values of the overexposed pixels shown in the hatched graph are shown in the dotted line graph for reference.

The output images from the HDR image generation unit 109 shown in FIGS. 7(a), 7(b), and 7(c) and the brightness shown in FIGS. 8(a), 8(b), and 8(c) The distribution graphs correspond to each other. In the above description, it is assumed that the frequency of occurrence of brightness values is counted on a pixel basis, but one frame of an image may be divided into regions and the average value for each region may be used. Since the amount of data to be processed can be reduced by using the average value of each divided area, the image processing load can be reduced.

FIG. 7(a) is an image when the tunnel exit is quite far away from the driving position of the own vehicle. In FIG. 7(a), since the tunnel exit is far away from the driving position of the vehicle, the number of pixels in the overexposed area shown by 7A is overwhelmingly larger than the number of pixels in the dark area shown by 7B. few. The brightness distribution graph at this time is as shown in FIG. 8(a). As shown in FIG. 8(a), when the tunnel exit is quite far away, the frequency of occurrence of low-luminance pixels is overwhelmingly high, and the frequency of occurrence of pixels at 100%, which is the saturation luminance level, is low.

FIG. 7(b) is an image taken when the tunnel exit is relatively close as seen from the driving position of the own vehicle. In FIG. 7(b), since the tunnel exit is getting closer to the driving position of the own vehicle, the dark area shown by 7B gradually decreases compared to FIG. 7(a), and instead the white area shown by 7A is The area gradually becomes wider. The brightness distribution graph at this time is as shown in FIG. 8(b). As shown in FIG. 8(b), as the tunnel exit approaches, the frequency of occurrence of low-luminance pixels decreases compared to FIG. 8(a), and the frequency of occurrence of pixels at 100%, which is the saturated luminance level, increases.

FIG. 7(c) is an image when the tunnel exit is nearby when viewed from the driving position of the own vehicle. In FIG. 7(c), since the tunnel exit is close to the driving position of the own vehicle, the dark area indicated by 7B is further reduced compared to FIG. 7(b), and instead the overexposed area indicated by 7A is further reduced. It becomes wider. The brightness distribution graph at this time is as shown in FIG. 8(c). As shown in FIG. 8C, when the tunnel exit is nearby, the frequency of occurrence of low-luminance pixels decreases greatly, and the frequency of occurrence of pixels at 100%, which is the saturated luminance level, increases greatly.

As described above, by generating a brightness distribution graph using the range calculation unit 110 for the image signal output from the HDR image generation unit 109, it is possible to generate a luminance distribution graph using the range calculation unit 110. You can know the brightness distribution of the subject.

Next, a brightness distribution range is calculated from the information on the brightness distribution graph. Specifically, the brightness values that occur more than a predetermined number of times in the brightness distribution graph are extracted, and the difference between the maximum brightness value and the minimum brightness value among the extracted brightness values is determined. In this embodiment, the luminance range in which the frequency of occurrence is equal to or higher than Th(8α) is used as a target for calculating the luminance distribution range. In FIG. 8(a), luminance values of 0 to 9%, 10 to 19%, 20 to 29%, and 30 to 39% are extracted as targets. In other words, since luminance values of 0 to 39% occur with high frequency, the luminance distribution range is calculated to be 39%. In FIGS. 8(b) and 8(c), the occurrence frequency is at least Th(8α) or higher in both the 0 to 9% and 100% luminance ranges. In other words, the brightness distribution range, which is the difference between the maximum brightness value and the minimum brightness value, is 100%. By calculating the brightness distribution range, it is possible to understand the extent of the dynamic range of the photographic subject. When the process of calculating the brightness distribution range is completed in S604, the process advances to S605.

In S605, it is determined whether the brightness distribution range calculated using the range calculation unit 110 is equal to or greater than a predetermined value. In this embodiment, whether the luminance distribution range is 90% or more is used as a determination threshold for determining whether the dynamic range is wide. In FIG. 8A, the brightness distribution range is 39%, which is less than the predetermined value, and it is determined that the dynamic range is not wide. On the other hand, in FIGS. 8(b) and 8(c), since the luminance distribution range is 100%, the luminance distribution range is greater than the predetermined value, and the dynamic range is determined to be wide. The purpose of performing this processing is not only to suppress overexposure of bright objects such as tunnel exits, but also to express the gradation of dark objects such as inside tunnels. If you only want to suppress overexposure of the subject, it can be achieved by controlling the gain or other mechanisms such as the aperture or ND filter if there are any, but this will impair the gradation and noise in dark subject areas. It gets lost. By performing the processing in this flow and determining whether to expand the dynamic range, the aim is to preserve the gradation of dark areas and express bright subjects without blown out highlights. Further, in this embodiment, the threshold value for determining the width of the dynamic range is set to 90%, but other settings may be used. If you want to actively switch to the dynamic range extension function, you may set the threshold value low, or if you want to suppress false detections, you may set the threshold value high. If it is determined in S605 that the brightness distribution range is equal to or greater than the predetermined value, the process proceeds to S606, and if it is determined that the brightness distribution range is less than the predetermined value, the process returns to S602 again. In other words, when the distance to the tunnel exit approaches within a certain distance as shown in FIGS. 8(b) and 8(c), the process proceeds to S606.

In S606, it is determined whether the frequency of occurrence of pixels at 100%, which is the saturated brightness level, in the brightness distribution range calculated using the range calculation unit 110 is equal to or higher than a predetermined value. The purpose of performing this processing is to determine the danger to driving due to loss of subject information due to overexposure. For example, if the tunnel exit is still far away, the loss of road traffic sign information in the tunnel due to deactivation of the flicker reduction function is more important than the risk of driving due to the occurrence of whiteout areas. There is a high possibility that it could be dangerous. I want to cancel the flicker reduction function and give priority to the dynamic range expansion function only when it is determined that the risk to driving due to the occurrence of blown-out areas exceeds the risk of the disappearance of traffic lights, road traffic signs, etc. In this embodiment, when the frequency of occurrence of pixels at the saturated brightness level is equal to or higher than Th (8β), it is determined that the own vehicle will soon reach the tunnel exit, and a decision is made to switch from the flicker reduction function to the dynamic range expansion function. .

If it is determined in S606 that the frequency of occurrence of pixels at the saturated luminance level is equal to or greater than Th(8β), the process advances to S607. Furthermore, if it is determined that the frequency of occurrence of pixels at the saturated luminance level is less than Th(8β), the process returns to S602 again. In this implementation, the occurrence frequency of 100% of pixels is less than Th (8β) in the brightness distribution graph of FIG. 8(b), and is greater than Th(8β) in the brightness distribution graph of FIG. The process proceeds to S607 only when the distance to the exit meets the condition shown in FIG. 8(c).

In S607, it is determined whether the number of frames in which it was determined in S606 that the frequency of occurrence of pixels at the saturated luminance level is equal to or higher than Th(8β) continues for a predetermined number of frames or more. The purpose of performing this processing is to prevent the flicker reduction function from being erroneously canceled when an image with a wide brightness distribution range happens to be acquired for one frame. For example, if you are passing through an intersection at night and the headlights of an oncoming vehicle making a right turn illuminate your vehicle for a moment, the flicker reduction function may be unintentionally disabled. If the flicker reduction function is canceled by mistake, information such as traffic lights and road traffic signs may be lost due to LED flicker, creating a risk for driving. In this embodiment, the headlights of an oncoming vehicle are used as an example, but false detections may also occur in various other situations, such as when sunlight shines through the trees for a moment while driving on a dark mountain road. In order to prevent such a situation, frame count control is performed in this embodiment.

In this embodiment, the above-mentioned predetermined number of frames is set to 30 frames. The predetermined number of frames does not necessarily have to be 30 frames, and may be changed as appropriate based on the aforementioned traveling speed information, the brightness distribution range, the calculation result of the frequency of occurrence of pixels at the saturated brightness level, and the like. In S607, if it is determined that the number of frames in which it is determined that the frequency of occurrence of pixels at the saturated luminance level is equal to or higher than Th(8β) is 30 or more consecutive frames, the process advances to S608. Furthermore, if it is determined that 30 frames or more are not consecutive, the process returns to S602 again.

In S608, processing is performed to switch from the flicker reduction function to the dynamic range extension function. Specifically, the imaging system control unit 103 disables the flicker reduction unit 105 and enables the dynamic range extension unit 106 via the imaging function switching unit 107.

In this embodiment, the switching of the imaging function is determined based on information on the frequency of occurrence of pixels at the saturation luminance level, assuming that there will be blown-out highlights at the end of the tunnel. The switching of the imaging function may be determined using brightness level information near the brightness level.

The above is the control flow when switching from a state in which the flicker reduction function is enabled to a dynamic range expansion function in this embodiment.

Here, a method of changing the exposure time when switching from a state in which the flicker reduction function is enabled to the dynamic range extension function will be explained using FIG. 9.

The basic idea of the exposure time switching method in this embodiment is to gradually widen the dynamic range instead of rapidly widening it all at once. The reason is that if the dynamic range is suddenly expanded, the output brightness from the imaging unit 104 will change rapidly even though the brightness of the subject remains the same. If the output brightness from the imaging unit 104 changes rapidly, erroneous recognition may occur during automatic driving, or images with poor visibility may occur even when used for visual recognition. This embodiment will be described on the assumption that the dynamic range is increased by a factor of √2 per frame.

In FIG. 9, the horizontal axis indicates the brightness value as in FIG. Further, the saturation brightness level when the flicker reduction function is enabled is shown as Sat (LFM), and the saturation brightness level when the dynamic range extension function is enabled is shown as Sat (HDR).

The vertical axis indicates the frequency of occurrence of each luminance value, as in FIG. Further, a threshold value for determining whether to extend the dynamic range is indicated by Th(9β).

FIG. 9A shows a brightness distribution graph at the timing when the process advances from S607 to S608. At this time, since the frequency of occurrence of pixels at 100%, which is the saturation brightness level, exceeds Th(9β), the imaging function is switched from the flicker reduction function to the dynamic range expansion function. The exposure time at this timing is 11 ms.

FIG. 9(b) is a brightness distribution graph in the next frame of FIG. 9(a). As described above, in this embodiment, the dynamic range is expanded by a factor of √2 per frame, so Sat (HDR) is approximately 140%. Further, the exposure time at this time was 7.8 ms. In FIG. 9B, the frequency of occurrence of pixels at the saturation luminance level in Sat (HDR) is still higher than Th (9β). Therefore, in the next frame, it is necessary to further widen the dynamic range by a factor of √2.

FIG. 9(c) is a brightness distribution graph in the next frame of FIG. 9(b). In the frame of FIG. 9(c), Sat (HDR) is 200%. Further, the exposure time at this time is 5.5 ms. In FIG. 9C, the frequency of occurrence of the saturated luminance level in Sat (HDR) does not exceed Th (9β), so the dynamic range is not expanded in the next frame.

In this embodiment, we have explained how to widen the dynamic range based on the frequency of occurrence of the saturated luminance level. However, if the dynamic range has been widened too much, the dynamic range can be readjusted to narrow it based on the generation results of the luminance distribution graph. Also good. Further, in this embodiment, the dynamic range is changed in steps of √2 times, but the dynamic range may be changed in other enlargement ratios.

Furthermore, although the present embodiment has been described on the premise that the exposure time for short-second exposure is determined, if the exposure time for short-second exposure is made too short, the noise of the output image from the imaging system 101 will decrease. In order to suppress this, the exposure time of the long exposure may be changed at the same time as the exposure time of the short exposure.

The above is an example of the exposure time switching method in this embodiment.

As described above, when it is determined that the environment requires a wide dynamic range with the flicker reduction function enabled, the flicker reduction function is temporarily canceled. By switching to the dynamic range extension function, it is possible to reduce the risk of driving due to subject information being lost due to overexposure.

Next, the control flow when switching from the dynamic range expansion function to the flicker reduction function in this embodiment will be explained using the flowchart of FIG. 10. First, as a premise, the image capturing unit 104 is in a state in which the flicker reduction unit 105 is disabled and the dynamic range extension unit 106 is enabled.

In S1001, the traveling speed acquisition unit 113 is used to acquire the traveling speed information of the host vehicle from the tire rotational speed information as described above. When the process in S1001 is completed, the process advances to S1002.

In S1002, it is determined whether the traveling speed acquired in S1001 is less than a predetermined speed. The reason for performing the determination process in S1002 is as described in the explanation of S603. In S1002, if the traveling speed is less than a predetermined speed, the process advances to S1003, and if the traveling speed is equal to or higher than the predetermined speed, the process advances to S1004.

In S1003, processing is performed to switch the imaging function from the dynamic range extension function to the flicker reduction function. Specifically, the imaging system control unit 103 disables the dynamic range extension unit 106 and enables the flicker reduction unit 105 via the imaging function switching unit 107.

In S1004, the brightness distribution range is calculated using the range calculation unit 110 for the image signal output from the HDR image generation unit 109 using the same method as in S604. When the process of calculating the brightness distribution range is completed in S1004, the process advances to S1005.

In S1005, it is determined whether the brightness distribution range calculated using the range calculation unit 110 is less than a predetermined value using a method similar to S605. If it is determined in S1005 that the brightness distribution range is less than the predetermined value, the process advances to S1006, and if it is determined that the brightness distribution range is not less than the predetermined value, the process returns to S1001 again.

In S1006, in the same manner as in S607, it is determined whether the number of frames in which it was determined that the brightness distribution range calculated using the range calculation unit 110 in S1005 is less than a predetermined number continues for a predetermined number of frames or more. do. In S1006, if it is determined that the number of frames in which the luminance distribution range is determined to be less than the predetermined number continues to be equal to or greater than the predetermined number of frames, the process advances to S1003. If it is determined that the predetermined number of frames or more are not consecutive, the process returns to S1001 again.

As described above, when the dynamic range expansion function is enabled and the vehicle's driving speed is lower than the specified speed, or when there is no nearby environment that requires a wide dynamic range, the dynamic range Deactivate extensions. By prioritizing the flicker reduction function again, it is possible to reduce the dangers associated with driving due to the disappearance of information such as traffic lights and road traffic signs.

As explained above, the flicker reduction function is usually prioritized to prevent LED information from being lost in traffic lights, etc., and the flicker reduction function is temporarily canceled in environments where a wide dynamic range is required. . By giving priority to expanding the dynamic range, it is possible to reduce driving risks caused by subject information being lost due to overexposure or the like.

Although the present invention has been described above in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and the present invention may take various forms without departing from the gist of the present invention. included. Some of the embodiments described above may be combined as appropriate.

Further, a case where a software program that implements the functions of the above-described embodiments is supplied directly from a recording medium or using wired/wireless communication to a system or device having a computer capable of executing the program, and the program is executed. Also included in the present invention.

Therefore, in order to realize the functional processing of the present invention on a computer, the program code itself that is supplied and installed in the computer also realizes the present invention. In other words, the present invention also includes a computer program itself for realizing the functional processing of the present invention.

In this case, the form of the program does not matter, such as an object code, a program executed by an interpreter, or script data supplied to the OS, as long as it has the function of a program.

The recording medium for supplying the program may be, for example, a hard disk, a magnetic recording medium such as a magnetic tape, an optical/magnetic optical storage medium, or a nonvolatile semiconductor memory.

Further, as a method of supplying the program, a method may be considered in which a computer program forming the present invention is stored in a server on a computer network, and a connected client computer downloads and programs the computer program.

100 Vehicle-mounted camera system 101 Imaging system 102 Vehicle-mounted system 103 Imaging system control section 104 Imaging section 105 Flicker reduction section 106 Dynamic range expansion section 107 Imaging function switching section 108 Imaging system image processing section 109 HDR image generation section 110 Range calculation section 111 Vehicle-mounted system Control unit 112 In-vehicle system image processing unit 113 Traveling speed acquisition unit 501 Long exposure image memory 502 Short exposure image memory 503 Brightness correction unit 504 Image composition unit

Claims (12)

an imaging means for switching between the first imaging function and the second imaging function, the imaging means having a first imaging function for reducing flicker and a second imaging function for expanding a dynamic range; a range calculation means for calculating a brightness distribution range of the image;
An imaging system comprising: an imaging system control unit that determines which of the first imaging function and the second imaging function to use based on information on the calculated brightness distribution range. The imaging system control means communicates with an in-vehicle system,
The imaging system according to claim 1, wherein the first imaging function and the second imaging function are switched when the traveling speed acquired from the in-vehicle system is less than a predetermined speed. comprising high dynamic range image generation means for performing dynamic range expansion processing by combining a plurality of images output from the imaging means;
The imaging according to claim 1 or 2, wherein the range calculation means counts the frequency of occurrence for each brightness from the image signal output from the high dynamic range image generation means, and generates brightness distribution information based on the count result. system. The range calculation means calculates a brightness distribution range from the difference between the maximum brightness value and the minimum brightness value among the brightness distribution information,
The imaging system according to claim 3, wherein the imaging system control means determines which of the first imaging function and the second imaging function to use based on information on the brightness distribution range. The range calculating means calculates a brightness distribution range using only brightness values whose occurrence frequency is a predetermined number of times or more among the brightness distribution information,
The imaging system control means selects which of the first imaging function and the second imaging function is selected based on information on the luminance distribution range using only luminance values whose occurrence frequency is equal to or greater than a predetermined number of luminance distribution information. The imaging system according to claim 3 or 4, wherein the imaging system determines whether to use the imaging system. The imaging system control means determines which of the first imaging function and the second imaging function to use based on information on the frequency of occurrence of the maximum brightness value calculated by the range calculation means. The imaging system according to any one of the above. When switching the imaging function, the imaging system control means determines whether the traveling speed is a predetermined speed or higher, or whether the brightness distribution range is less than a predetermined value, and determines the frequency of occurrence of the maximum brightness value. 6. The switching from the first imaging function to the second imaging function is performed when at least one determination result of the determination of whether or not the number of times or not is the same for a predetermined number of frames or more consecutively. The imaging system described in . The imaging system includes high dynamic range image generation means for generating an expanded dynamic range image by combining short exposure and long exposure images captured by the imaging means,
The imaging means expands the dynamic range by using the second imaging function to make the exposure time of the short exposure less than a predetermined value, and the imaging system control means controls the first imaging function and the second imaging function. 8. The imaging system according to claim 6, wherein when switching, the exposure time of the short exposure is determined based on information about the frequency of occurrence of the maximum brightness value. When switching between the first imaging function and the second imaging function, the imaging system control means shortens the exposure time of the short exposure when the maximum brightness value occurs more frequently than a predetermined value. The imaging system according to claim 8. The imaging system control means is configured to switch the first imaging function from the first imaging function when a frequency of occurrence of pixels at a saturated brightness level in an image taken by the imaging means at a specific position where a dynamic range of a predetermined value or more is required exceeds a threshold. The imaging system according to any one of claims 1 to 9, wherein the imaging system switches to a second imaging function. A first step of reducing flicker, a second step of expanding the dynamic range, a switching step of switching between the first step and the second step, and calculating a brightness distribution range of the image output from the imaging means. Range calculation process,
Based on the information on the brightness distribution range calculated in the range calculation step, it is determined which of the first step and the second step is to be used, and a control signal for switching between the first step and the second step is sent to the imaging function. An imaging system control method, comprising: an imaging system control step of transmitting data to a switching means. A program that causes a computer to function as each means of the imaging system according to any one of claims 1 to 10.

Images