JP7492599B2 - Vehicle-mounted camera device - Google Patents

Description

The present invention relates to an in-vehicle camera device that eliminates the ranging error phenomenon in which distance measurements by a stereo camera are distorted due to the high dynamic range of image sensors.

As social awareness of safe driving and autonomous driving increases, the required accuracy for object recognition and distance measurement in vehicle-mounted camera devices is increasing. Stereo camera devices simultaneously measure visual information from images and distance information to objects in the images, allowing them to grasp various objects around the vehicle (people, cars, three-dimensional objects, white lines/road surfaces, signs, etc.) in detail, and are devices that contribute to improving safety during driving assistance. Another characteristic of stereo cameras is that they have high spatial resolution and accuracy when it comes to measuring the distance to the target object. This is because, unlike monocular cameras, they can measure the distance to any object based on triangulation.

The evolution of in-vehicle camera equipment is driven by the evolution of image sensors. In particular, high dynamic range imaging (HDR) is expected to become an important technological trend in the future. HDR imaging uses ingenuity in image sensors and image processing to capture a wider range of subject brightness in a single image than with normal CMOS sensors. This is a major benefit for in-vehicle cameras, which are particularly susceptible to the effects of external light and the environment, as it can capture a wide range of subject brightness in a single image, from dark to bright areas, without causing shadow crush or white blowout.

On the other hand, problems have also emerged when applying high dynamic range imaging elements to stereo cameras. This problem is due to the high level of accuracy required for measuring the distance to an object for stereo cameras. The distance measurement range of an in-vehicle stereo camera can be as long as 100 to 200 m, depending on the device's specifications such as the baseline length. In this case, it is necessary to examine the correspondence (parallax) between the images captured by the left and right cameras at a sub-pixel level of less than one pixel. When calculating parallax at the sub-pixel level, there are various optical disturbance factors. For example, refraction of the light path due to the influence of the windshield and blurring of the projected light due to the focal characteristics of the lens. It has also been found that high dynamic range also affects distance measurement.

High dynamic range imaging is a common technique for expanding the imaging sensitivity range of a camera, and various processing methods and devices have been proposed. This is generally referred to as the aiming process. One example is disclosed in Patent Document 1.

Patent document 1 is a document relating to a solid-state imaging device that generates an image with an expanded dynamic range based on imaging pixel signals using low sensitivity pixels and high sensitivity pixels in a pixel array section.

JP 2015-201834 A

Patent Document 1 describes the capturing and synthesis of high dynamic range images that take advantage of the characteristics of the image sensor. However, it does not discuss the issues specific to stereo cameras that come with a high dynamic range, namely the occurrence of distance measurement errors that come with a high dynamic range, and how to deal with these issues.

The present invention has been made in consideration of the above problems, and its purpose is to reduce the impact of high dynamic range on distance measurement errors using only an in-vehicle stereo camera device, for example, by focusing on the characteristics of the in-vehicle stereo camera device, and to provide an in-vehicle camera device having such image correction and image sensor arrangement configuration.

The in-vehicle camera device of the present invention, which solves the above problem, has a first HDR imaging element including a first sensitivity area and a second sensitivity area around the first sensitivity area as a pair, and a second HDR imaging element including a third sensitivity area and a fourth sensitivity area around the third sensitivity area as a pair, and is characterized in that at least one of the weights for the signals from the first sensitivity area and the second sensitivity area, or the weights for the signals from the third sensitivity area and the fourth sensitivity area, is changed according to the bias of the arrangement of the first sensitivity area and the second sensitivity area or the arrangement of the third sensitivity area and the fourth sensitivity area, or the pair of signals to be output is changed according to the arrangement of the first sensitivity area and the second sensitivity area or the arrangement of the third sensitivity area and the fourth sensitivity area, or the arrangement of the first sensitivity area and the second sensitivity area and the arrangement of the third sensitivity area and the fourth sensitivity area are the same arrangement.

According to the present invention, for example in an in-vehicle stereo camera device, the effect of absorbing distance measurement errors caused by high dynamic range, which vary depending on the lateral position, distance, and brightness of the object, is achieved, thereby making distance measurement more accurate.

In addition, issues, configurations and effects other than those described above will become clear from the description of the embodiments below.

1 is a block diagram showing a schematic configuration of an in-vehicle stereo camera system according to an embodiment of the present invention; FIG. 4 is a process flow diagram inside the vehicle-mounted stereo camera device. A diagram showing the challenges of achieving a high dynamic range. 1 shows an overall view of data processing according to this embodiment. FIG. 4 is a process flow diagram of data processing inside the vehicle-mounted stereo camera device according to the present embodiment. Image synthesis concept diagram. An image of the arrangement of light receiving sections on an image sensor. A diagram showing the prescribed arrangement of image sensors. An image of the arrangement of light receiving sections on an image sensor.

Next, an embodiment of the present invention will be described below with reference to the drawings. Note that in each drawing, parts having the same functions are designated by the same reference numerals, and repeated description may be omitted.

Figure 1 is a block diagram showing a schematic configuration of an in-vehicle system (in-vehicle stereo camera system) including an in-vehicle stereo camera device 100 according to one embodiment of the present invention. Figure 2 is a process flow diagram of a processing unit driven by a recognition application inside the in-vehicle stereo camera device 100 shown in Figure 1.

The in-vehicle stereo camera device 100 of this embodiment is a device that is mounted on a vehicle and recognizes the environment outside the vehicle based on image information of the area to be photographed in front of the vehicle. The in-vehicle stereo camera device 100 recognizes, for example, white lines on the road, pedestrians, vehicles, other three-dimensional objects, traffic lights, signs, and lit lamps, and adjusts the brakes, steering, and other adjustments of the vehicle (own vehicle) on which the in-vehicle stereo camera device 100 is mounted. In addition, after the in-vehicle stereo camera device 100 is mounted on the vehicle, the device also performs an aiming process in which the amount of correction for the optical axis and distance measurement error is calculated at the vehicle factory.

The in-vehicle stereo camera device 100 has two cameras 101, 102 (left camera 101, right camera 102) arranged side by side to acquire image information, and a processing device 110 that performs recognition processing of objects in an image based on the image information acquired by the cameras 101, 102. The processing device 110 is configured as a computer equipped with a processor such as a CPU (Central Processing Unit), and memories such as a ROM (Read Only Memory), RAM (Random Access Memory), and HDD (Hard Disk Drive). Each function of the processing device 110 is realized by the processor executing a program stored in the ROM. The RAM stores data including intermediate data of calculations performed by the program executed by the processor.

The processing device 110 has an image input interface 103 for controlling the imaging of the cameras 101 and 102 and importing the captured images. The image data imported through this image input interface 103 is sent via an internal bus 109 and processed by an image processing unit 104 and an arithmetic processing unit 105, and the intermediate results and image data that are the final results are stored in a storage unit 106.

The image processing unit 104 compares a first image (left image) obtained from the imaging element of the camera 101 with a second image (right image) obtained from the imaging element of the camera 102, performs image correction such as correction of device-specific deviations caused by the imaging element and noise interpolation on each image, and stores the results in the memory unit 106. Furthermore, it calculates the corresponding points between the first and second images to calculate disparity information, and stores this in the memory unit 106 as before.

The arithmetic processing unit 105 uses the images and parallax information (distance information for each point on the image) stored in the memory unit 106 to recognize various objects necessary for perceiving the environment around the vehicle. The various objects include people, cars, other obstacles, traffic lights, signs, car taillights and headlights, etc. These recognition results and some of the intermediate calculation results are recorded in the memory unit 106 as before. After performing various object recognition on the captured images, the arithmetic processing unit 105 uses these recognition results to calculate the vehicle control policy.

The vehicle control policy obtained as a result of the calculation and some of the object recognition results are transmitted to the in-vehicle network CAN 111 via the CAN interface 107, which causes the vehicle to brake. In addition, the control processing unit 108 monitors these operations to see if any of the processing units are operating abnormally or if any errors have occurred during data transfer, thus preventing any abnormal operations.

The image processing unit 104 is connected to the control processing unit 108, the storage unit 106, the arithmetic processing unit 105, and the image input interface 103, which is an input/output unit between the image pickup elements of the cameras 101 and 102, and the CAN interface 107, which is an input/output unit between the external in-vehicle network CAN 111, via the internal bus 109. The control processing unit 108, the image processing unit 104, the storage unit 106, the arithmetic processing unit 105, and the input/output units 103 and 107 are composed of a single or multiple computer units. The storage unit 106 is composed of a memory that stores, for example, image information obtained by the image processing unit 104 and image information created as a result of scanning by the arithmetic processing unit 105. The CAN interface 107, which is an input/output unit with the external in-vehicle network CAN 111, outputs information output from the in-vehicle stereo camera device 100 to the control system of the vehicle via the in-vehicle network CAN 111.

The functions of each processing unit have been described above. Next, we will describe the processing flow within the in-vehicle stereo camera device 100 with reference to Figure 2.

First, images are captured by the left and right cameras 101 and 102 (S201, S202), and image processing such as correction to absorb the inherent characteristics of the image sensor is performed on each of the image data 121 and 122 captured by each camera (S203). Next, the left and right images obtained from the left camera 101 and the right camera 102 are compared, and a parallax calculation is performed to calculate corresponding points in the left and right images (S204). The results are stored in the memory unit 106 in FIG. 1. Object detection is performed using the results of the parallax calculation and the images captured by the left and right cameras (S205). Furthermore, object recognition is performed on the detected object, i.e., a mass of some three-dimensional object (S206), and it is determined whether the object is a pedestrian or a vehicle. Objects to be recognized include people, cars, bicycles, motorcycles, other three-dimensional objects, signs, traffic lights, taillights, and other self-luminous objects, but when performing object recognition using the captured images and the parallax calculation results, a recognition dictionary is used as necessary. The recognition dictionary is data that has been created in advance by means of machine learning or the like, and is stored in the storage unit 106 of the in-vehicle stereo camera device 100 .

Taking into consideration the object recognition results and the state of the vehicle (speed, steering angle, etc.), vehicle control processing is performed (S207). The vehicle control processing (S207) is, for example, issuing a warning to the occupants and braking the vehicle, adjusting the steering angle, etc. Information related to vehicle control is output from the in-vehicle stereo camera device 100 through the CAN interface 107. Various object recognition processing (S206) and vehicle control processing (S207) are performed by the arithmetic processing unit 105 in FIG. 1, and output to the in-vehicle network CAN 111 is performed by the CAN interface 107. Each of these processing and means is, for example, configured by a single or multiple computer units, and is configured to be able to exchange data with each other.

Figure 3 shows the effect on a stereo camera when an image sensor (HDR image sensor) compatible with a high dynamic range is used. There are various types and methods of image sensors compatible with a high dynamic range, but here we focus on an image sensor in which elements (sensitivity regions) with different light receiving sensitivities are arranged on the image sensor. 301 is an image of an image sensor, on which two light receiving sections A (302) (first sensitivity region, third sensitivity region) and light receiving section B (303) (second sensitivity region, fourth sensitivity region) with different light receiving sensitivities are arranged. Assuming that light receiving section A is a normal sensitivity light receiving section suitable for capturing images of normal brightness, and light receiving section B is a low sensitivity light receiving section suitable for capturing images of bright objects. Generally, two light receiving sections A and B are paired as shown in 304, and the amount of light received by these two light receiving sections A and B is stored as electrons, and the electronic data (signals) obtained from the two light receiving elements are somehow combined to obtain the pixel value for one pixel.

When applying an image sensor compatible with such a high dynamic range to the stereo vision of a stereo camera, a problem occurs in that the arrangement of the image sensor and the brightness of the target object affect the parallax calculation. 305 and 307 shown in the lower part of Figure 3 are examples of the case where the image sensor is arranged symmetrically on the left and right of the stereo camera. Here, the image of the image sensor of the left camera 101 is 305, and light projected from an object in the outside world is projected onto it at 306. Similarly, 307 is the image of the image sensor of the right camera 102, and the arrangement of light receiving parts A and B above it is line-symmetrical with respect to 305. The projection position of the object onto the right camera 102 is 308.

A problem that arises in such an arrangement of imaging elements is the shift in parallax due to the luminance of the object. For example, when the luminance of the object is low, the signal strength captured by the light receiving unit A (302) becomes the luminance value of the image. If the luminance value of the object increases from here, the amount of light received by the light receiving unit B (303) in the upper right of the left camera 101 will be reflected in the luminance value of the image. On the other hand, when the light receiving unit is received by the right camera 102, the paired light receiving unit is the light receiving unit B (303) in the upper left, so the luminance value received here will be reflected. This means that as the luminance value of the object increases, the physical positions at which light is received by the left and right cameras 101 and 102 move in different directions on the left and right. This leads to the calculation of different distances in the parallax calculation in stereo vision. As a specific example, when detecting a vehicle at dusk or night, if the parallax value, i.e., the distance, differs between the body part and the lamp part of the vehicle, it may not be possible to group them together as one object, or it may have a negative effect on the calculation of the inclination of the vehicle body.

To solve this problem, when creating images from the left and right cameras 101 and 102 for stereo vision, the processing must be changed according to the arrangement of the image sensors. Figure 4 shows an overview of this data processing.

First, the process for manufacturing a stereo camera (S401) involves assembling an imaging module combining an imaging element and a lens, etc. (S402), then assembling two imaging modules into a stereo camera (S403), and checking whether the assembly is correct and whether the imaging elements function (S404). At this time, the type of imaging element to be arranged on the left and right and how it will be arranged are determined by the process and design of the imaging module assembly in S402 and the stereo camera assembly in S403. This is recorded on the stereo camera as arrangement characteristic information as in S410.

Next, when a vehicle equipped with the in-vehicle stereo camera device 100 travels on a real road (S405), images are captured by the light receiving units A (302) and B (303) of the left and right cameras 101 and 102 (S406, S407), the captured images are temporarily stored in a local buffer or memory (S408), and the image and brightness values obtained by the light receiving unit A (302) and the image and brightness values obtained by the light receiving unit B (303) are synthesized (S409) according to the arrangement characteristic information (S410) determined based on the information at the time of manufacturing the stereo camera, and this is sent to the subsequent image processing and parallax calculation processing (S411). In other words, in order to respond to the high dynamic range of the stereo camera, some kind of cooperation or constraint with the arrangement characteristics of the left and right image pickup elements is required.

The image synthesis S409 process according to the arrangement characteristic information S410 will be described later in the explanation of Figures 6 to 8. The basic idea is to take into account the arrangement positions of light receiving elements (sensitivity areas) with different sensitivities and the arrangement area of the light receiving elements of each sensitivity, and to obtain a new synthesized image by a light-intensive calculation such as a weighted linear calculation from each image obtained by light receiving elements with different sensitivities. The coefficients used in the weighted linear calculation are determined taking into account the arrangement positions and arrangement area of the light receiving elements.

Figure 5 is a diagram in which the above-mentioned processing is positioned in the data processing inside the in-vehicle stereo camera device 100. Images obtained from the light receiving unit A group (multiple light receiving units A) or the light receiving unit B group (multiple light receiving units B) in the left camera 101 or the right camera 102 (S501, S502) are synthesized (S503, S504) using the arrangement characteristic information of the imaging element obtained as manufacturing information or design information as described above (S505), and the image of the light receiving unit A and the image of the light receiving unit B are output from the left camera 101 and the right camera 102. This image synthesis may be processed in the camera module, or after sending the data to memory, a process of synthesizing the left and right images may be performed immediately before the parallax calculation. Furthermore, in S506, image correction is performed to correct lens distortion, etc. After the above processing, parallax calculation (S507) is performed based on the normalized left and right images obtained by the left camera 101 and the right camera 102. Based on the disparity image obtained as a result of the disparity calculation, object detection (S508) and distance measurement/recognition (S509) are performed, and the results are used to perform vehicle control calculations (S510), and the results are output via the in-vehicle network CAN 111 or the like.

Figure 6 shows an image synthesis. As mentioned earlier, a synthetic image 603 is created based on a light receiving unit A image 601 and a light receiving unit B image 602. This is performed on the images from the left and right cameras 101 and 102, respectively, and disparity calculation is performed on the normalized left and right images, which makes it possible to address problems in disparity calculation caused by the high dynamic range of the stereo camera (such as distance discrepancies due to the brightness of the object).

Figure 7 shows an image of the arrangement of light receiving parts on the image sensor, and Figure 7, Equation 1, and Equation 2 show the weights used in image synthesis. Image synthesis in this example is a process that determines the luminance value of an image by taking into account the luminance value obtained by light receiving part A (302) and the luminance value obtained by the surrounding light receiving part B (303). A simple method would be to change the arrangement of the pair of light receiving parts A/B (corresponding to the output luminance value) to be calculated depending on the arrangement of the image sensor of the stereo camera (i.e., depending on the arrangement of light receiving parts A and B) (for example, in the right camera 102 shown in the lower part of Figure 3, the light receiving part B (303) paired with light receiving part A (302) is changed from the upper left to the upper right, so that the physical position of receiving light is in the same direction), but a more advanced process would be to change the weight for the luminance value depending on the arrangement of light receiving part B (303) around the four corners of light receiving part A (302) (i.e., depending on the bias in the arrangement of light receiving part A and light receiving part B). The influence on the parallax is reduced if the weights are determined taking into consideration the actual physical positions and areas of the light receiving section A (302) and the light receiving section B (303) on the image sensor.

For example, 701 is an example where light receiving section B (303) is evenly arranged at the four corners relative to light receiving section A (302) (see area 702). In this case, it is considered that the arrangement is such that the luminance value of light receiving section B (303) is equal to the luminance value of light receiving section A (302) at a certain ratio, as shown in the following formula 1.

On the other hand, 703 is an example where the physical arrangement of the light receiving unit B (303) is uneven with respect to the light receiving unit A (302) (see region 704). In this case, it is possible to change the weight for the luminance value according to the area of the light receiving unit B (303) and the respective positions of the light receiving unit B (303) (i.e., according to the bias of the arrangement of the light receiving unit B with respect to the light receiving unit A). The following formula 2 is an example. There are various approaches to designing the weight, for example, the idea of integral images can be used, or weight arrangement based on sampling theory can be considered. In addition, when the process of matching pairs of light receiving elements in the left and right cameras 101 and 102 is performed by weighted image synthesis, the same effect can be obtained by setting the coefficient for the light receiving elements set as a pair to 1 and the other coefficients to 0.

Sampling by integral image will be described with reference to FIG. 8. FIG. 8 is a standard image of the arrangement of image sensors. As shown in 801, focus on the image sensor 802 with sensitivity B, and assume that an image P(i,j) is obtained from it. If it is possible to estimate the luminance value obtained when an image sensor with sensitivity B exists at the position of the star region 803 where the image sensor with sensitivity A exists, a distance measurement error in stereo vision due to the luminance difference of the object should not occur. Therefore, the luminance value of the image sensor with sensitivity B in the star region 803 is obtained by weighting, and at that time, the weighting can be derived from the concept of integral images. First, the integral image value S(i,j) of the image P(i,j) obtained from the image sensor with sensitivity B is obtained by the following formula 3.

For the integral image 804 obtained in this way, the luminance value corresponding to the star region 803 can be calculated as the luminance value within a rectangular area 805 by the following Equation 4.

In this formula 4, the brightness value can be calculated by linear summation on the integral image from the coordinates of the four corners of the rectangular area 805 (upper left point, lower left point, upper right point, lower right point) using the characteristics of the integral image. The coordinates of the four corners of the rectangular area 805 are derived from the position and area of the image sensor with sensitivity B on the image sensor and the position and area of the image sensor with sensitivity A, and generally do not become integer values. Therefore, the integral image value S(x,y) for the non-integer coordinate value (x,y) is obtained by linear interpolation from the integer values around it. The integral image value S(x,y) for the non-integer coordinate value (x,y) can be expressed as a linear function of the integral image value S(i,j) for the integer coordinates around it, and further, the integral image value S(i,j) for the integer coordinates is expressed as a linear sum of the original image P(i,j) obtained from the sensitivity B. Therefore, the estimated brightness value by the sensitivity B for the star region 803 can be expressed as a linear sum of the original image P(i,j). The coefficients at this time can be used as weights when synthesizing images.

In addition, if you want to reduce the amount of calculation required for these weight change processes, you can arrange the image pickup elements on the left and right cameras 101 and 102 so that the arrangement of the elements with multiple light receiving sensitivities matches (more specifically, so that the elements that form a pair with different light receiving sensitivities are in the same direction). This is shown in Figure 9. Here, we list examples of the arrangement of image pickup elements on the left and right cameras 101 and 102 that are assumed when the arrangement of the light receiving elements on the left and right is unified (the same arrangement) in advance, rather than weighted image synthesis (four examples in Figure 9). If weighted image synthesis is difficult, it is possible to address the issues in the high dynamic range in-vehicle stereo camera device 100 by devising the arrangement of the light receiving elements in the left and right cameras 101 and 102 as shown in Figure 9.

As described above, the in-vehicle stereo camera device (in-vehicle camera device) 100 of this embodiment includes a first HDR imaging element including a first sensitivity region and a second sensitivity region surrounding the first sensitivity region as a pair (when synthesizing images) (i.e., a signal obtained from the first sensitivity region (brightness value of the image) and a signal obtained from the second sensitivity region (brightness value of the image) are synthesized to output as a pixel value for one pixel of the image), and a second HDR imaging element including a third sensitivity region and a fourth sensitivity region surrounding the third sensitivity region as a pair (when synthesizing images) (i.e., a signal obtained from the third sensitivity region (brightness value of the image) and a signal obtained from the fourth sensitivity region (brightness value of the image) are synthesized to output as a pixel value for one pixel of the image). and a weighting factor for the signal from the first sensitivity region and the signal from the second sensitivity region, or a weighting factor for the signal from the third sensitivity region and the signal from the fourth sensitivity region (when synthesizing an image/brightness value) is changed in accordance with a bias in an arrangement of the first sensitivity region and the second sensitivity region or an arrangement of the third sensitivity region and the fourth sensitivity region, or a pair of signals (brightness values) to be output is changed in accordance with an arrangement of the first sensitivity region and the second sensitivity region or an arrangement of the third sensitivity region and the fourth sensitivity region, or an arrangement of the first sensitivity region and the second sensitivity region and an arrangement of the third sensitivity region and the fourth sensitivity region are the same arrangement.

Furthermore, when the arrangement of the first sensitivity area and the second sensitivity area and the arrangement of the third sensitivity area and the fourth sensitivity area are the same arrangement, the arrangement (of the sensitivity areas) of the first and second HDR imaging elements is designed so that the pair of sensitivity areas having different light receiving sensitivities are oriented in the same direction.

That is, the in-vehicle stereo camera device (in-vehicle camera device) 100 of this embodiment is an in-vehicle stereo camera device that processes images captured by a pair of cameras, and is characterized in that when the image sensor section is compatible with high dynamic range exposure and disparity calculation is performed from images captured by both cameras, the arrangement of the image sensors must satisfy certain regulations, or if the regulations are not satisfied, image synthesis parameters (pair characteristics, weights, etc.) are designed according to the characteristics of the light receiving sections of the image sensors, and the left and right images are corrected using this to make them suitable for stereoscopic viewing, and then stereoscopic viewing is performed.

In other words, this embodiment is characterized in that, in order to reduce the effect of distance measurement errors associated with the high dynamic range of the image sensor, the arrangement of the light receiving sections in the image sensor with different light receiving sensitivities is matched in the left and right cameras, or the weights and calculation methods are changed based on the arrangement characteristic information of the image sensor in the image synthesis process for converting images obtained in light receiving sections with different light receiving sensitivities according to the arrangement characteristic information of the image sensor in the left and right cameras into left and right camera images for stereoscopic vision.

In other words, this embodiment is characterized by the following: in order to reduce the effect of distance measurement errors caused by the high dynamic range of the image sensor, 1) when manufacturing the stereo camera, certain restrictions are placed on the direction of the image sensor, or correction parameters are prepared according to the characteristics of the arrangement of the light receiving element of the image sensor, and 2) when recognizing the outside world with a stereo camera attached to a vehicle, the input images from the left and right cameras for stereo vision are processed according to specific parameters determined when the stereo camera was manufactured, thereby dealing with the high dynamic range.

According to this embodiment, for example in an in-vehicle stereo camera device, the effect is to absorb distance measurement errors caused by high dynamic range, which vary depending on the lateral position, distance, and brightness of the target object, thereby making distance measurement more accurate.

Note that the present invention is not limited to the above-described embodiment, but includes various modified forms. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to having all of the configurations described.

In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in hardware, in part or in whole, for example by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the programs, tables, files, etc. that realize each function can be stored in a memory, a storage device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

In addition, the control lines and information lines shown are those considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be assumed that almost all components are interconnected.

100...vehicle-mounted stereo camera device (vehicle-mounted camera device)
Reference Signs List 101: Left camera 102: Right camera 103: Image input interface 104: Image processing unit 105: Arithmetic processing unit 106: Storage unit 107: CAN interface 108: Control processing unit 109: Internal bus 110: Processing device 111: In-vehicle network CAN
301: Image sensor 302: Light receiving section A (first sensitivity area, third sensitivity area)
303: Light receiving section B (second sensitivity area, fourth sensitivity area)
305: Imaging element of left camera 307: Imaging element of right camera

Claims (4)

A first HDR imaging element including a first sensitivity region and a second sensitivity region around the first sensitivity region as a pair, and a second HDR imaging element including a third sensitivity region and a fourth sensitivity region around the third sensitivity region as a pair,
At least one of weights for the signals from the first sensitivity region and the second sensitivity region, or weights for the signals from the third sensitivity region and the fourth sensitivity region is changed in accordance with the bias of the arrangement of the first sensitivity region and the second sensitivity region or the arrangement of the third sensitivity region and the fourth sensitivity region, or a pair of signals to be output is changed in accordance with the arrangement of the first sensitivity region and the second sensitivity region or the arrangement of the third sensitivity region and the fourth sensitivity region, or
an arrangement of the first sensitive region and the second sensitive region and an arrangement of the third sensitive region and the fourth sensitive region are the same arrangement,
The weight of the first HDR imaging element is changed based on at least one of the placement positions or placement areas of the first sensitivity area and the second sensitivity area in the first HDR imaging element, and the weight of the second HDR imaging element is changed based on at least one of the placement positions or placement areas of the third sensitivity area and the fourth sensitivity area in the second HDR imaging element . A first HDR imaging element including a first sensitivity region and a second sensitivity region around the first sensitivity region as a pair, and a second HDR imaging element including a third sensitivity region and a fourth sensitivity region around the third sensitivity region as a pair,
At least one of weights for the signals from the first sensitivity region and the second sensitivity region, or weights for the signals from the third sensitivity region and the fourth sensitivity region is changed in accordance with the bias of the arrangement of the first sensitivity region and the second sensitivity region or the arrangement of the third sensitivity region and the fourth sensitivity region, or a pair of signals to be output is changed in accordance with the arrangement of the first sensitivity region and the second sensitivity region or the arrangement of the third sensitivity region and the fourth sensitivity region, or
an arrangement of the first sensitive region and the second sensitive region and an arrangement of the third sensitive region and the fourth sensitive region are the same arrangement,
An in-vehicle camera device , wherein the weights are set using sampling with an integral image . The vehicle-mounted camera device according to claim 1 or 2 ,
An in-vehicle camera device characterized in that, when the arrangement of the first sensitivity area and the second sensitivity area and the arrangement of the third sensitivity area and the fourth sensitivity area are the same arrangement, the arrangement of the first and second HDR imaging elements is designed so that pairs of sensitivity areas having different light receiving sensitivities are oriented in the same direction. The vehicle-mounted camera device according to claim 1,
An in-vehicle camera device, wherein the weights are set using sampling with an integral image.

Images