JP6668975B2 - Electronics and vehicles - Google Patents

Description

The present invention relates to an electronic device and automotive.

  A camera mounted on a vehicle is known (see Patent Document 1). In a conventional camera mounted on a vehicle, there is a problem that an imaging condition for detecting a target object and an imaging condition for generating a display image cannot be set.

JP 2010-79424 A

An electronic device according to a first aspect of the present invention is an electronic device mounted on a car, wherein a first region for imaging a subject under a first imaging condition and a second imaging condition different from the first imaging condition. An imaging unit having a plurality of second regions for imaging a subject, a detection unit for detecting an object based on a first image signal generated by the first region, and a second unit generated by the second region. A display control unit that displays an image on the display unit based on the second image signal.
  An automobile according to a second aspect of the present invention includes the electronic device according to the first aspect.

1 is a schematic configuration diagram of a driving support device for a vehicle. FIG. 3 is a diagram illustrating a pixel array and a unit area of an imaging chip. FIG. 2 is a block diagram illustrating a configuration of a camera. FIGS. 4A to 4C are diagrams illustrating the divided first region and second region. It is a figure which shows typically the image on the imaging surface of an imaging chip. It is a figure which shows typically the image on the imaging surface of an imaging chip. FIGS. 7A and 7B are diagrams schematically illustrating an image on an imaging surface of an imaging chip, wherein FIG. 7A is a diagram illustrating a straight traveling state, and FIG. It is a figure which shows typically the image on the imaging surface of an imaging chip. It is a figure which shows typically the image on the imaging surface of an imaging chip. 5 is a flowchart illustrating a flow of a camera control process executed by a control unit. 9 is a flowchart illustrating details of an imaging condition setting process. It is a figure which shows typically the image on the imaging surface of an imaging chip. It is a figure which shows typically the image on the imaging surface of an imaging chip. It is a figure which shows typically the image on the imaging surface of an imaging chip. It is a figure which illustrates a display image. 9 is a flowchart illustrating details of an imaging condition setting process. FIG. 3 is a diagram schematically illustrating an image formed on an image sensor. FIG. 3 is a diagram schematically illustrating an image formed on an image sensor. FIG. 3 is a diagram schematically illustrating an image formed on an image sensor. FIG. 3 is a diagram schematically illustrating an image formed on an image sensor. 5 is a flowchart illustrating a flow of a camera control process. 9 is a flowchart illustrating details of an imaging condition setting process. 9 is a flowchart illustrating details of an imaging condition setting process. FIGS. 24A and 24B are diagrams illustrating a frame rate according to the image plane movement amount. FIG. 3 is a diagram illustrating a yaw angle of a camera. 9 is a flowchart illustrating details of an imaging condition setting process. FIG. 27A is a diagram illustrating the third embodiment. FIG. 3 is a diagram illustrating a projection pattern by an auxiliary light source. It is a figure which illustrates the object irradiated with infrared light. FIGS. 28A and 28B are diagrams for explaining the fourth embodiment. FIG. 29A is a diagram for explaining the fifth embodiment. FIG. 29B is a diagram illustrating a first area and a second area on the imaging surface.

Hereinafter, embodiments for implementing the present invention will be described with reference to the drawings.
(First embodiment)
<Scene where the camera is used>
FIG. 1 is a schematic configuration diagram of a driving support device 2 of a vehicle 1 equipped with cameras 3A and 3B according to the first embodiment. In FIG. 1, a driving support device 2 is mounted on a vehicle 1 such as an automobile. The driving support device 2 includes a camera 3A, a camera 3B, a control device 4, a first travel control unit 5, a second travel control unit 6, and the like. In this embodiment, an example in which an internal combustion engine is used as a drive source will be described, but a motor may be used as a drive source.

  In the present embodiment, the cameras 3A and 3B are configured by the same camera. The camera 3A (3B) includes an imaging optical system having a plurality of lenses, and a stacked imaging device described later. The camera 3 </ b> A is attached, for example, in front of the ceiling in the vehicle compartment, and acquires an image in front of the vehicle 1. The camera 3 </ b> B is attached, for example, behind the ceiling in the vehicle interior, and acquires an image behind the vehicle 1.

  Each of the cameras 3A and 3B measures the distance to each subject (object) at a plurality of positions on the shooting screen based on the acquired image (an image acquired based on a first region in the image sensor 100 described later). Distance measurement). The distance measurement is calculated by a distance measurement operation using an image signal from a focus detection pixel provided in the image sensor. Image data and distance measurement data acquired by the camera 3A (3B) are sent to the control device 4. Note that the camera 3A (3B) may be provided outside the vehicle.

  Control device 4 includes a CPU 4a and a storage unit 4b. The CPU 4a performs various calculations based on various programs stored in the storage unit 4b, using control parameters stored in the storage unit 4b, detection signals from sensors described below, and the like.

  The first travel control unit 5 performs constant-speed travel control and follow-up travel control based on an instruction from the control device 4. The constant speed traveling control is control for causing the vehicle 1 to travel at a constant speed based on a predetermined control program. When the speed of the preceding vehicle recognized by the control device 4 is equal to or lower than the target speed set in the vehicle 1 during the constant speed traveling control, the following traveling control is performed with respect to the preceding vehicle. This is control for running the vehicle while maintaining the inter-vehicle distance.

  The second travel control unit 6 performs driving support control based on an instruction from the control device 4. The driving support control outputs a steering control signal to the steering control device 9 so that the vehicle 1 travels along the road based on a predetermined control program, or avoids the vehicle 1 from colliding with an object. Is a control for outputting a brake control signal to the brake control device 8.

FIG. 1 further shows a throttle control device 7, a brake control device 8, a steering control device 9, a steering wheel 10, a turn signal switch 11, a vehicle speed sensor 12, a yaw rate sensor 13, a display / reproduction device 14, , A GPS device 15 and a visual line detection device 16 are illustrated.
The communication unit 17 is used in a modified example 4 described later, and the radar 18 is used in the second embodiment. These are not essential components in the first embodiment.

  The throttle control device 7 controls the opening of a throttle valve (not shown) according to the amount of depression of an accelerator pedal 7a. The throttle control device 7 also controls the opening of the throttle valve in accordance with a throttle control signal sent from the first travel control unit 5. The throttle control device 7 further sends a signal indicating the depression amount of the accelerator pedal 7a to the control device 4.

  The brake control device 8 controls the opening of a brake valve (not shown) according to the amount of depression of the brake pedal 8a. The brake control device 8 also controls the opening of the brake valve according to a brake control signal from the second travel control unit 6. The brake control device 8 further sends a signal indicating the depression amount of the brake pedal 8a to the control device 4.

  The steering control device 9 controls the steering angle of a steering device (not shown) according to the rotation angle of the steering wheel 10. The steering control device 9 also controls the steering angle of the steering device according to a steering control signal from the second travel control unit 6. The steering control device 9 further sends a signal indicating the rotation angle of the steering wheel 10 to the first travel control unit 5 and the control device 4, respectively.

  The turn signal switch 11 is a switch for operating a turn signal (turn signal) device (not shown). The turn signal device is a blinking light emitting device that indicates a change in the course of the vehicle 1. When the occupant of the vehicle 1 operates the turn signal switch 11, an operation signal from the turn signal switch 11 is sent to the turn signal device, the second travel control unit 6, and the control device 4, respectively. The vehicle speed sensor 12 detects the vehicle speed V of the vehicle 1 and sends out a detection signal to the first travel control unit 5, the second travel control unit 6, and the control device 4, respectively.

  The yaw rate sensor 13 detects a yaw rate of the vehicle 1 and sends a detection signal to the second travel control unit 6 and the control device 4, respectively. The yaw rate is a change speed of the rotation angle of the vehicle 1 in the turning direction. The display / reproduction device 14 displays information indicating a control state of the first travel control unit 5 and the second travel control unit 6, and the like. The display / reproduction device 14 is configured by, for example, a HUD (Head Up Display) that projects information on a windshield. As the display / reproducing device 14, information may be displayed on a room mirror (not shown), or a display unit and a reproducing unit of a navigation device (not shown) may be used.

  The GPS device 15 receives a radio wave from a GPS satellite and performs a predetermined calculation using information on the radio wave to calculate the position (latitude, longitude, and the like) of the vehicle 1. The position information calculated by the GPS device 15 is sent to a navigation device and the control device 4 (not shown).

The line-of-sight detection device 16 is built in, for example, the steering wheel 10. The line-of-sight detection device 16 detects the line-of-sight position of the occupant (for example, the driver), and sends line-of-sight information indicating an area where the driver is gazing to the control device 4. For gaze detection, a corneal reflection method in which infrared light is reflected by a driver's cornea to detect a gaze direction, a limbus tracking method using a difference in reflectance of light between the cornea and the sclera, and a driver's eyeball There is an image analysis method or the like in which the video is captured by a camera and the line of sight is detected by image processing, and any line of sight line detection method may be used. In the case of an autonomous vehicle, the occupant may not drive, and the target for detecting the line of sight may be an occupant other than the driver.
Alternatively, the eye-gaze detecting device 16 may be built in a glasses-type wearable device, and the eye-gaze information may be transmitted from the wearable device to the control device 4 by wireless communication.

<Detection of object>
An image acquired based on a first region described later is used for detecting the target. The control device 4 performs image processing on an image from the camera 3A (3B) as follows in order to detect a traveling path of the vehicle 1 and an object. First, the control device 4 generates a distance image (depth distribution image) based on the distance measurement data at a plurality of positions in the shooting screen. The control device 4 performs a well-known grouping process based on the data of the distance image, and stores frames (windows) of three-dimensional road shape data, side wall data, object data, and the like stored in the storage unit 4b in advance. In addition to detecting white lines (including white line data along the road and white lines crossing the road (stop line: intersection information) data), guardrails, curbs and other side walls existing along the road, Obstacles are detected by classifying them into other objects such as motorcycles, ordinary vehicles, large vehicles, pedestrians, telephone poles, and the like.
In the present description, the white or yellow line drawn on the traveling path is called a white line. In addition, the solid line and the broken line are called white lines.

<Driving support>
The control device 4 recognizes an object / obstacle which becomes a running path or an obstacle based on each information detected as described above, that is, each data of the white line data, the guardrail side wall data, and the object data, and the recognition result. The second driving control unit 6 performs the driving support control on the basis of the above. That is, the vehicle 1 is made to travel along the road, and the collision of the vehicle 1 with the target is avoided.

<Driving control>
The control device 4 estimates the own vehicle traveling route in the following four ways, for example.
(1) Estimation of own vehicle traveling path based on white line White line data of both the left and right sides or one of the right and left sides of the traveling path is obtained from the image acquired by camera 3A (3B), and vehicle 1 is determined from these white line data. When the shape of the traveling lane can be estimated, the control device 4 estimates that the own vehicle traveling path is parallel to the white line in consideration of the width of the vehicle 1 and the position of the vehicle 1 in the current lane. .

(2) Estimation of own-vehicle traveling route based on sidewall data of guardrails, curbs, etc. Sidewall data on both the left and right sides or one of the left and right sides of the traveling route is obtained from the image acquired by camera 3A (3B). When the shape of the lane in which the vehicle 1 is traveling can be estimated from the side wall data, the control device 4 considers the width of the vehicle 1 and the position of the vehicle 1 in the current lane, and determines that the traveling path of the own vehicle is the side wall. Presumed to be parallel.

(3) Estimation of the own vehicle traveling route based on the preceding vehicle trajectory The control device 4 estimates the own vehicle traveling route based on the past traveling trajectory of the preceding vehicle stored in the storage unit 4b. The preceding vehicle refers to a forward vehicle closest to the vehicle 1 among objects traveling in the same direction as the vehicle 1.

(4) Estimation of Own Car Running Path Based on Running State of Vehicle 1 The control device 4 estimates the own vehicle traveling path based on the operating state of the vehicle 1. For example, the own vehicle traveling path is estimated using the turning curvature based on the detection signal from the yaw rate sensor 13 and the detection signal from the vehicle speed sensor 12. The turning curvature Cua is calculated by Cua = dψ / dt / V. dψ / dt is the yaw rate (the speed of change of the rotation angle in the turning direction), and V is the vehicle speed of the vehicle 1.

  The control device 4 estimates the travel area of the vehicle 1 at the position where the target object exists based on the own vehicle traveling path for each target object according to a predetermined travel control program stored in the storage unit 4b. The traveling area and the position of the object are compared to determine whether or not each object is within the traveling area. The control device 4 further recognizes the preceding vehicle based on the imaging result of the camera 3A (3B). That is, the control device 4 sets the vehicle closest to the vehicle 1 to the preceding vehicle among the objects existing in the traveling area and traveling in the forward direction (the same direction as the vehicle 1).

  The control device 4 outputs inter-vehicle distance information between the preceding vehicle and the vehicle 1 and vehicle speed information of the preceding vehicle to the first travel control unit 5 as outside-vehicle information. Here, the vehicle speed information of the preceding vehicle is obtained based on the vehicle speed V of the vehicle 1 acquired every predetermined time and the image acquired by the camera 3A (3B) every predetermined time in synchronization with the acquisition timing of the vehicle speed V. It is calculated based on the change in the distance to the preceding vehicle (inter-vehicle distance) within the measured shooting screen.

  The first travel control unit 5 sends a throttle control signal to the throttle control device 7 so that the vehicle speed V detected by the vehicle speed sensor 12 converges to a predetermined vehicle speed (target speed) set in advance. As a result, the throttle control device 7 feedback-controls the opening of the throttle valve (not shown), and the vehicle 1 automatically runs at a constant speed.

  Further, the first travel control unit 5 is configured to control the vehicle speed information of the preceding vehicle input from the control device 4 during the travel control in the constant speed state when the vehicle speed information is equal to or lower than the target speed set in the vehicle 1. And sends a throttle control signal to the throttle control device 7 based on the inter-vehicle distance information input from the control device 4. Specifically, an appropriate inter-vehicle distance target value is set based on the inter-vehicle distance from the vehicle 1 to the preceding vehicle, the vehicle speed of the preceding vehicle, and the vehicle speed V of the vehicle 1, and is acquired by the camera 3A (3B). A throttle control signal is sent to the throttle control device 7 so that the inter-vehicle distance measured based on the image converges to the target value of the inter-vehicle distance. As a result, the throttle control device 7 performs feedback control of the opening of the throttle valve (not shown), and causes the vehicle 1 to follow the preceding vehicle.

<Explanation of the stacked image sensor>
The stacked imaging device 100 provided in the camera 3A (3B) described above is described in International Publication WO13 / 164915 filed and published by the present applicant before. The image pickup device 100 includes a back-illuminated image pickup chip that outputs pixel signals corresponding to incident light, a signal processing chip that processes pixel signals, and a memory chip that stores pixel signals. The imaging device 100 is configured to be able to set imaging conditions for each unit area.

  FIG. 2 is a diagram illustrating the pixel array of the imaging chip 111 and the unit area 131. Light incident on the image sensor 100 is incident in the positive Z-axis direction. As shown by the coordinate axes, the right direction on the paper orthogonal to the Z axis is defined as the plus direction of the X axis, and the upward direction of the paper orthogonal to the Z axis and the X axis is defined as the plus direction of the Y axis. In the following figures, coordinate axes are displayed so that the directions of the respective figures can be understood with reference to the coordinate axes in FIG.

  In the pixel region of the imaging chip 111, for example, 20 million or more pixels are arranged in a matrix. In the example of FIG. 2, four adjacent pixels of 2 × 2 pixels form one unit area 131. The grid lines in the figure show the concept of forming the unit area 131 by grouping adjacent pixels. The number of pixels forming the unit area 131 may be any number, for example, 1000 pixels or 1 pixel. Further, the number of pixels may be different between the unit areas.

  As shown in the partial enlarged view of the pixel region, the unit region 131 includes a so-called Bayer array including four pixels of green pixels Gb and Gr, blue pixels B and red pixels R. The green pixels Gb and Gr are pixels having a green filter as a color filter, and receive light in a green wavelength band of incident light. Similarly, a blue pixel B is a pixel having a blue filter as a color filter and receives light in a blue wavelength band, and a red pixel R is a pixel having a red filter as a color filter and emits light in a red wavelength band. Receive light.

  In the present embodiment, a plurality of blocks are defined so as to include at least one unit area 131 per block. That is, the minimum unit of one block is one unit area 131. As described above, among the possible values of the number of pixels forming one unit region 131, the smallest number of pixels is one pixel. Therefore, when one block is defined in pixel units, the minimum number of pixels among the number of pixels that can define one block is one pixel. Each block can control pixels included in each block with different control parameters. In each block, all the unit areas 131 in the block, that is, all the pixels in the block are controlled under the same imaging conditions. That is, photoelectric conversion signals having different imaging conditions can be obtained for a pixel group included in a certain block and a pixel group included in another block. Examples of the control parameters include a frame rate, a gain, a thinning rate, the number of rows or columns to be added for adding the photoelectric conversion signal, the charge accumulation time or the number of accumulations, and the number of digitization bits (word length). The imaging element 100 can perform not only thinning in the row direction (X-axis direction of the imaging chip 111) but also thinning in the column direction (Y-axis direction of the imaging chip 111). Further, the control parameter may be a parameter in image processing.

<Explanation of distance measurement>
In the present embodiment, a plurality of luminous fluxes passing through different pupil positions of the imaging optical system 31 (FIG. 3) based on image signals for distance measurement from focus detection pixels discretely provided on the imaging chip 111. The defocus amount of the imaging optical system 31 is obtained by detecting the shift amount (phase difference) of a plurality of images due to the following. Since the defocus amount and the distance to the object correspond one-to-one, the distance from the camera 3A to the object can be obtained.
Note that normal imaging pixels are provided at pixel positions other than the focus detection pixels in the imaging chip 111. The imaging pixel outputs an image signal for monitoring outside the vehicle.

<Explanation of camera>
FIG. 3 is a block diagram illustrating the configuration of the camera 3A (3B) having the above-described image sensor 100. As described above, the configurations of the cameras 3A and 3B are the same. 3, the camera 3A (3B) includes an imaging optical system 31, an imaging unit 32, an image processing unit 33, a work memory 34, a control unit 35, and a recording unit 36. Since the driving unit 37 is used in the second embodiment, it is not an essential component in the first embodiment.

  The imaging optical system 31 guides a light beam from the object scene to the imaging unit 32. The imaging unit 32 includes the imaging element 100 and the driving unit 32a, and photoelectrically converts the image of the object formed on the imaging chip 111 by the imaging optical system 31. The drive unit 32a generates a drive signal necessary for causing the image sensor 100 (image pickup chip 111) to perform independent accumulation control in the above-described block units. Instructions such as the position and shape of the block, its range, and the accumulation time are transmitted from the control unit 35 to the drive unit 32a.

  The image processing unit 33 includes a first image processing unit 30a and a second image processing unit 30b. In the present embodiment, as will be described later, the control unit 35 divides the imaging surface of the imaging chip 111 of the imaging device 100 into a first area and a second area, and different imaging conditions for the first area and the second area. To perform imaging. In this case, for example, the first image processing unit 30a performs image processing on an image signal output from the first region of the image sensor 100. In addition, the second image processing unit 30b performs image processing on an image signal output from the second region of the image sensor 100.

  FIGS. 4A to 4C are diagrams illustrating a first area and a second area divided by the control unit 35. FIG. According to FIG. 4A, the first area is constituted by the blocks in the odd columns, and the second area is constituted by the blocks in the even columns. That is, the blocks in the imaging chip 111 are divided into odd columns and even columns.

  According to FIG. 4B, the first area is constituted by the blocks in the odd rows, and the second area is constituted by the blocks in the even rows. That is, the blocks in the imaging chip 111 are divided into odd rows and even rows.

  According to FIG. 4 (c), the first area is constituted by the blocks of the odd rows in the odd columns and the blocks of the even rows in the even columns. Further, the second region is constituted by the blocks of the odd rows in the even columns and the blocks of the even rows in the odd columns. That is, in the imaging chip 111, the blocks are divided so as to have a checkered pattern.

  Returning to FIG. 3, the first image processing unit 30a and the second image processing unit 30b each use the work memory 34 as a work space for an image signal output from the first area and an image signal output from the second area. Perform image processing. The image processing includes processing for generating an RGB image signal by performing color signal processing (color tone correction) on a signal obtained in the Bayer array, white balance adjustment for the RGB image signal, sharpness adjustment, gamma correction, and gradation. Including processing for performing adjustment and the like. Further, a process of compressing in a predetermined compression format (JPEG format, MPEG format, etc.) as necessary is also included in the image processing. The first image processing unit 30a and the second image processing unit 30b also detect the color of an object included in the image.

  The work memory 34 is a memory for temporarily storing image data before and after image processing. The recording unit 36 records image data and the like on a storage medium including a nonvolatile memory. The control unit 35 includes, for example, a CPU 35a and a storage unit 35b. The CPU 35a controls the entire operation of the camera 3A (3B) according to a control signal from the control device 4. For example, a predetermined exposure calculation is performed based on the image signal captured by the imaging unit 32, and the accumulation time of the imaging chip 111 required for proper exposure is instructed to the driving unit 32a. The image data acquired by the camera 3A (3B) and the calculated distance measurement data are sent to the control device 4 (FIG. 1). The storage unit 35b stores a program executed by the CPU 35a and necessary information.

<Block control of image sensor>
The control device 4 causes the image sensor 100 (imaging chip 111) of the camera 3A (3B) to perform the independent accumulation control in the above-described block unit. For example, the control device 4 sends an instruction to the control unit 35, divides the imaging surface of the imaging chip 111 as described above, sets the first region and the second region, and sets the distance between the first region and the second region. , Charge accumulation (imaging) is performed under different imaging conditions. Although the imaging conditions are different between the first area and the second area, the image acquired based on the first area and the image acquired based on the second area each include the same subject (object).

In the first region (or the second region), the control device 4 sets a fourth region and a third region whose attention level is to be raised more than the fourth region, and sets the fourth region and the fourth region between the third region and the fourth region. , Charge accumulation (imaging) is performed under different conditions. In this case, the subjects (objects) acquired in the third area and the fourth area are different.
Here, in the third area and the fourth area, in addition to the control parameters described above, the size and position of the third area and the fourth area are one of the imaging conditions.

<Front camera>
In the present embodiment, the control device 4 sets a first region and a second region for the image sensor 100 of the camera 3A that images the front of the vehicle 1 as shown in FIG. 4A, for example. The control device 4 uses the image acquired based on the first region to detect the above-described traveling path of the vehicle 1 and the target. Further, the control device 4 uses the image acquired based on the second area for display on the display / reproduction device 14 (FIG. 1).
The control device 4 further sets, as the third region, a region in which the driver is not gazing, or a region in which the degree of gazing by the driver is low, in the first region.

<When turning right>
FIG. 5 is a diagram schematically illustrating an image of a subject (object) formed on the imaging chip 111 of the front camera 3A provided in the vehicle 1 that is about to turn right at an intersection. Although an inverted inverted image is actually formed, it is illustrated as an erect upright image for easy understanding. In FIG. 5, an imaging surface (imaging area) 70 of the imaging chip 111 includes an image of a road at an intersection, an image of an oncoming vehicle 91, and an image of a pedestrian 92.
When the image is divided as shown in FIG. 4A, the odd columns in the image in FIG. 5 correspond to the first region, and the even columns correspond to the second region.

  The control device 4 sets different conditions between the first area and the second area on the imaging surface 70 to cause charge accumulation (imaging). For example, the camera 3A is controlled so that the frame rate is set to 120 fps with respect to the unit area 131 (FIG. 2) corresponding to the first area, and the frame is set to the unit area 131 corresponding to the second area. The camera 3A is controlled so as to capture an image at a rate set to 30 fps.

Generally, the first region where the frame rate is set to be higher is limited to have a shorter charge accumulation time than the second region where the frame rate is set to be lower. Therefore, in a dark environment, the gain is increased to prevent the image from becoming dark. By increasing the gain, the image becomes brighter, and the object can be appropriately detected. At this time, the object can be appropriately detected even when the noise is increased by increasing the gain. On the other hand, in the second region where the frame rate is set lower, the charge accumulation time can be longer than in the first region, so that an image suitable for display, which is bright and has less noise without increasing the gain, can be obtained.
In the above description, the frame rate is set to be different between the first area and the second area. However, the gain between the first area and the second area may be changed without changing the frame rate. For example, if the gain of the first area is set higher than the gain of the second area, an object can be appropriately detected in the first area even in dark conditions such as evening or night.

  In general, the driver of the right-turn vehicle turns his or her eyes to an area 90 including an oncoming vehicle 91 and a pedestrian 92 who walks on a crosswalk of a road to which the vehicle turns right. Therefore, the control device 4 switches between the left third region 81 not including the region 90 and the right fourth region 82 (shaded) excluding the third region 81 among the first regions on the imaging surface 70. Different conditions are set and charge accumulation (imaging) is performed. For example, the camera 3 </ b> A is controlled so that the third condition is set for each of the unit areas 131 (FIG. 2) corresponding to the third area 81 of the imaging chip 111, and the unit area 131 corresponds to the fourth area 82. The camera 3A is controlled such that the fourth condition is set for each of the unit areas 131 to capture an image.

  Note that a plurality of third regions 81 and fourth regions 82 may be provided, or a plurality of regions having different charge accumulation controls (imaging conditions) may be provided in the third region 81 or the fourth region 82. Further, in the third region 81 and the fourth region 82, a pause region in which charge accumulation (imaging) is not performed may be provided.

<When turning left>
FIG. 6 is a diagram schematically illustrating an image of a subject (object) formed on the imaging chip 111 of the front camera 3A provided in the vehicle 1 about to turn left at the intersection. In FIG. 6, an imaging surface (imaging area) 70 of the imaging chip 111 includes an image of a road at an intersection, an image of an oncoming vehicle 91, and an image of a pedestrian 92.
When the image is divided as shown in FIG. 4A, the odd columns in the image of FIG. 6 correspond to the first region, and the even columns correspond to the second region. The control device 4 controls the camera 3A to set different frame rates between the first area and the second area to capture an image, as in the case of turning right.

  In general, the driver of the left-turn vehicle turns his or her gaze to an area 90 including a pedestrian 92 who walks a pedestrian crossing the left turn road among the objects, so that the degree of attention to the area 90 increases. Therefore, the control device 4 sets the distance between the right third region 81 not including the region 90 and the left fourth region 82 (shaded) excluding the third region 81 in the first region on the imaging surface 70. Different conditions are set and charge accumulation (imaging) is performed.

<When going straight>
FIG. 7 is a diagram schematically illustrating an image of a subject (object) formed on the imaging chip 111 of the front camera 3A provided in the vehicle 1 traveling straight on the road. In FIG. 7A, an imaging surface (imaging area) 70 of the imaging chip 111 includes an image of a straight road, an image of a preceding vehicle 93, and an image of another vehicle 94 traveling in an adjacent lane.
When the image is divided as shown in FIG. 4A, the odd columns in the image of FIG. 7A correspond to the first region, and the even columns correspond to the second region. The control device 4 controls the camera 3A so as to set different frame rates between the first area and the second area and capture an image, as in the case of turning right or turning left.

  In general, a driver of a straight-ahead vehicle turns his or her gaze exclusively to the area 90 including the front center, so that the degree of attention to the area 90 increases. Therefore, the control device 4 sets the distance between the third region 81 at the left and right ends not including the region 90 and the central fourth region 82 (shaded) excluding the third region 81 in the first region on the imaging surface 70. And charge accumulation (imaging) is performed by setting different conditions.

<High speed>
FIG. 7B schematically shows an image of a subject (object) formed on the imaging chip 111 of the front camera 3A provided in the vehicle 1 having a higher vehicle speed V than that of FIG. 7A. FIG. In FIG. 7B, the imaging surface (imaging area) 70 of the imaging chip 111 includes an image of a straight road, an image of the preceding vehicle 93, and an image of another vehicle 95 traveling in the adjacent lane.
In the image of FIG. 7B, the odd rows correspond to the first area, the even rows correspond to the second area, and the control device 4 determines that the frame rate differs between the first area and the second area. Is set in such a manner that the camera 3A is controlled so as to take an image, as in the case of the image in FIG.

  In general, the driver moves his / her line of sight to a distant place as the speed of the vehicle increases, so that the range of the region 90 of high interest becomes narrower than at low speed. Therefore, the control device 4 controls the third area 81 of the first area on the imaging surface 70, which is the peripheral part of the screen not including the area 90 (around the area 90), and the fourth area 81 at the center excluding the third area 81. Different conditions are set between the region 82 (shaded) and charge accumulation (imaging) is performed.

<When the sun is included in the image>
FIG. 8 is a diagram schematically illustrating an image formed on the imaging chip 111 of the front camera 3A provided in the vehicle 1 traveling toward the morning sun or the sunset. In FIG. 8, an image of the sun 96 is included in the imaging surface (imaging area) 70 of the imaging chip 111.
In the image of FIG. 8, the odd columns correspond to the first region, the even columns correspond to the second region, and the controller 4 sets different frame rates between the first region and the second region. The point that the camera 3A is controlled so as to pick up an image is the same as in the above-described examples.

  In general, the driver not only looks away from the sun 96 when he / she cannot directly look at the sun 96, but also looks further away from the high brightness area 90 around the sun 96. As a result, the degree of attention to the high-luminance area 90 decreases. Therefore, the control device 4 includes, among the first regions on the imaging surface 70, a third region 81 that is a high-luminance region 90 (around the sun 96) and a fourth region 82 (shaded) excluding the third region 81. During this period, different conditions are set and charge accumulation (imaging) is performed.

<When an object moving on the screen is included>
FIG. 9 is a diagram schematically illustrating an image formed on the imaging chip 111 of the front camera 3A provided in the vehicle 1 approaching the intersection where there is no traffic light. In FIG. 9, the imaging surface (imaging area) 70 of the imaging chip 111 includes a vehicle 97 traveling straight from the right to the left of the intersection, and an oncoming vehicle 91 traveling straight at the intersection.
In the image of FIG. 9, the odd rows correspond to the first area, the even rows correspond to the second area, and the control device 4 sets different frame rates between the first area and the second area. The point that the camera 3A is controlled so as to pick up an image is the same as in the above-described examples.

  Generally, the driver moves his or her line of sight in the moving direction of the moving object, so that the degree of attention to the region 90 including the vehicle 97 that has proceeded to the left increases. Therefore, the control device 4 controls the right (front right) third region 81 not including the region 90 and the left fourth region 82 (shaded) excluding the third region 81 among the first regions on the imaging surface 70. And charge accumulation (imaging) is performed by setting different conditions.

<Explanation of flowchart>
Hereinafter, how to determine the third area 81 and the fourth area 82 will be described with reference to flowcharts (FIGS. 10 and 11). FIG. 10 is a flowchart illustrating the overall flow of the control process of the camera 3A executed by the control device 4. A program for executing the processing according to the flowchart of FIG. 10 is stored in the storage unit 4b of the control device 4. For example, when power supply is started from the vehicle 1 (system is turned on) or the engine is started, the control device 4 starts a program for performing the processing in FIG.

  In step S10 in FIG. 10, the control device 4 determines whether or not the flag a = 0. The flag a is a flag that is set to 1 when the initialization has been completed and set to 0 when the initialization has not been completed. When the flag a = 0, the control device 4 makes an affirmative decision in step S10 and proceeds to step S20, and when the flag a ≠ 0, makes a negative decision in step S10 and proceeds to step S30.

  In step S20, the control device 4 performs an initial setting for the camera 3A, and proceeds to step S30. In the initial setting, a predetermined setting for causing the camera 3A to perform a predetermined operation is performed, and 1 is set to the flag a. As a result, the camera 3A sets the same imaging conditions over the entire imaging surface of the imaging device 100, and starts imaging at a frame rate of, for example, 60 frames per second (60 fps).

In step S30, the control device 4 performs an imaging condition setting process, and proceeds to step S40. The imaging condition setting process sets, for the imaging device 100 of the camera 3A, a first area formed by odd-numbered row blocks and a second area formed by even-numbered row blocks. , A process of setting the third area 81 and the fourth area 82 and determining the respective imaging conditions for each area. Details of the imaging condition setting process will be described later. In the present embodiment, the frame rate of the first area is set higher than that of the second area, the gain is increased, the thinning rate is reduced, and the accumulation time is set shorter. In the third area 81, the frame rate is increased, the gain is increased, the thinning rate is reduced, and the accumulation time is set shorter than in the fourth area 82. The camera 3A performs imaging based on this setting, and performs the above-described distance measurement (distance measurement).
Note that it is not necessary to change all of the frame rate, the gain, the thinning rate, the accumulation time, and the like between the first area and the second area, and at least one may be different. Further, it is not necessary to change all of the frame rate, the gain, the thinning rate, the accumulation time, and the like between the third area 81 and the fourth area 82, and at least one of them may be made different.

  In step S40 in FIG. 10, the control device 4 acquires the image data, the distance measurement data, and the information from each unit in the vehicle 1 acquired by the camera 3A after the imaging condition setting process, and proceeds to step S50. In step S50, the control device 4 determines whether or not a setting for displaying information has been made. When the display setting has been performed, the control device 4 makes an affirmative decision in step 50 and proceeds to step S60. When the display setting is not performed, the control device 4 makes a negative determination in step 50 and proceeds to step S70.

  In step S60, the control device 4 sends out display information to the display / playback device 14 (FIG. 1), and proceeds to step S70. The control device 4 causes the display / reproduction device 14 (FIG. 1) to display, for example, an image acquired based on the second area of the image sensor 100 as display information. In addition, when detecting an object (for example, the oncoming vehicle 91 in FIG. 6) from the image acquired based on the first area, the control device 4 displays the image (second image) displayed on the display / reproduction device 14 (FIG. 1). The corresponding target object (an oncoming vehicle in the above example) is highlighted in the images acquired in the two regions. As a mode of making the object stand out, for example, a frame surrounding the object is displayed, the display luminance of the object is increased to display it brightly, blinking, or colored.

The control device 4 may cause the display / playback device 14 to display the following message in addition to the image (the image acquired in the second area) displayed on the display / playback device 14 (FIG. 1). As an example of the message, for example, when a target object (an oncoming vehicle in the above example) is detected from an image acquired based on the first area, a message “an oncoming vehicle is ahead” is provided.
Instead of displaying the message on the display / reproducing device 14 or together with the display of the message, an audio signal for reproducing the message may be transmitted. A sound device of a navigation device (not shown) may be used as the sound reproducing device.

  In step S70, the control device 4 determines whether or not an off operation has been performed. Upon receiving, for example, an off signal (for example, a system off signal or an engine off signal) from the vehicle 1, the control device 4 makes an affirmative determination in step S70, performs a predetermined off process, and ends the process in FIG. For example, when not receiving the off signal from the vehicle 1, the control device 4 makes a negative determination in step S70 and returns to step S30. When returning to step S30, the above processing is repeated.

<Imaging condition setting process>
The details of the imaging condition setting process (S30) will be described with reference to the flowchart in FIG. In step S31 of FIG. 11, the control device 4 inputs the line-of-sight information from the line-of-sight detection device 16 (FIG. 1), and proceeds to step S32A.

  In step S32A, the control device 4 sets a first area composed of odd-numbered row blocks and a second area composed of even-numbered row blocks on the imaging surface 70 of the imaging chip 111, and proceeds to step S32B. Proceed to. In step S32B, the control device 4 sets the area not including the driver's line of sight among the first areas as the third area 81, and sets the area other than the third area 81 as the fourth area 82, and proceeds to step S33. For example, in a case where the driver of the vehicle 1 is gazing exclusively at the front front, as in the case illustrated in FIGS. A fourth region 82 is defined, and a region excluding the central portion is defined as a third region 81.

  In step S33, the control device 4 inputs the line-of-sight information from the line-of-sight detection device 16 (FIG. 1), and determines whether or not there is a line-of-sight movement. For example, when the amount of movement of the line of sight is larger than the predetermined value, the control device 4 makes an affirmative determination in step S33 and proceeds to step S34. When the amount of movement of the line of sight is not larger than the predetermined value, the control device 4 makes a negative determination in step S33 and proceeds to step S36.

  In step S34, the control device 4 determines whether or not the moving direction of the line of sight matches the turning direction of the vehicle 1. The turning direction of the vehicle 1 is detected based on, for example, one of the operation direction of the turn signal switch 11, the rotation operation direction of the steering wheel 10, and a comparison with the image of the previous frame acquired by the camera 3A. When the moving direction of the line of sight matches the turning direction of the vehicle 1, the control device 4 makes an affirmative determination in step S34 and proceeds to step S35. When the moving direction of the line of sight does not match the turning direction of the vehicle 1, the control device 4 makes a negative determination in step S34 and proceeds to step S39.

  In step S35, as described with reference to FIGS. 5 and 6, the control device 4 sets the first area on the imaging surface 70 in the direction opposite to the direction in which the driver's line of sight turns when turning right or turning left. A region located is set as a third region 81, and a region other than the third region 81 (that is, a region in the direction in which the line of sight is directed) is set as a fourth region 82, and the process proceeds to step S3E.

  In step S39, to which the operation proceeds after making a negative determination in step S34, the control device 4 determines whether or not the moving direction of the line of sight matches the moving direction of the image of the object moving on the imaging surface 70. The moving direction of the image of the target object is detected based on, for example, comparison with the image of the previous frame acquired by the camera 3A. When the moving direction of the line of sight matches the moving direction of the image of the target object, the control device 4 makes an affirmative determination in step S39 and proceeds to step S35. When the moving direction of the line of sight does not match the moving direction of the image of the target object, the control device 4 makes a negative determination in step S39 and proceeds to step S3A. In step S35 in which a positive determination is made in step S39 and the process proceeds, as described above with reference to FIG. 9, the control device 4 determines the direction opposite to the left direction in which the line of sight follows the vehicle 97 in the first area on the imaging surface 70. Is set as the third area 81, and an area other than the third area 81 (that is, an area in the direction in which the line of sight is directed) is set as the fourth area 82, and the process proceeds to step S3E.

  In step S3A, the control device 4 determines whether or not the subject image formed on the imaging surface 70 includes a sun image. For example, when there is an area where the value of the image data exceeds the predetermined value, the control device 4 proceeds to step S3B by making an affirmative determination in step S3A, and proceeds to step S3B. If not, the process makes a negative determination in step S3A and proceeds to step S3E. When a negative determination is made in step S3A, the control device 4 maintains the third area 81 and the fourth area 82 set in step S32B.

  In step S3B, the control device 4 determines whether or not the moved line-of-sight position is apart from the position of the image of the sun 96 (FIG. 8). When the line of sight is away from the position corresponding to the sun 96 (the driver is deflecting the line of sight from the sun), the control device 4 makes an affirmative determination in step S3B and proceeds to step S3C. When the gaze position is at a position corresponding to the sun 96 (the driver is looking at the sun), the control device 4 makes a negative determination in step S3B and proceeds to step S3E. When a negative determination is made in step S3B, the control device 4 maintains the third area 81 and the fourth area 82 set in step S32B.

  In step S3C, as described with reference to FIG. 8, the control device 4 controls the high-brightness area 90 (the position of the driver higher than the driver's line of sight) The area near the position of the image 96 is set as the third area 81, and the area other than the third area 81 is set as the fourth area 82, and the process proceeds to step S3D.

  In step S3D, the control device 4 sends an instruction to the camera 3A to cause the area of the image of the sun 96 to be set lower than the gain of the third area 81, and then proceeds to step S3E. Specifically, a gain lower than the gain set for the unit area 131 corresponding to the third area 81 is set for the unit area 131 (FIG. 2) corresponding to the image of the sun 96.

  In step S36 to which the process proceeds after making a negative determination in step S33 described above, the control device 4 determines whether the vehicle 1 is traveling straight and the driver's line of sight is substantially at the center of the front scene. When the vehicle 1 is traveling straight and the driver's line of sight is at a position corresponding to substantially the center of the front scene, the control device 4 makes an affirmative determination in step S36 and proceeds to step S37. If the vehicle 1 is not traveling straight or the driver's line of sight is not at a position corresponding to substantially the center, the control device 4 makes a negative determination in step S36 and proceeds to step S38.

  In step S37, the control device 4 is located in the direction opposite to the central direction of the line of sight in the first area on the imaging surface 70, as described with reference to FIGS. 7A and 7B. The peripheral area is set as a third area 81, and an area other than the third area 81 (that is, a substantially central area of the screen) is set as a fourth area 82, and the process proceeds to step S3E.

  In step S38, to which the operation proceeds after making a negative determination in step S36 described above, the control device 4 determines whether the vehicle 1 is decelerating and the driver's line of sight is substantially at the center of the front scene. The deceleration of the vehicle 1 is detected based on, for example, one of a depression signal of the brake pedal 8a, a decrease in the vehicle speed V, and a comparison with an image of a previous frame acquired by the camera 3A. When the vehicle 1 is decelerating and the driver's line of sight is at a position corresponding to substantially the center of the front subject, the control device 4 makes an affirmative determination in step S38 and proceeds to step S3E. When the vehicle 1 is not decelerating or the driver's line of sight is not at a position corresponding to substantially the center, the control device 4 makes a negative determination in step S38 and proceeds to step S3E.

  In step S3E, the control device 4 sends an instruction to the camera 3A to set the frame rate of the third area 81 higher than the frame rate of the fourth area 82, and proceeds to step S40 in FIG. For example, the frame rate of the third area 81 is 120 frames per second (120 fps), and the frame rate of the fourth area 82 is 30 fps. This is to increase the frequency of acquiring image information for an area where the degree of attention of the driver is low.

<Rear camera>
The control device 4 sets the third area and the fourth area for the camera 3B that images the rear of the vehicle 1 based on the setting information of the third area 81 and the fourth area 82 for the camera 3A that images the front of the vehicle 1. Is performed as follows.
The control device 4 also sets the first region and the second region as shown in FIG. 4A in the camera 3B that captures an image of the rear, and displays the image acquired based on the first region while the vehicle 1 is running. Used to detect roads and objects. Further, the control device 4 uses the image acquired based on the second area for display on the display / reproduction device 14 (FIG. 1). The control device 4 further sets a fourth region and a third region whose attention level is to be raised more than the fourth region in the first region, and stores electric charges (imaging) under different conditions between the third region and the fourth region. ). For example, the frame rate of the third area of the rear camera 3B is 120 fps, and the frame rate of the fourth area of the rear camera 3B is 30 fps.

<When turning left>
FIG. 12 is a diagram schematically illustrating an image of a subject (object) formed on the imaging chip 111 of the rear camera 3B provided in the vehicle 1 about to turn left at the intersection. As shown on the left side of the left side of the vehicle 1, the left and right sides are shown inverted from those in FIGS. In FIG. 12, an imaging surface (imaging area) 70 of the imaging chip 111 includes an image of a road behind the vehicle 1, an image of a motorcycle 98 following the vehicle 1, and an image of a vehicle 99 following the vehicle 1.

  As described above with reference to FIG. 6, the driver of the left-turn vehicle has a higher degree of attention to the front left region 90. Therefore, the control device 4 increases the frequency of obtaining image information of the two-wheeled vehicle 98 that may pass through the left side of the vehicle 1 particularly in the rear where the degree of attention of the driver is low, so as to increase the frequency of obtaining the image information on the imaging surface 70 illustrated in FIG. Charge accumulation (imaging) is performed by setting different conditions between the third area 81 on the rear left side and the fourth area 82 (shaded area) on the right side excluding the third area 81 in one area. According to FIG. 6 and FIG. 12, the front camera 3A when turning left has the third area on the right side toward the front, and the rear camera 3B when turning left has the third area on the right toward the rear.

<When turning right>
In the case of a right turn, the left and right settings may be interchanged with those of a left turn. That is, as described above with reference to FIG. 5, the driver of the right-turn vehicle has a higher degree of attention to the area 90 on the front right side. Therefore, the control device 4 increases the frequency of acquiring the image information of the vehicle passing through the right side of the vehicle 1 particularly in the rear region where the driver's attention level is low, so that the rear right side of the first region on the imaging surface 70 is increased. Different conditions are set between the third region 81 and the left fourth region 82 excluding the third region 81, and charge accumulation (imaging) is performed. That is, the front camera 3A when turning right makes the left area toward the front a third area (FIG. 5), and the rear camera 3B when turning right makes the left area a third area from behind.

  The setting of the third area 81 and the fourth area 82 for the rear camera 3B at the time of left turn and right turn is performed before or after the processing of step S35 (FIG. 11) for the front camera 3A (may be simultaneous). .

<During deceleration>
FIG. 21 is a diagram schematically illustrating an image of a subject (object) formed on the imaging chip 111 of the rear camera 3B provided in the vehicle 1 that is decelerating on a highway, for example. In FIG. 21, an imaging surface (imaging area) 70 of the imaging chip 111 includes an image of a road behind the vehicle 1 and an image of a following vehicle 99. In general, a driver of a vehicle that has applied a brake has a high degree of attention to a region 90 (for example, FIG. 7B) including the front center only. Accordingly, the control device 4 increases the frequency of acquiring image information of the vehicle 99 that may possibly approach the vehicle 1 when the vehicle 1 decelerates, particularly in the rear area where the degree of attention of the driver is low, as illustrated in FIG. Charge accumulation (imaging) by setting different conditions between the substantially central third region 81 and the peripheral fourth region 82 (shaded) excluding the third region 81 among the first regions on the surface 70. Is performed.

The setting of the third area 81 and the fourth area 82 for the rear camera 3B at the time of deceleration is performed before or after the processing of step S37 for the front camera 3A, which proceeds with an affirmative determination in step S38 (FIG. 11). (May be simultaneous).
In the above-described embodiment, the line of sight of the occupant is detected by the line of sight detection device 16, but the frame rate of the entire first area is set high without detecting the line of sight of the occupant to detect the target. It may be performed.

According to the above-described first embodiment, the following operation and effect can be obtained.
(1) An imaging device as an example of an electronic apparatus captures a first moving image and a second moving image outside a vehicle in a first area on an imaging surface and a second area different from the first area under different imaging conditions. 3A, a control device 4 for detecting an object in the first moving image, and a display / reproduction device 14 for displaying information of the object detected by the control device 4 in a second moving image. Thereby, it is possible to set an imaging condition suitable for the detection of the target in the first area. Further, the detected object can be visually notified to the occupant.

(2) An imaging device, which is an example of an electronic device, captures a first moving image and a second moving image outside a vehicle in a first area on an imaging surface and a second area different from the first area under different imaging conditions. 3A, a gaze detection device 16 that detects the gaze position of the driver, and a third region 81 that does not include the gaze position of the first region in the first moving image based on the detection result by the gaze detection device 16. And a display / reproducing device 14 for displaying information of the object detected by the control device 4 as a second moving image. This makes it possible to set an imaging condition suitable for the detection of the target object in the third region 81 with a low degree of attention by the driver. Further, the detected object can be visually notified to the occupant.

(3) Since the camera 3A sets the frame rate of the first area higher than the frame rate of the second area, the camera 3A obtains image information of the first moving image used for detecting the target object as compared with the second moving image used for display. Frequency can be increased.

(4) The imaging device makes the imaging condition of the first moving image different between the third region 81 of the first region that does not include the line of sight and the fourth region 82 that includes the line of sight of the first region. And a control device 4 for setting. Accordingly, it is possible to set imaging conditions suitable for each of the fourth region where the driver's line of sight is directed and the third region where the driver's line of sight is not directed.

(5) The control device 4 sets the frame rate of the third area 81 not including the line-of-sight position of the first area higher than the frame rate of the fourth area 82 including the line-of-sight position of the first area. . This makes it possible to increase the frequency of acquiring image information for the third region where the driver's line of sight is not directed, as compared to the fourth region where the driver's line of sight is directed.

  Although the camera 3A (3B) is controlled by the control device 4 in the above-described embodiment, a part of the control of the camera 3A (3B) is performed by the control unit 35 of the camera 3A (3B). You may.

(Modification 1)
The control device 4 controls the third region 81 and the fourth region 82 with respect to the first region of the camera 3B that images the rear of the vehicle 1 based on the image information acquired by the camera 3A that images the front of the vehicle 1. The setting may be performed.

  For example, a case where an image of the sun 96 as illustrated in FIG. 8 is detected on the right side of the image ahead of the vehicle 1 acquired by the camera 3A will be described. In the first modification, the control device 4 sets the gain for the right side (front right) of the first region on the imaging surface 70 for the camera 3A that images the front of the vehicle 1 on the left side of the first region on the imaging surface 70. Set lower than the gain for the area (front left). On the other hand, for the camera 3B that captures an image of the rear of the vehicle 1, the control device 4 sets the gains of the first area on the imaging surface 70 to the left and right sides (rear left and rear right) equally.

  When the vehicle 1 makes a left turn in the above-described state, there is a possibility that an image of the sun 96 appears in the camera 3B that captures the rear of the vehicle 1 after the vehicle 1 makes a left turn. More specifically, an image of the sun 96 appears in the left area toward the rear of the vehicle 1 after the left turn (that is, the rear right of the vehicle 1). Therefore, the control device 4 sets the gain for the left region (rear right) of the first region on the imaging surface 70 of the camera 3B lower than the gain for the right region (rear left) of the first region on the imaging surface 70. The control device 4 further sets the gain for the left and right sides (front left and front right) of the first area on the imaging surface 70 of the imaging surface 70 of the camera 3A that images the front of the vehicle 1 so as to be equal. Back.

  According to the first modification, the left and right side regions are set in the first region of the camera 3B that captures the rear of the vehicle 1 based on the image information acquired by the camera 3A that captures the front of the vehicle 1, For example, the risk of the image of the sun 96 appearing in the rear camera 3B is predicted in advance (before turning left in this example), and the gain is set immediately low when turning left. Coping is possible.

(Modification 2)
Generally, when the driver's degree of tension increases, the range of movement of the driver's line of sight decreases. Therefore, the control device 4 may increase the range of the third region 81 in the first region on the imaging surface 70 of the front camera 3A in a situation where the driver's tension increases. Examples of a scene in which the driver's degree of tension increases include running at night and running during rainfall. When the headlight is turned on or the wiper is turned on, the control device 4 determines that the driver's degree of tension is increased, and in the first area of the front camera 3A, the third area is higher than the normal area. 81 is expanded.

  Alternatively, the driver's tension may be determined based on the driver's body temperature or heart rate. For example, when a temperature sensor or a heart rate sensor is provided on the driver's seat, and the temperature is detected to be equal to or higher than a predetermined temperature or the heart rate is detected to be equal to or higher than a predetermined number, the control device 4 determines that the driver's tension has increased. Then, control is performed such that the third area 81 is expanded in the first area of the front camera 3A from the normal time.

  According to the second modification, in a case where the region where the driver's attention is low is widened due to the driver's tension (in other words, the region where the driver's attention is high becomes narrow), the third region in the first region is used. 81 can be expanded. If the third region 81 is expanded and the imaging frequency in the third region 81 is made higher than the imaging frequency in the fourth region 82, the decrease in the degree of attention by the driver is determined by the image based on the third region 81. And make up for it.

(Modification 3)
In the above embodiment, the frame rate of the first area of the camera 3A (camera 3B) is set to 120 fps, and the frame rate of the fourth area 82 is set to 30 fps. The example of making the difference is explained. Instead, a buffer area may be provided between the third area 81 and the fourth area 82 so that the frame rate is varied in multiple stages. For example, by setting the frame rate of the buffer area to an intermediate value (for example, 75 fps) between the frame rate of the third area 81 and the frame rate of the fourth area 82, the frame rate is changed in multiple stages.

  Further, the frame rate in the buffer area may be gradually changed. For example, the frame rate of a portion of the buffer region near the boundary with the third region 81 is set to 90 fps, and the frame rate of the portion of the buffer region near the boundary with the fourth region 82 is set to 60 fps. By providing such a buffer area, it is possible to avoid a state where the value of the frame rate greatly changes at the boundary.

(Modification 4)
In Modification 4, a communication unit 17 is added in FIG. The communication unit 17 is, for example, a wireless communication device that performs short-range communication called DSRC (Dedicated Short Range Communication). In the fourth modification, the communication unit 17 acquires information from an information providing system installed on a road, for example.

  The information providing system includes a camera similar to the camera 3A (3B) mounted on the vehicle 1, and is installed beforehand, for example, at an intersection with poor visibility or at a corner of a road. When the vehicle 1 enters a place with poor visibility, the information providing system determines a range that cannot be photographed by the camera 3A (3B) of the vehicle 1 (for example, a blind spot hidden behind a fence or a building when viewed from the vehicle 1). Shoot at a predetermined frame rate. The information providing system transmits to the vehicle 1 a radio signal on which the photographed image information is placed. The communication unit 17 of the vehicle 1 receives the wireless signal transmitted from the information providing system and sends out image information included in the wireless signal to the control device 4.

  For example, when the third region 81 set on the imaging surface 70 of the camera 3A corresponds to the above-described non-capturable range, the control device 4 transmits information not based on the image acquired based on the third region of the camera 3A. An object (pedestrian, two-wheeled vehicle, other vehicle, etc.) is detected based on the image transmitted from the providing system.

  According to the fourth modification, even when a blind spot occurs in the camera 3A (3B) of the vehicle 1, detection of the target object is performed based on an image acquired by an external camera different from the camera 3A (3B) of the vehicle 1. Will be possible.

(Modification 5)
FIG. 14 is a diagram schematically illustrating an image of a subject (object) formed on the imaging chip 111 of the front camera 3 </ b> A provided in the vehicle 1. In FIG. 14, the imaging surface (imaging area) 70 of the imaging chip 111 includes an image of a T-shaped road located in the traveling direction of the vehicle 1. Further on the imaging surface 70, a curve mirror 88A and a curve mirror 88B provided in the T-shaped path are shown.
In the image of FIG. 14, the odd columns correspond to the first region, the even columns correspond to the second region, and the controller 4 sets different frame rates between the first region and the second region. The point that the camera 3A is controlled so as to capture an image is the same as in the above-described embodiment.

  In the fifth modification, when detecting the curve mirror from the image based on the first area acquired by the camera 3A, the control device 4 displays the image of the curve mirror based on the second area acquired by the camera 3A. Display on the playback device 14 (FIG. 1). At this time, the mirror surfaces of the curved mirrors 88A and 88B are greatly enlarged and displayed. When the mirror portions of the curved mirrors 88A and 88B are greatly extended, correction may be made so that the curved images of the mirror surfaces become linear.

  For the enlarged display of the mirror surfaces of the curved mirrors 88A and 88B, a method of lowering the thinning rate of the pixel data used for display by so-called electronic zoom processing may be used. By changing the thinning rate to a low value, the resolution can be increased as compared to before the change.

However, in the fifth modification, the area where the mirror surfaces of the curved mirrors 88A and 88B appear in the second area (the fifth area) is higher than the area other than the mirror surface (the sixth area). An imaging condition suitable for the fine display is set. For example, the thinning rate of the pixels included in the fifth area is set lower than the thinning rate of the pixels included in the sixth area, and the frame rate in the fifth area is set higher than the frame rate in the sixth area. The lower the thinning rate, the higher the resolution. By increasing at least one of the resolution and the frame rate, imaging conditions suitable for high definition display are obtained.
Note that it is not necessary to change all the imaging conditions such as the frame rate and the thinning rate between the fifth area and the sixth area, and at least one may be different.

  FIG. 15 is a diagram exemplifying a display screen of the display / reproduction device 14 (FIG. 1) for displaying an image based on the fifth area and the sixth area. In FIG. 15, for example, a curved mirror 88A is displayed on the left side of the display screen in a greatly expanded state, and a curved mirror 88B is displayed on the right side of the display screen in a greatly expanded state. The shaded area indicates that the image quality is coarser than the area corresponding to the mirror surfaces of the curved mirrors 88A and 88B. The difference in image quality is caused by setting different imaging conditions between the fifth area and the sixth area.

  The control device 4 further displays and displays an object (for example, the motorcycle 98 and the vehicle 99 in FIG. 15) from an area where the curve mirrors 88A and 88B appear in the image acquired based on the fifth area. In the image (FIG. 15) displayed on the playback device 14 (FIG. 1), the corresponding objects (the two-wheeled vehicle 98 and the vehicle 99 in the above example) stand out. As a mode of making the object stand out, for example, a frame surrounding the object is displayed, the display luminance of the object is increased to display it brightly, blinking, or colored.

The control device 4 may cause the display / reproduction device 14 to display the following message on the image (FIG. 15) displayed on the display / reproduction device 14 (FIG. 1). As an example of the message, for example, a message such as "a motorcycle is reflected" or "a vehicle is reflected".
Instead of displaying the message on the display / reproducing device 14 or together with the display of the message, an audio signal for reproducing the message may be transmitted. A sound device of a navigation device (not shown) may be used as the sound reproducing device.

<Explanation of flowchart>
Hereinafter, a method of determining the third area 81 and the fourth area 82 in Modification Example 5 will be described with reference to a flowchart (FIG. 16). FIG. 16 is a flowchart illustrating details of the imaging condition setting process (S30B). The control device 4 executes an imaging condition setting process (S30B) following the imaging condition setting (S30) of FIG. 11 or instead of the imaging condition setting (S30) of FIG.

In step S301 in FIG. 16, the control device 4 sets a first area formed by odd-numbered row blocks and a second area formed by even-numbered row blocks on the imaging surface 70 of the imaging chip 111. To step S302.
Note that, when the imaging condition setting of FIG. 16 is performed subsequent to the imaging condition setting (S30) of FIG. 11, step S301 is omitted.

  In step S302, the control device 4 determines whether a curve mirror has been detected from an image based on the first region of the image sensor 100. The detection of the curve mirror is performed by using a template matching method, or by determining that the mirror surface of the curve mirror is located at a predetermined height from the road surface. When detecting the curve mirror from the image based on the first area, the control device 4 makes an affirmative determination in step S302 and proceeds to step S303. If the control device 4 does not detect the curve mirror from the image based on the first area, the control device 4 makes a negative determination in step S302 and proceeds to step S306.

  In step S303, the control device 4 sets the area corresponding to the mirror surface of the curve mirror in the second area as the fifth area, and sets the area other than the fifth area as the sixth area, and proceeds to step S304.

  In step S304, the control device 4 sends an instruction to the camera 3A to set the frame rate of the fifth area higher than the frame rate of the sixth area, and proceeds to step S305. For example, the frame rate of the fifth area is set to 120 frames per second (120 fps), and the frame rate of the sixth area is set to 30 fps. This is to increase the frequency of acquiring image information reflected on the mirror surface of the curve mirror which is difficult for the driver to understand.

  In step S305, the control device 4 sends an instruction to the camera 3A to set the thinning rate of the pixels included in the fifth area to be lower than the thinning rate of the pixels included in the frame rate of the sixth area. Proceed to step S40. This is to obtain high-definition image information reflected on the mirror surface of the curve mirror which is difficult for the driver to understand.

  In step S306, to which the operation proceeds after making a negative determination in step S302, the control device 4 sends an instruction to the camera 3A to set uniform imaging conditions for the entire second area, and then proceeds to step S40 in FIG. For example, the frame rate is set to 30 fps, and a normal thinning rate is set.

  In the fifth modification, the control device 4 performs the following step 60B instead of step S60 in FIG. That is, in step S60B, the control device 4 sends display information to the display / playback device 14 (FIG. 1). The control device 4 causes the display / playback device 14 (FIG. 1) to display, for example, an image acquired based on the fifth region and the sixth region of the image sensor 100 as display information. Specifically, as illustrated in FIG. 15, the curve mirrors 88A and 88B of the image acquired in the fifth area are displayed on the display screen of the display / playback apparatus 14 (FIG. 1) while being greatly stretched.

The control device 4 further detects and displays an object (for example, the motorcycle 98 and the vehicle 99 in FIG. 15) from an area in which the curve mirrors 88A and 88B appear in an image acquired based on the fifth area. The corresponding objects (the two-wheeled vehicle 98 and the vehicle 99 in the above example) are highlighted in the image displayed on 14 (FIG. 1).
Since the image reflected on the mirror surface of the curve mirror is a mirror image, the control device 4 may display an image obtained by inverting the image displayed on the display / reproduction device 14 (FIG. 1).

  According to the fifth modification, it is possible to increase the frequency of acquiring image information reflected on the mirror surfaces of the curve mirrors 88A and 88B which are difficult for the driver to understand. Further, it is possible to obtain image information reflected on the mirror surfaces of the curve mirrors 88A and 88B, which are difficult for the driver to understand, with high definition. Then, the acquired image information reflected on the mirror surfaces of the curve mirrors 88A and 88B can be displayed on the display / reproducing device 14 (FIG. 1) in an easy-to-understand manner.

(Modification 6)
In the above description, as the distance measurement performed by the cameras 3A and 3B, a method of calculating by distance measurement using an image signal from a focus detection pixel provided in the image sensor 100 is used. A method of performing distance measurement using two images may be used. Alternatively, a method of performing distance measurement using a millimeter wave radar separately from the cameras 3A and 3B may be used.

(Second embodiment)
In the second embodiment, a radar 18 is added to the driving support device 2 in FIG. The radar 18 is constituted by, for example, a millimeter wave radar device. The radar 18 detects an object or an obstacle in each area in association with a plurality of areas (referred to as areas) on the shooting screen of the camera 3A. As a method of detecting for each area, a method of scanning the area to be detected while changing the direction of the radar antenna (not shown) with an actuator, or the area of the detection target while switching multiple receiving antennas corresponding to each area with a switch etc. , But any of these methods may be used.

The communication unit 17 according to the second embodiment performs wireless communication (optical beacon, radio wave) with an external device such as an information provision system installed on a road or a vehicle information and communication system (VICS). Beacon and visible light communication). Any communication system may be used.
Note that, in the second embodiment, the camera 3B for photographing the rear is not an essential component.

<Explanation of camera>
In the second embodiment, a driving unit 37 is added in the block diagram of FIG. The drive unit 37 is configured by, for example, a motor. The drive unit 37 changes the horizontal direction (yaw angle) of the camera 3A according to an instruction from the control unit 35. The direction of the camera 3A is adjusted so that the camera 3A captures an object far in front of the vehicle 1 substantially at the center of the screen, that is, with the vanishing point at substantially the center of the screen. The control unit 35 controls the yaw angle based on this direction.

<Block control of image sensor>
The control device 4 causes the imaging device 100 (imaging chip 111) of the camera 3A to perform independent accumulation control for each of the unit areas 131 described above. Specifically, the control device 4 sets a non-priority area, a priority area, and a top priority area on the imaging surface of the imaging chip 111, and causes charge accumulation (imaging) to be performed under different conditions among these areas. Here, in addition to the above-described control parameters at the time of imaging, the size and position of the non-priority area, the priority area, the highest priority area, whether or not to perform distance measurement, and the yaw angle of the imaging unit 32 are also one of the imaging conditions. It is.

FIG. 17 is a diagram schematically illustrating an image of a subject (object) formed on the image sensor 100 of the camera 3A that images the front of the vehicle 1. Although an inverted inverted image is actually formed, it is illustrated as an erect upright image for easy understanding.
In some drawings referred to in the description of the second embodiment, reference numerals are common to those used in the above-described drawings, but the description of the present embodiment is applied.

FIG. 17 illustrates an imaging surface of the imaging chip 111, a non-priority area 81, a priority area 82, and a top priority area 83 in the imaging chip 111. The non-priority area 81, the priority area 82, and the highest priority area 83 perform charge accumulation (imaging) under different conditions.
Note that a pause area in which charge accumulation (imaging) is not performed in the imaging chip 111 may be provided.

  The control device 4 sets the non-priority area 81, the priority area 82, and the highest priority area 83 based on the above-described depth distribution, the classification result of the object and the obstacle, and the like. For example, the non-priority area 81 is set so as to reduce the degree of attention to a distant object, and the area including the preceding vehicle 70 is set as the highest priority area 83 so as to increase the degree of attention to the preceding vehicle 70. The priority area 82 is set so that the area other than the non-priority area 81 and the highest priority area 83 has a medium degree of attention.

  FIG. 18 is a diagram schematically illustrating an image of a subject (object) when the entire area of the imaging surface of the imaging element 100 is set as the priority area 82. In FIG. 18, the control device 4 sets the entire area of the imaging surface of the imaging chip 111 as the priority area 82 (common control for all areas). The control device 4 determines that the distance from the vehicle 1 to the subject is equal to or greater than a predetermined threshold (for example, 300 m) based on the distance image (depth distribution image) generated based on the distance measurement data at a plurality of positions in the shooting screen. Region to be extracted.

  FIG. 19 is a diagram illustrating a case where a part of the imaging surface is changed to the non-priority area 81 from the state of FIG. The control device 4 changes the extraction area (that is, the distance from the vehicle 1 to the subject to be equal to or more than 300 m) to the non-priority area 81. The control device 4 sets different control parameters between the non-priority area 81 and the priority area 82, and also sets not to read image signals for distance measurement in the non-priority area 81. When reading out the image signal for distance measurement is not performed, power consumption is reduced.

For example, the control device 4 sets a lower frame rate for the unit area 131 (FIG. 2) of the image sensor 100 included in the non-priority area 81 than in the case of the priority area 82. It is based on the idea of reducing the degree of attention for distant places.
Further, the control device 4 may be set so that image processing such as tracking processing by the image processing unit 33 is not performed on an image acquired in the non-priority area 81.

FIG. 20 is a diagram illustrating a case where a part of the imaging surface is changed to the highest priority area 83 from the state of FIG. When the vehicle 1 travels at the intersection (approaches the intersection, enters the intersection, and exits the intersection), the control device 4 changes the entire area of the imaging surface from the priority area 82 to the highest priority area 83. It is based on the idea of increasing the degree of attention when traveling at an intersection. However, an area including the pedestrian crossing 92 that passes through the intersection when turning left (or when turning right) is left as the priority area 82. The reason why the pedestrian crossing 92 is not included in the highest priority area 83 is as follows. That is, in general, the occupant driving the vehicle 1 pays attention to the pedestrian crossing 92 passing therethrough, and is based on the idea that it is not necessary to increase the degree of attention to the pedestrian crossing 92 by the camera 3A. In the present embodiment, rather, the area of low attention by the driving occupant is set as the highest priority area 83, thereby increasing the degree of attention by the camera 3A. Thereby, the degree of attention to another vehicle such as the vehicle 91 turning right from the oncoming lane in FIG. 20 can be increased.
The control device 4 may change an area in which the distance from the vehicle 1 to the subject is equal to or greater than a predetermined threshold (for example, 300 m) to the non-priority area 81 based on the distance image (depth distribution image). .

  For example, the control device 4 sets a higher frame rate for the unit area 131 (FIG. 2) of the image sensor 100 included in the highest priority area 83 compared to the case of the priority area 82. When the non-priority area 81 is set, the frame rate of the unit area 131 (FIG. 2) of the image sensor 100 included in the non-priority area 81 is set lower than that of the priority area 82. .

  In addition to the area including the pedestrian crossing 92 recognized based on the image acquired by the camera 3A as the priority area 82, the control device 4 also pays attention to the occupant based on the line-of-sight information sent from the line-of-sight detection device 16. The area may be the priority area 82.

  The depth distribution image used when the control device 4 sets the non-priority area 81, the priority area 82, and the top priority area 83 is the first depth distribution image generated based on the distance measurement data based on the distance measurement image signal. Alternatively, a second depth distribution image generated based on detection information of each area by the radar 18 may be used. In the present embodiment, the first depth distribution image is used when the movement of the image on the imaging surface of the image sensor 100 (referred to as the image plane movement amount) is equal to or smaller than a predetermined value, and when the image plane movement amount exceeds the predetermined value. The second depth distribution image is used. Details of the image plane movement amount will be described later.

<Explanation of flowchart>
Hereinafter, the flow of the control process of the camera 3A executed by the control device 4 will be described with reference to flowcharts (FIGS. 21, 22, 23, and 26). The program executed by the control device 4 is stored in the storage unit 4b of the control device 4. For example, when power supply is started from the vehicle 1 or the engine is started, the control device 4 starts a program for performing the processing in FIG. 21.

  In step S10 of FIG. 21, the control device 4 determines whether or not the flag a = 0. The flag a is a flag that is set to 1 when the initialization has been completed and set to 0 when the initialization has not been completed. When the flag a = 0, the control device 4 makes an affirmative decision in step S10 and proceeds to step S20, and when the flag a ≠ 0, makes a negative decision in step S10 and proceeds to step S30.

  In step S20, the control device 4 performs an initial setting for the camera 3A, and proceeds to step S30. The initial setting is to perform a predetermined setting for causing the camera 3A to perform a predetermined operation. In this example, the camera 3A sets the same imaging conditions over the entire area of the imaging surface of the imaging element 100, and starts imaging at a frame rate of, for example, 60 frames per second (60 fps).

In step S30, the control device 4 performs an imaging condition setting process, and proceeds to step S40. In the imaging condition setting processing, a non-priority area 81 and a top priority area 83 are set in addition to the priority area 82, and imaging conditions are set for the unit area 131 corresponding to each area. Details of the imaging condition setting process will be described later.
When setting the respective imaging conditions for the non-priority area 81, the priority area 82, and the highest priority area 83, it is not necessary to make all of the frame rate, the gain, the thinning rate, the accumulation time, and the like different between the respective areas. However, at least one may be different.

  In step S40, the control device 4 sends an instruction to the camera 3A, and drives the unit areas 131 of the image sensor 100 corresponding to the non-priority area 81, the priority area 82, and the top-priority area 83 under predetermined conditions, respectively. Acquisition and ranging calculation are performed. Thereby, distance measurement information based on the image data and the image signal for distance measurement is obtained. Further, the control device 4 causes the image processing unit 33 or the control unit 35 of the camera 3A to perform predetermined image processing and image analysis.

  In step S50, the control device 4 determines whether or not a setting for displaying information has been made. When the display setting is performed, the control device 4 makes an affirmative determination in step S50, and proceeds to step S60. When the display setting is not performed, the control device 4 makes a negative determination in step S50 and proceeds to step S70.

In step S60, the control device 4 sends out display information to the display / playback device 14 (FIG. 1), and proceeds to step S70. The display information is information according to the state of the vehicle 1 analyzed in the imaging condition setting process (S30), for example, "an oncoming vehicle is waiting for a right turn", "emergency stop", "turn right". , "Turn left" is displayed on the display / reproduction device 14, a navigation device (not shown), or the like.
Instead of transmitting the display information or together with the transmission of the display information, an audio signal for reproducing the message may be transmitted to an audio reproducing device (not shown). Also in this case, a sound reproducing unit of a navigation device (not shown) may be used as the sound reproducing device.

  In step S70, the control device 4 determines whether or not an off operation has been performed. When receiving an off signal (for example, an engine off signal) from vehicle 1, for example, control device 4 makes an affirmative determination in step S70, performs a predetermined off process, and ends the process in FIG. For example, when not receiving the off signal from the vehicle 1, the control device 4 makes a negative determination in step S70 and returns to step S30. When returning to step S30, the above processing is repeated.

<Imaging condition setting process>
The details of the imaging condition setting process (S30) will be described with reference to the flowchart in FIG. In the present embodiment, the image plane movement amount and depth distribution of an object captured by the image sensor 100, the driving state of the vehicle 1 (change in vehicle speed V, operating state of a steering wheel, and the like), and traveling environment (road conditions, running conditions, etc.) The imaging conditions are set for the unit areas 131 of the image sensor 100 corresponding to the non-priority area 81, the priority area 82, and the top priority area 83, respectively, in correspondence with the lanes.

In step S31 of FIG. 22, the control device 4 acquires distance information based on an image signal acquired in a state where the same imaging condition is set over the entire imaging surface of the imaging element 100, and sets 0 to a flag b. Proceed to step S32. The distance information is the first depth distribution image. The flag b is a flag that is set to 1 when a predetermined value (for example, 1.0) is substituted for a correction value α described later, and is set to 0 when a 0 is substituted for the correction value α. Here, a flag b = 0 is set as an initial condition.
Note that the second depth distribution image based on detection information of each area by the radar 18 may be used as the distance information.

  In step S32, the control device 4 determines whether or not the distance d from the vehicle 1 to the target is longer than a predetermined distance D (for example, 300 m). The control device 4 divides the shooting screen into a plurality of areas (for example, 8 × 8 = 64 areas) based on the distance information, and determines whether d> D holds for each area. I do. When a plurality of objects exist in the area, the distance to the closest subject is set as the distance d in the area.

  When at least one area where d> D is satisfied, the control device 4 makes an affirmative determination in step S32 and proceeds to step S33. When there is no area where d> D is satisfied, the control device 4 makes a negative determination in step S32 and proceeds to step S34. If a negative determination is made in step S32, the setting of the priority area 82 is maintained for the entire area of the imaging surface of the image sensor 100 (common control for all areas).

  In step S33, the control device 4 performs area-specific control for setting a priority area 82 and a non-priority area 81 on the imaging surface of the imaging element 100. The control device 4 sets the area where d> D is satisfied as the non-priority area 81, and maintains the setting of the priority area 82 for the area where d> D is not satisfied, and proceeds to step S34.

  In step S34, the control device 4 determines whether the vehicle 1 is outside the area of the intersection. The control device 4 refers to position information from the GPS device 15, map information from a navigation device (not shown), and external information such as a white line, a pedestrian crossing, and a traffic light recognized from an image acquired by the camera 3 </ b> A. If it is determined that No. 1 is outside the intersection, a positive determination is made in step S34, and the process proceeds to step S35 in FIG. When determining that the vehicle 1 is inside the intersection, the control device 4 makes a negative determination in step S34 and proceeds to step S301 in FIG.

FIG. 23 is a flowchart illustrating a process when the vehicle 1 travels outside the intersection. In step S35 of FIG. 23, the control device 4 calculates the image plane movement amount Δdiff. The image plane moving amount Δdiff is the moving amount of the image on the imaging surface of the image sensor 100 as described above. The control device 4 calculates the image plane movement amount Δdiff as in the following equation (1) using, for example, the vehicle speed V, the rotation angle θ of the steering wheel, and the predetermined coefficient μ, and proceeds to step S36.
Δdiff = (μ × V) × θ (1)

  In step S36, the control device 4 determines whether or not the turn signal device is off. For example, when the control device 4 determines that the turn signal device is off based on the operation signal of the turn signal switch 11 (FIG. 1), the control device 4 makes an affirmative determination in step S36 and proceeds to step S37. If the control device 4 determines that the turn signal device is ON, the control device 4 makes a negative determination in step S36 and proceeds to step S38.

When proceeding to step S37, it is assumed that the vehicle 1 travels along a road outside the intersection. In step S37, the control device 4 sets the imaging conditions of the imaging device 100 as follows. That is, the frame rate is set for each of the non-priority area 81, the priority area 82, and the top priority area 83 based on the image plane movement amount Δdiff calculated in step S35 with reference to FIG.
When the three areas of the non-priority area 81, the priority area 82, and the highest-priority area 83 are not set, for example, when the entire area of the imaging surface is set as the priority area 82, the control device 4 performs the processing shown in FIG. The setting is performed for the corresponding area in (A).

  According to FIG. 24A, four frame rates can be set so that different degrees of attention can be set for the object depending on the difference in the absolute value of the image plane movement amount Δdiff. Further, four frame rates can be set for the non-priority area 81, the priority area 82, and the top priority area 83, respectively. In general, as the vehicle speed V of the vehicle 1 increases, and as the radius of curvature of the vehicle 1 traveling decreases (the rotation angle θ of the steering wheel increases), the flow of an image (optical flow) captured on the image sensor 100 increases. By setting the frame rate in accordance with the magnitude of the absolute value of the image plane movement amount Δdiff, it becomes possible to set a frame rate suitable for a change in an image captured on the image sensor 100.

  In addition, the frame rate can be individually set for each of the non-priority area 81, the priority area 82, and the top priority area 83, so that each of the non-priority area 81, the priority area 82, and the top priority area 83 It is possible to set a frame rate suitable for each time.

The movable range of the yaw angle Y in FIG. 24A is a movable range when the above-described driving unit 37 changes the yaw angle Y of the camera 3A in accordance with an instruction from the control unit 35. Generally, when the vehicle 1 turns to the left, the flow of an image (optical flow) reflected on the image sensor 100 increases from left to right on the screen. Conversely, when the vehicle 1 turns right, the flow of the image shown on the image sensor 100 increases from right to left on the screen. For this reason, the control device 4 performs yaw angle control that moves the shooting direction of the camera 3A according to the magnitude and direction of the rotation angle θ of the steering wheel so that the image on the upstream side of the optical flow is quickly captured on the image sensor 100. Do. Using the rotation angle θ of the steering wheel and the predetermined coefficient λ, the yaw angle Y is expressed by the following equation (2).
Y = λ × θ (2)
However, since excessive movement of the yaw angle Y is not appropriate, in the present embodiment, the movable range of the yaw angle Y is limited by defining the movable range of the yaw angle Y.

  According to FIG. 24A, the yaw angle movable range is set to 4 so that the limit value of the movable range of the yaw angle Y increases stepwise as the absolute value of the image plane movement amount Δdiff calculated in step S35 increases. Determined in stages.

  Since the flag b = 0 in step S37, the correction value α in FIG. 24A is zero. That is, the correction for narrowing the movable range of the yaw angle Y is not performed.

  FIG. 25 is a schematic diagram illustrating the yaw angle 30 of the camera 3A. Generally, when the turning speed of the vehicle 1 is high, the image plane movement amount Δdiff (absolute value) becomes large, so that the imaging unit yaw angle 30 (absolute value) is also greatly changed. FIG. 25 illustrates the case where the vehicle 1 turns to the left, and the image plane movement amount Δdiff is positive in the right direction. The control device 4 controls the yaw angle 30 so as to direct the camera 3A in the turning direction of the vehicle 1 (left in this example).

  Further, as illustrated in FIG. 24A, when the absolute value of the image plane movement amount Δdiff is equal to or more than a predetermined value (for example, 20), the control device 4 obtains distance information using the radar 18. That is, the second depth distribution image based on the detection information of each area by the radar 18 is used for the determination processing in step S32. As described above, when the absolute value of the image plane movement amount Δdiff is large, the distance information is obtained by using the radar 18 having better responsiveness.

  After changing the imaging conditions for setting the imaging conditions of the imaging device 100 as described above, the control device 4 ends the processing in FIG. 23 and proceeds to step S40 in FIG.

When the determination in step S36 is negative and the process proceeds to step S38, it is assumed that the vehicle 1 changes the course by operating a turn signal device outside the intersection, for example, in an alley. In step S38, the control device 4 sets the imaging conditions of the imaging device 100 as follows. That is, a frame rate is set for each of the non-priority area 81, the priority area 82, and the top priority area 83 based on the image plane movement amount Δdiff calculated in step S35 with reference to FIG.
When the three areas of the non-priority area 81, the priority area 82, and the highest-priority area 83 are not set, for example, when the entire area of the imaging surface is set as the priority area 82, the control device 4 performs the processing shown in FIG. The setting is performed for the corresponding area in (B).

  According to FIG. 24B, attention is paid to the priority area 82 and the highest priority area 83 by setting the frame rate for the priority area 82 and the highest priority area 83 higher than in the case of FIG. The configuration is such that the degree is higher than in the case of FIG.

  The movable range of the yaw angle Y in FIG. 24B is a movable range when the above-described driving unit 37 changes the yaw angle Y of the camera 3A according to an instruction from the control unit 35. According to FIG. 24B, the limit value of the movable range of the yaw angle Y is set as shown in FIG. 24B so that the image on the upstream side of the optical flow is captured earlier on the image sensor 100 than in the case of FIG. It is configured to be larger than the case of A).

  Since the flag b = 0 in step S38, the correction value α in FIG. 24B is zero. That is, the correction for narrowing the movable range of the yaw angle Y is not performed.

  FIG. 26 is a flowchart illustrating a process when the vehicle 1 runs inside an intersection. In step S301 of FIG. 26, the control device 4 calculates the image plane movement amount Δdiff as in step S35, and proceeds to step S302.

  In step S302, the control device 4 acquires traffic information transmitted from an information providing system (not shown) installed on the road via the communication unit 18 and proceeds to step S303. As a result, the vehicle 1 acquires the presence or absence of a vehicle in the oncoming lane at the intersection, information on turning left and right of the vehicle, information on a signal displayed on a traffic signal at the intersection, and the like.

  In step S303, the control device 4 refers to position information from the GPS device 15, map information from a navigation device (not shown), and external information such as a white line, a pedestrian crossing, and a traffic light recognized from the image acquired by the camera 3A. Then, it is determined whether the vehicle 1 is in the straight lane, the left turn lane, or the right turn lane of the intersection, and the process proceeds to step S304, S307, or S313, respectively.

<For straight lane>
In step S304, the control device 4 determines whether there is an oncoming right turn vehicle at the intersection. The controller 4 determines whether there is a right-turning vehicle in the oncoming lane based on the traffic information acquired in step S302, and also determines whether there is a right-turning vehicle in the oncoming lane based on the image acquired by the camera 3A. When the control device 4 determines that there is a right-turning vehicle in the oncoming lane in at least one of the above determinations, the control device 4 makes an affirmative determination in step S304 and proceeds to step S305. If the control device 4 does not determine that a right-turning vehicle exists in the oncoming lane, the control device 4 makes a negative determination in step S304 and proceeds to step S306.

  In step S305, the control device 4 sets the imaging conditions of the imaging device 100 as follows. That is, the entire area of the imaging surface is set as the highest priority area 83 (all area common control), and based on the image plane movement amount Δdiff calculated in step S301, referring to FIG. Set the frame rate for.

  The movable range of the yaw angle Y and the change of the yaw angle Y are the same as in the case of step S38. Since the flag b = 0 in step S305, the correction for narrowing the movable range of the yaw angle Y is not performed as in the case of step S38.

  Further, as illustrated in FIG. 24B, when the absolute value of the image plane movement amount Δdiff is equal to or larger than a determination threshold (for example, 10) smaller than that in FIG. To obtain distance information. That is, when the image plane movement amount Δdiff is smaller than that in FIG. 24A, switching is performed so as to obtain the distance information using the radar 18.

  After changing the imaging conditions for setting the imaging conditions of the image sensor 100 as described above, the control device 4 ends the processing in FIG. 26 and proceeds to step S40 in FIG.

  In step S306, the control device 4 sets the imaging conditions of the imaging device 100 as follows. That is, the entire area of the imaging surface is set as the highest priority area 83 (common control for all areas), and based on the image plane movement amount Δdiff calculated in step S301, referring to FIG. Set the frame rate for.

The movable range of the yaw angle Y and the change of the yaw angle Y are the same as in the case of step S37. Since the flag b = 0 in step S306, the correction for narrowing the movable range of the yaw angle Y is not performed similarly to the case of step S37. Further, as exemplified in FIG. 24A, the control device 4 determines whether the radar 18 has the absolute value of the image plane moving amount Δdiff which is equal to or larger than a determination threshold (for example, 20) larger than that in FIG. To obtain distance information. That is, when the image plane movement amount Δdiff is larger than that in FIG. 24B, switching is performed so as to obtain distance information using the radar 18.
After changing the imaging conditions for setting the imaging conditions of the image sensor 100 as described above, the control device 4 ends the processing in FIG. 26 and proceeds to step S40 in FIG.

<In case of left turn lane>
In step S307, the control device 4 determines whether or not the oncoming right turn vehicle exists at the intersection. The controller 4 determines whether there is a right-turning vehicle in the oncoming lane based on the traffic information acquired in step S302, and also determines whether there is a right-turning vehicle in the oncoming lane based on the image acquired by the camera 3A. When the control device 4 determines that there is a right-turning vehicle in the oncoming lane in at least one of the above determinations, the control device 4 makes an affirmative determination in step S307 and proceeds to step S308. If the control device 4 does not determine that there is a right-turning vehicle in the oncoming lane, the control device 4 makes a negative determination in step S307 and proceeds to step S309.

  In step S308, the control device 4 performs a yaw angle correction setting. The control device 4 sets 1 to the flag b and proceeds to step S309. As a result, the process of correcting the value of the yaw angle Y with the correction value α becomes effective. When the movable range of the yaw angle Y is increased as the absolute value of the image plane movement amount Δdiff increases, the direction of the camera 3A moves in the turning direction of the vehicle 1, while the other vehicle in the oncoming lane is shot by the camera 3A. There is also a high risk of departure. Therefore, when another vehicle is present in the oncoming lane, the control device 4 provides a correction value α so that the movable range of the yaw angle Y is narrower than when the other vehicle does not exist, and the yaw angle Y Correct the movable range. This is because the camera 3A surely acquires the information of the vehicle in the oncoming lane while turning the camera 3A in the turning direction of the vehicle 1.

  In step S309, the control device 4 sets the imaging conditions of the imaging device 100 as follows. That is, the entire area of the imaging surface is set as the highest priority area 83 (all area common control), and based on the image plane movement amount Δdiff calculated in step S301, referring to FIG. Set the frame rate for.

  Since the flag b = 1, as described above, the control device 4 corrects the limit value of the movable range of the yaw angle with the correction value α to narrow it.

  In step S310, the control device 4 determines whether or not the pedestrian signal of the pedestrian crossing to be passed when turning left is a crossing permission signal (for example, a green signal). The control device 4 determines the presence or absence of a crossing permission signal based on the traffic information acquired in step S302, and also determines the presence or absence of a green signal based on an image acquired by the camera 3A. If the control device 4 determines that the signal is a crossing permission signal in at least one of the above determinations, the control device 4 makes an affirmative determination in step S310 and proceeds to step S311. If the control device 4 does not determine that the signal is a traversing permission signal, the control device 4 makes a negative determination in step S311 and ends the processing in FIG. 26 and proceeds to step S40 in FIG.

In step S311, the control device 4 performs area-specific control for setting a priority area 82 and a highest priority area 83 on the imaging surface of the image sensor 100 based on the line-of-sight information sent from the line-of-sight detection device 16. The control device 4 sets the area corresponding to the occupant's line of sight as the priority area 82, and the area not corresponding to the occupant's line of sight maintains the highest priority area 83, and proceeds to step S312.
As described above, the control device 4 may set the area including the crosswalk 92 recognized based on the image acquired by the camera 3A as the priority area 82 (FIG. 20).

  In step S312, the control device 4 sets the imaging conditions of the imaging device 100 as follows. That is, based on the image plane movement amount Δdiff calculated in step S301, the frame rates for the priority area 82 and the highest priority area 83 are set with reference to FIG.

  Since the flag b = 1, as described above, the control device 4 corrects the limit value of the movable range of the yaw angle Y by using the correction value α to narrow it. The control device 4 ends the process in FIG. 26 and proceeds to step S40 in FIG.

<In case of right turn lane>
In step S313, the control device 4 determines whether an oncoming straight-ahead vehicle exists at the intersection. The control device 4 determines whether there is a straight-ahead vehicle in the oncoming lane based on the traffic information acquired in step S302, and also determines whether there is a straight-ahead vehicle in the oncoming lane based on the image acquired by the camera 3A. If the control device 4 determines that at least one of the above determinations indicates that a straight-ahead vehicle exists in the oncoming lane, it makes an affirmative determination in step S313 and proceeds to step S314. If the control device 4 does not determine that a straight-ahead vehicle exists in the oncoming lane, the control device 4 makes a negative determination in step S313 and proceeds to step S315.

  In step S314, the control device 4 performs a yaw angle correction setting. The control device 4 sets 1 to the flag b and proceeds to step S315. As a result, the process of correcting the value of the yaw angle Y with the correction value α becomes effective. This is because the camera 3A surely acquires the information of the vehicle in the oncoming lane while turning the camera 3A in the turning direction of the vehicle 1.

  In step S315, the control device 4 sets the imaging conditions of the imaging device 100 as follows. That is, the entire area of the imaging surface is set as the highest priority area 83 (all area common control), and based on the image plane movement amount Δdiff calculated in step S301, referring to FIG. Set the frame rate for.

  Since the flag b = 1, as described above, the control device 4 corrects the limit value of the movable range of the yaw angle with the correction value α to narrow it.

  In step S316, the control device 4 determines whether or not the pedestrian signal of the pedestrian crossing to be passed when turning right is a crossing permission signal (for example, a green signal). The control device 4 determines the presence or absence of a crossing permission signal based on the traffic information acquired in step S302, and also determines the presence or absence of a green signal based on an image acquired by the camera 3A. When the control device 4 determines that the signal is the crossing permission signal in at least one of the above determinations, the control device 4 makes an affirmative determination in step S316 and proceeds to step S317. If the control device 4 does not determine that the signal is a crossing permission signal, the control device 4 makes a negative determination in step S316, ends the process in FIG. 26, and proceeds to step S40 in FIG.

In step S317, the control device 4 performs area-specific control for setting the priority area 82 and the highest priority area 83 on the imaging surface of the image sensor 100 based on the visual line information sent from the visual line detection device 16. The control device 4 sets the area corresponding to the occupant's line of sight as the priority area 82, and maintains the highest priority area 83 for the area not corresponding to the occupant's line of sight, and proceeds to step S318.
As described above, the control device 4 may set the area including the pedestrian crossing recognized based on the image acquired by the camera 3A as the priority area 82.

  In step S318, the control device 4 sets the imaging conditions of the imaging device 100 as follows. That is, based on the image plane movement amount Δdiff calculated in step S301, the frame rates for the priority area 82 and the highest priority area 83 are set with reference to FIG.

  Since the flag b = 1, as described above, the control device 4 corrects the limit value of the movable range of the yaw angle Y by using the correction value α to narrow it. The control device 4 ends the process in FIG. 26 and proceeds to step S40 in FIG.

According to the above-described second embodiment, the following operation and effect can be obtained.
(1) The vehicle 1 has a first depth distribution image based on the output from the pixels of the imaging device 100 and the imaging unit 32 including the imaging device 100 capable of independently setting the imaging conditions of the plurality of unit areas 131. And a control device 4 for setting the imaging conditions of the plurality of unit areas 131 based on the first depth distribution image. Accordingly, for example, an area in which the distance from the vehicle 1 to the subject is equal to or greater than a predetermined threshold (for example, 300 m) is set as the non-priority area 81, and the other areas are set as priority areas 82. Conditions can be set.

(2) The movable vehicle 1 includes the steering wheel 10, and the control device 4 controls the plurality of unit areas 131 based on the vehicle speed V of the vehicle 1, the rotation angle θ of the steering wheel, and the first depth distribution image. Is set. This makes it possible to set appropriate imaging conditions for the imaging unit 32 based on the image plane movement amount Δdiff and the first depth distribution image.

(3) The vehicle 1 moves the imaging unit 32 based on the steering wheel 10 provided on the movable vehicle 1, the imaging unit 32 that performs imaging, and the vehicle speed V of the vehicle 1 and the operation amount of the steering wheel 10. And a control device 4 for performing the control. Thereby, the yaw angle control of the imaging unit 32 can be appropriately performed so that the image on the upstream side of the image flow (optical flow) captured on the image sensor 100 is captured quickly.

(4) Since the imaging unit 32 includes the imaging element 100 that can independently set the imaging conditions of the plurality of unit areas 131, different imaging conditions can be set for each unit area 131.

(5) Since the control device 4 moves the imaging unit 32 in accordance with the operation of the steering wheel 10, the yaw angle control of the imaging unit 32 can be performed in real time with respect to the rotation operation of the steering wheel 10.

(6) The control device 4 determines the amount of movement of the imaging unit 32 based on the vehicle speed V and the rotation angle θ of the steering wheel 10, so that the yaw angle control of the imaging unit 32 can be appropriately performed.

(7) The vehicle 1 includes the imaging unit 32 including the imaging element 100 capable of independently setting the imaging conditions of the plurality of unit areas 131, the control device 4 for detecting approach to an intersection, and the detection by the control device 4. The control device 4 is provided for setting imaging conditions of the plurality of unit areas 131 of the image sensor 100 based on the result. Thereby, different imaging conditions can be set when the vehicle 1 approaches the intersection and when the vehicle 1 is away from the intersection. Furthermore, appropriate imaging conditions can be set for each of the vehicles 1 traveling on the left-turn lane, straight-ahead lane, and right-turn lane at the intersection.

(8) Since the radar 18 is provided for detecting an object around the vehicle 1, a second depth image can be obtained separately from the first depth distribution image calculated by the control unit 35.

(9) The vehicle 1 includes the line-of-sight detection device 16 that detects the line of sight of the occupant. Thereby, for example, the imaging conditions of the imaging unit 32 can be set in consideration of the occupant's line of sight, such as increasing the frame rate for an area where the occupant is not paying attention.

(10) The control device 4 of the vehicle 1 sets the highest priority area 83 to an area other than the attention area 82 including the pedestrian crossing 92 when the vehicle 1 turns left (or right) at the intersection, and sets the frame rate. Is increased or the thinning rate is reduced. As a result, an area where the driving occupant has a low degree of attention is raised as the highest priority area 83 by the imaging unit 32. More specifically, another vehicle such as a vehicle 91 turning right from the opposite lane when turning left as shown in FIG. 20 is monitored by the imaging unit 32.

(11) The control device 4 of the vehicle 1 sets the imaging conditions of the imaging unit 32 based on the image plane movement amount Δdiff representing the size of the flow of the image captured on the imaging element 100 of the imaging unit 32. This makes it possible to appropriately set imaging conditions for the imaging unit 32, such as changing the level of the frame rate according to the magnitude of the image plane movement amount Δdiff.

(12) Since the control device 4 calculates the image plane movement amount Δdiff based on the product of the vehicle speed V and the rotation angle θ of the steering wheel, the control device 4 can appropriately calculate the size of the flow of the image captured on the image sensor 100.

(13) The control device 4 controls the yaw angle Y of the imaging unit 32 based on the direction and size of the image plane movement amount Δdiff. Thereby, the yaw angle control of the imaging unit 32 can be appropriately performed so that the image on the upstream side of the optical flow is quickly captured on the imaging surface of the imaging unit 32.

(14) The control device 4 changes the frame rate of the imaging unit 3 based on the magnitude of the image plane movement amount Δdiff. This makes it possible to appropriately set imaging conditions for the imaging unit 32, such as increasing the frame rate as the magnitude of the image plane movement amount Δdiff increases.

(15) The control device 4 corrects the movement range of the yaw angle of the imaging unit 32 to be narrow, depending on the presence of a right-turning vehicle in the oncoming lane or a straight-ahead vehicle in the oncoming lane. Thereby, the information of the vehicle in the oncoming lane can be reliably acquired by the imaging unit 32 while the direction of the imaging unit 32 is directed in the turning direction of the vehicle 1.
Although the camera 3A is controlled by the control device 4 in the above-described embodiment, a part of the control of the camera 3A may be performed by the control unit 35 of the camera 3A.

(Modification 1 of Second Embodiment)
In the above-described embodiment, an example in which the frame rates are different between the priority area 82 and the highest priority area 83 has been described. Instead, a buffer area may be provided between the priority area 82 and the highest priority area 83, and the frame rate may be varied in multiple stages. For example, by setting the frame rate of the buffer area to an intermediate value between the frame rate of the priority area 82 and the frame rate of the highest priority area 83, the frame rate is changed in multiple stages. By providing such a buffer area, it is possible to avoid a state where the value of the frame rate greatly changes at the boundary.

(Modification 2 of the second embodiment)
In the above description, as the distance measurement performed by the camera 3A, a method of calculating by distance measurement using an image signal from a focus detection pixel provided in the image sensor 100 is used. A method of performing distance measurement using an image may be used.

(Third embodiment)
The third embodiment is an example in which distance measurement is performed using two images obtained by a stereo camera. In particular, a case where the auxiliary light source 22 is used will be described. FIG. 27A is a diagram illustrating the third embodiment. In FIG. 27 (a), the auxiliary light source 22 is provided at the front of the vehicle 1 and irradiates the front of the vehicle 1 traveling parallel to the road surface at a predetermined ground height (for example, 50 cm). The auxiliary light source 22 emits an infrared light 22B having a predetermined projection pattern within the angle of view of the stereo camera 3S. The vehicle 1 is the same as the vehicle illustrated in FIG. 1, and a stereo camera 3S is provided instead of the camera 3A.

  FIG. 27B is a diagram illustrating a projection pattern by the auxiliary light source 22. In FIG. 27B, a low-contrast wall 40 provided in front of the vehicle 1 is irradiated with infrared light 22B. The projection pattern is bilaterally symmetric with respect to the center of the wall 40, and the intervals between the stripes of the pattern are uneven. In the example of FIG. 27B, the interval between the stripes is gradually increased from the center of the wall 40 toward the left and right ends.

  For example, when an infrared LED is used as the auxiliary light source 22, the pattern of the light emitting portion of the LED chip can be configured in a stripe pattern, and the pattern can be projected. Further, the infrared light 22B from the auxiliary light source 22 may be projected in a stripe shape by using a lens and a prism.

On the other hand, the left and right cameras constituting the stereo camera 3S are equipped with the same image sensor as the image sensor 100 described above. In each image sensor, among RGB pixels having a predetermined arrangement such as a Bayer arrangement, for example, IR pixels in which a filter for receiving infrared light is arranged at a position of a G pixel are arranged at a predetermined ratio. Thereby, the stereo camera 3S can acquire two right and left infrared images based on the image signals from the IR pixels. The filter disposed in the IR pixel has a characteristic of transmitting the wavelength of the infrared light 22B emitted from the auxiliary light source 22 and blocking other wavelength ranges.
The position at which the IR pixel is arranged is not limited to the position of the G pixel in the Bayer array, but may be the position of the R pixel or the position of the B pixel.

  The control device 4 performs distance measurement using a stereo image of the infrared light 22B acquired by the stereo camera 3S. FIG. 27C is a diagram illustrating a target object (two persons) irradiated with the infrared light 22B. When the outside of the vehicle 1 has a predetermined brightness, the control device 4 drives all the RGB pixels of the respective image pickup devices of the left and right cameras and acquires an RGB stereo image at a predetermined frame rate. The control device 4 detects a target object ahead of the vehicle 1 based on the RGB stereo image.

  The control device 4 drives an IR pixel corresponding to the detected object, the object being irradiated with the infrared light 22B, and an IR pixel not corresponding to the object irradiated with the infrared light 22B. Do not drive. The control device 4 measures the distance based on the RGB stereo image, and when the distance measurement based on the RGB stereo image is difficult, measures the distance to the target based on the stereo image using the infrared light 22B.

  When the outside of the vehicle 1 does not reach the predetermined brightness, the control device 4 drives all the IR pixels of the image sensors of the left and right cameras to acquire a stereo image by the infrared light 22B at a predetermined frame rate. . The control device 4 detects an object ahead of the vehicle 1 on the basis of the stereo image of the first frame by the infrared light 22B. Then, the control device 4 drives the detected target object and the IR pixels corresponding to a predetermined range including the target object for the next frame and thereafter, and does not drive the other IR pixels. The control device 4 measures the distance to the object based on the stereo image by the infrared light 22B.

  The control device 4 further increases the frame rate of the corresponding IR pixel as the distance to the detected object is shorter, and lowers the frame rate of the corresponding IR pixel as the distance to the detected object is longer. As described above, since the IR pixels are selectively driven and the frame rate of the IR pixels is made different according to the distance to the object, all of the IR pixels of each of the image sensors of the left and right cameras are set to a high level. Power consumption can be reduced as compared with the case of driving at a frame rate.

The light emission timing of the auxiliary light source 22 that emits the infrared light 22 </ b> B may be always emitted, or may be intermittent light emission emitted at the timing of driving the IR pixels of the image sensors of the left and right cameras.
Alternatively, the infrared light 22B may be emitted when a predetermined condition is determined based on an RGB stereo image. For example, when there is a wall in front of the vehicle 1 (T-junction, dead end, etc.), when the object (person) is wearing plain clothes, when the object is a car, the object has a regular pattern. When it has (such as a shutter), the auxiliary light source 22 emits infrared light 22B.

According to the third embodiment, the following operation and effect can be obtained. That is, the imaging apparatus includes a stereo camera 3S having an imaging element capable of independently setting imaging conditions of a plurality of regions, and an infrared light 22B of an object irradiated with the infrared light 22B of a predetermined pattern. The control device 4 controls the imaging conditions of an area of the imaging element that captures an area to be irradiated and the imaging conditions of an area of the imaging element that captures an area not irradiated with the infrared light 22B to be different. Thereby, based on the stereo image by the stereo camera 3S, distance measurement can be appropriately performed even at night or in backlight. Further, since the interval between stripes of the projection pattern by the infrared light 22B is made non-uniform, it is easy to detect parallax even when the projection is performed on a plane directly in front.
It is also effective not only at night but also in fog.

(Fourth embodiment)
FIGS. 28A and 28B are diagrams for explaining the fourth embodiment. In FIG. 28 (a), the vehicle 1 is the same as the vehicle illustrated in FIG. 1, except that a transmission type liquid crystal filter 21 is laminated on a windshield glass 20 (excluding a region through which a light beam to the camera 3 passes). I have. In the fourth embodiment, the camera 3A acquires information (required light) that is difficult for the driver to visually recognize.

  In FIG. 28B, a head-up display as a display / playback device 14 is provided in the dashboard. The head-up display 14 calls attention to the driver 30. Further, the transmissive liquid crystal filter 21 (FIG. 28A) suppresses dazzling light (unnecessary light) for the driver 30.

  The transmission type liquid crystal filter 21 is a filter capable of controlling the visible light transmittance for each area. In the present embodiment, when the control device 4 detects that an object (the sun 35 such as the sunrise or sunset) having a predetermined brightness or more is present, the liquid crystal elements constituting the transmissive liquid crystal filter 21 are detected. The transmittance of the liquid crystal element of the transmission type liquid crystal filter 21 corresponding to the sun 35 and the surrounding area in the scenery observed by the driver 30 is reduced. Thereby, excessive unnecessary light to the eyes of the driver 30 is suppressed.

  In addition to the sun 35, the reflected light reflected from the building or the headlight of an oncoming vehicle during night driving may be set as a target to be suppressed as excessive unnecessary light. Here, it is assumed that the relationship between the position of the eyes of the driver 30 based on the physique and the sitting position of the driver 30 and the scenery observed by the driver 30 via the transmissive liquid crystal filter 21 is stored in advance.

  The head-up display 14 is a display device that displays information displayed on a liquid crystal panel or the like as a virtual image on the windshield glass 20. The driver 30 observes the information on the head-up display 14 projected so as to overlap the scenery outside the vehicle 1. In the present embodiment, when the control device 4 detects the presence of an object (a pedestrian, a vehicle (including a car or a motorcycle), or the like), a frame including the object in a scene observed by the driver 30 ( P2 may be displayed on the head-up display 14.

In addition, even when the control device 4 detects the presence of an object (a road sign, a signal, a pedestrian crossing, a gutter, etc.), the frame P1 including the object in the scenery observed by the driver 30 (may be a border). May be displayed on the head-up display 14.
The relationship between the position of the eyes of the driver 30 based on the physique and the sitting position of the driver 30, the scene observed by the driver 30 through the windshield glass 20, and the projection position of the head-up display 14 is stored in advance. It is assumed that

(Segmentation)
The control device 4 performs control of the transmittance of the transmissive liquid crystal filter 21, display control of the head-up display 14, and control of the gain and the like of the camera 3A by performing classification based on the image acquired by the camera 3A.

(Excess light area)
The control device 4 detects an object having a predetermined brightness or higher as follows. For example, position information and time information of the vehicle 1 calculated by the GPS device 15, azimuth information of the vehicle 1 detected by the navigation device 18, position information of the sun 35 prepared in advance (observation point and altitude (elevation angle) and Based on the information indicating the azimuth) and the angle of view information of the camera 3A, an area corresponding to unnecessary excess light due to the sun 35 or the like (referred to as an excess light area) in the image acquired by the camera 3A is detected. As described above, the control device 4 lowers the transmittance of the liquid crystal element corresponding to the excess light region in the transmission type liquid crystal filter 21. Further, the control device 4 sets a low gain corresponding to the excessive light area on the imaging surface for the imaging device 100 of the camera 3A.

(Insufficient light area)
When the object of the image acquired by the camera 3A is darker than a predetermined brightness, the control device 4 determines that the light from the object is insufficient light. The control device 4 sets a high gain in an area corresponding to the insufficient light in the image sensor 100 of the camera 3A (referred to as an insufficient light area). Thereby, the object of the image acquired by the camera 3A becomes bright, and the object (pedestrian, vehicle (including car and motorcycle), sign, etc.) can be easily recognized.
Note that, not only the gain of the image sensor 100 may be partially increased but also the frame rate may be partially increased. When it is determined that there is no target object, the gain and the frame rate may be returned to the values before the change.

(Necessary light area)
When an image (a road sign, a signal, a pedestrian crossing, a gutter, etc.) is included in the image acquired by the camera 3A, the control device 4 determines that the light from the object is necessary light. The control device 4 partially sets the gain of an area (referred to as a necessary light area) corresponding to the necessary light in the image sensor 100 of the camera 3A to a high value. Thereby, the object of the image acquired by the camera 3A becomes bright, and the information of the object (a road sign, a signal, a pedestrian crossing, a gutter, etc.) becomes easy to understand.
Note that when the image acquired by the camera 3A is brighter than the predetermined brightness, the control device 4 may return the gain and the frame rate to the values before the change.

The control device 4 further causes the head-up display 14 to display frames (may be borders) P1 and P2 including the respective objects in the insufficient light area and the required light area, as described above. As a result, the driver 30 can be notified of the presence of the target object or can be alerted.
In addition, instead of the display of the frame or the outline, or in addition to the display of the frame or the outline, a mark stored in advance (for example, the same mark as a road sign, a mark imitating a person, a mark imitating a vehicle, or the like) is displayed. It may be displayed.

  The control device 4 may perform pixel addition on the imaging area of the imaging device 100 instead of increasing the gain of the insufficient light area or the necessary light area, or while increasing the gain. Pixel addition means that a plurality of pixel signals are added and treated as one pixel signal. The control device 4 obtains one image by, for example, adding pixels of a plurality of frames (for example, four images) of which acquisition timing is continuous. In the pixel addition, corresponding pixel signals between a plurality of frames are added.

  After detecting the target by temporarily increasing the gain in the insufficient light area or the required light area or performing pixel addition between a plurality of frames as described above, the control device 4 controls the movement of the target object. The gain may be reduced to the extent that tracking can be performed, or the number of frames for which pixel addition is performed may be reduced.

  According to the above-described fourth embodiment, the following operation and effect can be obtained. That is, the electronic device includes an imaging unit 32 including an imaging element 100 capable of independently setting imaging conditions of a plurality of regions, a head-up display 14 provided on a front surface of a driver 30 of the vehicle 1, an imaging unit, A control device 4 is provided for setting imaging conditions for a plurality of areas and controlling the display of the head-up display 14 based on the information on the brightness in front of the vehicle 1 obtained from the image captured at 32. For example, by setting the gain of the region corresponding to the insufficient light in the image sensor 100 to be high, the object of the image acquired by the camera 3A becomes bright, and the object (including a pedestrian, a vehicle (including a car and a motorcycle), Signs, etc.). In addition, it is possible to notify the driver 30 of the presence of the target object or to call attention.

(Fifth embodiment)
In the fifth embodiment, the wearable camera 19 worn by the driver 30 and the camera 3A mounted on the vehicle 1 cooperate. FIG. 29A is a diagram for explaining the fifth embodiment. In FIG. 29A, the camera 3A has a function as a drive recorder. For example, when the power supply is started from the vehicle 1 (system on) or the engine is started, the camera 3A starts imaging at a predetermined frame rate. I do. The wearable camera 19 starts imaging at a predetermined frame rate, for example, when the wearable camera 19 is worn by the driver 30. The wearable camera 19 sequentially transmits captured images to the control device 4 via a built-in wireless communication unit. The control device 4 can record images acquired by the camera 3A and the wearable camera 19 in the storage unit 4b of the control device 4.

The camera 3A divides the imaging surface of the imaging chip 111 of the imaging device 100 into a first area (normal imaging area) and a second area (special imaging area), and performs different imaging between the first area and the second area. Imaging can be performed by setting conditions. According to FIG. 29 (b), the first area is constituted by the blocks in the odd columns, and the second area is constituted by the blocks in the even columns. That is, the blocks in the imaging chip 111 are divided into odd columns and even columns.
Note that the manner of dividing the first area and the second area is not limited to the example of FIG. 29B, but may be divided into odd rows and even rows as shown in FIG. It can be divided in a checkered pattern like this.

  For example, at the time of traveling at night, the control device 4 causes the camera 3A to perform normal imaging in the first area and sets the gain in the second area in the camera 3A to be higher than the gain in the first area to perform imaging. Let

  For example, in a backlight state during traveling in the daytime, the control device 4 causes the camera 3A to perform normal imaging in the first area, and sets the gain in the second area in the camera 3A to be lower than the gain in the first area. To perform imaging. In addition to setting the gain low, the accumulation time may be set short.

  The control device 4 does not drive the pixels in the second area in the camera 3A when the vehicle does not correspond to the night driving or daylight backlight state. The control device 4 causes only normal imaging to be performed using the first area in the camera 3A.

  The wearable camera 19 is equipped with a line-of-sight detection unit, and can detect the line of sight of the driver 30. The line-of-sight information detected by the line-of-sight detection unit is transmitted by wireless communication from the wearable camera 19 to the control device 4 together with the captured image via the wireless communication unit.

  The control device 4 determines whether or not the imaging range of the wearable camera 19 is included in the imaging range of the camera 3A based on the line of sight of the driver 30. Then, when the imaging range of the wearable camera 19 is included in the imaging range of the camera 3A, the control device 4 records only the image acquired by the camera 3A in the storage unit 4b of the control device 4.

  When a part of the imaging range of the wearable camera 19 is included in the imaging range of the camera 3A, the control device 4 controls both the image acquired by the camera 3A and the image acquired by the wearable camera 19. In the storage unit 4b.

  Furthermore, when the imaging range of the wearable camera 19 is not included in the imaging range of the camera 3A, the control device 4 transmits both the image acquired by the camera 3A and the image acquired by the wearable camera 19 to the control device 4. The information is recorded in the storage unit 4b.

According to the above-described fifth embodiment, the following operation and effect can be obtained.
(1) The image recording device includes a camera 3A that captures an image around the vehicle 1, a control device 4 that inputs a second image captured by a wearable camera 19 mounted on a driver 30 of the vehicle 1, and a camera 3A. The control device 4 that compares the captured range of the captured first image and the captured range of the second image, and at least one of the first image and the second image is stored in the storage unit 4b based on the comparison result of the control device 4. And a control device 4 for recording. For example, when at least a part of the imaging range of the wearable camera 19 is not included in the imaging range of the camera 3A, both the first image acquired by the camera 3A and the second image acquired by the wearable camera 19 are displayed. By recording the information in the storage unit 4b of the control device 4, information that cannot be obtained by the camera 3A alone can be left.
On the other hand, when the entire imaging range of the wearable camera 19 is included in the imaging range of the camera 3A, only the first image is recorded in the storage unit 4b of the control device 4, so that the usage amount of the storage unit 4b is reduced. It can be kept low.

(2) In the case where the camera 3A performs imaging in the second area, an image obtained by normal imaging in the first area and an image obtained by imaging in the second area under different conditions are used. Since both are recorded, it is possible to leave information that is difficult to understand in normal imaging. Furthermore, when imaging is not performed in the second area of the camera 3A, only images obtained by normal imaging in the first area are recorded, so that the amount of use of the storage unit 4b can be reduced.

(Modification)
In each of the above-described embodiments, the image acquired by the camera 3A mounted on the vehicle 1 may be shared with another vehicle other than the vehicle 1, or may be shared with a traffic control center. For example, the image acquired by the camera 3A is transmitted to another vehicle or a traffic control center by the wireless communication device. Transmission from the vehicle 1 to another vehicle may be performed by inter-vehicle communication, or may be transmitted from the vehicle 1 via a traffic control center.

  When an image is transmitted from the vehicle 1 to the traffic control center, information generated by the traffic control center based on the image received from the vehicle 1 may be distributed to other vehicles. For example, when an obstacle is included in the image received from the vehicle 1, the control device of the traffic control center analyzes the image and issues a message "There is a falling object near the prefectural road XX line @ 4-chome." Generate and distribute to other vehicles.

  When the image including the above-described obstacle on the road is acquired by the camera 3A of the vehicle 1, the control device 4 sets a region of interest in the imaging chip 111 with respect to the obstacle, and sets another region (forward region of interest). (For example, increasing the resolution and the frame rate). Thereby, detailed information on the obstacle in the attention area can be obtained. Further, by capturing only the region of interest (that is, a part of the image) with high definition, the data of the acquired image can be compared with the case of capturing the entire region (all of the image) with the imaging chip 111 with high definition. The amount can be kept low.

  By reducing the data amount, the communication amount when transmitting an image from the vehicle 1 to another vehicle or a traffic control center can be reduced. When an image obtained by the camera 3A is stored as a drag recorder, by reducing the data amount, the capacity of a storage medium for recording can be reduced.

  In the case where the communication amount when transmitting an image from the vehicle 1 to another vehicle or the traffic control center is to be reduced, all areas in the imaging chip 111 are imaged with high definition, and only an area corresponding to an obstacle is cut out. Some of the images may be transmitted from the vehicle 1 to another vehicle or a traffic control center.

  Examples of sharing an image acquired by the camera 3A mounted on the vehicle 1 with vehicles other than the vehicle 1 and the traffic control center include, in addition to the presence or absence of the above-mentioned obstacle, presence or absence of an accident, occurrence of a disaster, It can also be applied to the presence or absence of construction, the presence or absence of construction, the presence or absence of congestion, the presence or absence of inspection, and the presence or absence of chain attachment restrictions.

  REFERENCE SIGNS LIST 1 vehicle, 2 driving assistance device, 3A, 3B camera, 4 control device, 4a, 35a CPU, 4b, 35b storage unit, 10 steering wheel, 12 vehicle speed sensor, 14 display / reproduction device , 16: eye-gaze detecting device, 17: communication unit, 18: radar, 31: imaging optical system, 32: imaging unit, 35: control unit, 70: imaging surface, 81: first area, non-priority area, 82: 2 areas, priority area, 83: highest priority area, 100: imaging element, 111: imaging chip

Claims (11)

In electronic devices mounted on cars,
An imaging unit having a plurality of first regions for imaging the subject under the first imaging condition and a plurality of second regions for imaging the subject under a second imaging condition different from the first imaging condition;
A detection unit that detects an object based on a first image signal generated by the first region;
A display control unit that displays an image on a display unit based on a second image signal generated by the second region;
Electronic equipment with. The electronic device according to claim 1,
  The electronic device, wherein the imaging unit has the first region and the second region alternately arranged in at least a first direction. The electronic device according to claim 2,
  The electronic device, wherein the imaging unit includes the first region and the second region alternately arranged in a second direction that intersects the first direction. The electronic device according to any one of claims 1 to 3 ,
The display controller, an electronic apparatus for displaying information of the object detected in the previous danger out portion on the display unit. The electronic device according to any one of claims 1 to 4 ,
The electronic device, wherein the imaging unit is configured such that a frame rate of the first area is higher than a frame rate of the second area. The electronic device according to any one of claims 1 to 5 ,
The electronic device, wherein the imaging unit has a gain in the first area higher than a gain in the second area. The electronic device according to any one of claims 1 to 6 ,
A line-of-sight detector that detects a line-of-sight position of the occupant of the vehicle,
Prior dangerous out unit, based on a detection result by the visual line detection part, an electronic device to detect the object at the viewpoint position does not include the area of said first region in said first image signal. The electronic device according to claim 7 ,
A setting unit that sets the imaging conditions of the first image signal differently in an area that does not include the line-of-sight position in the first area and in an area that includes the line-of-sight position in the first area;
Electronic equipment provided with. The electronic device according to claim 8 ,
The electronic device, wherein the setting unit sets a frame rate of an area of the first area that does not include the line-of-sight position higher than a frame rate of an area of the first area that includes the line-of-sight position. The electronic device according to any one of claims 1 to 9,
The electronic device, wherein the imaging unit captures an image of a common target using the first image signal and the second image signal.   An automobile equipped with the electronic device according to any one of claims 1 to 10.

Images