CN116263987A - Method for capturing images in a vehicle interior and vehicle interior camera system - Google Patents

Abstract

The invention relates to a method for capturing images in a vehicle interior space (102) using a camera (124, 324), a first lighting device (126) and a second lighting device (128), the method comprising: -manipulating the first lighting device (126) in a first lighting mode (220) to illuminate a first region (140) of the vehicle interior space, the first lighting mode being selected according to a first application for which an image to be taken with the camera (124) at the first region (140) is provided; manipulating the second lighting device (128, 328) in a second lighting mode to illuminate a second region (142) of the vehicle interior space, the second lighting mode being selected according to a second application for which an image to be taken with the camera (124) at the second region (142) is provided; and the camera (124) is operated to take an image of the vehicle interior space (102). The invention also relates to a vehicle interior space camera system (120).

Description

Method for capturing images in a vehicle interior and vehicle interior camera system

Technical Field

The present invention relates to a method for capturing images in a vehicle interior with a camera, a first lighting device and a second lighting device, as well as to a computing unit for performing the method, to a vehicle interior camera system for performing the method, and to a computer program for performing the method.

Background

A camera for sensing an interior space may be used in a vehicle. For this purpose, the driver can be illuminated, for example, in the case of a so-called driver monitoring system ("Driver Monitoring Systems"), by means of an illumination device with typical infrared light. Light reflected by the driver is detected by a camera or by a sensor (detector) and evaluated, for example, within the framework of image processing; thereby, it is possible to determine, for example, where the driver's head is located or what the driver's line of sight is.

Typically, such a system may also be used for interior space monitoring. In this case, it is intended, for example, to be able to sense an abnormal behavior in the sharing of automobiles (e.g., illegal smoking) or to analyze the state of the internal space of the vehicle (e.g., forgotten articles, dirt).

Disclosure of Invention

According to the invention, a method for capturing images in a vehicle interior as well as a computing unit for performing the method, a vehicle interior camera system for performing the method and a computer program for performing the method are proposed with the features of the independent patent claims. Advantageous embodiments are the subject matter of the dependent claims and the following description.

The invention relates to a system with a camera (or sensor or detector) for recording or capturing images of the interior of a vehicle or also comprising an image sequence, i.e. video, and thereby in particular monitoring or sensing the interior of the vehicle. Of particular interest here is the illumination for such cameras. For this purpose, two illumination devices can be provided for illumination, each for illuminating a specific region of the vehicle interior. In this way, the first illumination device may be arranged to illuminate a first region of the vehicle interior space and the second illumination device may be arranged to illuminate a second region of the vehicle interior space. The two regions may be different from each other, but may also overlap at least partially. The first area may for example comprise a driver area of the vehicle, i.e. the first lighting means may be arranged for lighting the driver; while the second area may include, for example, a secondary driver area and/or a backseat area.

Such lighting devices typically emit infrared light so that images can be taken with a camera even at night, but do not disturb the occupants of the vehicle. However, the camera or detector is especially set up for detecting infrared light and visible light.

The two illumination devices may for example be arranged beside the camera (e.g. left and right side), respectively; the camera and the two illumination devices may then be arranged, for example, in a common housing and form part of a vehicle interior space camera system. Such a vehicle interior camera system or at least the camera and, if appropriate, the two illumination devices can be arranged in or on a rear view mirror, in or on a central display or dashboard, or in or on a roof module. At least advantageously: the camera is arranged in the middle or in the center of the vehicle in order to have a good overview of the interior space of the vehicle. However, these illumination means can in principle also be arranged remote from the camera.

In this respect, it can also be said to be a single-phase system. However, a dual camera or a multi-camera system is also conceivable, wherein at least one further camera is then provided, which should be arranged in a suitable position in the vehicle or in the vehicle interior.

The use of two illumination devices allows for more efficient illumination or illumination of different areas in the vehicle interior space. However, as has now been demonstrated, the illumination required for these different areas may be different, for example for different applications. Thus, for example, a different illumination may be required for the driver than for the remaining areas. In other words, the same illumination is not always required for every region.

In this context, it is proposed that: the first lighting device is operated in a first lighting mode to illuminate the first area, the first lighting mode being selected according to the first application. The first application is an application for which an image to be photographed with a camera in the first area is provided. Correspondingly, the second lighting device is then operated in a second lighting mode to illuminate the area, the second lighting mode being selected according to the second application. The first illumination mode and/or the second illumination mode may then be changed, in particular, as the case may be. Namely, it is also possible in particular that: the two lighting devices are actuated independently of one another and thereby individually. In this way, for example, the driver may be illuminated differently, more strongly or more frequently in the first region than, for example, the secondary driver, since attention monitoring is to be carried out at the driver, whereas, for example, occupancy monitoring is sufficient at the secondary driver.

Preferably, the first illumination mode comprises emitting light pulses having a first illumination frequency and/or a first pulse duration; and/or the second illumination mode comprises emitting light pulses having a second illumination frequency and/or a second pulse duration. A light pulse is to be understood here as a light emission within a specific pulse duration of, for example, a few milliseconds, for example, a brief switching on of the light source; this may be repeated at a particular frequency, the illumination frequency. In this way, the total illumination time period and thus also the heat generation in the lighting device or also in the control or drive circuit therefor can be significantly reduced by reducing the illumination frequency, while the type and the time period of the light pulses remain unchanged. For driving these lighting devices, pulse Width Modulation (PWM) driving with a correspondingly selected duty cycle can be used, for example.

The camera may then be operated as well to take images at a taking frequency (or image taking rate) that corresponds to at least the maximum of the first and second illumination frequencies. If, for example, the first illumination frequency is 60 Hz, but the second illumination frequency is, for example, only 30 Hz, 10 z or also only 1 Hz, the shooting frequency of the camera can likewise be 60 Hz. Thereby, the driver can be photographed with a sufficiently high frequency in a sufficiently illuminated image, while the energy supply and heat generation at the illumination device for the co-driver can be (significantly) reduced.

If another camera is used, one of the two lighting devices may be used for each camera. Then, for example, one camera may then be matched with respect to the shooting frequency to the (possibly higher) illumination frequency of the first illumination device, while the shooting frequency of the other camera may be matched with the (possibly lower) illumination frequency of the second illumination device. In this case, the other camera can also operate more efficiently.

The principle can also be applied to other lighting devices and/or other cameras and any combination thereof.

Preferably, the first illumination frequency and/or the pulse duration are selected to be at least temporarily different from the second illumination frequency or the pulse duration in order to reduce heating of the second illumination means. In principle, however, for example, the two illumination frequencies or pulse durations can also be selected to be temporarily identical, as long as this should be necessary in certain cases or for corresponding applications, which are not necessarily identical per se, but may require the same type of illumination.

For example, at least one of the following applications may be considered as the first application: driver observation, driver gaze tracking, and driver eyelid observation. For the second application, for example, at least one of the following applications may be considered: presence monitoring, video conferencing, gesture control, and data logging. The data record is to be understood as meaning in particular a so-called "event data record (Event Data Recording)", in order to store, in the event of an accident, important data, in particular images, about the state and/or number of vehicle occupants in the respective accident phases (before, during and after the accident, also referred to as "pre-crash", "post-crash)").

The computing unit according to the invention, for example a control device of a motor vehicle or a control unit for a camera or a vehicle interior camera system, is designed in particular in a program-oriented manner to carry out the method according to the invention.

The invention also relates to a vehicle interior camera system having a camera, a first lighting device, a second lighting device and a computing unit according to the invention. If necessary, another camera can also be provided. The camera, the first lighting device and the second lighting device may be arranged in a common housing, while the computing unit may be separate from the housing and connected thereto, in particular via power and/or signal lines, i.e. in particular to the camera and the lighting devices, in order to operate the camera and the lighting devices and to obtain images from the camera. The control or drive circuit for at least one of the camera, the first lighting device and the second lighting device can likewise be contained in the computing unit, but can also be arranged in the housing if necessary.

In particular, when the control device to be implemented is also used for other tasks and is thus always present, an implementation of the method according to the invention in the form of a computer program or a computer program product having a program code for executing all method steps is also advantageous, since this results in particularly low costs. Finally, a machine readable storage medium is provided, on which a computer program as described above is stored. In particular, storage media or data carriers suitable for providing the computer program are magnetic, optical and electrical memory, such as hard disk, flash memory, EEPROM, DVD and the like. It is also possible to download the program via a computer network (internet, intranet, etc.). Such downloading may be effected here in a wired manner or in a cable manner or in a wireless manner (e.g. via a WLAN network, a 3G, 4G, 5G or 6G connection, etc.).

Further advantages and embodiments of the invention emerge from the description and the attached drawing.

The invention is schematically illustrated in the drawings and is described below with reference to the drawings according to embodiments.

Drawings

Fig. 1 schematically shows a vehicle with a vehicle interior space camera system according to the invention in a preferred embodiment.

Fig. 2 shows the imaging and illumination frequencies that can be used in the method according to the invention in a preferred embodiment.

Fig. 3 schematically shows a vehicle interior space camera system in another preferred embodiment according to the invention.

Detailed Description

In fig. 1, a vehicle 100 having a vehicle interior space camera system 120 in a preferred embodiment according to the present invention is schematically illustrated. The vehicle interior camera system 120 is for detecting the vehicle interior 102 and has a camera 124, a first illumination device 126, and a second illumination device 128, which are arranged in a common housing 122. The housing with these components is in turn arranged in the middle of the vehicle, for example integrated on or also in the rear view mirror.

The vehicle interior space camera system 120 further comprises a computing unit 130, which may be, for example, a dedicated control device for the vehicle interior space camera system 120 or may also be a further control device of the vehicle 100 that is otherwise used. The computing unit 130 is connected via power and/or signal lines 132a, 132b, 132c with the camera 124, the first lighting device 126 and the second lighting device 128 in order to be able to handle these components and to be able to obtain images acquired by the camera 124. These power and/or signal lines 132b, 132c may be simplified or combined into a common control line if there are additional coded signals for operating the corresponding lighting device 126 or 128 in order to then switch on an electronic switch, for example a MOSFET or MOSFET switch, in the camera 124 to activate the respective desired lighting device 126 or 128.

The first illumination device 126 is arranged to illuminate a first area 140 (here only indicated); the first region 140 includes, for example, an operator region having the operator's seat 104. The second illumination means 128 is arranged to illuminate a second area 142 (again only indicated here); the second region 142 includes, for example, a secondary driver region having the driver's seat 106. However, the second region 142 may also, additionally or alternatively, also cover the rear seat 108.

The camera 124 has a photographing region 144, which covers, for example, the driver's seat 104, the passenger's seat 106 and the rear seat 108, i.e. in particular substantially the entire vehicle interior 102, at least to the extent that a person can be located there.

Fig. 2 shows the recording and illumination frequencies that can be used in the method according to the invention in a preferred embodiment. This should also be explained in more detail later with reference to the vehicle interior space camera system 120 in fig. 1.

For this purpose, a shooting mode 210 of the camera, a first illumination mode 220 of the first illumination device and a second illumination mode 230 of the second illumination device are shown on a time or time axis 200.

The photographing mode 210 of the camera includes an exposure time of a duration or length 240, for example, of 4 ms. The interval 242 of the start times of the successive, individual exposure times is for example 16.67, ms. Thereby, a frequency of 60 Hz shots, i.e. 60 exposures per second of the detector of the camera, 4 ms each, is provided. For this purpose, the camera can be actuated accordingly. Also conceivable are: the camera or the vehicle interior camera system and the subsequent image processing operate at a higher frame refresh rate than 60 Hz, for example at 90 Hz, wherein, with the maximum image data bandwidth transmission capability of the image sensor maintained, instead of transmitting the complete image at a time, only important partial region regions, so-called regions of interest (Regions of Interest, ROI), are transmitted.

The illumination pattern 220 of the first illumination means comprises an illumination time of duration or length 240, e.g. also 4 ms; i.e. they are light pulses with a pulse duration of 4 ms. The interval 242 of the start times of the successive, individual light pulses is for example likewise 16.67, ms. Thereby, a first illumination frequency of 60 Hz is provided, i.e. 60 times per second, illuminating the first area, 4 ms at a time. For this purpose, the first illumination device can be actuated accordingly. In this way, for each individual exposure of the detector of the camera, a first area, i.e. for example the driver, is illuminated by the first illumination means in order to obtain an (infrared) image of the driver.

The illumination pattern 230 of the second illumination device includes an illumination time having a duration or length 240, such as 4 ms; i.e. they are light pulses with a pulse duration of 4 ms. However, the interval 244 of the start times of the successive, individual light pulses is here for example 33.333 ms. Thereby, a second illumination frequency of 30 Hz is provided, i.e. 30 times per second, illuminating the second area, 4 ms at a time. For this purpose, the second illumination device can be actuated accordingly. In this way, for every second exposure of the detector of the camera, a second area, i.e. for example the secondary driver, is illuminated by means of the second illumination means in order to obtain an (infrared) image of the secondary driver.

As mentioned, not only the first illumination frequency but also the second illumination frequency may be changed, wherein it should be noted that: the shooting frequency of the camera then corresponds to the higher of the two illumination frequencies (if the two illumination frequencies differ). For example, it is conceivable to: if it is sufficient for the second application, i.e. for it to provide an image to be captured with the camera in the second area, i.e. for this purpose an application for which the second illumination means is provided, the second illumination frequency is set to only 10 Hz or also to only 1 Hz. However, the two illumination frequencies may likewise be selected to be the same, or may be selected to be the same only temporarily if necessary. Then, the shooting frequency may be selected to be the same as the two illumination frequencies.

Based on functional requirements for e.g. driver observations as first application, high sampling rates and processing rates up to 60 Hz may be required, so that the first or driver-side lighting means are active for each image capture in synchronization with the camera or its integration time (exposure). Typically, the second lighting device would then likewise be activated at 60 Hz, which would cause: while the second area, i.e. for example the right side of the vehicle interior space, would be illuminated at 60 Hz, a second application or function important for this, such as a video conference of only 30 Hz or presence monitoring (presence detection) of only 10 Hz or even 1 Hz is required.

As explained in more detail with reference to fig. 2, the advantage of the proposed approach is in particular the on-demand activation of the (infrared) illumination, so that the illumination as well as the camera remain available at high ambient temperatures despite the thermal load in the camera, which is then low. Also, less computation or ECU resources are required (in the case of the mentioned computation units), in particular in the case of so-called System-on-Chip (SoC), more precisely by intelligent allocation of computation power. Such allocation includes, inter alia, the partitioning of so-called perceptual and video stacks. This relates to two different image streams for various applications, namely, for example, driver viewing (perception stack) and video conferencing and/or event data recording (video stack). Thereby, costs are also saved.

For this purpose, a vehicle interior space camera system 320 according to the invention in a further preferred embodiment is schematically shown in fig. 3. The vehicle interior space camera system includes a camera 324, a first lighting device 326, and a second lighting device 328, which are disposed in a common housing 322. A computing unit 330 is also provided, which is illustratively connected via a power or supply line 332a (possibly multiple) for the lighting devices and a power and signal line 332b for the camera.

The computing unit 330 has a microcontroller 331 designed as a system-on-chip, which has a memory designed as RAM 332 and ROM 333, in which images (or data) obtained via power and signal lines 332b and processed in a deserializer 336 (where, for example, serial data blocks can be converted into parallel data blocks) are obtained and forwarded as a video stack 334 and a perception stack 335. In the present case, the video stack 334 is handed over via the ethernet 340 to a control device 350 in the vehicle, which is for example provided for manipulating the man-machine interface and thereby providing the video conference as the second application 360. In the present case, the perception stack 335 is handed over to the CAN gateway 351 via the CAN 341 or another bus and is further handed over to a control device 352 in the vehicle, which is provided, for example, for controlling the partially automated driving or other safety-relevant functions and which requires, for example, a driver's observation as the first application 361.

Furthermore, the microcontroller 331 can specifically control the lighting devices via a suitable control circuit 337 and a drive circuit 338.

Subsequently, possible processing steps of the perceptual and video stack should be exemplarily described. Here, a frame rate of 60 Hz should be required for the sense stack, while only a frame rate of 30 Hz is required for the video stack.

First, color and infrared data in an image obtained from a camera may be divided. This is a Processing step that may be in the SoC or in a dedicated ISP chip (ISP stands for Image-Signal-Processing), and that divides the original input Image into: for example a first image in RGB format, i.e. a color image; and a second image, an IR (infrared) image, or both images are extracted therefrom. The color image with RGB pixels (or the detector used to capture the color image) is itself also sensitive to infrared light, and thus the infrared image is preferably subtracted from the color image in order to compensate for the effects of the "faded color image".

Various computer vision functions, such as head and eye tracking, face ID (face recognition or face identification) and also attention detection, head gesture recognition and recognition of a reaction to the driver or passenger, etc., may be included in the perception stack, not just the mentioned driver observations. These computer vision functions may be comprised by the mentioned first application.

The mentioned extracted infrared image, which typically has one quarter of the original resolution of the original input image, may be used as input signal for the perceptual stack processing path. However, additionally, the raw input image may be used for eye tracking ("eye tracking") in order to achieve a higher accuracy for tracking eyelids and gaze. In this case, the face found, for example, from head tracking can be used as an initial temporal and spatial search area in the current image for eye recognition in the next image, as long as eye tracking has not been performed.

Functions such as head tracking also function at a lower frequency, e.g. 30 Hz, whereas eye tracking should be implemented at a higher frequency, e.g. 60 Hz. In response, 60 Hz may be suitable for the sense stack.

The video stack processing path or the second application becomes active, in particular when the video conference is activated by a user, for example by the driver or co-driver. The color image is guided here, for example, through a so-called Image Signal Processing (ISP) pipeline, so that in a subsequent step the image can be improved with respect to image quality, color and image brightness and contrast. Additionally, the background may be masked or replaced with a background image, for example, according to user wishes. The image may then be encoded and/or compressed and sent via ethernet to another control device as mentioned for outputting the image to a human machine interface.

Furthermore, special camera adjustments may be provided which manipulate or adjust the camera according to the function, in particular by means of camera parameters such as analog gain, digital gain and exposure time corresponding to the requirements of the relevant application, in particular the first application. For computer vision functions (of the second application) it is possible to manipulate or adjust with good image brightness and good contrast and sharpness with a maximum exposure time of e.g. 4 ms in order to avoid motion blur.

For a good presentation on a man-machine interface (e.g. with a display), a long exposure time of up to 6 ms is required if necessary for a color image (with RGB), which is applicable in any case as long as e.g. the video conferencing function is activated by the user. Depending on the circumstances, especially the lighting conditions, the infrared image may be balanced with respect to the RGB or color image depending on the circumstances, to obtain a higher image quality. If, for example, the tracking confidence is high, i.e. an eye or other tracking with IR can just be reliably achieved, the image quality can be improved, for example, in the color image, or vice versa.

Namely, in this way, it is realized that: different applications requiring different frame rates are realized by different lighting means, but here too excessive heating of the vehicle interior space camera system is still avoided.

Claims (15)

1. A method for acquiring images in a vehicle interior space (102) using a camera (124, 324), a first lighting device (126, 326) and a second lighting device (128, 328), the method comprising:

-manipulating the first lighting device (126, 326) in a first lighting mode (220) to illuminate a first area (140) of the vehicle interior space, the first lighting mode being selected according to a first application (361) for which an image to be taken with the camera (124, 324) at the first area (140) is provided;

-manipulating the second lighting device (128, 328) in a second lighting mode (230) to illuminate a second area (142) of the vehicle interior space, the second lighting mode being selected according to a second application (360) for which an image to be taken with the camera (124, 324) at the second area (142) is provided; and also

-manipulating the camera (124, 324) to take an image of the vehicle interior space (102).

2. The method of claim 1, wherein the first illumination pattern (210) comprises emitting light pulses having a first illumination frequency pulse duration; and/or wherein the second illumination pattern (230) comprises emitting light pulses having a second illumination frequency.

3. The method of claim 2, wherein the camera (124, 324) is manipulated to capture images at a capture frequency that corresponds to at least a maximum of the first illumination frequency and the second illumination frequency.

4. A method according to claim 2 or 3, wherein the first illumination frequency is selected to be at least temporarily different from the second illumination frequency, wherein preferably the first illumination frequency is selected to be 60 Hz and the second illumination frequency is selected to be up to 30 Hz.

5. The method according to any one of claims 2 to 4, in case of additional use of another camera, the method further comprising:

the further camera is operated to capture an image of the vehicle interior space (102) at a capture frequency, which corresponds to at least a minimum of the first illumination frequency and the second illumination frequency.

6. The method of any of the preceding claims, wherein the first illumination mode (210) comprises emitting light pulses having a first pulse duration; and/or wherein the second illumination mode (230) comprises emitting light pulses having a second pulse duration.

7. The method according to any of the preceding claims, wherein the first lighting device (126, 326) and/or the second lighting device (128, 328) is/are set up for emitting infrared light.

8. The method of any of the preceding claims, wherein the first application (361) comprises at least one of the following applications: driver observation, driver gaze tracking, and driver eyelid observation.

9. The method of any of the preceding claims, wherein the second application (360) comprises at least one of the following applications: presence monitoring, video conferencing, gesture control, and data logging.

10. The method according to any of the preceding claims, wherein the first illumination pattern (220) and/or the second illumination pattern (230) is changed as the case may be.

11. The method of any of the above claims, wherein the first area (140) comprises a driver area of the vehicle, and wherein the second area (142)) comprises at least one of a co-driver area and a backseat area.

12. A computing unit (130, 330) which is set up to perform all method steps of the method according to any of the preceding claims.

13. A vehicle interior space camera system (120, 320) having a camera (124, 324), a first lighting device (126, 326), a second lighting device (128, 328) and a computing unit (130, 330) according to claim 12 and preferably having a further camera.

14. A computer program which, when executed on a computing unit (130, 330), causes the computing unit (130, 330) to perform all the method steps of the method according to any one of claims 1 to 11.

15. A machine readable storage medium having stored thereon the computer program of claim 14.

Images