WO2021121540A1

WO2021121540A1 - Depth estimation system

Info

Publication number: WO2021121540A1
Application number: PCT/EP2019/085292
Authority: WO
Inventors: Radu Ciprian Bilcu; Mikko Muukki
Original assignee: Huawei Technologies Co., Ltd.
Priority date: 2019-12-16
Filing date: 2019-12-16
Publication date: 2021-06-24
Also published as: CN114829977A

Abstract

A depth estimation system (1) for capturing color and distance information from a point (P), the depth estimation system (1) comprising a light emitter (2) and a multi-layered imaging sensor (3), the multi-layered imaging sensor (3) being adapted for absorbing electromagnetic radiation (R) within a plurality of discrete electromagnetic spectrums. This enables capturing different types of information such as distance information and color information for a 3D data point.

Description

DEPTH ESTIMATION SYSTEM

TECHNICAL FIELD

The disclosure relates to a depth estimation system comprising a light emitter.

BACKGROUND

Depth sensing systems are becoming common imaging features in electronic devices such as smartphones. The depth sensing systems enhance camera operations, for instance laser-base focusing, for scene three-dimensional (3D) mapping where the 3D model of the surrounding scene is built, in AR/VR applications and for different scenery objects analysis solutions. The 3D coordinates of the scene points can be calculated from the depth information when knowing the intrinsic parameters of the capturing system.

One example of such depth sensing solutions comprises time-of- flight (ToF) sensors where an infrared (IR) wave is projected onto the scene. The reflections from the scene are captured by an IR sensor which outputs the time passing as the light travels from the IR generator to the IR sensor, for each pixel. A further example comprises structured light (SL) sensors. These modules calculate the depth based on the position, onto an IR sensor, of a certain IR pattern when reflected by the scene, also known as the triangulation principle.

However, these solutions comprise disadvantages. ToF solutions often encounter multipath problems, due to the continuous wave detection method used. SL solutions usually comprise a laser emitter and an IR camera which captures light only in the narrow wavelength band around the wavelength of the laser emitter in order to minimize the effect of ambient light and prevent sensor saturation. As a consequence, the IR camera module, which contains the IR sensor, must be equipped with a narrow IR passband filter. To get other information, such as color information, a further imaging sensor is placed near the IR sensor. Due to the IR sensor and the color sensor seeing slightly different views of the scene, a method to identify the correspondence between the detected dots in the IR image and their position in the color image needs to be used. Calculating the correspondence between the detected dots in the IR image and the geometric calibration of the two cameras introduces extra computations and extra sources of errors. SUMMARY

It is an object to provide an improved depth estimation system. The foregoing and other objects are achieved by the features of the independent claim. Further implementation forms are apparent from the dependent claims, the description, and the figures.

According to a first aspect, there is provided a depth estimation system for capturing color and distance information from a point, the depth estimation system comprising a light emitter and a multi-layered imaging sensor, the multi-layered imaging sensor being adapted for absorbing electromagnetic radiation within a plurality of discrete electromagnetic spectrums.

Such a system enables capturing different types of information for each 3D data point, without having to align the different types of information. By using the same sensor, there is no need for extra calibration, except the calibration steps of a standard SL system.

In a possible implementation form of the first aspect, layers of the multi-layered imaging sensor are stacked on top of each other, each discrete layer absorbing electromagnetic radiation within a predefined, discrete spectrum. This allows each color component, for a certain pixel, to be collected from one and the same horizontal and vertical position, but at different levels inside the sensor structure. As a consequence, the color information will be aligned with the distance information, and pixel saturation is eliminated.

In a further possible implementation form of the first aspect, the electromagnetic radiation comprises first electromagnetic radiation and second electromagnetic radiation, the first electromagnetic radiation originating directly from the object, and the second electromagnetic radiation being emitted by the light emitter and reflected off the point. This enables capturing color information for each 3D data point, without having to align the 3D data and the color image. By using same sensor for capturing the color information and the distance information, a solution which is robust to the pixel saturation effect is achieved, since the detection of each color component happens at different levels of the pixel.

In a further possible implementation form of the first aspect, the first electromagnetic radiation is visible light. In a further possible implementation form of the first aspect, the depth estimation system further comprises a diffractive optical element, diffracting a beam of electromagnetic radiation emitted by the light emitter into a plurality of individual rays, one of the rays being reflected off the point towards the multi-layered imaging sensor.

In a further possible implementation form of the first aspect, the depth estimation system is a structured light depth sensing system.

In a further possible implementation form of the first aspect, the light emitter is a laser emitter.

In a further possible implementation form of the first aspect, the second electromagnetic radiation is infrared radiation.

In a further possible implementation form of the first aspect, the multi-layered imaging sensor comprises a plurality of organic layers.

In a further possible implementation form of the first aspect, the multi-layered imaging sensor comprises a first organic layer, a second organic layer, a third organic layer, and a silicon layer, the first organic layer absorbing electromagnetic radiation within a first spectrum, the second organic layer absorbing electromagnetic radiation within a second spectrum, the third organic layer absorbing electromagnetic radiation within a third spectrum, and the silicon layer absorbing electromagnetic radiation within a fourth spectrum.

In a further possible implementation form of the first aspect, the multi-layered imaging sensor comprises a first organic layer, a second organic layer, a third organic layer, and a fourth organic layer, the first organic layer absorbing electromagnetic radiation within a first spectrum, the second organic layer absorbing electromagnetic radiation within a second spectrum, the third organic layer absorbing electromagnetic radiation within a third spectrum, the fourth organic layer absorbing electromagnetic radiation within a fourth spectrum.

In a further possible implementation form of the first aspect, the first electromagnetic spectrum, the second electromagnetic spectrum, and the third electromagnetic spectrum are within the range of 400-700 nm. In a further possible implementation form of the first aspect, the fourth electromagnetic spectrum is within the range of 800-2500 nm.

According to a second aspect, there is provided an electronic device comprising the depth estimation system according to the above.

These and other aspects will be apparent from the embodiments described below.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following detailed portion of the present disclosure, the aspects, embodiments and implementations will be explained in more detail with reference to the example embodiments shown in the drawings, in which:

Fig. 1 shows a schematic illustration of a depth estimation system in accordance with an embodiment of the present invention;

Fig. 2 shows a schematic perspective view of a multi-layered imaging sensor comprised in a depth estimation system in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION

The present invention relates to an electronic device, such as a camera, smartphone, or tablet, comprising a depth estimation system 1 used for capturing both color information and distance information from a point P, point P being a point located on an object in the surroundings. The depth estimation system 1 may be a structured light depth sensing system.

The depth estimation system 1 comprises a light emitter 2 and a multi-layered imaging sensor 3, as shown in Fig. 1. The light emitter 2 may be a laser emitter.

The multi-layered imaging sensor 3 is adapted for absorbing electromagnetic radiation R within a plurality of discrete electromagnetic spectrums such as the visible light spectrum and the infrared spectrum. The multi-layered imaging sensor 3 comprises a plurality of layers stacked on top of each other, each layer absorbing light only from a certain wavelength, or narrow part of the electromagnetic spectrum. The sensor 3 may comprise any suitable number of layers, but Fig. 2 shows embodiments comprising four layers 3a, 3b, 3c, 3d, 3e. Each discrete layer 3a, 3b, 3c, 3d, 3e absorbs electromagnetic radiation R within a predefined, discrete spectrum. Due to these layers, for a certain pixel, each color component can be collected from one and the same horizontal and vertical position, but at different levels inside the sensor structure. As a consequence, the color information will be aligned with the distance information, and pixel saturation is eliminated.

The electromagnetic radiation R may comprise first electromagnetic radiation R1 and second electromagnetic radiation R2. The first electromagnetic radiation R1 originates directly from the object P, and may comprise color information. The second electromagnetic radiation R2 is initially emitted by the light emitter 2 and thereafter reflected off the point P, and may be used for generating distance information. The first electromagnetic radiation R1 may be visible light, and the second electromagnetic radiation R2 may be infrared radiation.

The depth estimation system 1 may comprise a diffractive optical element 4, which element diffracts a beam of electromagnetic radiation, emitted by the light emitter 2, into a plurality of individual rays. One of the rays R2 are subsequently reflected off the point P in a direction towards the multi-layered imaging sensor 3.

The multi-layered imaging sensor 3 may comprise a plurality of organic layers 3a, 3b, 3c, 3d, i.e. any suitable number of organic layers. The multi-layered imaging sensor 3 may furthermore comprise at least one inorganic layer, such as a silicon layer.

In one embodiment, the multi-layered imaging sensor 3 comprises a first organic layer 3a, a second organic layer 3b, a third organic layer 3c, and a silicon layer 3e. The first organic layer 3a absorbs electromagnetic radiation within a first spectrum, the second organic layer 3b absorbs electromagnetic radiation within a second spectrum, the third organic layer 3c absorbs electromagnetic radiation within a third spectrum, and the silicon layer 3e absorbs electromagnetic radiation within a fourth spectrum.

In a further embodiment, the multi-layered imaging sensor 3 comprises a first organic layer 3a, a second organic layer 3b, a third organic layer 3c, and a fourth organic layer 3d. The first organic layer 3a absorbs electromagnetic radiation within a first spectrum, the second organic layer 3b absorbs electromagnetic radiation within a second spectrum, the third organic layer 3c absorbs electromagnetic radiation within a third spectrum, and the fourth organic layer 3d absorbs electromagnetic radiation within a fourth spectrum.

The fourth electromagnetic spectrum may be within the range of 800-2500 nm.

The first electromagnetic spectrum, the second electromagnetic spectrum, and the third electromagnetic spectrum may be within the range of 400-700 nm. The first spectrum may comprise blue or green wavelengths, the second spectrum green or blue wavelengths, the third spectrum red wavelengths, and the fourth spectrum infrared wavelengths.

In a further embodiment, the fourth layer may effectively record bandpass wavelength for example at nominal wavelength 850 nm, 940 nm, or 1340 nm and having for example 20 nm, 50 nm or 100 nm bandpass width. This may be achieved by having an infrared cut off filter located in front of a first organic layer 3a, where the filter may pass blue, green, and red wavelengths, and has additional bandpass wavelength at e.g. 850 nm, 940 nm, or 1340 nm and a width of for example 20 nm, 50 nm, or 100 nm.

The various aspects and implementations have been described in conjunction with various embodiments herein. However, other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed subject-matter, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measured cannot be used to advantage. A computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

The reference signs used in the claims shall not be construed as limiting the scope. Unless otherwise indicated, the drawings are intended to be read (e.g., cross-hatching, arrangement of parts, proportion, degree, etc.) together with the specification, and are to be considered a portion of the entire written description of this disclosure. As used in the description, the terms “horizontal”, “vertical”, “left”, “right”, “up” and “down”, as well as adjectival and adverbial derivatives thereof (e.g., “horizontally”, “rightwardly”, “upwardly”, etc.), simply refer to the orientation of the illustrated structure as the particular drawing figure faces the reader. Similarly, the terms “inwardly” and “outwardly” generally refer to the orientation of a surface relative to its axis of elongation, or axis of rotation, as appropriate.

Claims

1. A depth estimation system (1) for capturing color and distance information from a point (P), said depth estimation system (1) comprising a light emitter (2) and a multi-layered imaging sensor (3), said multi-layered imaging sensor (3) being adapted for absorbing electromagnetic radiation (R) within a plurality of discrete electromagnetic spectrums.

2. The depth estimation system (1) according to claim 1, wherein layers (3a, 3b, 3c, 3d, 3e) of said multi-layered imaging sensor (3) are stacked on top of each other, each discrete layer (3a, 3b, 3c, 3d, 3e) absorbing electromagnetic radiation (R) within a predefined, discrete spectrum.

3. The depth estimation system (1) according to claim 1 or 2, wherein said electromagnetic radiation (R) comprises first electromagnetic radiation (Rl) and second electromagnetic radiation (R2), said first electromagnetic radiation (Rl) originating directly from said object (P), and said second electromagnetic radiation (R2) being emitted by said light emitter (2) and reflected off said point (P).

4. The depth estimation system (1) according claim 3, wherein said first electromagnetic radiation (Rl) is visible light.

5. The depth estimation system (1) according to any one of the previous claims, further comprising a diffractive optical element (4), diffracting a beam of electromagnetic radiation emitted by said light emitter (2) into a plurality of individual rays, one of said rays (R2) being reflected off said point (P) towards said multi-layered imaging sensor (3).

6. The depth estimation system (1) according to any one of the previous claims, wherein said depth estimation system (1) is a structured light depth sensing system.

7. The depth estimation system (1) according to any one of the previous claims, wherein said light emitter (2) is a laser emitter.

8. The depth estimation system (1) according to any one of claims 3 to 7, wherein said second electromagnetic radiation (R2) is infrared radiation.

9. The depth estimation system (1) according to any one of the previous claims, wherein said multi-layered imaging sensor (3) comprises a plurality of organic layers (3a, 3b, 3c, 3d).

10. The depth estimation system (1) according to any one of the previous claims, wherein said multi-layered imaging sensor (3) comprises a first organic layer (3a), a second organic layer (3b), a third organic layer (3c), and a silicon layer (3e), said first organic layer (3a) absorbing electromagnetic radiation within a first spectrum, said second organic layer (3b) absorbing electromagnetic radiation within a second spectrum, said third organic layer (3c) absorbing electromagnetic radiation within a third spectrum, and said silicon layer (3e) absorbing electromagnetic radiation within a fourth spectrum.

11. The depth estimation system (1) according to any one claims 1 to 9, wherein said multi layered imaging sensor (3) comprises a first organic layer (3a), a second organic layer (3b), a third organic layer (3c), and a fourth organic layer (3d), said first organic layer (3a) absorbing electromagnetic radiation within a first spectrum, said second organic layer (3b) absorbing electromagnetic radiation within a second spectrum, said third organic layer (3c) absorbing electromagnetic radiation within a third spectrum, said fourth organic layer (3d) absorbing electromagnetic radiation within a fourth spectrum.

12. The depth estimation system (1) according to claim 10 or 11, wherein said first electromagnetic spectrum, said second electromagnetic spectrum, and said third electromagnetic spectrum are within the range of 400-700 nm.

13. The depth estimation system according to at least one of claims 10 to 12, wherein said fourth electromagnetic spectrum is within the range of 800-2500 nm.

14. An electronic device comprising the depth estimation system (1) according to any one of claims 1 to 13.