Detailed Description
The embodiments of the present application will be described in detail below. It should be emphasized that the following description is merely exemplary in nature and is not intended to limit the scope of the application and its applications.
FIG. 1 is a schematic diagram of a depth camera based on structured light technology. The depth camera comprises a structured light projection module 11 and a collection module 12, the structured light projection module is used for projecting structured light patterns in a space projection area A, the collection module is used for collecting structured light images on objects in a collection area B, the general projection area A is not lower than the collection area B, and therefore it is guaranteed that the objects in the collection area corresponding to the collection module can be covered by the structured light patterns. In addition, the depth camera also comprises a processor which is used for receiving the structured light image collected by the collection module and carrying out depth calculation on the structured light image to obtain a depth image.
The depth calculation is generally to perform matching calculation on the acquired current structured light image and a pre-acquired and saved reference structured light image to obtain a deviation value of a pixel in the current structured light image relative to a corresponding pixel in the reference structured light image, and based on the deviation value, a depth value can be calculated. The depth values of the plurality of pixels constitute a depth image. The deviation value here generally refers to a deviation value in the direction along the baseline. Therefore, it is generally required that the structured light image has very high irrelevancy along the baseline direction to prevent the occurrence of the mismatch phenomenon.
In one embodiment, the structured light projection module is used for projecting the infrared speckle pattern, the acquisition module is a corresponding infrared camera, and the processor is a special SOC chip. When the depth camera is integrated into other computing terminals as an embedded device, such as a computer, a tablet, a mobile phone, a television, a game machine, an internet of things device, and the like, the functions implemented by the processor described above may be implemented by a processor or an application in the terminal, for example, the depth calculation function is stored in a memory in the form of a software module and called by the processor in the terminal to implement depth calculation.
The structured light projection module mainly comprises a VCSEL array chip, a lens and a DOE, and a structured light pattern, such as a spot pattern, is emitted to a space after the light source chip is converged by the lens and modulated by the DOE.
FIG. 2 shows a schematic view of a single beam passing through a DOE to form a speckle pattern. The light beam 21 is perpendicularly incident on the DOE22, and the spot pattern 24 is projected on a plane with a distance D after diffraction, so that the plane where the DOE is located is an xoy plane, and the direction where the optical axis is located is a z direction, where it is assumed that a connecting line direction between the structured light projection module 11 and the collection module 12 is parallel to the x direction, and in some embodiments, the baseline direction may be any other direction. The spot pattern forms an area 23, which refers to the smallest rectangular area 23 that can contain all the spots 24, with adjacent sides of the rectangular area 23 being parallel to the x and y axes, respectively, and at least one spot on each side, typically the diffraction order of the spot being the highest order along the direction. The Z axis of the optical axis is taken as a starting point, and the included angles formed by the Z axis and the four sides of the rectangular area are thetaxa、θxb、θya、θybThe four included angles can be used to represent the diffraction divergence angle of the DOE22, and also define the angle range of the diffraction spot pattern region 23 after the light beam 21 passes through the DOE22, with the optical axis as the center.
The position of each spot 24 within the spot area 24 is determined by the diffraction equation:
sinθx=mxλ/Px(1)
sinθy=mxλ/Py(2)
in the above equation, θx、θyRespectively, the diffraction angles in the x and y directions, mx、mxDenotes the number of diffraction orders in the x, y directions, respectively, λ denotes the wavelength of the light beam 21, Px、PyRefer to the period of DOE22 in the x and y directions, respectively, i.e., the size of the basic cell.
The arrangement of the speckle pattern 24 depends on the diffraction angles of the beams of the respective diffraction orders of the DOE, which are determined by the performance of the DOE itself.
FIG. 2 shows the case of a light beam incident normal to the DOE, it being understood that when the light beam is incident at an angle, the diffractive regions 23 are offset from normal incidence; furthermore, when a single light source is replaced with a plurality of sub-light sources, such as a VCSEL array chip, the resulting diffraction regions can be viewed as a combination of sub-diffraction regions formed by individual single light beams.
FIG. 3 is a schematic diagram of a structured light projection module according to one embodiment of the present application. The structured light projection module includes an array 31 of multiple light sources (such as VCSEL array chips), a lens 32, and a DOE 33. For convenience of illustration, 3 sub-light sources (sub-light source 1, sub-light source 2 and sub-light source 3 from bottom to top in the figure) are drawn in the figure only in one-dimensional x direction, in an actual embodiment, the number of light sources may reach dozens or even tens of thousands, the light sources may be arranged in two dimensions, the arrangement form may be regular or irregular, in the following description, only the case of one-dimensional regular arrangement is described, and other cases are also applicable to the following description.
The array of light sources 31 emits light beams that form a patterned beam corresponding to the arrangement of light sources, the patterned beam is focused by the lens 32 and then incident on the DOE33, and the DOE33 projects a spot-patterned beam into space, which projects a spot pattern when incident on a plane at a distance D. The convergence refers to that the lens converges an incident light beam with a certain divergence angle and then emits the incident light beam with a smaller divergence angle. The sub-light sources 31 have a pitch nxThe size of the area where the sub-light source is located is sxIn the case of two-dimensional arrangement, the pitch may be n (n)x,ny) To indicate that, likewise, the sub-light sources are located in the regionSize s(s)x,sy) The distance between the light source 31 and the DOE33 is d, which in some embodiments is approximately equal to the focal length of the lens 32.
The lens 32 may be a single lens or a combination of lenses, and in some embodiments is used to collimate the light beam emitted by the light source 31.
The sub-light sources may be non-correlated light sources, in which case, the interference effect between the sub-light sources is negligible, so that the light beams emitted by the sub-light sources 1, 2, 3 form sub-spot patterns 351, 352, and 353 (shown by the dashed elliptic lines in the figure) respectively after passing through the DOE33, and the final spot pattern is formed by combining the sub-spot patterns 351, 352, and 353, in the embodiment shown in fig. 3, since the diffraction divergence angle of the DOE33 is large, the sub-spot patterns overlap with each other, in some embodiments, the diffraction divergence angle of the DOE33 may be set to adjust the overlapping degree, and in the embodiment shown in fig. 4, the sub-spot patterns 451, 452, and 453 do not overlap with each other.
In fig. 3, each sub-spot pattern is composed of 3 spots (for illustration purposes only, there may be any number of spots, and the two-dimensional distribution is possible), corresponding to-1, 0, and 1 order diffracted beams, respectively. For grating diffraction, when the light source moves in a direction parallel to the grating plane, the diffracted beam moves accordingly, i.e. the light spot moves accordingly, and the following relationship is provided:
T=tD/d (3)
in the formula, T and T respectively represent the translation amounts of the light source and the diffraction spot. Therefore, the speckle pattern formed by a sub-light source can be regarded as a speckle pattern formed by translating other sub-light sources, and the relationship between the distance between diffraction spots of corresponding orders in two sub-speckle patterns and the distance between the two sub-light sources is determined by the above formula.
Therefore, the size of the region (361, 362 and 363, shown by the dotted rectangle in the figure) composed of the light spots of the same diffraction order corresponding to the plurality of sub-light sources and the distance between the light spots in the region can be calculated by the following formula:
Nx=nxD/d (4)
Sx=sxD/d (5)
in the formula, Sx、NxRespectively representing the area size formed by the light spots of the same order and the space between the light spots in the area. As can be seen from the above equations, the relationship between the size of the area and the spot pitch within the area, and the size of the light source 31 and the pitch of the sub-light sources is consistent with the lens pinhole imaging model, and therefore the areas 361, 362, and 363 can be regarded as images formed by the light source 31. That is, the finally projected diffracted light beam is a combination of patterned light beams emitted from a plurality of light sources, in other words, the speckle pattern finally formed on the plane 34 is a combination of image patterns of a plurality of light source arrangement patterns. In fig. 3, there is no overlap between image patterns 361, 362, and 363. While in some embodiments there may be overlap between image patterns, such as in the embodiment shown in fig. 4, there may be overlap between image patterns 461, 462, and 463. One of the factors that determines whether the image patterns overlap is the spacing M between diffraction spots of adjacent ordersxThis distance is determined by the performance of the DOE itself. It is understood that the imaging relationship between the image patterns and the light source arrangement patterns may be centrosymmetric, or other imaging relationships may be implemented by designing the lens, such as the same relationship (e.g., the relationship between the image patterns constitutes a duplicate (replica)), mirror images (e.g., the relationship between the image patterns constitutes an axisymmetric mirror image (mirror)), rotation (e.g., there is a certain rotation angle between the image patterns, such as 30 degrees, 45 degrees, 60 degrees, or other suitable angles), and so on.
In the embodiments shown in fig. 3 and 4, the light sources are distributed near the optical axis of the lens, so that the light beam centers of the sub-light sources far away from the optical axis are no longer parallel to the optical axis after being converged by the lens. This deviation will distort the image pattern away from the optical axis, making the overall speckle pattern density non-uniform. Therefore, on the one hand, it is possible to eliminate the non-uniformity as much as possible by making the size of the light source smaller, and on the other hand, it is also possible to reduce the distortion of the image pattern by changing the lens form, such as the embodiment shown in fig. 5, in which the lens 51 is a microlens array. It should be noted that although the distortion is small in the embodiment shown in fig. 5, the diffraction angle of the entire embodiment is also small as compared with the diffraction angles of the embodiments shown in fig. 3 and 4.
In addition to the above distortion affecting the density distribution, the more important factors are the arrangement pitch of the light sources and the performance of the DOE (the pitch between the spots of different diffraction orders), which can make the final spot pattern density distribution uniform only if certain mutual constraint conditions are met. As will be described in detail below.
As can be seen from the above analysis of the embodiment shown in fig. 3, the speckle pattern projected by the structured light projection module may be a combination of sub-speckle patterns formed by diffracting the respective sub-light sources, or may be a combination of image patterns of a plurality of light source array patterns. In the following description, the present application will be explained taking the latter as an example.
A plane at a distance D from the DOE and a regular arrangement of two-dimensional light sources are still chosen for illustration. The size of the light source array is s(s)x,sy) The pitch of the sub-light sources is n (n)x,ny) Thus, the size of the image pattern formed on the plane at the distance D is S (S)x,Sy) The pitch of the spots in the pattern is N (N)x,Ny) As shown in fig. 6, and has the following relationship:
N=nD/d (6)
S=sD/d (7)
the final speckle pattern formed on the plane with the distance D is formed by arranging a plurality of image patterns at a certain interval, wherein the interval refers to the interval between the speckles with different diffraction orders after the DOE diffraction, and therefore, the density distribution of the speckle pattern is determined by the speckle distribution and the interval of the image patterns. FIG. 6 is a schematic view of an image pattern with a dimension S (S)x,Sy) Average spot spacing in the pattern is N (N)x,Ny) It is understood that the two-dimensional arrangement of the light sources may be an irregular two-dimensional arrangement in the present embodiment; FIG. 7 shows a pattern of spots formed in a plane at a distance D from a single beam of light incident on a DOE, the different spots representing different orders of diffraction, the adjacent orders of diffraction being spaced by a distance M (M)x,My). The speckle pattern finally formed by the structured light projection module can be seen as the combination of the image pattern shown in fig. 6 according to the arrangement shown in fig. 7, namely, the combination of the center of the image pattern and each speckle in fig. 7, so as to form the final speckle pattern, as shown in the combination schematic diagram of fig. 8.
In the speckle pattern formed by the projection module shown in fig. 8, M ═ S (((M)x=Sx)&(My=Sy) G) between adjacent image patterns, i.e. the edges just overlap, with a gap G (G) between adjacent image patternsx,Gy) M-S-0. Since the edges of each image pattern generally have diffraction spots of high orders, when the edges are overlapped, the high-order spots in the adjacent patterns may overlap or approach each other, thereby causing the problems of reduced spot number, enlarged individual spot area, or increased partial area density.
Fig. 7 corresponds to DOE diffraction performance, in this embodiment, in order to generate a speckle pattern with relatively uniform density, the arrangement in fig. 7 is a regular arrangement, and the finally generated speckle pattern shown in fig. 8 has high randomness in a local area (single image pattern area), but if the image pattern is regarded as a unit, the overall speckle pattern can still be regarded as a regular arrangement. It will be appreciated that the overall speckle pattern is less uncorrelated, since any small region in a single replicated region can find multiple speckle arrangements identical to that of the small region, in the x-direction or other directions.
Therefore, how to solve the contradiction between uniform density distribution and high degree of irrelevance is an important problem for designing the speckle pattern.
FIG. 9 is a schematic view of a pattern generated by a single beam of light via a DOE according to an embodiment of the present application. In relation to the pattern shown in fig. 7, the arrangement here adds some randomness on a regular basis, thereby increasing the degree of irrelevancy. The average spot spacing in fig. 9 is nearly constant relative to the embodiment shown in fig. 7. Fig. 10 shows a speckle pattern formed by combining the image patterns shown in fig. 6 in the manner shown in fig. 9. The degree of uncorrelation of the spot pattern in this embodiment is improved as compared with fig. 8. For example, due to the interleaving of randomly generated image patterns, the degree of uncorrelation of arbitrarily chosen sub-regions in the baseline x-direction is increased. Fig. 10 is similar to fig. 8 in that the distribution of the spots in the pattern is dense overall, and the random arrangement is such that there are three situations, namely slight gaps, abutment or slight overlap between adjacent image patterns. Since the three connection methods are simultaneously present, the distribution density of the speckle pattern is reduced compared to the uniformity shown in fig. 8, and the problem of overlapping of the speckles is easily caused.
Fig. 11 is a schematic diagram of a speckle pattern with regular gaps according to an embodiment of the present application, and compared with fig. 8, fig. 11 provides a speckle pattern with a significant space between adjacent image patterns, so as to increase the area of the speckle pattern, i.e. increase the field angle.
Fig. 12 is a schematic diagram of a speckle pattern with irregular gaps according to an embodiment of the present application, in which the average distance between adjacent image patterns is larger and the randomness is higher than that in fig. 10, so that the adjacent image patterns are in a spaced state, thereby further improving the degree of irrelevance, and in addition, compared with the embodiments in fig. 8 and 10, the speckle overlapping problem is solved and the field angle is improved.
It should be noted that the above description of "adjacent image patterns are spaced apart from each other" means that there is a significant gap (gap not caused by error) between the adjacent image patterns, i.e. the boundary of the adjacent image patterns can be easily distinguished from the speckle pattern; in other words, this spacing is a design goal of the projection module, and is intended to prevent adjacent image patterns from abutting each other (i.e., the gaps between the image patterns are not significant, such as the edge spots of adjacent image patterns overlap, or the edge spot spacing between adjacent image patterns is close to the inner spot spacing of the image patterns, or there is some overlap between adjacent image patterns). In general, due to the randomness, the image pattern spacing cannot be equal, and in general, an average of the image pattern spacing not less than the average spacing of the spots in the image pattern will produce a spot pattern with significant gaps as shown in FIG. 11 or FIG. 12, i.e., G > N. Moreover, it is not excluded that a small amount of adjacent image patterns may deviate from the design target and generate a small amount of overlap or abutment due to process reasons such as local processing accuracy or component mounting accuracy, and such process errors are allowed in the protection scope of the present application, which does not mean that the technical solution of the present application must unconditionally exclude such errors, as long as a large portion of adjacent image patterns can have a sufficiently significant gap from the overall view.
Due to the existence of the obvious gap, when the depth calculation is carried out based on the speckle pattern, the selection of the matching window generally needs to consider the size of the gap, and theoretically, the size of the matching window in the same direction is not smaller than that of the gap.
In the above description, the square shape of the image pattern and the overall shape of the final speckle pattern are substantially aligned with the x-axis and the y-axis respectively (i.e. two sides of the square are parallel to the x-axis and the y-axis respectively), and the baseline of the structured light projection module and the structured light collection module is in the x-direction. In some embodiments, to further improve the degree of irrelevance, the shape of the speckle pattern may be rotated such that the baseline is not parallel to any side of the speckle pattern and/or the image pattern, that is, an included angle is generated between the side of the speckle pattern and/or the image pattern that is originally parallel to the baseline direction and the baseline, so that the degree of irrelevance of the speckle pattern in the baseline direction may be further improved to improve the depth calculation accuracy. Fig. 13 is a schematic view of the spot pattern after rotation. It is verified that in order to ensure that the speckle pattern will fill the entire field of view of the collection module, the included angle should not be too large, and preferably the included angle is 2 to 10 degrees. It will be appreciated that, because of the randomness, the sides of the speckle pattern referred to herein are not straight sides, and may be understood as the general direction in which the image pattern is arranged in the speckle pattern. It will also be appreciated that the rotation described herein is applicable to all of the embodiments described previously in this application.
The foregoing is a further detailed description of the present application in connection with specific/preferred embodiments and is not intended to limit the present application to that particular description. For a person skilled in the art to which the present application pertains, several alternatives or modifications to the described embodiments may be made without departing from the concept of the present application, and these alternatives or modifications should be considered as falling within the scope of the present application.