[go: up one dir, main page]

WO2023098323A1 - 深度图像的获取方法及装置、系统、计算机可读存储介质 - Google Patents

深度图像的获取方法及装置、系统、计算机可读存储介质 Download PDF

Info

Publication number
WO2023098323A1
WO2023098323A1 PCT/CN2022/125991 CN2022125991W WO2023098323A1 WO 2023098323 A1 WO2023098323 A1 WO 2023098323A1 CN 2022125991 W CN2022125991 W CN 2022125991W WO 2023098323 A1 WO2023098323 A1 WO 2023098323A1
Authority
WO
WIPO (PCT)
Prior art keywords
infrared
image
pixel
depth
information
Prior art date
Application number
PCT/CN2022/125991
Other languages
English (en)
French (fr)
Inventor
吴佳杰
孙牵宇
张宣彪
许亮
王勇
Original Assignee
上海商汤智能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海商汤智能科技有限公司 filed Critical 上海商汤智能科技有限公司
Publication of WO2023098323A1 publication Critical patent/WO2023098323A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image

Definitions

  • the present application relates to the technical field of image processing, and in particular to a depth image acquisition method, device, system, and computer-readable storage medium.
  • the TOF system obtains a 3D image from the distance or depth between the camera and the object to be identified, and uses light emission to illuminate the object to be identified.
  • the time difference between the time and the light reception time of the reflected light from the identified object measures distance or depth;
  • the second is RGB binocular, binocular stereo vision is an important form of machine vision, it is based on the principle of parallax and uses imaging equipment Obtain two images of the measured object from different positions, and obtain the three-dimensional geometric information of the object by calculating the position deviation between the corresponding points of the image.
  • Binocular stereo vision fuses the images obtained by the two eyes and observes the difference between them. , so that we can obtain an obvious sense of depth; the third is the structured light system, the structured light system obtains a 3D image from the depth of the identified object, and the depth is obtained by emitting patterned infrared structured light to the identified object and analyzing the received from the identified object Infrared pattern to measure.
  • RGB binoculars use two sets of cameras, which require two sets of complete hardware structures, so the cost of the system is high, and the power consumption of the system is also high.
  • the system is relatively large; At present, there are few suppliers' solutions, the system consumes high power and lacks RGB information, while structured light requires the cooperation of laser emitting equipment, and the system cost is high.
  • This application provides at least one depth image acquisition method, device, system, and computer-readable storage medium.
  • the first aspect of the present application provides a method for acquiring a depth image, the method comprising: acquiring multiple frames of original images collected by a monocular color-infrared image acquisition module, wherein each frame described The original image includes infrared pixel information, the phase shift angle between the original images of adjacent frames is 2 ⁇ /N, and N is a positive integer greater than 2; based on the infrared pixels in the original images of at least N consecutive frames information, determining depth information corresponding to the infrared pixel information; generating a depth image based on the depth information corresponding to the infrared pixel information.
  • the infrared pixel information includes pixel value information of infrared pixel points; determining the depth information corresponding to the infrared pixel information based on the infrared pixel information in at least N consecutive frames of the original image includes: using The pixel value information of each infrared pixel in at least N consecutive frames of the original image, and calculate the phase offset of the infrared pixel; determine the phase offset of each infrared pixel based on the phase offset of each infrared pixel The depth information is used as the depth information corresponding to the infrared pixel information.
  • the original image includes a plurality of pixel units arranged in a matrix, and each pixel unit includes one infrared pixel point; the generating the depth image based on the depth information corresponding to the infrared pixel information includes: based on each The depth information of the infrared pixel point generates the depth information of the pixel unit where the infrared pixel point is located; the depth image is generated based on the depth information of each pixel unit.
  • each of the pixel units further includes at least one color pixel point; the generating the depth information of the pixel unit where the infrared pixel point is located based on the depth information of each infrared pixel point includes: based on the infrared pixel point corresponding The depth information of the infrared pixel determines the depth information corresponding to the color pixel in the pixel unit where the infrared pixel is located.
  • the generating the depth image based on the depth information corresponding to the infrared pixel information includes: generating a frame of depth image based on the depth information corresponding to the infrared pixel information in N frames of consecutive original images.
  • N 4.
  • the acquisition of multi-frame original images collected by the monocular color-infrared image acquisition module includes: sending an image frame acquisition signal to the monocular color-infrared image acquisition module, the image frame acquisition signal includes control acquisition The phase shift angle between adjacent frame images is a control signal of 2 ⁇ /N, and N is a positive integer greater than 2; receiving the multi-frame acquired by the monocular color-infrared image acquisition module under the control of the above image frame acquisition signal The original image.
  • the second aspect of the present application provides a depth image acquisition device, including: an acquisition module, used to acquire multiple frames of original images collected by a monocular color-infrared image acquisition module, wherein each frame The original image includes infrared pixel information, the phase shift angle between the original images of adjacent frames is 2 ⁇ /N, and N is a positive integer greater than 2; the determination module is used to determine based on at least N consecutive frames of the original The infrared pixel information in the image is used to determine depth information corresponding to the infrared pixel information; a generation module is configured to generate a depth image based on the depth information corresponding to the infrared pixel.
  • the third aspect of the present application provides an image processing device, including a memory and a processor coupled to each other, and the processor is used to execute the program instructions stored in the memory, so as to realize the first aspect or The acquisition method of the depth image in the second aspect.
  • the fourth aspect of the present application provides an image processing system, including: a monocular color-infrared image acquisition module, used to acquire multiple frames of original images, wherein each frame of the original image includes infrared pixels Information, the phase shift angle between the original images of adjacent frames is 2 ⁇ /N, where N is a positive integer greater than 2; the image processing device may include the device in the fourth aspect above, for obtaining The multiple frames of the original image collected by the color-infrared image acquisition module; based on the infrared pixel information in at least N consecutive frames of the original image, determine the depth information corresponding to the infrared pixel information; and based on the The depth information corresponding to the infrared pixels is used to generate a depth image.
  • a monocular color-infrared image acquisition module used to acquire multiple frames of original images, wherein each frame of the original image includes infrared pixels Information, the phase shift angle between the original images of adjacent frames is 2 ⁇ /N, where N is a positive integer greater than 2
  • the above-mentioned image processing device includes a system-on-a-chip SoC and a deserializer
  • the above-mentioned monocular color-infrared image acquisition module also includes a serializer; the above-mentioned monocular color-infrared image acquisition module uses the serializer to collect The original image is transmitted to the deserializer in the form of serial data, and the deserializer deserializes the serial data to obtain the original image and transmits it to the SoC.
  • the fifth aspect of the present application provides a computer-readable storage medium, on which program instructions are stored, and when the program instructions are executed by the processor, the acquisition method of the depth image in the first aspect or the second aspect above is realized .
  • the present application collects multiple frames of original images through the monocular color-infrared image acquisition module. Since each frame of original images includes infrared pixel information, the difference between the original images of adjacent frames The phase shift angle between them is 2 ⁇ /N, and N is a positive integer greater than 2. Therefore, a group of images is composed of at least N frames of continuous original images with a preset phase shift angle, and then it can be based on a set of images with a preset phase shift angle.
  • the infrared pixel information in the original image of the shifted angle determines the depth information corresponding to the infrared pixel information, and then generates a depth image, so the image including depth information can be obtained through a monocular method, the cost of the system is lower, and the power consumption of the system is lower.
  • the use of a monocular camera allows the hardware to have a smaller size, which is conducive to the miniaturization and thinning of the device.
  • FIG. 1 is a schematic flow diagram of an embodiment of a method for acquiring a depth image in the present application
  • Fig. 2 is a schematic flow chart of an embodiment of step S12 in Fig. 1;
  • Fig. 3 is a schematic flow chart of an embodiment of step S13 in Fig. 1;
  • Fig. 4 is a schematic diagram of the corresponding relationship between color images and depth images in the application scene of the acquisition method of depth images in the present application;
  • FIG. 5 is a schematic flowchart of another embodiment of the method for acquiring a depth image in the present application.
  • FIG. 6 is a schematic frame diagram of an embodiment of a depth image acquisition device of the present application.
  • Fig. 7 is a schematic frame diagram of an embodiment of an image processing device of the present application.
  • Fig. 8 is a schematic framework diagram of an embodiment of the image processing system of the present application.
  • Fig. 9 is a schematic diagram of the sensor pixel structure arrangement of the monocular color-infrared image acquisition module in Fig. 8;
  • Fig. 10 is a schematic diagram of an embodiment of a computer-readable storage medium of the present application.
  • system and “network” are often used interchangeably herein.
  • the term “and/or” in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist simultaneously, and there exists alone B these three situations.
  • the character "/” in this article generally indicates that the contextual objects are an “or” relationship.
  • “many” herein means two or more than two.
  • FIG. 1 is a schematic flowchart of an embodiment of a method for acquiring a depth image in the present application. Specifically, the following steps may be included:
  • Step S11 Obtain multiple frames of original images collected by the monocular color-infrared image collection module.
  • each frame of the original image includes infrared pixel information, and the phase shift angle between the original images of adjacent frames is 2 ⁇ /N, where N is a positive integer greater than 2.
  • the identification and monitoring of related targets can be carried out through two-dimensional image recognition, for example, the two-dimensional image information of related targets can be collected by a color image acquisition module such as an RGB image acquisition module, which can be used in this application Color pixel information for identifying and judging related targets.
  • the relevant target may have a complex geometric shape, and when the RGB image acquisition module is used to collect two-dimensional image information, the depth information of the relevant target will be lost, and the recognition accuracy is not high.
  • the infrared image perceives and reflects the difference in radiated energy between the target and the background, which can overcome some visual obstacles and detect the target, and has a large operating distance and anti-interference.
  • the pixels of the infrared image It has good spatial correlation; therefore, in order to overcome the low recognition accuracy of the target in the two-dimensional image, the depth information of the relevant target can be obtained by simultaneously acquiring infrared pixel information.
  • the monocular color-infrared image acquisition module can be an RGB-IR module. Since the RGB-IR module can simultaneously collect color image information and infrared image information, the collected image has color pixels (red pixels Point R, green pixel point G, blue pixel point B) and infrared pixel point IR, so each frame of original image collected by RGB-IR module includes color pixel information recorded by color pixel points, and infrared pixel point Point recorded infrared pixel information.
  • RGB-IR module can simultaneously collect color image information and infrared image information
  • the collected image has color pixels (red pixels Point R, green pixel point G, blue pixel point B) and infrared pixel point IR, so each frame of original image collected by RGB-IR module includes color pixel information recorded by color pixel points, and infrared pixel point Point recorded infrared pixel information.
  • the infrared image information is generated by reflecting the infrared light emitted by the infrared light source in the RGB-IR module, wherein the infrared light emitted by the infrared light source can be a sinusoidal modulation signal, and the difference between the original images of the above-mentioned adjacent frames
  • the phase shift angle may be a phase shift of infrared light between acquisitions of adjacent frames of raw images.
  • the infrared light reflected back from the same position in the original image of each frame The intensity may be different, so the intensity of infrared reflected light sensed by the sensor unit in the RGB-IR module may be different, and the infrared pixel information in the original image collected by the RGB-IR module may be different.
  • the phase shift angle between the original images of adjacent frames taking 90° as an example, after the infrared sinusoidal grating light source emits laser light into the scene, the light receiver can receive the reflected light and sample it every 90° phase way to obtain multiple frames of original images.
  • Step S12 Based on the infrared pixel information in at least N consecutive frames of the original image, determine depth information corresponding to the infrared pixel information.
  • the number of frames of the above-mentioned multiple frames of original images is at least N frames, and the phase shift angles of at least N frames of original images are different.
  • a group of images is composed of at least N frames of original images with phase shift angle differences.
  • the infrared pixel information can obtain multiple frames of infrared images, and then the depth information corresponding to the infrared images can be obtained through the method of monocular depth estimation. It can be understood that the acquired video signal is subjected to signal modulation to obtain a sine wave modulation signal, the original image of this application is each frame of image in the modulation signal, and each frame of image in the modulation signal corresponds to a phase of the sine wave , which is the phase shift angle of the original image.
  • Step S13 Generate a depth image based on the depth information corresponding to the infrared pixel information.
  • a frame of depth image can be generated, and a frame of depth image can be obtained by collecting multiple frames of original images in a monocular way, so that the cost of the system is lower, and the system can have a smaller volume.
  • the above step S13 specifically includes: generating a frame of depth image based on depth information corresponding to infrared pixel information in N consecutive frames of the original image. Therefore, by setting the phase shift angle between the original images of adjacent frames as 2 ⁇ /N, a group of images is composed of N consecutive original images, so all the original images that make up a group of images are located in the same frame period Therefore, based on the infrared pixel information in a group of original images in the same frame period, the depth information corresponding to the infrared pixel information can be determined, and then a frame of depth image can be generated.
  • each frame of original image includes infrared pixel information
  • the phase shift angle between the original images of adjacent frames is 2 ⁇ /N, and N is greater than A positive integer of 2
  • a group of images is composed of at least N frames of continuous original images with a preset phase shift angle, and then based on the infrared pixel information in a group of original images with a preset phase shift angle, the infrared
  • the depth information corresponding to the pixel information and then generate the depth image, so the image including the depth information can be obtained through the monocular method, the cost of the system is lower, and the power consumption of the system is also lower.
  • the monocular color infrared image acquisition mode is adopted
  • the group enables the hardware to have a smaller size, which is conducive to the miniaturization and thinning of the device.
  • FIG. 2 is a schematic flowchart of an embodiment of step S12 in FIG. 1 .
  • the infrared pixel information includes pixel value information of infrared pixel points, and the above step S12 may specifically include the following steps:
  • Step S121 Using the pixel value information of each infrared pixel in at least N consecutive frames of the original image, calculate the phase shift of the infrared pixel.
  • Step S122 Determine the depth information of each infrared pixel based on the phase shift of each infrared pixel, as the depth information corresponding to the infrared pixel information.
  • the corresponding infrared pixel information in multiple frames of continuous original images collected in one cycle may be different, so , using the infrared pixel information of each infrared pixel in at least N consecutive frames of the original image, the depth information of the corresponding infrared pixel can be obtained.
  • the above step S121 may specifically include: based on the difference between the infrared pixel information of the infrared pixel in different frames of the original image, obtaining the phase shift of the corresponding infrared pixel.
  • the infrared pixel information in the original image may be different, so based on the difference between the infrared pixel information of the infrared pixel in different frames of the original image, the phase shift of the corresponding infrared pixel is obtained.
  • the above step S122 may specifically include: using the phase shift of each infrared pixel and the modulation frequency of the original image to obtain the depth information of the corresponding infrared pixel as the depth information corresponding to the infrared pixel information in the original image of each frame .
  • the depth information of infrared pixels is related to the phase offset of infrared pixels and the modulation frequency of the original image.
  • the modulation frequency of the original image is known, so the calculated phase of each infrared pixel can be used The offset and the known modulation frequency of the original image make it more convenient to obtain the depth information of the infrared pixels.
  • the above solution uses the pixel value information of an infrared pixel in each frame of the original image to obtain the phase offset of the infrared pixel, and then obtains the corresponding depth information through the phase offset of the infrared pixel, so that the infrared pixel
  • the depth information of a point is associated with its phase offset, which facilitates the acquisition of depth information.
  • FIG. 3 is a schematic flowchart of an embodiment of step S13 in FIG. 1 .
  • the original image includes a plurality of pixel units arranged in a matrix, and each pixel unit includes one infrared pixel point; the above-mentioned step S13 may specifically include the following steps:
  • Step S131 Generate depth information of the pixel unit where the infrared pixel is located based on the depth information of each infrared pixel.
  • Step S132 Generate a depth image based on the depth information of each pixel unit.
  • each infrared pixel point can be used to generate
  • the pixel value information in the image can obtain the depth information of each infrared pixel, and based on the depth information of each infrared pixel, the depth information of the pixel unit where each infrared pixel is located can be generated.
  • each of the pixel units further includes at least one color pixel point
  • the color pixel point may include a red pixel point R, a green pixel point G or a blue pixel point B
  • the above step S131 may specifically include Determining, based on the depth information corresponding to the infrared pixel, the depth information corresponding to the color pixel in the pixel unit where the infrared pixel is located.
  • each pixel unit also includes at least one color pixel point
  • the depth information corresponding to the color pixel point in the pixel unit where the infrared pixel point is located can be determined based on the depth information corresponding to a certain infrared pixel point.
  • the multi-frame original image collected by the infrared image acquisition module can not only generate a frame of depth image, but also generate multi-frame color images correspondingly according to the color pixels of the multi-frame original image, that is, generate a corresponding frame of depth at the same time image and multi-frame color images, so the monocular color-infrared image acquisition module can realize the acquisition of color images and depth images at the same time, and the corresponding one-frame depth image and multi-frame color images are synchronized.
  • the monocular color-infrared image collection module can collect color information by receiving color signals, and collect depth information by emitting infrared pulse signals and receiving infrared reflection signals.
  • the monocular color-infrared image acquisition module can include an infrared emitting component to transmit infrared pulse signals to the environment ahead. After the infrared pulse signals reach objects in the environment, a part of them is absorbed and emitted in the form of radiation, and the other A part is reflected back; the monocular color-infrared image acquisition module can also include an infrared reflection receiving component, which receives the infrared reflection signal and processes it into a depth image based on the calculation of the phase shift angle difference.
  • color pixel information and infrared pixel information can be obtained, and a depth image can be obtained from the infrared pixel information.
  • At least N frames of continuous original images collected by the monocular color-infrared image acquisition module can not only generate a frame of depth image, but also can correspond to the color pixel information of at least N frames of continuous original images.
  • Generating at least N frames of color images means simultaneously generating a corresponding frame of depth images and at least N frames of color images, so the corresponding frame of depth images and at least N frames of color images are synchronized. It can be understood that, if the synchronous depth image and the color image are fused, the registration process during the fusion of the depth image and the color image can be omitted, and the obtained fusion effect is better.
  • FIG. 4 is a schematic diagram of a corresponding relationship between a color image and a depth image in an application scenario of the depth image acquisition method of the present application.
  • the number of frames of at least one original image is four frames, and the phase shift angle difference between every two adjacent frames of original images is 90 degrees, as shown in the figure, since each frame of original image includes color pixel information , therefore, each frame of original image corresponds to a frame of color image, that is, RGB frame, and since each frame of original image includes infrared pixel information, and a frame of depth image is generated according to the infrared pixel information in four frames of original images, that is, depth frame , so the generated one-frame depth image and four-frame color image are synchronized.
  • the frame rate of the color image is 4 times that of the depth image.
  • the frame rate of the original image frame is 120fps
  • the frame rate of the color image is also 120fps.
  • the frame rate of the depth image is 30fps.
  • N 4.
  • the number of frames of the original images constituting a group of images is four frames, and the phase shift angle between every two adjacent frames of original images is ⁇ /2.
  • the four frames of original images are divided into two groups of original image groups; wherein, each group of original image groups includes two frames of non-adjacent original images;
  • the difference value of the infrared pixel information in the image, and arctangent processing is performed on the ratio of the infrared pixel point between the difference values of the two original image groups to obtain the phase shift of the infrared pixel point.
  • the preset phase shift angle is 90 degrees, that is, the phase shift angle between two adjacent frames of the original image is 90 degrees; for example, the first frame of the original image The phase shift angle is 0 degree, the phase shift angle of the second frame original image is 90 degree, the phase shift angle of the third frame original image is 180 degree, the phase shift angle of the fourth frame original image is 270 degree, the first frame The original image and the third frame original image are divided into the first group of original image groups, and the second frame original image and the fourth frame original image are divided into the second group of original image groups.
  • the infrared pixel information in one frame of original image is A0°
  • the infrared pixel information in the second frame of original image is A90°
  • the infrared pixel information in the third frame of original image is A180°
  • the infrared pixel information in the fourth frame of original image The infrared pixel information of the infrared pixel is A270°, so the difference value of the infrared pixel information of the infrared pixel in the first group of original image groups can be obtained A0°-A180°
  • the infrared pixel information of the second group of original image groups The difference value is A270°-A90°
  • arctangent processing is performed on the ratio of the infrared pixel point between the difference values of the two original image groups to obtain the phase shift of the infrared pixel point.
  • the infrared pixel information of the infrared pixel in each frame of the original image is the integrated value of the light intensity within the exposure time of each frame of the original image.
  • the depth information of the infrared pixel is directly proportional to the phase shift of the infrared pixel, and is inversely proportional to the modulation frequency.
  • the relationship between the depth information of infrared pixels, the phase shift of infrared pixels and the modulation frequency is shown in formula (2):
  • D is the depth information of infrared pixels
  • c is the light speed value (3*108m/s)
  • fmod is the modulation frequency
  • fmod can be preset by the user, and is a known quantity.
  • the phase shift and depth information of the infrared pixel point can be calculated by the above formulas (1) and (2), avoiding complex depth calculation formulas, and realizing fast and accurate acquisition of phase shift angle and depth information.
  • the number of frames of the original images constituting a group of images can also be other numbers of frames, that is, N can also be other positive integers, such as 3, 5, 8, etc.; for example, a group of The number of frames of the original image is eight frames, and the phase shift angle between two adjacent frames of the original image is ⁇ /4.
  • a frame of depth image corresponding to a group of original images can also be obtained based on the above-mentioned similar principle.
  • FIG. 5 is a schematic flowchart of another embodiment of a method for acquiring a depth image in the present application. Specifically, the following steps may be included:
  • Step S51 Send an image frame acquisition signal to the monocular color-infrared image acquisition module, the image frame acquisition signal includes a control signal that controls the phase shift angle between the images of adjacent frames collected to be 2 ⁇ /N, and N is A positive integer greater than 2.
  • Step S52 Receive multiple frames of original images acquired by the monocular color-infrared image acquisition module under the control of the image frame acquisition signal, wherein each frame of the original image includes infrared pixel information.
  • Step S53 Based on the infrared pixel information in at least N consecutive frames of the original image, determine depth information corresponding to the infrared pixel information.
  • Step S54 Generate a depth image based on the depth information corresponding to the infrared pixels.
  • steps S52-S54 are basically similar to steps S11-S13 in the above-mentioned embodiment of the present application, and will not be repeated here.
  • the execution subject of the method for acquiring a depth image in the embodiment of the present application may be an image processing device.
  • the method for acquiring a depth image may be executed by a terminal device or a server or other electronic equipment, wherein the terminal device may be a user equipment (User Equipment, UE), mobile device, user terminal, terminal, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device, etc.
  • the method for acquiring a depth image may be implemented by a processor invoking computer-readable instructions stored in a memory.
  • the image processing device is connected to the monocular color-infrared image acquisition module.
  • the image processing device sends the image frame acquisition signal to the monocular color-infrared image acquisition module, since the image frame acquisition signal includes adjacent
  • the phase shift angle between frame images is a control signal of 2 ⁇ /N, and N is a positive integer greater than 2, so the monocular color-infrared image acquisition module can collect signals according to the image frame, acquire multiple frames of original images and send them to
  • each frame of the original image includes infrared pixel information, the phase shift angle between the original images of adjacent frames is 2 ⁇ /N, and N is a positive integer greater than 2;
  • the image processing device receives the monocular color -
  • the infrared image acquisition module acquires multiple frames of original images under the control of the image frame acquisition signal, it can determine the depth information corresponding to the infrared pixel information based on the infrared pixel information in at least N frames of continuous original images, and based on the infrared pixel
  • each frame of original image includes infrared pixel information
  • the phase shift angle between the original images of adjacent frames is 2 ⁇ /N
  • N is greater than 2 is a positive integer
  • a group of images is composed of at least N frames of continuous original images with a preset phase shift angle
  • the infrared pixel can be determined based on the infrared pixel information in a group of original images with a preset phase shift angle
  • the depth information corresponding to the information, and then generate the depth image so the image including the depth information can be obtained through the monocular method, the cost of the system is lower, and the power consumption of the system is also lower.
  • the hardware can have a smaller size, which is useful It is beneficial to realize the miniaturization and thinning of the equipment.
  • FIG. 6 is a schematic frame diagram of an embodiment of an apparatus for acquiring a depth image according to the present application.
  • the acquisition device 60 of the depth image includes: an acquisition module 600, a determination module 602, and a generation module 604; the acquisition module 600 is used to acquire multi-frame original images collected by a monocular color-infrared image acquisition module, wherein each frame described
  • the original image includes infrared pixel information, the phase shift angle between the original images of adjacent frames is 2 ⁇ /N, and N is a positive integer greater than 2;
  • the determination module 602 is used in the original images based on at least N consecutive frames
  • the infrared pixel information determine the depth information corresponding to the infrared pixel information; the generating module 604 is configured to generate a depth image based on the depth information corresponding to the infrared pixel information.
  • the acquisition module 600 collects multiple frames of original images through the monocular color-infrared image acquisition module. Since each frame of original image includes infrared pixel information, the phase shift angle between the original images of adjacent frames is 2 ⁇ /N, N is a positive integer greater than 2. Therefore, a group of images is composed of at least N frames of continuous original images with a preset phase shift angle, and then the determination module 602 can be based on a group of original images with a preset phase shift angle.
  • the infrared pixel information determines the depth information corresponding to the infrared pixel information, so the generation module 604 can further generate a depth image, so the acquisition of an image including depth information through a monocular method is realized, the cost of the system is lower, and the power consumption of the system is also lower , At the same time, the hardware can have a smaller size, which is conducive to the miniaturization and thinning of the device.
  • the infrared pixel information includes pixel value information of infrared pixel points; the determination module 602 executes to determine the infrared pixel information corresponding to the infrared pixel information based on the infrared pixel information in at least N consecutive frames of the original image.
  • the step of depth information includes: using the pixel value information of each infrared pixel in at least N consecutive frames of the original image to calculate the phase offset of the infrared pixel; The depth information of each infrared pixel point is used as the depth information corresponding to the infrared pixel information.
  • the original image includes a plurality of pixel units arranged in a matrix, and each pixel unit includes one infrared pixel point; the generating module 604 executes the process of generating a depth image based on the depth information corresponding to the infrared pixel information
  • the steps include: generating depth information of the pixel unit where the infrared pixel is located based on the depth information of each infrared pixel; generating a depth image based on the depth information of each pixel unit.
  • each pixel unit further includes at least one colored pixel point; the step of generating the depth information of the pixel unit where the infrared pixel point is located based on the depth information of each infrared pixel point by the generation module 604 specifically includes: The depth information corresponding to the infrared pixel determines the depth information corresponding to the color pixel in the pixel unit where the infrared pixel is located.
  • the step of generating the depth image based on the depth information corresponding to the infrared pixel information by the generation module 604 includes: generating a frame of depth image based on the depth information corresponding to the infrared pixel information in N frames of consecutive original images .
  • N 4.
  • the acquisition module 600 executes the step of acquiring multi-frame original images acquired by the monocular color-infrared image acquisition module comprising: sending an image frame acquisition signal to the monocular color-infrared image acquisition module, the The image frame acquisition signal includes a control signal that controls the phase shift angle between the adjacent frame images collected to be 2 ⁇ /N, and N is a positive integer greater than 2; receiving the monocular color-infrared image acquisition module in the image Multi-frame original images acquired under the control of the frame acquisition signal.
  • FIG. 7 is a schematic frame diagram of an embodiment of an image processing device of the present application.
  • the image processing device 70 includes a memory 71 and a processor 72 coupled to each other.
  • the processor 72 is configured to execute program instructions stored in the memory 71 to implement the steps of any one of the above embodiments of the method for acquiring a depth image.
  • the image processing device 70 may include, but is not limited to: a microcomputer, a server.
  • the image processing device 70 may also include a part of an electronic device with a display function such as a notebook computer, a smart phone, and a vehicle computer. It is not limited here.
  • the processor 72 is configured to control itself and the memory 71 to implement the steps in any one of the above embodiments of the method for acquiring a depth image.
  • the processor 72 may also be called a CPU (Central Processing Unit, central processing unit).
  • the processor 72 may be an integrated circuit chip with signal processing capabilities.
  • the processor 72 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
  • a general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.
  • the processor 72 may be jointly implemented by an integrated circuit chip.
  • each frame of original image includes infrared pixel information
  • the phase shift angle between the original images of adjacent frames is 2 ⁇ /N, and N is greater than A positive integer of 2
  • a group of images is composed of at least N frames of continuous original images with a preset phase shift angle, and then based on the infrared pixel information in a group of original images with a preset phase shift angle, the infrared
  • the depth information corresponding to the pixel information and then generate the depth image, so the image including the depth information can be obtained through the monocular method, the cost of the system is lower, and the power consumption of the system is also lower.
  • the hardware can have a smaller size. It is beneficial to realize the miniaturization and thinning of the equipment.
  • FIG. 8 is a schematic frame diagram of an embodiment of an image processing system of the present application.
  • the image processing system 80 includes an interconnected monocular color-infrared image acquisition module 81 and an image processing device 82; the monocular color-infrared image acquisition module 81 is used to acquire multiple frames of original images, wherein each frame of the original The image includes infrared pixel information, and the phase shift angle between the original images of adjacent frames is 2 ⁇ /N, where N is a positive integer greater than 2; the image processing device 82 is the above-mentioned image processing device 70, and the image processing device 82 It is used to obtain multiple frames of the original image collected by the monocular color-infrared image acquisition module; based on the infrared pixel information in the original image in at least N consecutive frames, determine the corresponding infrared pixel information depth information; and generating a depth image based on the depth information corresponding to the infrared pixels.
  • the monocular color-infrared image acquisition module 81 is an RGB-IR module, and the monocular color-infrared image acquisition module 81 includes an RGB-IR sensor 810, through which RGB-IR sensor 810 can acquire multiple Frames of original images, each frame of original images includes infrared pixel information, the phase shift angle between the original images of adjacent frames is 2 ⁇ /N, and N is a positive integer greater than 2;
  • the image processing device 82 includes a system-on-chip SoC (System On Chip) 820, the system level chip 820 can process at least N frames of continuous original images, based on the infrared pixel information in at least N frames of continuous original images, determine the depth information corresponding to the infrared pixel information, and based on the corresponding infrared pixel information The depth information generates a depth image.
  • SoC System On Chip
  • FIG. 9 is a schematic diagram of the sensor pixel structure arrangement of the monocular color-infrared image acquisition module in FIG. 8.
  • the RGB-IR sensor 810 every four pixels arranged in a matrix contain one infrared pixel IR and three color pixels, and every eight pixels serve as a group of pixel basic units, including a complete Color pixel information and infrared pixel information (including red pixel point R, green pixel point G, blue pixel point B and infrared pixel point IR).
  • RGB-IR RGB-IR sensor
  • the system-on-chip 820 acquires at least N frames of continuous After the original image, the depth information of 500,000 infrared pixels IR is extracted. Since the phase offset ⁇ of each infrared pixel IR may be different, the depth information is also inconsistent, so that 500,000 infrared pixels can be obtained. A full depth image with one-to-one correspondence of depth information.
  • the monocular color-infrared image acquisition module 81 also includes a serializer 812, and the RGB-IR sensor 810 is connected to the serializer 812 through an I2C interface and a MIPI interface;
  • the image processing device 82 also includes a deserializer 822.
  • the deserializer 822 is connected to the SoC 820 through the MIPI interface and the I2C interface;
  • the serializer 812 is connected to the deserializer 822 through the HSD interface.
  • the monocular color-infrared image acquisition module 81 transmits the acquired original image to the deserializer 822 in the form of serial data through the serializer 812, and the deserializer 822 deserializes the serial data to obtain the original image and transmits it to SoC 820 .
  • the monocular color-infrared image acquisition module 81 is connected to the image processing device 82 through the HSD connector, and the HSD connector is used for data transmission and equipment power supply.
  • the SoC 820 of the image processing device 82 configures the RGB-IR sensor 810 through the deserializer 822 and the serializer 812 to start the image acquisition function.
  • the RGB-IR sensor 810 can generate a video signal, and the video signal is transmitted to the serializer 812 through the MIPI interface, and then the serializer 812 packages the data and sends it to the deserializer 822 through the HSD interface, and finally sends it to the system level Chip 820, at this time, the data received by the SoC 820 is the original image including color pixel information and infrared pixel information, so the SoC 820 can use any of the above methods for acquiring depth images to perform image processing to obtain the corresponding After that, the color image and the depth image can be displayed on the screen, and the corresponding depth image and the color image have synchronization.
  • FIG. 10 is a schematic frame diagram of an embodiment of a computer-readable storage medium of the present application.
  • the computer-readable storage medium 10 stores program instructions 100 that can be executed by a processor, and the program instructions 100 are used to implement the steps in any of the above-mentioned embodiments of the depth image acquisition method.
  • each frame of original image includes infrared pixel information
  • the phase shift angle between the original images of adjacent frames is 2 ⁇ /N, and N is greater than A positive integer of 2
  • a group of images is composed of at least N frames of continuous original images with a preset phase shift angle, and then based on the infrared pixel information in a group of original images with a preset phase shift angle, the infrared
  • the depth information corresponding to the pixel information and then generate the depth image, so the image including the depth information can be obtained through the monocular method, the cost of the system is lower, and the power consumption of the system is also lower.
  • the hardware can have a smaller size. It is beneficial to realize the miniaturization and thinning of the equipment.
  • the disclosed methods, devices and systems can be implemented in other ways.
  • the device implementations described above are only illustrative.
  • the division of modules or units is only a logical function division. In actual implementation, there may be other division methods.
  • units or components can be combined or integrated. to another system, or some features may be ignored, or not implemented.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
  • a unit described as a separate component may or may not be physically separated, and a component shown as a unit may or may not be a physical unit, that is, it may be located in one place, or may also be distributed to network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.
  • the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
  • the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) execute all or part of the steps of the methods in various embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc., which can store program codes. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Studio Devices (AREA)
  • Measurement Of Optical Distance (AREA)

Abstract

本申请公开了一种深度图像的获取方法及装置、系统、计算机可读存储介质，其中，深度图像的获取方法包括：获取利用单目彩色-红外图像采集模组采集得到的多帧原始图像，其中，每帧所述原始图像包括红外像素信息，相邻帧的所述原始图像之间的相移角为2π/N，N为大于2的正整数；基于至少N帧连续的所述原始图像中的所述红外像素信息，确定所述红外像素信息对应的深度信息；基于所述红外像素信息对应的深度信息生成深度图像。上述方案，能够在低成本条件下获取深度图像。

Description

深度图像的获取方法及装置、系统、计算机可读存储介质
本申请要求在2021年11月30日提交中国专利局、申请号为202111445318.7、申请名称为“深度图像的获取方法及装置、系统、计算机可读存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理技术领域，特别是涉及一种深度图像的获取方法及装置、系统、计算机可读存储介质。
背景技术
随着光学测量技术以及计算机视觉的发展，已经发展了用于获得深度图像的各种技术和产品。目前行业主流的深度相机有三种：一是TOF(Time Of Flight，飞行时间)系统，TOF系统从摄像机与被识别对象之间的距离或深度获得3D图像，使用向被识别对象照射光的光发射时间与从被识别对象反射光的光接收时间之间的时间差分测量距离或深度；二是RGB双目，双目立体视觉是机器视觉的一种重要形式，它是基于视差原理并利用成像设备从不同的位置获取被测物体的两幅图像，通过计算图像对应点间的位置偏差，来获取物体三维几何信息的方法，双目立体视觉融合两只眼睛获得的图像并观察它们之间的差别，使我们可以获得明显的深度感；三是结构光系统，结构光系统从被识别对象的深度获得3D图像，深度通过向被识别对象发射图案化的红外结构光并分析从被识别对象接收的红外线的图案来测量。
但是，上述几种方式都存在一定的不足。例如，RGB双目采用两组摄像头，需要两套完整硬件结构，所以系统的成本高、系统的功耗也高，同时，系统体积较大；虽然TOF和结构光可以采用单目方式，但TOF当前供应商方案较少，系统功耗高且缺失RGB信息，而结构光需要激光发射设备配合，系统成本高。
发明内容
本申请提至少供一种深度图像的获取方法及装置、系统、计算机可读 存储介质。
为了解决上述问题，本申请第一方面提供了一种深度图像的获取方法，所述方法包括：获取利用单目彩色-红外图像采集模组采集得到的多帧原始图像，其中，每帧所述原始图像包括红外像素信息，相邻帧的所述原始图像之间的相移角为2π/N，N为大于2的正整数；基于至少N帧连续的所述原始图像中的所述红外像素信息，确定所述红外像素信息对应的深度信息；基于所述红外像素信息对应的深度信息生成深度图像。
可选地，所述红外像素信息包括红外像素点的像素值信息；所述基于至少N帧连续的所述原始图像中的所述红外像素信息，确定红外像素信息对应的深度信息，包括：利用至少N帧连续的所述原始图像中的各红外像素点的像素值信息，计算所述红外像素点的相位偏移；基于各个所述红外像素点的相位偏移确定各个所述红外像素点的深度信息，作为所述红外像素信息对应的深度信息。
可选地，所述原始图像包括呈矩阵排列的多个像素单元，每个像素单元包括一个所述红外像素点；所述基于所述红外像素信息对应的深度信息生成深度图像，包括：基于各红外像素点的深度信息生成所述红外像素点所在像素单元的深度信息；基于各所述像素单元的深度信息生成深度图像。
可选地，每个所述像素单元还包括至少一个彩色像素点；所述基于各红外像素点的深度信息生成所述红外像素点所在像素单元的深度信息，包括：基于所述红外像素点对应的深度信息确定所述红外像素点所在像素单元中的彩色像素点对应的深度信息。
可选地，所述基于所述红外像素信息对应的深度信息生成深度图像，包括：基于N帧连续的所述原始图像中的红外像素信息对应的深度信息生成一帧深度图像。
可选地，N＝4。
可选地，所述获取利用单目彩色-红外图像采集模组采集得到的多帧原始图像，包括：向单目彩色-红外图像采集模组发送图像帧采集信号，图像帧采集信号包括控制采集的相邻帧图像之间的相移角为2π/N的控制信号，N为大于2的正整数；接收单目彩色-红外图像采集模组在上述图像帧采集信号控制下采集得到的多帧原始图像。
为了解决上述问题，本申请第二方面提供了一种深度图像的获取装置，包括：获取模块，用于获取利用单目彩色-红外图像采集模组采集得到的多帧原始图像，其中，每帧所述原始图像包括红外像素信息，相邻帧的所述原始图像之间的相移角为2π/N，N为大于2的正整数；确定模块，用于基于至少N帧连续的所述原始图像中的所述红外像素信息，确定所述红外像素信息对应的深度信息；生成模块，用于基于所述红外像素对应的深度信息生成深度图像。
为了解决上述问题，本申请第三方面提供了一种图像处理装置，包括相互耦接的存储器和处理器，所述处理器用于执行所述存储器中存储的程序指令，以实现上述第一方面或第二方面中的深度图像的获取方法。
为了解决上述问题，本申请第四方面提供了一种图像处理系统，包括：单目彩色-红外图像采集模组，用于采集得到多帧原始图像，其中，每帧所述原始图像包括红外像素信息，相邻帧的所述原始图像之间的相移角为2π/N，N为大于2的正整数；图像处理装置，可以包括上述第四方面中的装置，用于获取利用所述单目彩色-红外图像采集模组采集得到的多帧所述原始图像；基于至少N帧连续的所述原始图像中的所述红外像素信息，确定所述红外像素信息对应的深度信息；以及基于所述红外像素对应的深度信息生成深度图像。
可选地，上述图像处理装置包括系统级芯片SoC和解串器，上述单目彩色-红外图像采集模组还包括串化器；上述单目彩色-红外图像采集模组通过串化器将采集的原始图像以串行数据的方式传输至解串器，解串器对串行数据进行解串以获得原始图像并传输至系统级芯片。
为了解决上述问题，本申请第五方面提供了一种计算机可读存储介质，其上存储有程序指令，程序指令被处理器执行时实现上述第一方面或第二方面中的深度图像的获取方法。
本申请的有益效果：区别于现有技术的情况，本申请通过单目彩色-红外图像采集模组采集得到多帧原始图像，由于每帧原始图像包括红外像素信息，相邻帧的原始图像之间的相移角为2π/N，N为大于2的正整数，因此，由至少N帧具有预设相移角的连续的原始图像组成一组图像，进而可以基于一组具有预设相移角的原始图像中的红外像素信息，确定红外像素 信息对应的深度信息，进而生成深度图像，故实现了通过单目方式获取包括深度信息的图像，系统的成本更低，系统的功耗也更低，同时，采用单目摄像头使得硬件可以拥有更小巧的体积，有利于实现设备的小型化和轻薄化。
附图说明
为了更清楚地说明本发明实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其它的附图，其中：
图1是本申请深度图像的获取方法一实施例的流程示意图；
图2是图1中步骤S12一实施例的流程示意图；
图3是图1中步骤S13一实施例的流程示意图；
图4是本申请深度图像的获取方法一应用场景中彩色图像和深度图像的对应关系示意图；
图5是本申请深度图像的获取方法另一实施例的流程示意图；
图6是本申请深度图像的获取装置一实施例的框架示意图；
图7是本申请图像处理装置一实施例的框架示意图；
图8是本申请图像处理系统一实施例的框架示意图；
图9是图8中单目彩色-红外图像采集模组的传感器像素点结构排布示意图；
图10是本申请计算机可读存储介质一实施例的框架示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅是本申请的一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
本文中术语“系统”和“网络”在本文中常被可互换使用。本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系， 例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。此外，本文中的“多”表示两个或者多于两个。
请参阅图1，图1是本申请深度图像的获取方法一实施例的流程示意图。具体而言，可以包括如下步骤：
步骤S11：获取利用单目彩色-红外图像采集模组采集得到的多帧原始图像。其中，每帧所述原始图像包括红外像素信息，相邻帧的所述原始图像之间的相移角为2π/N，N为大于2的正整数。
在一个实施场景中，对相关目标的识别与监测，可以通过二维图像识别进行，例如通过彩色图像采集模组如RGB图像采集模组采集相关目标的二维图像信息，可以是本申请中的彩色像素信息，以进行相关目标的识别判断。然而，相关目标可能具有复杂的几何形状，而在利用RGB图像采集模组采集二维图像信息时，相关目标的深度信息会丢失，识别精度不高。而红外图像感受和反映的是目标及背景向外辐射能量的差异，其可以克服部分视觉上的障碍而探测到目标，具有较大的作用距离和抗干扰性，另外，红外图像的像素之间具有良好的空间相关性；因此，为了克服二维图像对目标的识别精度不高的情况，可以通过同时获取红外像素信息，来得到相关目标的深度信息。
在一个实施场景中，单目彩色-红外图像采集模组可以为RGB-IR模组，由于RGB-IR模组可以同时采集彩色图像信息和红外图像信息，采集的图像具有彩色像素点(红色像素点R、绿色像素点G和蓝色像素点B)以及红外像素点IR，因此通过RGB-IR模组采集得到的每帧原始图像中，包括有彩色像素点记录的彩色像素信息，和红外像素点记录的红外像素信息。其中，红外图像信息是由RGB-IR模组中的红外光源发射出的红外光经过反射后产生的，其中红外光源发射的红外光可以是正弦调制信号，上述相邻帧的原始图像之间的相移角可以是采集相邻帧的原始图像之间的红外光的相位偏移。，由于相邻帧的原始图像之间的相移角为2π/N，N为大于2的正整数，故在一个相移角周期内，每帧原始图像中的同一个位置反射回的红外光强可能不同，由此RGB-IR模组中的传感单元感应到的红外反射光强度可能不同，进而在RGB-IR模组采集到的原始图像中的红外像素信 息可能不同。
相邻帧的原始图像之间的相移角，以90°为例，在红外正弦光栅光源向场景中发射激光后，光接收器可以接收反射光线并通过每隔90°的相位进行一次采样的方式，获得多帧原始图像。
步骤S12：基于至少N帧连续的所述原始图像中的所述红外像素信息，确定所述红外像素信息对应的深度信息。
上述多帧原始图像的帧数为至少N帧，至少N帧原始图像的相移角不相同，由至少N帧具有相移角差的原始图像组成一组图像，通过这一组图像中的红外像素信息可以得到多帧红外图像，然后可以通过单目深度估计的方法来获得红外图像所对应的深度信息。可以理解的是，将采集得到的视频信号经过信号调制而得到正弦波调制信号，本申请的原始图像即为调制信号中的每帧图像，而调制信号中的每帧图像在正弦波对应一个相位，即为原始图像的相移角。
步骤S13：基于所述红外像素信息对应的深度信息生成深度图像。
基于红外像素信息对应的深度信息，可以生成一帧深度图像，实现了通过单目方式采集多帧原始图像来获取一帧深度图像，使系统的成本更低，系统可以拥有更小巧的体积。
在一实施例中，上述步骤S13具体包括：基于N帧连续的所述原始图像中的红外像素信息对应的深度信息生成一帧深度图像。因此，通过将相邻帧的原始图像之间的相移角设置为2π/N，由N帧连续的原始图像组成一组图像，故组成一组图像的所有原始图像位于同一个帧周期内，于是可以基于同一个帧周期内的一组原始图像中的红外像素信息，确定红外像素信息对应的深度信息，进而生成一帧深度图像。
上述方案，通过单目彩色-红外图像采集模组采集得到多帧原始图像，由于每帧原始图像包括红外像素信息，相邻帧的原始图像之间的相移角为2π/N，N为大于2的正整数，因此，由至少N帧具有预设相移角的连续的原始图像组成一组图像，进而可以基于一组具有预设相移角的原始图像中的红外像素信息，确定红外像素信息对应的深度信息，进而生成深度图像，故实现了通过单目方式获取包括深度信息的图像，系统的成本更低，系统的功耗也更低，同时，采用单目彩色红外图像采集模组使得硬件可以拥有 更小巧的体积，有利于实现设备的小型化和轻薄化。
请参阅图2，图2是图1中步骤S12一实施例的流程示意图。本实施例中，所述红外像素信息包括红外像素点的像素值信息，上述步骤S12具体可以包括如下步骤：
步骤S121：利用至少N帧连续的所述原始图像中的各红外像素点的像素值信息，计算所述红外像素点的相位偏移。
步骤S122：基于各个所述红外像素点的相位偏移确定各个所述红外像素点的深度信息，作为所述红外像素信息对应的深度信息。
可以理解的是，对于每个红外像素点来说，由于每帧原始图像之间存在相移角差，其在一个周期内采集的多帧连续的原始图像中对应的红外像素信息可能不同，因此，利用每个红外像素点在至少N帧连续的所述原始图像中的红外像素信息，可以获得对应红外像素点的深度信息。
具体地，上述步骤S121具体可以包括：基于红外像素点在不同帧原始图像中的红外像素信息之间的差异，得到对应红外像素点的相位偏移。可以理解的是，以RGB-IR模组为例，由于像素点在调制的正弦波发出至接收反射波之间的过程中存在相移角，而相移角的存在导致红外像素点在每帧原始图像中的红外像素信息可能不同，于是，基于红外像素点在不同帧原始图像中的红外像素信息之间的差异，得到对应红外像素点的相位偏移。此时，上述步骤S122具体可以包括：利用每个红外像素点的相位偏移和原始图像的调制频率，获得对应红外像素点的深度信息，作为各帧原始图像中的红外像素信息对应的深度信息。可以理解的是，红外像素点的深度信息与红外像素点的相位偏移以及原始图像的调制频率相关，原始图像的调制频率是已知的，因此可以利用计算出的每个红外像素点的相位偏移和已知的原始图像的调制频率，使得红外像素点的深度信息的获取较为方便。
上述方案，利用某个红外像素点在每帧原始图像中的像素值信息，可以得到该红外像素点的相位偏移，再通过该红外像素点的相位偏移得到对应的深度信息，使得红外像素点的深度信息与其相位偏移相关联，便于深度信息的获取。
请参阅图3，图3是图1中步骤S13一实施例的流程示意图。本实施例中，所述原始图像包括呈矩阵排列的多个像素单元，每个像素单元包括 一个所述红外像素点；上述步骤S13具体可以包括如下步骤：
步骤S131：基于各红外像素点的深度信息生成所述红外像素点所在像素单元的深度信息。
步骤S132：基于各所述像素单元的深度信息生成深度图像。
由于原始图像包括呈矩阵排列的多个像素单元，每个像素单元包括一个红外像素点，于是在获取到一组具有预设相移角的原始图像后，可以利用各红外像素点在每帧原始图像中的像素值信息，得到各红外像素点的深度信息，而基于各红外像素点的深度信息可以生成各红外像素点所在像素单元的深度信息，对于所有的红外像素点来说，都可以获取对应的深度信息，对于一组原始图像来说，由于每个红外像素点的深度信息可能不一致，使得各像素单元的深度信息不一致，从而根据所有像素单元的深度信息可以得到一帧完整的深度图像，使深度图像具有较好的准确性。
进一步地，在一实施例中，每个所述像素单元还包括至少一个彩色像素点，彩色像素点可以包括红色像素点R、绿色像素点G或蓝色像素点B；上述步骤S131具体可以包括：基于所述红外像素点对应的深度信息确定所述红外像素点所在像素单元中的彩色像素点对应的深度信息。
由于各所述像素单元还包括至少一个彩色像素点，于是基于某个红外像素点对应的深度信息可以确定该红外像素点所在像素单元中的彩色像素点对应的深度信息，因此，利用单目彩色-红外图像采集模组采集得到的多帧原始图像，不仅可以生成一帧深度图像，且根据多帧原始图像的彩色像素点还可以对应生成多帧彩色图像，即同时生成相对应的一帧深度图像和多帧彩色图像，因此利用单目彩色-红外图像采集模组能够同时实现对彩色图像和深度图像的获取，而且相对应的一帧深度图像和多帧彩色图像具有同步性。
作为一种可实施方式，单目彩色-红外图像采集模组可以通过接收彩色信号来采集彩色信息，并通过发射红外脉冲信号、接收红外反射信号来采集深度信息。例如，该单目彩色-红外图像采集模组可以包括红外发射组件，以向前方环境中发射红外脉冲信号，红外脉冲信号到达环境中的物体后，一部分被吸收后以辐射的形式发散出来，另一部分被反射回来；该单目彩色-红外图像采集模组还可以包括红外反射接收组件，通过接收红外反射信 号，基于相移角差的计算将其处理为深度图像。由此，可以得到彩色像素信息和红外像素信息，并由红外像素信息得到深度图像。
可以理解的是，利用单目彩色-红外图像采集模组采集得到的至少N帧连续的原始图像，不仅可以生成一帧深度图像，且根据至少N帧连续的原始图像的彩色像素信息还可以对应生成至少N帧彩色图像，即同时生成相对应的一帧深度图像和至少N帧彩色图像，因此相对应的一帧深度图像和至少N帧彩色图像具有同步性。可以理解的是，若对具有同步性的深度图像和彩色图像进行融合时，可以免去深度图像和彩色图像融合时的配准过程，且得到的融合效果更佳。
请结合图4，图4是本申请深度图像的获取方法一应用场景中彩色图像和深度图像的对应关系示意图。在一应用场景中，至少一帧原始图像的帧数为四帧，每相邻两帧原始图像之间的相移角差为90度，如图所示，由于每帧原始图像包括彩色像素信息，因此，每帧原始图像对应可以生成一帧彩色图像，即RGB帧，又由于每帧原始图像包括红外像素信息，并根据四帧原始图像中的红外像素信息生成一帧深度图像，即深度帧，于是，所生成的一帧深度图像和四帧彩色图像具有同步性。此时彩色图像的帧率为深度图像的帧率的4倍，当原始图像帧的帧率为120fps时，彩色图像的帧率也为120fps，对应的，深度图像的帧率为30fps。
在一个实施场景中，N＝4。此时，组成一组图像的原始图像的帧数为四帧，每相邻两帧原始图像之间的相移角为π/2。具体地，将四帧原始图像划分为两组原始图像组；其中，每组原始图像组包括两帧不相邻的原始图像；对于每个红外像素点，获取红外像素点在每组原始图像组中的红外像素信息的差异值，并对红外像素点在两组原始图像组的差异值之间的比值进行反正切处理，得到红外像素点的相位偏移。可以理解的是，当原始图像的帧数为四帧时，预设相移角为90度，即相邻两帧原始图像之间的相移角为90度；例如，第一帧原始图像的相移角为0度，第二帧原始图像的相移角为90度，第三帧原始图像的相移角为180度，第四帧原始图像的相移角为270度，将第一帧原始图像和第三帧原始图像划分为第一组原始图像组，将第二帧原始图像和第四帧原始图像划分为第二组原始图像组，对于某个红外像素点而言，其在第一帧原始图像中的红外像素信息为A0°，在 第二帧原始图像中的红外像素信息为A90°，在第三帧原始图像中的红外像素信息为A180°，在第四帧原始图像中的红外像素信息为A270°，于是，可以获取该红外像素点在第一组原始图像组中的红外像素信息的差异值为A0°-A180°，在第二组原始图像组中的红外像素信息的差异值为A270°-A90°，然后对红外像素点在两组原始图像组的差异值之间的比值进行反正切处理，得到红外像素点的相位偏移。具体地，红外像素点的相位偏移与其在各帧原始图像中的红外像素信息之间的关系如公式(1)所示：
Figure PCTCN2022125991-appb-000001
其中，
Figure PCTCN2022125991-appb-000002
为红外像素点的相位偏移，红外像素点在各帧原始图像中的红外像素信息为各帧原始图像的曝光时间内的光强积分值。
在一个实施场景中，所述红外像素点的深度信息与所述红外像素点的相位偏移成正比例关系，且与所述调制频率成反比例关系。具体地，红外像素点的深度信息与红外像素点的相位偏移以及调制频率之间的关系如公式(2)所示：
Figure PCTCN2022125991-appb-000003
其中，D为红外像素点的深度信息，c为光速值(3*108m/s)，fmod为调制频率，fmod可以通过用户进行预先设置，为已知量。
于是，对于每个红外像素点，可以通过上述公式(1)和(2)计算得到红外像素点的相位偏移和深度信息，避免了复杂的深度计算公式，实现了快速准确获取相移角和深度信息。可以理解的是，在其他应用场景中，组成一组图像的原始图像的帧数也可以为其他数量帧，即N也可以为其他正整数，例如3、5、8等；例如，一组原始图像的帧数为八帧，相邻两帧原始图像之间的相移角为π/4，此时基于上述类似的原理也可以获取一组原始图像对应的一帧深度图像。
请参阅图5，图5是本申请深度图像的获取方法另一实施例的流程示意图。具体而言，可以包括如下步骤：
步骤S51：向单目彩色-红外图像采集模组发送图像帧采集信号，所述图像帧采集信号包括控制采集的相邻帧的图像之间的相移角为2π/N的控制信号，N为大于2的正整数。
步骤S52：接收所述单目彩色-红外图像采集模组在所述图像帧采集信号控制下采集得到的多帧原始图像，其中，每帧所述原始图像包括红外像素信息。
步骤S53：基于至少N帧连续的所述原始图像中的所述红外像素信息，确定所述红外像素信息对应的深度信息。
步骤S54：基于所述红外像素对应的深度信息生成深度图像。
本实施例中，步骤S52-S54与本申请上述实施例的步骤S11-S13基本类似，此处不再赘述。
本申请实施例的深度图像的获取方法的执行主体可以为图像处理装置，例如，深度图像的获取方法可以由终端设备或服务器或其它电子设备执行，其中，终端设备可以为用户设备(User Equipment，UE)、移动设备、用户终端、终端、个人数字处理(Personal Digital Assistant，PDA)、手持设备、计算设备、车载设备、可穿戴设备等。在一些可能的实现方式中，该深度图像的获取方法可以通过处理器调用存储器中存储的计算机可读指令的方式来实现。
具体地，图像处理装置与单目彩色-红外图像采集模组连接，当图像处理装置向单目彩色-红外图像采集模组发送图像帧采集信号后，由于图像帧采集信号包括控制采集的相邻帧图像之间的相移角为2π/N的控制信号，N为大于2的正整数，因此单目彩色-红外图像采集模组可以根据该图像帧采集信号，采集得到多帧原始图像并发送给图像处理装置，其中，每帧原始图像包括红外像素信息，相邻帧的原始图像之间的相移角为2π/N，N为大于2的正整数；图像处理装置在接收到单目彩色-红外图像采集模组在图像帧采集信号控制下采集得到的多帧原始图像后，可以基于至少N帧连续的原始图像中的红外像素信息，确定红外像素信息对应的深度信息，并基于红外像素对应的深度信息生成深度图像。因此，通过单目彩色-红外图像采集模组采集得到多帧原始图像，由于每帧原始图像包括红外像素信息，相邻帧的原始图像之间的相移角为2π/N，N为大于2的正整数，因此，由至少N帧具有预设相移角的连续的原始图像组成一组图像，进而可以基于一组具有预设相移角的原始图像中的红外像素信息，确定红外像素信息对应的深度信息，进而生成深度图像，故实现了通过单目方式获取包括深度信 息的图像，系统的成本更低，系统的功耗也更低，同时，硬件可以拥有更小巧的体积，有利于实现设备的小型化和轻薄化。
请参阅图6，图6是本申请深度图像的获取装置一实施例的框架示意图。深度图像的获取装置60包括：获取模块600、确定模块602和生成模块604；获取模块600用于获取利用单目彩色-红外图像采集模组采集得到的多帧原始图像，其中，每帧所述原始图像包括红外像素信息，相邻帧的所述原始图像之间的相移角为2π/N，N为大于2的正整数；确定模块602用于基于至少N帧连续的所述原始图像中的所述红外像素信息，确定所述红外像素信息对应的深度信息；生成模块604用于基于所述红外像素信息对应的深度信息生成深度图像。
上述方案，获取模块600通过单目彩色-红外图像采集模组采集得到多帧原始图像，由于每帧原始图像包括红外像素信息，相邻帧的原始图像之间的相移角为2π/N，N为大于2的正整数，因此，由至少N帧具有预设相移角的连续的原始图像组成一组图像，进而确定模块602可以基于一组具有预设相移角的原始图像中的红外像素信息，确定红外像素信息对应的深度信息，于是生成模块604进而可以生成深度图像，故实现了通过单目方式获取包括深度信息的图像，系统的成本更低，系统的功耗也更低，同时，硬件可以拥有更小巧的体积，有利于实现设备的小型化和轻薄化。
在一些实施例中，所述红外像素信息包括红外像素点的像素值信息；确定模块602执行基于至少N帧连续的所述原始图像中的所述红外像素信息，确定所述红外像素信息对应的深度信息的步骤包括：利用至少N帧连续的所述原始图像中的各红外像素点的像素值信息，计算所述红外像素点的相位偏移；基于各个所述红外像素点的相位偏移确定各个所述红外像素点的深度信息，作为所述红外像素信息对应的深度信息。
在一些实施例中，所述原始图像包括呈矩阵排列的多个像素单元，每个像素单元包括一个所述红外像素点；生成模块604执行基于所述红外像素信息对应的深度信息生成深度图像的步骤包括：基于各红外像素点的深度信息生成所述红外像素点所在像素单元的深度信息；基于各所述像素单元的深度信息生成深度图像。
在一些实施例中，每个所述像素单元还包括至少一个彩色像素点；生 成模块604执行基于各红外像素点的深度信息生成所述红外像素点所在像素单元的深度信息的步骤具体包括：基于所述红外像素点对应的深度信息确定所述红外像素点所在像素单元中的彩色像素点对应的深度信息。
在一些实施例中，生成模块604执行基于所述红外像素信息对应的深度信息生成深度图像的步骤包括：基于N帧连续的所述原始图像中的红外像素信息对应的深度信息生成一帧深度图像。
在一些实施例中，N＝4。
在一些实施例中，获取模块600执行获取利用单目彩色-红外图像采集模组采集得到的多帧原始图像的步骤包括：向单目彩色-红外图像采集模组发送图像帧采集信号，所述图像帧采集信号包括控制采集的相邻帧图像之间的相移角为2π/N的控制信号，N为大于2的正整数；接收所述单目彩色-红外图像采集模组在所述图像帧采集信号控制下采集得到的多帧原始图像。
请参阅图7，图7是本申请图像处理装置一实施例的框架示意图。图像处理装置70包括相互耦接的存储器71和处理器72，处理器72用于执行存储器71中存储的程序指令，以实现上述任一深度图像的获取方法实施例的步骤。在一个具体的实施场景中，图像处理装置70可以包括但不限于：微型计算机、服务器，此外，图像处理装置70还可以包括笔记本电脑、智能手机、车载电脑等具有显示功能的电子设备的一部分，在此不做限定。
具体而言，处理器72用于控制其自身以及存储器71以实现上述任一深度图像的获取方法实施例中的步骤。处理器72还可以称为CPU(Central Processing Unit，中央处理单元)。处理器72可能是一种集成电路芯片，具有信号的处理能力。处理器72还可以是通用处理器、数字信号处理器(Digital Signal Processor,DSP)、专用集成电路(Application Specific Integrated Circuit,ASIC)、现场可编程门阵列(Field-Programmable Gate Array,FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。另外，处理器72可以由集成电路芯片共同实现。
上述方案，通过单目彩色-红外图像采集模组采集得到多帧原始图像，由于每帧原始图像包括红外像素信息，相邻帧的原始图像之间的相移角为 2π/N，N为大于2的正整数，因此，由至少N帧具有预设相移角的连续的原始图像组成一组图像，进而可以基于一组具有预设相移角的原始图像中的红外像素信息，确定红外像素信息对应的深度信息，进而生成深度图像，故实现了通过单目方式获取包括深度信息的图像，系统的成本更低，系统的功耗也更低，同时，硬件可以拥有更小巧的体积，有利于实现设备的小型化和轻薄化。
请参阅图8，图8是本申请图像处理系统一实施例的框架示意图。图像处理系统80包括相互连接的单目彩色-红外图像采集模组81和图像处理装置82；单目彩色-红外图像采集模组81用于采集得到多帧原始图像，其中，每帧所述原始图像包括红外像素信息，相邻帧的所述原始图像之间的相移角为2π/N，N为大于2的正整数；图像处理装置82，为上述的图像处理装置70，图像处理装置82用于获取利用所述单目彩色-红外图像采集模组采集得到的多帧所述原始图像；基于至少N帧连续的所述原始图像中的所述红外像素信息，确定所述红外像素信息对应的深度信息；以及基于所述红外像素对应的深度信息生成深度图像。
在一个实施场景中，单目彩色-红外图像采集模组81为RGB-IR模组，单目彩色-红外图像采集模组81包括RGB-IR传感器810，通过RGB-IR传感器810可以采集得到多帧原始图像，每帧原始图像包括红外像素信息，相邻帧的原始图像之间的相移角为2π/N，N为大于2的正整数；图像处理装置82包括系统级芯片SoC(System On Chip)820，系统级芯片820可以对至少N帧连续的原始图像进行处理，基于至少N帧连续的原始图像中的红外像素信息，确定红外像素信息对应的深度信息，并基于红外像素信息对应的深度信息生成深度图像。请结合图9，图9是图8中单目彩色-红外图像采集模组的传感器像素点结构排布的一种示意图。如图所示，RGB-IR传感器810中，每四个矩阵排列的像素点中包含一个红外像素点IR和三个彩色像素点，每八个像素点作为一组像素基本单元，包含有完整的彩色像素信息和红外像素信息(包含红色像素点R、绿色像素点G、蓝色像素点B和红外像素点IR)。在实际的深度图像的获取方法中，以200万像素点的RGB-IR传感器为例，其包括50万个红外像素点IR，在实际计算过程中，当系统级芯片820获取至少N帧连续的原始图像之后，提取其中的50万个 红外像素点IR的深度信息，由于每个红外像素点IR的相位偏移φ可能不同，使得其深度信息也不一致，从而可以得到包含50万个红外像素点一一对应的深度信息的完整深度图像。
在一个实施场景中，单目彩色-红外图像采集模组81还包括串化器812，RGB-IR传感器810通过I2C接口及MIPI接口与串化器812连接；图像处理装置82还包括解串器822，解串器822通过MIPI接口及I2C接口与系统级芯片820连接；串化器812通过HSD接口与解串器822连接。单目彩色-红外图像采集模组81通过串化器812将采集的原始图像以串行数据的方式传输至解串器822，解串器822对串行数据进行解串以获得原始图像并传输至系统级芯片820。
在一应用场景中，单目彩色-红外图像采集模组81通过HSD接插件与图像处理装置82进行连接，HSD接插件用于数据传输及设备供电。在上电之后，图像处理装置82的系统级芯片820通过解串器822和串化器812对RGB-IR传感器810进行配置以启动图像采集功能。于是，RGB-IR传感器810可以生成视频信号，视频信号通过MIPI接口传输给串化器812，然后串化器812对数据进行打包并通过HSD接口传送给解串器822，并最终发送给系统级芯片820，此时系统级芯片820接收到的数据为包括彩色像素信息和红外像素信息的原始图像，于是，系统级芯片820可以采用上述任一深度图像的获取方法来进行图像处理，得到相对应的彩色图像和深度图像，之后，可以将彩色图像和深度图像在屏幕上显示，相对应的深度图像和彩色图像具有同步性。
请参阅图10，图10是本申请计算机可读存储介质一实施例的框架示意图。计算机可读存储介质10存储有能够被处理器运行的程序指令100，程序指令100用于实现上述任一深度图像的获取方法实施例中的步骤。
上述方案，通过单目彩色-红外图像采集模组采集得到多帧原始图像，由于每帧原始图像包括红外像素信息，相邻帧的原始图像之间的相移角为2π/N，N为大于2的正整数，因此，由至少N帧具有预设相移角的连续的原始图像组成一组图像，进而可以基于一组具有预设相移角的原始图像中的红外像素信息，确定红外像素信息对应的深度信息，进而生成深度图像，故实现了通过单目方式获取包括深度信息的图像，系统的成本更低，系统 的功耗也更低，同时，硬件可以拥有更小巧的体积，有利于实现设备的小型化和轻薄化。
在本申请所提供的几个实施例中，应该理解到，所揭露的方法、装置和系统，可以通过其它的方式实现。例如，以上所描述的装置实施方式仅仅是示意性的，例如，模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性、机械或其它的形式。
作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施方式方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)或处理器(processor)执行本申请各个实施方式方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。

Claims (18)

  1. 一种深度图像的获取方法，其中，所述方法包括：
    获取利用单目彩色-红外图像采集模组采集得到的多帧原始图像，其中，每帧所述原始图像包括红外像素信息，相邻帧的所述原始图像之间的相移角为2π/N，N为大于2的正整数；
    基于至少N帧连续的所述原始图像中的所述红外像素信息，确定所述红外像素信息对应的深度信息；
    基于所述红外像素信息对应的深度信息生成深度图像。
  2. 根据权利要求1所述的方法，其中，所述红外像素信息包括红外像素点的像素值信息；
    所述基于至少N帧连续的所述原始图像中的所述红外像素信息，确定红外像素信息对应的深度信息，包括：
    利用至少N帧连续的所述原始图像中的各红外像素点的像素值信息，计算所述红外像素点的相位偏移；
    基于各个所述红外像素点的相位偏移确定各个所述红外像素点的深度信息，作为所述红外像素信息对应的深度信息。
  3. 根据权利要求2所述的方法，其中，所述原始图像包括呈矩阵排列的多个像素单元，每个像素单元包括一个所述红外像素点；
    所述基于所述红外像素信息对应的深度信息生成深度图像，包括：
    基于各红外像素点的深度信息生成所述红外像素点所在像素单元的深度信息；
    基于各所述像素单元的深度信息生成深度图像。
  4. 根据权利要求3所述的方法，其中，每个所述像素单元还包括至少一个彩色像素点；
    所述基于各红外像素点的深度信息生成所述红外像素点所在像素单元的深度信息，包括：
    基于所述红外像素点对应的深度信息确定所述红外像素点所在像素单元中的彩色像素点对应的深度信息。
  5. 根据权利要求1所述的方法，其中，所述基于所述红外像素信息对应的深度信息生成深度图像，包括：
    基于N帧连续的所述原始图像中的红外像素信息对应的深度信息生成一帧深度图像。
  6. 根据权利要求1所述的方法，其中，N＝4。
  7. 根据权利要求1所述的方法，其中，所述获取利用单目彩色-红外图像采集模组采集得到的多帧原始图像，包括：
    向单目彩色-红外图像采集模组发送图像帧采集信号，所述图像帧采集信号包括控制采集的相邻帧图像之间的相移角为2π/N的控制信号，N为大于2的正整数；
    接收所述单目彩色-红外图像采集模组在所述图像帧采集信号控制下采集得到的多帧原始图像。
  8. 一种深度图像的获取装置，其中，包括：
    获取模块，用于获取利用单目彩色-红外图像采集模组采集得到的多帧原始图像，其中，每帧所述原始图像包括红外像素信息，相邻帧的所述原始图像之间的相移角为2π/N，N为大于2的正整数；
    确定模块，用于基于至少N帧连续的所述原始图像中的所述红外像素信息，确定所述红外像素信息对应的深度信息；
    生成模块，用于基于所述红外像素对应的深度信息生成深度图像。
  9. 根据权利要求8所述的装置，其中，所述红外像素信息包括红外像素点的像素值信息；
    所述确定模块执行所述基于至少N帧连续的所述原始图像中的所述红外像素信息，确定所述红外像素信息对应的深度信息的步骤包括：利用至少N帧连续的所述原始图像中的各红外像素点的像素值信息，计算所述红外像素点的相位偏移；基于各个所述红外像素点的相位偏移确定各个所述红外像素点的深度信息，作为所述红外像素信息对应的深度信息。
  10. 根据权利要求9所述的装置，其中，所述原始图像包括呈矩阵排列的多个像素单元，每个像素单元包括一个所述红外像素点；
    所述生成模块执行所述基于所述红外像素信息对应的深度信息生成深度图像的步骤包括：基于各红外像素点的深度信息生成所述红外像素点所在像素单元的深度信息；基于各所述像素单元的深度信息生成深度图像。
  11. 根据权利要求10所述的装置，其中，每个所述像素单元还包括至少一个彩色像素点；
    所述生成模块执行所述基于各红外像素点的深度信息生成所述红外像素点所在像素单元的深度信息的步骤具体包括：基于所述红外像素点对应的深度信息确定所述红外像素点所在像素单元中的彩色像素点对应的深度信息。
  12. 根据权利要求8所述的装置，其中，所述生成模块执行所述基于所述红外像素信息对应的深度信息生成深度图像的步骤包括：基于N帧连续的所述原始图像中的红外像素信息对应的深度信息生成一帧深度图像。
  13. 根据权利要求8所述的装置，其中，N＝4。
  14. 根据权利要求8所述的装置，其中，所述获取模块执行所述获取利用单目彩色-红外图像采集模组采集得到的多帧原始图像的步骤包括：向单目彩色-红外图像采集模组发送图像帧采集信号，所述图像帧采集信号包括控制采集的相邻帧图像之间的相移角为2π/N的控制信号，N为大于2的正整数；接收所述单目彩色-红外图像采集模组在所述图像帧采集信号控制下采集得到的多帧原始图像。
  15. 一种图像处理装置，其中，包括相互耦接的存储器和处理器，所述处理器用于执行所述存储器中存储的程序指令，以实现权利要求1至7任一项所述的深度图像的获取方法。
  16. 一种图像处理系统，其中，包括：
    单目彩色-红外图像采集模组，用于采集得到多帧原始图像，其中，每帧所述原始图像包括红外像素信息，相邻帧的所述原始图像之间的相移角为2π/N，N为大于2的正整数；
    图像处理装置，所述图像处理装置为权利要求15所述的图像处理装置，用于获取利用所述单目彩色-红外图像采集模组采集得到的多帧所述原始图像；基于至少N帧连续的所述原始图像中的所述红外像素信息，确定所述红外像素信息对应的深度信息；以及基于所述红外像素对应的深度信息生成深度图像。
  17. 根据权利要求16所述的系统，其中，所述图像处理装置包括系统级芯片SoC和解串器，所述单目彩色-红外图像采集模组还包括串化器；
    所述单目彩色-红外图像采集模组通过所述串化器将采集的所述原始图像以串行数据的方式传输至所述解串器，所述解串器对所述串行数据进行解串以获得所述原始图像并传输至所述系统级芯片。
  18. 一种计算机可读存储介质，其存储有程序指令，其中，所述程序指令被处理器执行时实现权利要求1至7任一项所述的深度图像的获取方法。
PCT/CN2022/125991 2021-11-30 2022-10-18 深度图像的获取方法及装置、系统、计算机可读存储介质 WO2023098323A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111445318.7A CN114119696A (zh) 2021-11-30 2021-11-30 深度图像的获取方法及装置、系统、计算机可读存储介质
CN202111445318.7 2021-11-30

Publications (1)

Publication Number Publication Date
WO2023098323A1 true WO2023098323A1 (zh) 2023-06-08

Family

ID=80368946

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/125991 WO2023098323A1 (zh) 2021-11-30 2022-10-18 深度图像的获取方法及装置、系统、计算机可读存储介质

Country Status (2)

Country Link
CN (1) CN114119696A (zh)
WO (1) WO2023098323A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114119696A (zh) * 2021-11-30 2022-03-01 上海商汤临港智能科技有限公司 深度图像的获取方法及装置、系统、计算机可读存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104918034A (zh) * 2015-05-29 2015-09-16 深圳奥比中光科技有限公司 一种3d图像捕获装置、捕获方法及3d图像系统
US20150310622A1 (en) * 2014-04-24 2015-10-29 Avago Technologies General Ip (Singapore) Pte. Ltd. Depth Image Generation Utilizing Pseudoframes Each Comprising Multiple Phase Images
CN106576159A (zh) * 2015-06-23 2017-04-19 华为技术有限公司 一种获取深度信息的拍照设备和方法
CN109889799A (zh) * 2017-12-06 2019-06-14 西安交通大学 基于rgbir摄像头的单目结构光深度感知方法及装置
CN114119696A (zh) * 2021-11-30 2022-03-01 上海商汤临港智能科技有限公司 深度图像的获取方法及装置、系统、计算机可读存储介质

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108271012A (zh) * 2017-12-29 2018-07-10 维沃移动通信有限公司 一种深度信息的获取方法、装置以及移动终端
CN108921888B (zh) * 2018-07-03 2022-07-19 京东方科技集团股份有限公司 一种深度图像检测方法、系统及计算机可读存储介质
CN109798838B (zh) * 2018-12-19 2020-10-27 西安交通大学 一种基于激光散斑投射的ToF深度传感器及其测距方法
CN109951625B (zh) * 2019-04-12 2023-10-31 光微信息科技(合肥)有限公司 彩色深度图像传感器以及成像装置
CN110335211B (zh) * 2019-06-24 2021-07-30 Oppo广东移动通信有限公司 深度图像的校正方法、终端设备以及计算机存储介质
CN112363180A (zh) * 2020-10-28 2021-02-12 Oppo广东移动通信有限公司 一种成像测距传感器、方法、系统及存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150310622A1 (en) * 2014-04-24 2015-10-29 Avago Technologies General Ip (Singapore) Pte. Ltd. Depth Image Generation Utilizing Pseudoframes Each Comprising Multiple Phase Images
CN104918034A (zh) * 2015-05-29 2015-09-16 深圳奥比中光科技有限公司 一种3d图像捕获装置、捕获方法及3d图像系统
CN106576159A (zh) * 2015-06-23 2017-04-19 华为技术有限公司 一种获取深度信息的拍照设备和方法
CN109889799A (zh) * 2017-12-06 2019-06-14 西安交通大学 基于rgbir摄像头的单目结构光深度感知方法及装置
CN114119696A (zh) * 2021-11-30 2022-03-01 上海商汤临港智能科技有限公司 深度图像的获取方法及装置、系统、计算机可读存储介质

Also Published As

Publication number Publication date
CN114119696A (zh) 2022-03-01

Similar Documents

Publication Publication Date Title
CN113160328B (zh) 一种外参标定方法、系统、机器人和存储介质
EP3714437B1 (en) Three-dimensional (3d) reconstructions of dynamic scenes using reconfigurable hybrid imaging system
US9432593B2 (en) Target object information acquisition method and electronic device
WO2018086348A1 (zh) 双目立体视觉系统及深度测量方法
EP3665506B1 (en) Apparatus and method for generating a representation of a scene
CN107635129B (zh) 一种三维三目摄像装置及深度融合方法
WO2021203883A1 (zh) 三维扫描方法、三维扫描系统和计算机可读存储介质
CN110376602A (zh) 多模式深度计算处理器及3d图像设备
CN102997891B (zh) 场景深度的测量装置和方法
US11494925B2 (en) Method for depth image acquisition, electronic device, and storage medium
US20150009119A1 (en) Built-in design of camera system for imaging and gesture processing applications
JP2022160678A (ja) ３次元画像機器
CN112700484B (zh) 一种基于单目深度相机的深度图彩色化方法
CN110378971A (zh) 一种图像对齐精度的检测方法及装置、设备、存储介质
CN102316355A (zh) 3d机器视觉信号的生成方法及3d机器视觉传感器
WO2023098323A1 (zh) 深度图像的获取方法及装置、系统、计算机可读存储介质
Hahne et al. Depth imaging by combining time-of-flight and on-demand stereo
WO2022228461A1 (zh) 一种基于激光雷达的三维超声成像方法和系统
CN115908578A (zh) 多传感器数据融合的方法、装置和系统
CN109085603A (zh) 光学三维成像系统和彩色三维图像成像方法
CN111510700A (zh) 图像采集装置
US20210215827A1 (en) Using time-of-flight techniques for stereoscopic image processing
Lai et al. Single-pixel 3D imaging based on fusion temporal data of single-photon detector and millimeter-wave radar
CN106959747B (zh) 三维人体测量方法及其设备
CN113052884A (zh) 信息处理方法、信息处理装置、存储介质与电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22900128

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22900128

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 05.11.2024)