CN112802116B - Image processing method, camera calibration method, device, storage medium and electronic device - Google Patents

Abstract

The application relates to an image processing method, a camera calibration device, a computer readable storage medium and an electronic device. The method comprises the following steps: controlling the temperature of the camera module to reach at least two different designated temperatures, wherein the camera module comprises a light emitter and a camera; controlling the camera to acquire a reference image formed when the light emitter irradiates a reference plane at the specified temperature; storing the specified temperature in correspondence with a reference image; wherein the reference image carries reference depth information, which is used to calculate depth information of the object. The image processing method, the camera calibration device, the computer readable storage medium and the electronic equipment can realize the accuracy of image processing.

Description

Image processing method, camera calibration method, device, storage medium and electronic device

This application is a divisional application of a patent application with an application date of June 28, 2018, an invention name of “Camera calibration method, device, computer-readable storage medium and electronic device” and an application number of 201810690949.7.

Technical Field

The present application relates to the field of computer technology, and in particular to an image processing method, a camera calibration method, a device, a computer-readable storage medium, and an electronic device.

Background Art

Structured light can be used in applications such as unlocking, payment, and beautification. Specifically, a laser emitter can emit infrared light with certain structural features, and then a laser camera can collect images formed by these infrared lights. The images formed by these collected infrared lights can be used to calculate the depth information from the object to the camera. If the laser emitter and the laser camera are deformed, the collected image will change, which will cause errors in the calculated depth information.

Summary of the invention

The embodiments of the present application provide a camera calibration method, device, computer-readable storage medium, and electronic device, which can improve the accuracy of image processing.

A camera calibration method, the method comprising:

Controlling the temperature of a camera module to reach at least two different specified temperatures, wherein the camera module includes a light emitter and a camera;

Controlling the camera to collect a reference image formed when the light emitter illuminates the reference plane at the specified temperature;

The designated temperature is stored corresponding to a reference image; wherein the reference image carries reference depth information, and the reference depth information is used to calculate depth information of an object.

A camera calibration device, the device comprising:

A temperature control module, used to control the temperature of a camera module to reach at least two different specified temperatures, wherein the camera module includes a light emitter and a camera;

An image acquisition module, used for controlling the camera to collect a reference image formed when the light emitter illuminates the reference plane at the specified temperature;

The image storage module is used to store the specified temperature and the reference image in correspondence; wherein the reference image carries reference depth information, and the reference depth information is used to calculate the depth information of the object.

A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

Controlling the temperature of a camera module to reach at least two different specified temperatures, wherein the camera module includes a light emitter and a camera;

Controlling the camera to collect a reference image formed when the light emitter illuminates the reference plane at the specified temperature;

The designated temperature is stored corresponding to a reference image; wherein the reference image carries reference depth information, and the reference depth information is used to calculate depth information of an object.

An electronic device includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the instructions are executed by the processor, the processor performs the following steps:

Controlling the temperature of a camera module to reach at least two different specified temperatures, wherein the camera module includes a light emitter and a camera;

Controlling the camera to collect a reference image formed when the light emitter illuminates the reference plane at the specified temperature;

The designated temperature is stored corresponding to a reference image; wherein the reference image carries reference depth information, and the reference depth information is used to calculate depth information of an object.

The above-mentioned camera calibration method, device, computer-readable storage medium and electronic device can control the temperature of the camera module to reach at least two different specified temperatures, and control the camera module to collect reference images formed at different specified temperatures, and then store the reference images corresponding to the specified temperatures. Since the camera module will deform at different temperatures, and the temperature itself will also affect the image collected by the camera module, the camera module is controlled to collect images at different specified temperatures during camera calibration. In this way, the corresponding reference image can be obtained according to the temperature of the camera module, and the depth information of the object can be calculated according to the reference depth information in the reference image, thereby avoiding the error caused by the temperature change of the camera module and improving the accuracy of image processing.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

FIG1 is a diagram of an application environment of a camera calibration method according to an embodiment;

FIG2 is a schematic diagram of an electronic device equipped with a camera module according to an embodiment;

FIG3 is a flow chart of a camera calibration method in one embodiment;

FIG4 is a flow chart of a camera calibration method in another embodiment;

FIG5 is a schematic diagram showing a principle for calculating depth information in one embodiment;

FIG6 is a flow chart of a camera calibration method in another embodiment;

FIG7 is a flow chart of a camera calibration method in another embodiment;

FIG8 is a hardware structure diagram of a camera calibration method according to an embodiment;

FIG9 is an interaction diagram for implementing a camera calibration method in one embodiment;

FIG10 is a schematic diagram of the structure of a camera calibration device in one embodiment;

FIG. 11 is a schematic diagram of the structure of a camera calibration device in another embodiment.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first client may be referred to as a second client, and similarly, a second client may be referred to as a first client. Both the first client and the second client are clients, but they are not the same client.

FIG1 is an application environment diagram of a camera calibration method in an embodiment. As shown in FIG1 , the application environment includes a calibration device 10 and an electronic device 12. A camera module can be installed on the electronic device 12, and the camera module includes a light emitter and a camera. The electronic device 12 is fixed on the calibration device 10, and the camera module of the electronic device 12 is calibrated by the calibration device 10. Specifically, the calibration device 10 includes a surface light source 100, a reference plane 102, a laser sensor 104, an electric angle table 106, and an electric lifting platform 108. The electric angle table 106 can adjust the angle of the electronic device 10 so that the optical axis of the camera module of the electronic device 10 is perpendicular to the reference plane 102. The electric lifting platform 108 can adjust the vertical distance between the electronic device 12 and the reference plane 102, and measure the vertical distance through the laser sensor 104. The surface light source 100 is used to illuminate the coding area on the reference plane 102. The electronic device 12 can control the temperature of the camera module to reach at least two different specified temperatures. When the light source emitted by the light emitter at different specified temperatures irradiates the reference plane 102, the reference image formed on the reference plane 102 at different specified temperatures is collected by the camera, and a corresponding relationship between the specified temperature and the reference image is established, and then the specified temperature and the reference image are stored accordingly.

FIG2 is a schematic diagram of an electronic device with a camera module installed in an embodiment. As shown in FIG2 , the electronic device 20 is equipped with a camera module, including a light emitter 202 and a camera 204. In the process of calibrating the camera, the electronic device 20 can control the temperature of the camera module to reach different specified temperatures, and emit light through the light emitter 202 at different specified temperatures, and collect the reference image formed when the light is irradiated to the reference plane through the camera 204. Finally, the electronic device 20 can establish a corresponding relationship between the specified temperature and the reference image, and store the specified temperature and the reference image.

FIG3 is a flow chart of a camera calibration method in one embodiment. As shown in FIG3 , the camera calibration method includes steps 302 to 306. Among them:

Step 302: Control the temperature of a camera module to reach at least two different specified temperatures, wherein the camera module includes a light emitter and a camera.

In one embodiment, a camera can be installed on the electronic device, and an image can be acquired through the installed camera. Cameras can be divided into laser cameras, visible light cameras and other types according to the different images acquired. Laser cameras can acquire images formed by irradiating lasers onto objects, and visible light cameras can acquire images formed by irradiating visible light onto objects. Several cameras can be installed on an electronic device, and the installation locations are not limited. For example, a camera can be installed on the front panel of the electronic device, and two cameras can be installed on the back panel. The camera can also be installed inside the electronic device in an embedded manner, and then the camera can be opened by rotating or sliding. Specifically, a front camera and a rear camera can be installed on the electronic device. The front camera and the rear camera can acquire images from different perspectives. Generally, the front camera can acquire images from the front perspective of the electronic device, and the rear camera can acquire images from the back perspective of the electronic device.

The electronic device can measure the depth information from the object in the scene to the electronic device through the captured image, and the depth information can be measured specifically through structured light. When obtaining depth information through structured light, a camera module including a light emitter and a camera can be installed on the electronic device, and the process of obtaining depth information can specifically include a camera calibration stage and a measurement stage. In the camera calibration stage, the light emitter can emit light, and when the light is irradiated to the reference plane, a reference image can be formed, and then the reference image is obtained through the camera. The distance from the reference plane to the electronic device is known, and then the correspondence between the known distance and the reference image can be established. In the measurement stage, the actual distance of the object can be calculated based on the image obtained in real time and the above correspondence.

It is understandable that the camera module may generate heat during operation, and the parameters and shape of the camera module may be affected by temperature changes. Therefore, in order to reduce the error caused by temperature, the camera module can be controlled to reach different temperatures during the camera calibration process, and the camera module can be controlled at different temperatures to collect reference images. Specifically, the temperature of the camera module can be controlled to reach at least two different specified temperatures, and the camera module can be fully calibrated at different specified temperatures.

Step 304 , controlling the camera to collect a reference image formed when the light emitter illuminates the reference plane at a specified temperature.

The light emitter and camera in the camera module are generally on the same horizontal line. The calibration device first needs to adjust the position of the electronic device so that the optical axis formed by the light emitter and the camera is perpendicular to the reference plane, so that the vertical distance from the electronic device to the reference plane can be calculated. It is understandable that the above vertical distance is adjustable, and when the vertical distance from the electronic device to the reference plane is different, the reference image formed is also different. When the temperature of the camera module reaches different specified temperatures, the light source generator can be controlled to emit light. When the light is irradiated to the reference plane, the reference image formed is collected by the camera.

Step 306, storing the designated temperature and the reference image in correspondence; wherein the reference image carries reference depth information, and the reference depth information is used to calculate the depth information of the object.

The light transmitter can emit a laser containing a number of scattered spots, and then the camera can be used to collect a reference image formed by the laser containing scattered spots irradiating the reference plane. The reference depth information is the distance from the electronic device to the reference plane. The reference depth information is known, and a model for calculating the depth information can be obtained based on the reference image and the reference depth information. During the measurement process, the speckle image formed when the laser is irradiated onto the object can be obtained, and the depth information of the object contained in the speckle image can be calculated based on the above model.

During the camera calibration process, reference images corresponding to different specified temperatures are collected and stored. In the process of measuring depth information, the temperature of the camera module can be first obtained, and the corresponding reference image can be obtained according to the temperature, and the depth information of the object can be calculated according to the obtained reference image. For example, the camera module is controlled to collect a reference image at 30°C (Degree Celsius) and 80°C respectively, and then the reference image and the camera module are stored accordingly. During the measurement process, the current temperature of the camera module is first obtained, and the reference image corresponding to the specified temperature closest to the current temperature is obtained to calculate the depth information.

The camera calibration method provided in the above embodiment can control the temperature of the camera module to reach at least two different specified temperatures, and control the camera module to collect reference images formed at different specified temperatures, and then store the reference images corresponding to the specified temperatures. Since the camera module will deform at different temperatures, and the temperature itself will also affect the image collected by the camera module, the camera module is controlled to collect images at different specified temperatures during camera calibration. In this way, the corresponding reference image can be obtained according to the temperature of the camera module, and the depth information of the object can be calculated according to the reference depth information in the reference image, thereby avoiding the error caused by the temperature change of the camera module and improving the accuracy of image processing.

FIG4 is a flow chart of a camera calibration method in another embodiment. As shown in FIG4 , the camera calibration method includes steps 402 to 412. Among them:

Step 402: inputting at least two pulse width modulations (PWM) of different frequencies into the light emitter, and controlling the temperature of the light emitter to reach at least two different designated temperatures through the pulse width modulation (PWM).

In one embodiment, the light emitter can be connected to the processor, and the processor sends instructions to the light emitter to control the switch of the light emitter. Specifically, during the camera calibration process, the laser speckle can be emitted by the light emitter, and then the reference image formed by the laser speckle irradiated on the object can be collected by the laser camera. The operation of the light emitter can be controlled by a pulse wave, so that the higher the operating frequency, the higher the temperature of the light emitter will be, and the temperature of the camera module will also increase. Therefore, during the calibration process, the temperature of the camera module can be adjusted by controlling the operating frequency of the light emitter. Specifically, the light emitter is controlled to work at a specified frequency, and the temperature of the camera module is controlled to reach a specified temperature by the light emitter working at the specified frequency.

Specifically, the processor can be connected to the camera module and the operating frequency of the light emitter can be controlled by the processor. The processor will input a pulse signal to the light emitter and control the switch of the light emitter through the pulse signal. The above pulse signal can be PWM (Pulse Width Modulation). The processor can input PWM of different frequencies to the light emitter, so that the light emitter reaches different specified temperatures.

Step 404 , controlling the camera to collect a reference image formed when the light emitter illuminates the reference plane at a specified temperature.

Each time a reference image is acquired, the electronic device can establish a corresponding relationship between the acquired reference image and the specified temperature. After acquiring the reference image, the electronic device will store the reference image and the corresponding specified temperature. In this way, during the actual shooting process, the corresponding reference image can be acquired according to the temperature of the camera module.

Step 406: establish a corresponding relationship between the specified temperature and the reference image, and write the specified temperature and the reference image into the secure operation environment of the terminal for storage.

It is understandable that the correspondence between the specified temperature and the reference image can be directly established, or a temperature range can be defined according to the specified temperature, and the correspondence between the temperature range and the reference image can be established, and then the temperature range and the reference image can be written into the terminal. For example, the reference images formed by the light emitter at the specified temperatures of 30°C, 60°C and 90°C are respectively "pic-01", "pic-02" and "pic-03". If the temperature ranges corresponding to the above specified temperatures are [0,50°C], [50°C,90°C], [90°C,+∞), then in the ranging process, the temperature range into which the light emitter falls can be determined, and the corresponding reference image can be obtained according to the temperature range.

Generally, to ensure the security of image processing, electronic devices will calculate depth information in a safe operating environment. Therefore, the collected reference image and the corresponding specified temperature can be stored in a safe operating environment, and during the measurement process, the depth information can be calculated directly in the safe environment. For example, the upper-layer application of the electronic device initiates a face payment instruction. During the face payment process, the depth information can be obtained through the camera module, and the depth information is used to determine whether the face is alive. Therefore, it is necessary to ensure that the depth information is calculated in a safe operating environment.

In the embodiment provided in the present application, the safe operation environment in the terminal can be divided into a first safe operation environment and a second safe operation environment, and the storage space in the first safe operation environment is larger than the storage space in the second safe operation environment. In order to avoid excessive occupation of the storage space in the second safe operation environment and affect the processing of the image, the specified temperature and reference image can be written into the first safe operation environment of the terminal for storage during the calibration process. When it is detected that the terminal is turned on, the specified temperature and reference image are loaded from the first safe operation environment to the second safe operation environment for storage.

Step 408: When it is detected that the camera module is turned on, the camera module is controlled to collect a speckle image.

Specifically, the processing unit of the terminal can receive instructions from the upper-layer application. When the processing unit receives the image acquisition instruction, it can control the camera module to work and collect speckle images through the camera. The processing unit is connected to the camera, and the image acquired by the camera can be transmitted to the processing unit, and the processing unit performs processing such as cropping, brightness adjustment, face detection, and face recognition. Specifically, when the processing unit receives the image acquisition instruction, the processing unit controls the light emitter to work. When the light emitter is turned on, the laser camera collects the speckle image formed by the light emitter irradiating the object.

It is understandable that the above-mentioned light emitter can be a laser emitter. When the laser is irradiated on an optically rough surface whose average fluctuation is greater than the order of the wavelength, the sub-waves scattered by the randomly distributed surface elements on these surfaces are superimposed on each other so that the reflected light field has a random spatial light intensity distribution and presents a granular structure, which is the laser speckle. The formed laser speckle is highly random, so the laser speckles generated by lasers emitted by different laser emitters are different. When the formed laser speckle is irradiated on objects of different depths and shapes, the generated speckle images are different. The laser speckle formed by different laser emitters is unique, and thus the obtained speckle image is also unique. The laser speckle formed by the light emitter can be irradiated on an object, and then the speckle image formed by the laser speckle irradiated on the object is collected by a laser camera.

Image acquisition instructions refer to instructions for triggering image acquisition operations. For example, when a user unlocks a smartphone, he can verify the unlocking by acquiring a face image. The upper-layer application can initiate an image acquisition instruction and control the camera module to acquire images through the image acquisition instruction. Specifically, the first processing unit can receive an image acquisition instruction initiated by an upper-layer application. When the first processing unit detects the image acquisition instruction, it controls the camera module to turn on and then controls the camera module to acquire a speckle image. The speckle image acquired by the camera module can be sent to the second processing unit, and the second processing unit calculates the depth information based on the speckle image.

Step 410: When it is detected that the temperature change of the light emitter exceeds the temperature threshold, the current temperature of the light emitter is obtained.

After detecting that the camera module is turned on, a temperature sensor can be used to periodically detect the temperature of the light emitter, and the detected temperature can be sent to the second processing unit. The second processing unit determines whether the temperature change of the light emitter exceeds the temperature threshold. If so, the temperature is used as the current temperature of the light emitter, and the corresponding target reference image is obtained based on the current temperature, and the depth information is calculated based on the obtained target reference image. For example, the temperature threshold can be 5°C. When the temperature change of the light emitter exceeds 5°C, the corresponding target reference image can be determined based on the obtained temperature of the light emitter. It can be understood that in order to ensure accuracy, the time interval between collecting the speckle image and obtaining the current temperature cannot be too long.

Step 412: acquiring a corresponding target reference image according to the current temperature of the light emitter, and calculating a depth image according to the speckle image and the target reference image, wherein the depth image is used to represent the depth information of the object.

The specified temperature and the reference image are stored correspondingly. Then, in the process of measuring the depth information, the corresponding target reference image can be determined according to the current temperature of the light emitter, and then the depth image can be calculated according to the speckle image and the target reference image. Specifically, the target reference image can be compared with the speckle image to obtain the offset information. The offset information is used to indicate the horizontal offset of the speckle spots in the speckle image relative to the corresponding speckle spots in the target reference image. The depth image is calculated according to the offset information and the reference depth information.

In one embodiment, each pixel point (x, y) in the speckle image is traversed, and a pixel block of a preset size is selected with the pixel point as the center. For example, a pixel block of 31 pixels*31 pixels can be selected. Then, a matching pixel block is searched on the target reference image, and the horizontal offset between the coordinates of the matching pixel point on the target reference image and the coordinates of the pixel point (x, y) is calculated. The offset to the right is positive, and the offset to the left is negative. Then, the calculated horizontal offset is substituted into formula (1) to obtain the depth information of the pixel point (x, y). In this way, the depth information of each pixel point in the speckle image is calculated in turn, and the depth information corresponding to each pixel point in the speckle image can be obtained.

The depth image can be used to represent the depth information corresponding to the photographed object, and each pixel contained in the depth image represents a depth information. Specifically, each scattered speckle in the reference image corresponds to a reference depth information. After obtaining the horizontal offset between the scattered speckle in the reference image and the scattered speckle in the speckle image, the relative depth information from the object in the speckle image to the reference plane can be calculated based on the horizontal offset. Then, based on the relative depth information and the reference depth information, the actual depth information from the object to the camera can be calculated, that is, the final depth image can be obtained.

FIG5 is a schematic diagram of a principle for calculating depth information in one embodiment. As shown in FIG5 , the laser lamp 502 can generate laser speckles, and after the laser speckles are reflected by an object, the formed image is acquired by the laser camera 504. During the calibration process of the camera, the laser speckles emitted by the laser lamp 502 will be reflected by the reference plane 508, and then the reflected light will be collected by the laser camera 504, and the reference image will be obtained by imaging the imaging plane 510. The reference depth from the reference plane 508 to the laser lamp 502 is L, and the reference depth is known. In the actual process of calculating the depth information, the laser speckles emitted by the laser lamp 502 will be reflected by the object 506, and then the reflected light will be collected by the laser camera 504, and the actual speckle image will be obtained by imaging the imaging plane 510. The calculation formula for the actual depth information can be obtained as follows:

Wherein, L is the distance between the laser light 502 and the reference plane 508, f is the focal length of the lens in the laser camera 504, CD is the distance between the laser light 502 and the laser camera 504, and AB is the offset distance between the image of the object 506 and the image of the reference plane 508. AB can be the product of the pixel offset n and the actual distance p of the pixel point. When the distance Dis between the object 504 and the laser light 502 is greater than the distance L between the reference plane 506 and the laser light 502, AB is a negative value; when the distance Dis between the object 504 and the laser light 502 is less than the distance L between the reference plane 506 and the laser light 502, AB is a positive value.

In one embodiment, the camera module may include a first camera module and a second camera module. The first camera module is composed of a floodlight and a laser camera, and the second camera module is composed of a laser light and a laser camera. The laser camera of the first camera module and the laser camera of the second camera module may be the same laser camera or different laser cameras, which is not limited here. The laser light may emit laser speckle, and the speckle image may be collected by the first camera module. The floodlight may emit laser, and the infrared image may be collected by the second camera module.

Among them, the infrared image can represent the detailed information of the photographed object, and the depth information of the photographed object can be obtained according to the speckle image. In order to ensure that the infrared image and the speckle image collected by the electronic device are corresponding, it is necessary to control the camera module to collect the infrared image and the speckle image at the same time. Assuming that the first camera module and the second camera module work in time-sharing, it is necessary to ensure that the time interval between collecting the infrared image and the speckle image is very short. Specifically, according to the image acquisition instruction, the first camera module is controlled to collect the infrared image, and the second camera module is controlled to collect the speckle image; wherein the time interval between the first moment of collecting the infrared image and the second moment of collecting the speckle image is less than the first threshold.

The first threshold is generally a relatively small value. When the time interval is less than the first threshold, it is considered that the object has not changed, and the collected infrared image and speckle image are corresponding. It can be understood that it can also be adjusted according to the change law of the object being photographed. The faster the change of the object being photographed, the smaller the corresponding first threshold is. Assuming that the object being photographed is in a stationary state for a long time, the first threshold can be set to a larger value. Specifically, the change speed of the object being photographed is obtained, and the corresponding first threshold is obtained according to the change speed.

For example, when a mobile phone needs to be unlocked through face authentication, the user can click the unlock button to initiate the unlock command and aim the front camera at the face to take a picture. The mobile phone will send the unlock command to the processing unit, which will then control the camera to work. First, the first camera module collects infrared images, and after an interval of 1 millisecond, the second camera module is controlled to collect speckle images, and the collected infrared images and speckle images are used for authentication and unlocking.

Furthermore, at the first moment, the camera module is controlled to collect infrared images, and at the second moment, the camera module is controlled to collect speckle images; the time interval between the first moment and the target moment is less than the second threshold; the time interval between the second moment and the target moment is less than the third threshold. If the time interval between the first moment and the target moment is less than the second threshold, the camera module is controlled to collect infrared images; if the time interval between the first moment and the target moment is greater than the second threshold, a prompt message indicating a response timeout may be returned to the application, and the application is waited to re-initiate an image acquisition instruction.

After the camera module acquires the infrared image, the processing unit may control the camera module to acquire the speckle image, and the time interval between the second moment and the first moment of acquiring the speckle image is less than the first threshold, and the time interval between the second moment and the target moment is less than the third threshold. If the time interval between the second moment and the first moment is greater than the first threshold, or the time interval between the second moment and the target moment is greater than the third threshold, a prompt message of response timeout may be returned to the application, and the application is waited to re-initiate the image acquisition instruction. It can be understood that the second moment of acquiring the speckle image may be greater than the first moment of acquiring the infrared image, or may be less than the first moment of acquiring the infrared image, which is not limited here.

Specifically, the electronic device can be provided with a floodlight controller and a laser light controller respectively, and the floodlight controller and the laser light controller are respectively connected through two PWM (Pulse Width Modulation) channels. The processing unit can input PWM to the floodlight controller to control the floodlight to turn on, or input PWM to the laser light controller to control the laser light to turn on.

In the embodiment provided in the present application, the step of storing the reference image may further include:

Step 602, obtaining the module identification of the camera module, and establishing a corresponding relationship between the module identification, the specified temperature and the reference image.

It is understandable that in the process of calibrating the camera, the camera module installed on the terminal can be calibrated, or the camera module can be calibrated separately. In this way, if the camera module on the terminal is damaged, after replacing the camera module, the reference image of the calibrated camera module can be directly written into the terminal.

Specifically, each camera module has a corresponding module ID, which can be used to distinguish different camera modules. When calibrating the camera module, after obtaining the reference image, a correspondence between the module ID, the specified temperature, and the reference image can be established. In this way, after the terminal reinstalls the camera module, it can obtain the corresponding specified temperature and reference image according to the module ID.

Step 604, storing the module identification, the specified temperature and the reference image in the server.

In the process of calibrating the camera module alone, the obtained module identification, specified temperature and reference image can be stored in the server. The server can store the above module identification, specified temperature and reference image in a list form, and the specified temperature and reference image can be queried and obtained according to the module identification. After the camera module is calibrated, the terminal can obtain the reference image from the server when reinstalling the camera module. Specifically:

Step 702: When the server receives the reference image acquisition request sent by the terminal, the server acquires the corresponding specified temperature and reference image according to the module identifier included in the reference image acquisition request.

The terminal can reinstall the camera module, and after installing the camera module, read the module identification of the installed camera module. Then, a reference image acquisition request is generated according to the module identification, and the reference image acquisition request is sent to the server. Specifically, when the terminal sends the reference image acquisition request, the module identification contained therein can be encrypted, and then the encrypted reference image acquisition request can be sent to the server.

Step 704: Send the designated temperature and the reference image to the terminal.

After receiving the reference image acquisition request, the server can search for the corresponding specified temperature and reference image according to the module identifier, and send the specified temperature and reference image to the terminal after encryption. After receiving the specified temperature and reference image, the terminal performs decryption processing. Then the decrypted specified temperature and reference image are written to the terminal. Specifically, the algorithm for encrypting the module identifier, specified temperature and reference image is not limited. For example, it can be based on DES (Data Encryption Standard), MD5 (Message-Digest Algorithm 5), HAVAL (Diffie-Hellman, key exchange algorithm).

The camera calibration method provided in the above embodiment can control the temperature of the camera module to reach at least two different specified temperatures, and control the camera module to collect reference images formed at different specified temperatures, and then store the reference images corresponding to the specified temperatures. Since the camera module will deform at different temperatures, and the temperature itself will also affect the image collected by the camera module, the camera module is controlled to collect images at different specified temperatures during camera calibration. In this way, the corresponding reference image can be obtained according to the temperature of the camera module, and the depth information of the object can be calculated according to the reference depth information in the reference image, thereby avoiding the error caused by the temperature change of the camera module and improving the accuracy of image processing.

It should be understood that, although the various steps in the flowcharts of Fig. 3, Fig. 4, Fig. 6, and Fig. 7 are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in Fig. 3, Fig. 4, Fig. 6, and Fig. 7 may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

FIG8 is a hardware structure diagram of a camera calibration method in one embodiment. As shown in FIG8 , the electronic device may include a camera module 810, a central processing unit (CPU) 820 and a second processing unit 830, wherein the camera module 810 includes a laser camera 812, a floodlight 814, an RGB (Red/Green/Blue, red/green/blue color mode) camera 816 and a laser light 818. The second processing unit 830 includes a pulse width modulation module 832, an SPI/I2C (Serial Peripheral Interface/Inter-Integrated Circuit, Serial Peripheral Interface/Bidirectional Two-Wire Synchronous Serial Interface) module 834, a RAM (Random Access Memory, Random Access Memory) module 836, and a Depth Engine module 838. Among them, the first processing unit 822 may be a CPU core in a TEE (Trusted execution environment), and the second processing unit 830 is an MCU (Microcontroller Unit) processor. It is understandable that the CPU 820 can be in a multi-core operation mode, and the CPU core in the CPU 820 can run in TEE or REE (Rich Execution Environment). Both TEE and REE are operation modes of ARM modules (Advanced RISC Machines). Generally, operations with higher security in electronic devices need to be executed under TEE, and other operations can be executed under REE.

During the camera calibration process, the pulse width modulation module 932 in the second processing unit 830 can be used to control the laser lamp 818 to reach at least two different specified temperatures, and when reaching different specified temperatures, the laser camera 812 is controlled to collect the reference image formed when the laser lamp 818 illuminates the reference plane. The collected reference image and the specified temperature can be stored in the first processing unit 822 in the trusted operating environment (the first safe operating environment). When the electronic device is turned on, the specified temperature and the reference image are loaded from the first processing unit 822 to the second processing unit 830 for storage. It can be understood that the second processing unit 830 is a processing unit external to the central processing unit 820, and its input and output are controlled by the first processing unit 822 in the trusted operating environment, so it can be considered that the second processing unit 830 is in the second safe operating environment.

In the process of measuring depth information, when the central processor 820 receives an image acquisition instruction initiated by the target application, the CPU core running under the TEE, i.e., the first processing unit 822, sends the image acquisition instruction to the second processing unit 830 through the SECURE SPI/I2C to the SPI/I2C module 834 in the MCU830. After receiving the image acquisition instruction, the second processing unit 830 controls the floodlight 814 in the camera module 810 to turn on to collect infrared images and controls the laser light 818 in the camera module 810 to turn on to collect speckle images by emitting pulse waves through the pulse width modulation module 832. The camera module 810 can transmit the collected infrared image and speckle image to the Depth Engine module 838 in the second processing unit 830, and the Depth Engine module 838 can calculate the infrared parallax image according to the infrared image, calculate the depth image according to the speckle image and the reference image, and obtain the depth parallax image according to the depth image. Then, the infrared parallax image and the depth parallax image are sent to the first processing unit 822 running under the TEE. The first processing unit 822 will calibrate the infrared parallax image to obtain a calibrated infrared image, and calibrate the depth parallax image to obtain a calibrated depth image. Among them, because the laser camera 812 and the RGB camera 816 are installed in different positions, when collecting images, it is necessary to align and calibrate the images collected by the two cameras to avoid errors caused by shooting angles. That is, the infrared image and the depth image need to be calibrated to obtain a calibrated infrared image and a calibrated depth image, respectively.

In one embodiment, face recognition can be performed based on the corrected infrared image to detect whether there is a face in the corrected infrared image and whether the detected face matches the stored face; if the face recognition is successful, liveness detection can be performed based on the corrected infrared image and the corrected depth image to detect whether the face is a live face. After obtaining the corrected infrared image and the corrected depth image, liveness detection can be performed first and then face recognition, or face recognition and liveness detection can be performed simultaneously. After face recognition is successful and the detected face is a live face, the first processing unit 822 can send one or more of the corrected infrared image, the corrected depth image, and the face recognition result to the target application.

FIG9 is an interactive diagram of implementing a camera calibration method in an embodiment. As shown in FIG9 , the interactive process of the camera calibration method may include steps 902 to 920. Among them:

Step 902: The calibration device controls the temperature of the camera module to reach at least two different specified temperatures.

Step 904: The calibration device controls the camera to collect a reference image formed when the light emitter illuminates the reference plane at a specified temperature.

Step 906: The calibration device obtains the module identification of the camera module and establishes a corresponding relationship between the module identification, the specified temperature and the reference image.

Step 908: The calibration device sends the module identification, the specified temperature and the reference image to the server.

Step 910: The server receives and stores the module identification, the specified temperature, and the reference image.

Step 912: The terminal installs a camera module, obtains a module identifier of the installed camera module, and generates an image acquisition request according to the module identifier.

Step 914: The terminal sends the generated image acquisition request to the server.

Step 916: The server obtains the corresponding specified temperature and reference image according to the module identifier included in the reference image acquisition request.

Step 918: The server sends the acquired designated temperature and reference image to the terminal.

Step 920: The terminal receives the designated temperature and the reference image sent by the server, and writes the designated temperature and the reference image into the secure operation environment of the terminal for storage.

The camera calibration device provided in the above embodiment can obtain the corresponding reference image according to the temperature of the camera module, and calculate the depth information of the object according to the reference depth information in the reference image, thereby avoiding the error caused by the temperature change of the camera module and improving the accuracy of image processing.

FIG10 is a schematic diagram of the structure of a camera calibration device in one embodiment. As shown in FIG10 , the camera calibration device 1000 includes a temperature control module 1002, an image acquisition module 1004 and an image storage module 1006. Among them:

The temperature control module 1002 is used to control the temperature of the camera module to reach at least two different specified temperatures, wherein the camera module includes a light emitter and a camera.

The image acquisition module 1004 is used to control the camera to collect the reference image formed when the light emitter illuminates the reference plane at the specified temperature.

The image storage module 1006 is used to store the designated temperature and the reference image in correspondence; wherein the reference image carries reference depth information, and the reference depth information is used to calculate the depth information of the object.

The camera calibration device provided in the above embodiment can control the temperature of the camera module to reach at least two different specified temperatures, and control the camera module to collect reference images formed at different specified temperatures, and then store the reference images corresponding to the specified temperatures. Since the camera module will deform at different temperatures, and the temperature itself will also affect the image collected by the camera module, the camera module is controlled to collect images at different specified temperatures during camera calibration. In this way, the corresponding reference image can be obtained according to the temperature of the camera module, and the depth information of the object can be calculated according to the reference depth information in the reference image, thereby avoiding the error caused by the temperature change of the camera module and improving the accuracy of image processing.

FIG11 is a schematic diagram of the structure of a camera calibration device in another embodiment. As shown in FIG11 , the camera calibration device 1100 includes a temperature control module 1102, an image acquisition module 1104, an image storage module 1106 and a depth calculation module 1108. Among them:

The temperature control module 1102 is used to control the temperature of the camera module to reach at least two different specified temperatures, wherein the camera module includes a light emitter and a camera.

The image acquisition module 1104 is used to control the camera to collect the reference image formed when the light emitter illuminates the reference plane at the specified temperature.

The image storage module 1106 is used to store the designated temperature and the reference image in correspondence; wherein the reference image carries reference depth information, and the reference depth information is used to calculate the depth information of the object.

The depth calculation module 1108 is used to control the camera module to collect a speckle image when it is detected that the camera module is turned on; obtain the current temperature of the light emitter when it is detected that the temperature change of the light emitter exceeds the temperature threshold; obtain the corresponding target reference image according to the current temperature of the light emitter, and calculate a depth image according to the speckle image and the target reference image, wherein the depth image is used to represent the depth information of the object.

The camera calibration device provided in the above embodiment can obtain the corresponding reference image according to the temperature of the camera module, and calculate the depth information of the object according to the reference depth information in the reference image, thereby avoiding the error caused by the temperature change of the camera module and improving the accuracy of image processing.

In one embodiment, the temperature control module 1102 is further configured to input at least two pulse width modulation PWMs of different frequencies to the light emitter, and control the temperature of the light emitter to reach at least two different specified temperatures through the pulse width modulation PWM.

In one embodiment, the image storage module 1106 is further used to establish a corresponding relationship between the specified temperature and the reference image, and write the specified temperature and the reference image into the secure operation environment of the terminal for storage.

In one embodiment, the image storage module 1106 is also used to write the specified temperature and reference image into the first safe operating environment of the terminal for storage; when it is detected that the terminal is turned on, the specified temperature and reference image are loaded from the first safe operating environment to the second safe operating environment for storage.

In one embodiment, the image storage module 1106 is also used to obtain the module identification of the camera module, and establish a corresponding relationship between the module identification, the specified temperature and the reference image; and store the module identification, the specified temperature and the reference image in the server.

In one embodiment, the image storage module 1106 is also used to obtain the corresponding specified temperature and reference image according to the module identifier contained in the reference image acquisition request when the server receives the reference image acquisition request sent by the terminal; and send the specified temperature and reference image to the terminal.

The division of the various modules in the above-mentioned camera calibration device is only for illustration. In other embodiments, the camera calibration device can be divided into different modules as needed to complete all or part of the functions of the above-mentioned camera calibration device.

The embodiment of the present application further provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, the processors execute the camera calibration method provided in the above embodiment.

A computer program product comprising instructions, when running on a computer, enables the computer to execute the camera calibration method provided in the above embodiment.

Any reference to memory, storage, database or other medium used in this application may include non-volatile and/or volatile memory. Suitable non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM), which is used as an external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims (18)

1. An image processing method applied to an electronic device with a camera module, wherein the camera module comprises a light emitter and a camera, and the method is characterized by comprising the following steps:

receiving an image acquisition instruction;

controlling the camera module to acquire speckle images; the speckle image is an image formed by the light emitter irradiating an object;

Detecting the temperature of the light emitter and judging whether the temperature change of the light emitter exceeds a temperature threshold value;

when detecting that the temperature change of the light emitter exceeds a temperature threshold, acquiring the current temperature of the light emitter;

And acquiring a corresponding target reference image according to the current temperature of the light emitter, and calculating a depth image according to the speckle image and the target reference image, wherein the depth image is used for representing the depth information of the object.

2. The method of claim 1, wherein the acquiring the corresponding target reference image based on the current temperature of the light emitter comprises:

acquiring a corresponding target reference image from a safe operation environment of the terminal according to the current temperature of the light emitter;

the safe operation environment of the terminal stores the designated temperature and the corresponding reference image.

3. The method of claim 2, wherein the acquiring the corresponding target reference image from the secure operating environment of the terminal according to the current temperature of the light emitter comprises:

when the terminal is detected to be started, loading the designated temperature and the reference image into a second safe operation environment from a first safe operation environment for storage, and acquiring a corresponding target reference image from the second safe operation environment of the terminal according to the current temperature of the light emitter, wherein the storage space in the first safe operation environment is larger than the storage space in the second safe operation environment.

4. The method of claim 3, wherein the step of,

The receiving the image acquisition instruction comprises:

Transmitting the image acquisition instruction to a second processing unit in a second safe operation environment through a first processing unit in the first safe operation environment;

the computing a depth image from the speckle image and a target reference image comprises:

A second processing unit in the second secure operating environment calculates a depth image from the speckle image and a target reference image.

5. The method according to claim 4, wherein the method further comprises:

Controlling the camera module to acquire an infrared image, and sending the infrared image to a second processing unit, wherein the second processing unit calculates an infrared parallax image according to the infrared image;

Obtaining a depth parallax image according to the depth image;

and transmitting the infrared parallax image and the depth parallax image to the first processing unit, and correcting the infrared parallax image by the first processing unit to obtain a corrected infrared image and correcting the depth parallax image to obtain a corrected depth image.

6. The method of claim 5, wherein the method further comprises:

detecting whether a human face exists in the corrected infrared image or not and whether the detected human face is matched with a stored human face or not;

detecting whether the human face is a living human face according to the corrected infrared image and the corrected depth image;

After the face recognition is passed and the detected face is a living face, the first processing unit sends one or more of the corrected infrared image, the corrected depth image and the face recognition result to a target application program.

7. The method of claim 1, wherein the acquiring the corresponding target reference image based on the current temperature of the light emitter comprises:

sending a reference image acquisition request to a server, and acquiring a corresponding target reference image from the server according to the current temperature of the light emitter;

the server stores a module identifier, a designated temperature and a reference image of the camera module, and the module identifier, the designated temperature and the reference image establish a corresponding relation.

8. The method according to claim 1, wherein the method further comprises:

A reference image acquisition request is sent to a server, wherein the reference image acquisition request comprises a module identifier, the server stores the module identifier of the camera module, the designated temperature and the reference image, and the corresponding relation is established between the module identifier, the designated temperature and the reference image;

and receiving and storing the designated temperature and the reference image sent by the server.

9. A method of camera calibration, the method comprising:

Controlling the temperature of the camera module to reach at least two different designated temperatures, wherein the camera module comprises a light emitter and a camera;

controlling the camera to acquire a reference image formed when the light emitter irradiates a reference plane at the specified temperature;

Establishing a corresponding relation between the specified temperature and the reference image, and writing the specified temperature and the reference image into a safe operation environment of the terminal for storage; wherein the reference image carries reference depth information, which is used to calculate depth information of the object.

10. The method of claim 9, wherein controlling the temperature of the camera module to reach at least two different specified temperatures comprises:

At least two Pulse Width Modulation (PWM) of different frequencies are input to the light emitter, and the temperature of the light emitter is controlled to reach at least two different designated temperatures through the PWM.

11. The method of claim 9, wherein writing the specified temperature and reference image into a secure operating environment of a terminal for storage comprises:

writing the designated temperature and the reference image into a first safe operation environment of the terminal for storage;

and when the terminal is detected to be started, loading the designated temperature and the reference image from the first safe operation environment into a second safe operation environment for storage.

12. The method of claim 9, wherein storing the specified temperature in correspondence with a reference image comprises:

acquiring a module identifier of the camera module, and establishing a corresponding relation between the module identifier, the designated temperature and a reference image;

and storing the module identification, the designated temperature and the reference image into a server.

13. The method of claim 12, wherein after storing the module identification, the specified temperature, and the reference image in a server, further comprising:

when the server receives a reference image acquisition request sent by a terminal, acquiring a corresponding designated temperature and a reference image according to a module identifier contained in the reference image acquisition request;

And sending the designated temperature and the reference image to the terminal.

14. An image processing apparatus for use in an electronic device having a camera module including a light emitter and a camera, the apparatus comprising:

The instruction receiving module is used for receiving an image acquisition instruction;

The temperature detection module is used for detecting the temperature of the light emitter and judging whether the temperature change of the light emitter exceeds a temperature threshold value or not;

The depth calculation module is used for controlling the camera module to acquire speckle images; the speckle image is an image formed by the light emitter irradiating an object;

When detecting that the temperature change of the light emitter exceeds a temperature threshold, acquiring the current temperature of the light emitter; and acquiring a corresponding target reference image according to the current temperature of the light emitter, and calculating a depth image according to the speckle image and the target reference image, wherein the depth image is used for representing the depth information of the object.

15. A camera calibration apparatus, the apparatus comprising:

the temperature control module is used for controlling the temperature of the camera module to reach at least two different specified temperatures, wherein the camera module comprises a light emitter and a camera;

the image acquisition module is used for controlling the camera to acquire a reference image formed when the light emitter irradiates a reference plane at the specified temperature;

The image storage module is used for establishing a corresponding relation between the specified temperature and the reference image, and writing the specified temperature and the reference image into a safe operation environment of the terminal for storage; wherein the reference image carries reference depth information, which is used to calculate depth information of the object.

16. A computer readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, implements the method of any one of claims 1 to 13.

17. An electronic device comprising a memory and a processor, the memory having stored therein computer readable instructions that, when executed by the processor, cause the processor to perform the method of any of claims 1-13.

18. A computer program product comprising instructions which, when run on a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 13.