WO2022236492A1

WO2022236492A1 - Image processing method and apparatus

Info

Publication number: WO2022236492A1
Application number: PCT/CN2021/092444
Authority: WO
Inventors: 吕笑宇; 马莎; 罗达新
Original assignee: 华为技术有限公司
Priority date: 2021-05-08
Filing date: 2021-05-08
Publication date: 2022-11-17
Also published as: CN113924768B; CN113924768A

Abstract

Provided in the present application are an image processing method and apparatus, which relate to the field of image processing, can be applied to autonomous driving or intelligent driving scenarios, and are used for reducing the amount of calculation during the determination of polarization information. The method comprises: an image processing apparatus acquiring first feature information, wherein the first feature information is used for representing a scenario feature and indicating the acquisition of polarization information of at least one target, and the first feature information corresponds to a first illumination mode, a first collection mode and a first image processing mode; controlling illumination of polarized light according to the first illumination mode, and controlling, according to the first collection mode, the collection of a first polarization image of the at least one target in the first illumination mode; and processing the first polarization image according to the first image processing mode to acquire the polarization information of the at least one target. In this way, the image processing apparatus outputs the polarization information according to requirements, such that the amount of calculation of the image processing apparatus can be reduced.

Description

Image processing method and device

technical field

The present application relates to the technical field of image processing, and in particular to an image processing method and device.

Background technique

Currently, a terminal device can use a polarization camera to detect surrounding environment information, and determine information such as shapes and colors of objects (eg, vehicles, obstacles, roads, etc.) existing around the terminal device.

However, the polarization image acquired by the terminal device has polarization information in multiple dimensions. For example, the current common polarization information includes at least: polarization-modulated polarization image, Stokes vector, degree of polarization, polarization angle, and Mueller matrix. When it is necessary to obtain different polarization information of the target, the terminal device needs to adopt different processing methods. When the current terminal device obtains the polarization information, the terminal device will uniformly output all the polarization information of the target that can be obtained (the polarization information generally includes the above-mentioned polarization-modulated polarization image, Stokes vector, degree of polarization, polarization angle, and Muller matrix) , which will lead to a large amount of calculation when the terminal device processes the polarization image, which will affect the calculation performance of the terminal device.

Contents of the invention

The present application provides an image processing method and device, which solves the problem in the prior art that a terminal device has a large amount of calculation when processing a polarization image, which affects the calculation performance of the terminal device.

In order to solve the above technical problems, the application adopts the following technical solutions:

In a first aspect, an image processing method is provided, including: acquiring first feature information; the first feature information is used to characterize scene features and obtain polarization information indicating at least one target, the first feature information corresponds to the first lighting mode, The first acquisition mode and the first image processing mode. The polarized light illumination is controlled according to the first illumination mode, and, according to the first acquisition mode, the first polarized image of at least one target under the first illumination mode is controlled to be collected. According to the first image processing manner, the first polarization image is processed to acquire polarization information of at least one target.

Based on the above technical solution, in the image processing method provided by the embodiment of the present application, the image processing device can select the corresponding image processing method according to the acquisition of the polarization information of the target to obtain the polarization information of the target, avoiding the acquisition of all possible polarization information of the target The resulting image processing device has a large amount of calculation.

In addition, the image processing device can determine the Mueller matrix of the target according to the polarized light provided for the target and the collected polarized light reflected by the target, and then determine the quantitative polarization information of the target. The image processing device can also provide illumination for the target, which improves the imaging clarity of the target in dark and light scenes.

With reference to the first aspect above, in a possible implementation manner, the method further includes: determining a first illumination manner, a first collection manner, and a first image processing manner according to the first characteristic information.

Based on this, the image processing device can determine the station name method, image acquisition method, and image processing method for the target when different polarization information is obtained in each scene. In this way, the requirements for collecting different polarization information in different scenarios can be better met.

With reference to the first aspect above, in a possible implementation manner, the scene feature includes light intensity information of the environment.

Based on this, the image acquisition device can provide lighting for the scene when the ambient light intensity is low, so as to increase the ambient light intensity of the scene.

With reference to the foregoing first aspect, in a possible implementation manner, the polarization information includes at least one of the following: a second polarization image, qualitative polarization information, or quantitative polarization information.

Based on this, the image acquisition device can acquire the second polarization image of the target, qualitative polarization information, and one or more types of polarization information in the quantitative polarization information according to requirements.

In combination with the first aspect above, in a possible implementation manner, the first lighting method belongs to a set of polarized light lighting methods, and the set of polarized light lighting methods includes at least one of the following: synchronous lighting with multiple polarized light sources, asynchronous lighting with multiple polarized light sources, Or no lighting.

Among them, the synchronous illumination of multi-polarized light sources is: providing polarized illumination simultaneously through multiple polarized light sources of different polarization states; the asynchronous illumination of multi-polarized light sources is: providing polarized illumination sequentially through multiple polarized light sources of different polarization states; Provide lighting.

Based on this, different lighting methods can be applied to different scenes and polarization information acquisition requirements, and the image processing device determines to provide different lighting methods under different scenes and polarization information acquisition requirements, which can improve the application of the image processing method provided by this application. sex.

In combination with the first aspect above, in a possible implementation manner, the first acquisition mode belongs to a polarization acquisition mode set, and the polarization acquisition mode set includes: a synchronous acquisition operation of a single-frame polarization image, and an asynchronous acquisition operation of a multi-frame polarization image.

The synchronous acquisition operation of a single-frame polarization image includes: acquiring multiple polarization images of different polarization states from a single image by acquiring a single image of at least one target; the asynchronous acquisition operation of a multi-frame polarization image includes: acquiring at least one target A plurality of images, each image in the plurality of images includes a plurality of polarization images of different polarization states.

Based on this, different image acquisition methods can be applied to different scenarios and polarization information acquisition requirements, and the image processing device determines to provide different image methods under different scenarios and polarization information acquisition requirements, which can improve the image processing method provided by this application. applicability. In addition, the illumination method and the image acquisition method cooperate with each other. The illumination method provides polarized light illumination with different polarization states, and the image acquisition device collects polarization images under different polarization states of polarized light illumination conditions, which can determine the Mueller matrix of the target, and then determine the target Quantitative polarization information of .

In combination with the first aspect above, in a possible implementation manner, the first image processing method belongs to a set of polarization image processing methods, and the set of polarization image processing methods includes: polarization image acquisition operation, qualitative polarization information acquisition operation, and quantitative polarization information acquisition operation .

Wherein, the polarization image acquisition operation is used to acquire a second polarization image, and the second polarization image belongs to the polarization images of the first polarization image in different polarization states; the qualitative polarization information acquisition operation is used to acquire at least one of the following polarization information of at least one target : Stokes vector, degree of polarization, or angle of polarization; the quantitative polarization information acquisition operation is used to acquire the Mueller matrix of at least one target.

Based on this, different image processing methods can be applied to different polarization information acquisition requirements, and the image processing device determines to adopt different image processing methods under different polarization information acquisition requirements, which can improve the applicability of the image processing method provided in this application.

In combination with the first aspect above, in a possible implementation manner, the first feature information indicates that the ambient light intensity is less than a preset value and indicates to obtain the second polarization image, and the first lighting operation includes: synchronous lighting of multiple polarization light sources, polarization image acquisition The method includes: a single-frame polarization image synchronous acquisition operation, and the polarization image processing operation includes: a polarization image acquisition operation.

Based on this, when the image processing device is in a scene where the ambient light of the scene is weak and polarized images need to be acquired, the lighting device provides synchronous illumination of multi-polarized light sources for the scene, which can improve the ambient light intensity of the scene; the image acquisition device collects a single image of the target, which can Reducing the number of images to be processed; the image processing device processes the acquired second polarization image of the target in a single image, which can reduce the calculation amount of the image processing device.

In combination with the first aspect above, in a possible implementation, the first feature information indicates that the ambient light intensity is less than a preset value and indicates the acquisition of qualitative polarization information. The polarized light illumination operation includes: synchronous illumination of multiple polarized light sources, polarization image acquisition method Including: single-frame polarization image synchronous acquisition operation, polarization image processing operation includes: qualitative polarization information acquisition operation.

Based on this, when the image processing device is in a scene where the ambient light of the scene is weak and qualitative polarization information needs to be obtained, the lighting device provides synchronous illumination of multi-polarized light sources for the scene, which can improve the ambient light intensity of the scene; the image acquisition device collects a single image of the target, The number of processed images can be reduced; the image processing device processes the acquired single image to acquire the qualitative polarization information of the target, which can reduce the calculation amount of the image processing device.

In combination with the first aspect above, in a possible implementation manner, the first characteristic information indicates that the ambient light intensity is less than a preset value and indicates the acquisition of qualitative polarization information, and the polarized light illumination operation includes: asynchronous illumination of multiple polarized light sources, polarization image acquisition The method includes: multi-frame polarization image asynchronous acquisition operation, and the polarization image processing operation includes: quantitative polarization information acquisition operation.

Based on this, the image processing device provides asynchronous illumination of multi-polarized light sources for the scene where the ambient light of the scene is weak and qualitative polarization information needs to be obtained. The lighting device can improve the ambient light intensity of the scene and provide polarized light with different polarization states for the target. The image acquisition device collects multiple images of the target under different polarized lights, and the image processing device combines the polarized light provided by the polarized light source and the multiple images collected under the multiple polarized lights to determine the Mueller matrix of the target , so as to determine the quantitative polarization information of the target.

In combination with the first aspect above, in a possible implementation manner, the first feature information indicates that the ambient light intensity is greater than or equal to a preset value and indicates to obtain the second polarized image, and the polarized light illumination operation includes: no illumination, polarized image acquisition mode Including: single-frame polarization image synchronous acquisition operation, polarization image processing operation includes: polarization image acquisition operation.

Based on this, the image processing device does not need to provide multi-polarized light source synchronous lighting for the scene in the scene where the ambient light of the scene is strong and polarized images need to be obtained by the lighting device, which can reduce the energy consumption of the lighting device; the image acquisition device collects a single image of the target, The number of images to be processed can be reduced; the image processing device processes the acquired second polarization image of a single image acquisition target, which can reduce the calculation amount of the image processing device.

In combination with the first aspect above, in a possible implementation manner, the first feature information indicates that the ambient light intensity is greater than or equal to a preset value and indicates the acquisition of qualitative polarization information. The polarized light illumination operation includes: no illumination, and the polarized image acquisition method includes : Synchronous acquisition operation of single frame polarization image, polarization image processing operation includes: qualitative polarization information acquisition operation.

Based on this, the image processing device does not need to provide multi-polarized light source synchronous lighting for the scene where the image processing device needs to obtain qualitative polarization information under the scene environment light intensity, which can reduce the energy consumption of the lighting device; the image acquisition device collects a single image of the target , the number of processed images can be reduced; the image processing device can process the acquired qualitative polarization information of a single image to obtain the target, which can reduce the calculation amount of the image processing device.

In combination with the first aspect above, in a possible implementation manner, the first feature information indicates that the ambient light intensity is greater than or equal to a preset value and indicates the acquisition of quantitative polarization information. The polarized light illumination operation includes: asynchronous illumination of multiple polarized light sources, polarization The image acquisition method includes: multi-frame polarization image asynchronous acquisition operation, and the polarization image processing operation includes: quantitative polarization information acquisition operation.

Based on this, in the scene where the image processing device needs to obtain qualitative polarization information due to the ambient light intensity of the scene, the lighting device provides asynchronous illumination of multi-polarized light sources for the scene, and can provide polarized light with different polarization states for the target; the image acquisition device collects the target's For multiple images under different polarized lights, the image processing device can determine the Mueller matrix of the target by combining the polarized light provided by the polarized light source and the collected multiple images under the multiple polarized lights, thereby determining the quantitative polarization of the target information.

In a second aspect, an image processing device is provided, including: a processing unit and an acquisition unit. An acquisition unit, configured to acquire first feature information; the first feature information is used to characterize scene features and to indicate the acquisition of polarization information of at least one target, and the first feature information corresponds to the first lighting mode, the first acquisition mode, and the first image processing method. The processing unit is configured to control polarized light illumination according to the first illumination mode, and, according to the first acquisition mode, control to collect a first polarized image of at least one target under the first illumination mode. The processing unit is further configured to process the first polarization image according to the first image processing manner to acquire polarization information of at least one target.

With reference to the second aspect above, in a possible implementation manner, the processing unit is further configured to: determine a first lighting manner, a first acquisition manner, and a first image processing manner according to the first characteristic information.

With reference to the second aspect above, in a possible implementation manner, the processing unit is specifically configured to: generate a first instruction and a second instruction; the first instruction is used to indicate the first lighting mode; the second instruction is used to indicate the first Acquisition mode: instruct the acquisition unit to send the first instruction to the lighting device, and send the second instruction to the image acquisition device.

With reference to the second aspect above, in a possible implementation manner, the processing unit is further configured to: instruct the acquisition unit to receive the first polarization image from the image acquisition device.

With reference to the second aspect above, in a possible implementation manner, the scene feature includes light intensity information of the environment.

With reference to the second aspect above, in a possible implementation manner, the polarization information includes at least one of the following: a second polarization image, qualitative polarization information, or quantitative polarization information.

In combination with the second aspect above, in a possible implementation, the first lighting method belongs to a set of polarized light lighting methods, and the set of polarized light lighting methods includes at least one of the following: synchronous lighting with multiple polarized light sources, asynchronous lighting with multiple polarized light sources, Or no lighting; wherein, multi-polarized light source synchronous lighting is: providing polarized lighting through multiple polarized light sources with different polarization states at the same time; multi-polarized light source asynchronous lighting is: using multiple polarized light sources with different polarization states to provide polarized lighting sequentially; Lighting is: No lighting provided.

In combination with the second aspect above, in a possible implementation, the first collection method belongs to a collection of polarization collection methods, and the collection of polarization collection methods includes: a synchronous collection operation of a single-frame polarization image, an asynchronous collection operation of a multi-frame polarization image; The synchronous acquisition operation of polarization images includes: acquiring multiple polarization images of different polarization states from a single image by acquiring a single image of at least one target; the asynchronous acquisition operation of multi-frame polarization images includes: acquiring multiple polarization images of at least one target images, each of the plurality of images includes a plurality of polarization images of different polarization states.

In combination with the second aspect above, in a possible implementation, the first image processing method belongs to a set of polarization image processing methods, and the set of polarization image processing methods includes: polarization image acquisition operations, qualitative polarization information acquisition operations, and quantitative polarization information acquisition operations ; Wherein, the polarization image acquisition operation is used to acquire a second polarization image, and the second polarization image belongs to the polarization images of the first polarization image in different polarization states; the qualitative polarization information acquisition operation is used to acquire at least one of the following polarizations of at least one target Information: Stokes vector, degree of polarization, or angle of polarization; the quantitative polarization information acquisition operation is used to acquire the Mueller matrix of at least one target.

In combination with the above-mentioned second aspect, in a possible implementation manner, the first feature information indicates that the ambient light intensity is less than a preset value and indicates to obtain the second polarization image, and the first lighting operation includes: synchronous lighting of multiple polarization light sources, polarization image acquisition The method includes: a single-frame polarization image synchronous acquisition operation, and the polarization image processing operation includes: a polarization image acquisition operation.

In combination with the second aspect above, in a possible implementation, the first feature information indicates that the ambient light intensity is less than a preset value and indicates the acquisition of qualitative polarization information. The polarized light illumination operation includes: synchronous illumination of multiple polarized light sources, polarization image acquisition method Including: single-frame polarization image synchronous acquisition operation, polarization image processing operation includes: qualitative polarization information acquisition operation.

In combination with the second aspect above, in a possible implementation, the first feature information indicates that the ambient light intensity is less than a preset value and indicates the acquisition of qualitative polarization information. The polarized light illumination operation includes: asynchronous illumination of multiple polarized light sources, polarization image acquisition The method includes: multi-frame polarization image asynchronous acquisition operation, and the polarization image processing operation includes: quantitative polarization information acquisition operation.

In combination with the second aspect above, in a possible implementation manner, the first characteristic information indicates that the ambient light intensity is greater than or equal to a preset value and indicates to obtain the second polarized image, and the polarized light illumination operation includes: no illumination, polarized image acquisition mode Including: single-frame polarization image synchronous acquisition operation, polarization image processing operation includes: polarization image acquisition operation.

In combination with the second aspect above, in a possible implementation manner, the first characteristic information indicates that the ambient light intensity is greater than or equal to a preset value and indicates the acquisition of qualitative polarization information. The polarized light illumination operation includes: no illumination, and the polarized image acquisition method includes : Synchronous acquisition operation of single frame polarization image, polarization image processing operation includes: qualitative polarization information acquisition operation.

In combination with the second aspect above, in a possible implementation, the first feature information indicates that the ambient light intensity is greater than or equal to a preset value and indicates that quantitative polarization information is obtained, and the polarized light illumination operation includes: asynchronous illumination of multiple polarized light sources, polarization The image acquisition method includes: multi-frame polarization image asynchronous acquisition operation, and the polarization image processing operation includes: quantitative polarization information acquisition operation.

In a third aspect, an illuminating device is provided, including: a plurality of polarized light sources with different polarization states, at least one processor, and a communication interface. The communication interface is used to receive a first instruction from the image processing device, and the first instruction is used to indicate a first lighting mode; the first lighting mode is used to represent a way of providing illumination by using a plurality of polarized light sources with different polarization states. The processor is configured to control a plurality of polarized light sources with different polarization states to provide illumination according to the first instruction.

In combination with the third aspect above, in a possible implementation, the first lighting method belongs to a set of polarized light lighting methods, and the set of polarized light lighting methods includes at least one of the following: synchronous lighting with multiple polarized light sources, asynchronous lighting with multiple polarized light sources, Or no lighting; wherein, multi-polarized light source synchronous lighting is: providing polarized lighting through multiple polarized light sources with different polarization states at the same time; multi-polarized light source asynchronous lighting is: using multiple polarized light sources with different polarization states to provide polarized lighting sequentially; Lighting is: No lighting provided.

With reference to the third aspect above, in a possible implementation manner, the first lighting method includes: synchronous lighting of multiple polarized light sources; The light source provides polarized illumination.

In combination with the third aspect above, in a possible implementation manner, the first lighting method includes: multi-polarization light source asynchronous lighting; the processor is specifically used to: control each polarized light source among multiple polarized light sources with different polarization states to sequentially provide polarized lighting.

With reference to the third aspect above, in a possible implementation manner, the first illumination manner includes: no illumination; and the processor is specifically configured to: control multiple polarized light sources with different polarization states to provide no illumination.

It should be noted that, when the lighting device described in the third aspect is integrated with the image acquisition device described in the following fourth aspect and the image processing device described in the following fifth aspect, the lighting device and the image acquisition device, the image The processing devices may share at least one processor, and the at least one processor executes the processing operations in the lighting device, the image acquisition device, and the image processing device.

In a fourth aspect, an image acquisition device is provided, including: a polarized image acquisition device, at least one processor and a communication interface; the communication interface is used to receive a second instruction from the image processing device, and the second instruction is used to instruct the first acquisition mode; the first acquisition mode is used to characterize the way of collecting polarization images. The processor is configured to control the polarization image collector to collect polarization images according to the second instruction.

In combination with the fourth aspect above, in a possible implementation, the first collection method belongs to a collection of polarization collection methods, and the collection of polarization collection methods includes: a synchronous collection operation of a single-frame polarization image, an asynchronous collection operation of a multi-frame polarization image; The synchronous acquisition operation of polarization images includes: acquiring multiple polarization images of different polarization states from a single image by acquiring a single image of at least one target; the asynchronous acquisition operation of multi-frame polarization images includes: acquiring multiple polarization images of at least one target images, each of the plurality of images includes a plurality of polarization images of different polarization states.

In combination with the fourth aspect above, in a possible implementation manner, the first collection method includes: a single-frame polarization image synchronous collection operation; the processor is specifically configured to control the polarization image collector to collect at least one target according to the second instruction. A single image, and acquisition of multiple polarization images of different polarization states from a single image.

In combination with the fourth aspect above, in a possible implementation manner, the first collection method includes: an asynchronous collection operation of multi-frame polarization images; the processor is specifically configured to control the polarization image collector to collect at least a plurality of times according to the second instruction An image of an object, multiple images of at least one object are determined, and multiple polarization images of different polarization states are respectively obtained from each of the multiple images.

It should be noted that, when the image acquisition device described in the fourth aspect is integrated with the lighting device described in the third aspect above, and the image processing device described in the fifth aspect below, the lighting device and the image acquisition device, The image processing device may share at least one processor, and the at least one processor executes the processing operations in the lighting device, the image acquisition device, and the image processing device.

According to a fifth aspect, an image processing device is provided, including: at least one processor and a communication interface. A communication interface, used to acquire first feature information; the first feature information is used to characterize the scene feature and to indicate the acquisition of polarization information of at least one target, the first feature information corresponds to the first lighting mode, the first collection mode and the first image processing method. The processor is configured to control polarized light illumination according to the first illumination mode, and, according to the first collection mode, control to collect a first polarized image of at least one target under the first illumination mode. The processor is further configured to process the first polarization image according to the first image processing manner to acquire polarization information of at least one target.

With reference to the fifth aspect above, in a possible implementation manner, the processor is further configured to: determine a first lighting manner, a first acquisition manner, and a first image processing manner according to the first feature information.

With reference to the fifth aspect above, in a possible implementation manner, the processor is specifically configured to: generate a first instruction and a second instruction; the first instruction is used to indicate the first lighting mode; the second instruction is used to indicate the first Collection mode: instruct the communication interface to send the first instruction to the lighting device, and send the second instruction to the image collection device.

With reference to the fifth aspect above, in a possible implementation manner, the processor is further specifically configured to: instruct the communication interface to receive the first polarization image from the image acquisition device.

With reference to the fifth aspect above, in a possible implementation manner, the scene feature includes light intensity information of the environment.

With reference to the foregoing fifth aspect, in a possible implementation manner, the polarization information includes at least one of the following: a second polarization image, qualitative polarization information, or quantitative polarization information.

In combination with the fifth aspect above, in a possible implementation manner, the first lighting method belongs to a set of polarized light lighting methods, and the set of polarized light lighting methods includes at least one of the following: synchronous lighting with multiple polarized light sources, asynchronous lighting with multiple polarized light sources, Or no lighting; wherein, multi-polarized light source synchronous lighting is: providing polarized lighting through multiple polarized light sources with different polarization states at the same time; multi-polarized light source asynchronous lighting is: using multiple polarized light sources with different polarization states to provide polarized lighting sequentially; Lighting is: No lighting provided.

In combination with the fifth aspect above, in a possible implementation, the first collection method belongs to a collection of polarization collection methods, and the collection of polarization collection methods includes: a synchronous collection operation of a single-frame polarization image, an asynchronous collection operation of a multi-frame polarization image; The synchronous acquisition operation of polarization images includes: acquiring multiple polarization images of different polarization states from a single image by acquiring a single image of at least one target; the asynchronous acquisition operation of multi-frame polarization images includes: acquiring multiple polarization images of at least one target images, each of the plurality of images includes a plurality of polarization images of different polarization states.

In combination with the fifth aspect above, in a possible implementation, the first image processing method belongs to a set of polarization image processing methods, and the set of polarization image processing methods includes: polarization image acquisition operation, qualitative polarization information acquisition operation, and quantitative polarization information acquisition operation ; Wherein, the polarization image acquisition operation is used to acquire a second polarization image, and the second polarization image belongs to the polarization images of the first polarization image in different polarization states; the qualitative polarization information acquisition operation is used to acquire at least one of the following polarizations of at least one target Information: Stokes vector, degree of polarization, or angle of polarization; the quantitative polarization information acquisition operation is used to acquire the Mueller matrix of at least one target.

In combination with the fifth aspect above, in a possible implementation manner, the first feature information indicates that the ambient light intensity is less than a preset value and indicates to obtain the second polarization image, and the first lighting operation includes: synchronous lighting of multiple polarization light sources, polarization image acquisition The method includes: a single-frame polarization image synchronous acquisition operation, and the polarization image processing operation includes: a polarization image acquisition operation.

In combination with the fifth aspect above, in a possible implementation, the first feature information indicates that the ambient light intensity is less than a preset value and indicates the acquisition of qualitative polarization information. The polarized light illumination operation includes: synchronous illumination of multiple polarized light sources, polarization image acquisition method Including: single-frame polarization image synchronous acquisition operation, polarization image processing operation includes: qualitative polarization information acquisition operation.

In combination with the fifth aspect above, in a possible implementation, the first characteristic information indicates that the ambient light intensity is less than a preset value and indicates the acquisition of qualitative polarization information, and the polarized light illumination operation includes: asynchronous illumination of multiple polarized light sources, polarization image acquisition The method includes: multi-frame polarization image asynchronous acquisition operation, and the polarization image processing operation includes: quantitative polarization information acquisition operation.

In combination with the fifth aspect above, in a possible implementation manner, the first feature information indicates that the ambient light intensity is greater than or equal to a preset value and indicates to obtain the second polarized image, and the polarized light illumination operation includes: no illumination, polarized image acquisition mode Including: single-frame polarization image synchronous acquisition operation, polarization image processing operation includes: polarization image acquisition operation.

In combination with the fifth aspect above, in a possible implementation manner, the first feature information indicates that the ambient light intensity is greater than or equal to a preset value and indicates the acquisition of qualitative polarization information. The polarized light illumination operation includes: no illumination, and the polarized image acquisition method includes : Synchronous acquisition operation of single frame polarization image, polarization image processing operation includes: qualitative polarization information acquisition operation.

In combination with the fifth aspect above, in a possible implementation manner, the first characteristic information indicates that the ambient light intensity is greater than or equal to a preset value and indicates to obtain quantitative polarization information, and the polarized light illumination operation includes: asynchronous illumination of multiple polarized light sources, polarization The image acquisition method includes: multi-frame polarization image asynchronous acquisition operation, and the polarization image processing operation includes: quantitative polarization information acquisition operation.

It should be noted that, when the image processing device described in the fifth aspect and the lighting device described in the third aspect above are integrated with the image acquisition device described in the fourth aspect below, the lighting device and the image acquisition device, the image The processing devices may share at least one processor, and the at least one processor executes the processing operations in the lighting device, the image acquisition device, and the image processing device.

In a sixth aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium includes a computer program or an instruction, and when the computer program or instruction is run on a computer, the computer executes the computer according to the first aspect and the first aspect. The methods described in any one of the possible implementations of .

In a seventh aspect, the present application provides a computer program product containing instructions. When the computer program product is run on a computer, the computer executes the method described in the first aspect and any possible implementation of the first aspect. method.

It should be understood that descriptions of technical features, technical solutions, beneficial effects or similar language in this application do not imply that all features and advantages can be realized in any single embodiment. On the contrary, it can be understood that the description of features or beneficial effects means that specific technical features, technical solutions or beneficial effects are included in at least one embodiment. Therefore, descriptions of technical features, technical solutions or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in this embodiment may also be combined in any appropriate manner. Those skilled in the art will understand that the embodiments can be implemented without one or more specific technical features, technical solutions or advantageous effects of the specific embodiments. In other embodiments, additional technical features and beneficial effects may also be identified in certain embodiments that do not embody all embodiments.

Description of drawings

FIG. 1 is a functional block diagram of a vehicle provided in an embodiment of the present application;

FIG. 2 is a system architecture diagram of an image processing system provided by an embodiment of the present application;

Fig. 3 is a schematic diagram of a lighting device provided by an embodiment of the present application;

FIG. 4 is a schematic flow diagram of an image processing method provided in an embodiment of the present application;

Fig. 5 is a schematic flow chart of another image processing method provided by the embodiment of the present application;

Figures 6a-6d are schematic diagrams of a comparison of polarization images with different polarization directions provided by the embodiment of the present application;

Figure 7a and Figure 7b are schematic diagrams comparing a visible light image and a polarization degree image provided by the embodiment of the present application;

Figure 8a and Figure 8b are schematic diagrams comparing a visible light image and a polarization angle image provided by the embodiment of the present application;

FIG. 9 is a schematic structural diagram of a processing device provided in an embodiment of the present application;

FIG. 10 is a schematic diagram of a hardware structure of a processing device provided in an embodiment of the present application;

FIG. 11 is a schematic diagram of a hardware structure of another processing device provided by an embodiment of the present application.

Detailed ways

In the description of the present application, unless otherwise specified, "/" means "or", for example, A/B may mean A or B. The "and/or" in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B, which can mean: A exists alone, A and B exist at the same time, and B exists alone These three situations. In addition, "at least one" means one or more, and "plurality" means two or more. Words such as "first" and "second" do not limit the number and order of execution, and words such as "first" and "second" do not necessarily limit the difference.

It should be noted that, in this application, words such as "exemplary" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design described herein as "exemplary" or "for example" is not to be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner.

In order to make this application more clear, some concepts involved in this application will be briefly introduced below.

1. Polarization

Polarization refers to the vibration phenomenon that the transverse wave vibrates during its propagation and the vibration direction is perpendicular to the wave propagation direction. Polarization is a property of transverse waves, longitudinal waves are not polarized.

Since light is also a transverse wave, it will also be polarized during propagation. Items of different materials can reflect light of different polarization states after being irradiated by light (the polarization state can be polarization characteristics such as degree of polarization, polarization angle, etc., and in this application, the polarization state is taken as the degree of polarization as an example for illustration). And the polarization state of the reflected light after the same material is illuminated by light with different polarization states may also be different.

Based on the characteristic of different polarization states of reflected light from different objects, objects of different materials can be distinguished by detecting the polarization state of reflected light from objects. It can solve the problem that similar objects are difficult to identify caused by different objects with the same spectrum under visible light (a certain spectral range, different types of objects present the same spectral characteristics).

2. Quantitative polarization information

Quantitative polarization information refers to the polarization information of the object itself, which is a fixed polarization information that does not change with the change of the environment where the object is located. The parameter commonly used to characterize the quantitative polarization information of an object is the Mueller matrix. After the Mueller matrix of the object is determined, the material of the object can be determined by the look-up table method.

The look-up table method refers to pre-configuring objects of different materials and their corresponding Mueller matrices. In this way, after the Mueller matrix of the object is determined, the material of the object corresponding to the Mueller matrix can be queried according to the Mueller matrix.

3. Qualitative polarization information

Qualitative polarization information refers to the polarization information exhibited by an object in its current environment. Qualitative polarization information may change with changes in the environment of the object (light intensity, light irradiation direction, polarization information of the irradiated light, etc.).

Qualitative polarization information usually includes at least one of the following: Stokes vector (stokes vector), degree of polarization, or angle of polarization.

4. Pixel-level coated polarization sensor

After coating polarizers with different polarization directions on the sensor of the image acquisition device, it is possible to obtain polarization images modulated by different polarization angles, and after coating RGB filter layers, it is possible to obtain colored polarization images.

The following is an introduction to the working principle of the pixel-level coated polarization sensor:

In the image collected by the pixel-level coating sensor, the polarization states between every four (the number is not limited, and can be multiple) adjacent pixels are different. In one example, the polarization degrees of the four adjacent pixels are: 0°, 45°, 90°, 135°. In this way, after an image is captured by the image acquisition device, pixels with the same polarization state in the image are extracted, so that an image in one polarization state of the image can be obtained. Using the same method, the image acquisition device can obtain images of the 4 polarization states of the target.

The above is a brief introduction to some of the contents and concepts involved in this application.

Hereinafter, the application scenarios of the embodiments of the present application are briefly introduced.

Embodiments of the present application provide an image processing method and device, which are applied to a terminal device and used to determine polarization information of at least one target device.

A terminal device may also be referred to as user equipment (UE), terminal, access terminal, subscriber unit, subscriber station, mobile station, remote station, remote terminal, mobile device, user terminal, wireless communication device, user agent, or user device. The terminal device can be a vehicle to everything (V2X) device, for example, a smart car (smart car or intelligent car), a digital car (digital car), an unmanned car (unmanned car or driverless car or pilotless car or automobile), Automatic car (self-driving car or autonomous car), pure electric vehicle (pure EV or Battery EV), hybrid electric vehicle (hybrid electric vehicle, HEV), range extended electric vehicle (range extended EV, REEV), plug-in hybrid Power vehicle (plug-in HEV, PHEV), new energy vehicle (new energy vehicle), etc. The terminal device may also be a device to device (device to device, D2D) device. The terminal device can also be a mobile station (mobile station, MS), a subscriber unit (subscriber unit), a drone, an Internet of things (internet of things, IoT) device, a station (station, ST) in a WLAN, a cellular phone (cellular phone), smart phone (smart phone), cordless phone, wireless data card, tablet computer, session initiation protocol (session initiation protocol, SIP) phone, wireless local loop (wireless local loop, WLL) station, personal digital processing ( Personal digital assistant (PDA) equipment, laptop computer (laptop computer), machine type communication (machine type communication, MTC) terminal, handheld device with wireless communication function, computing device or other processing equipment connected to a wireless modem, vehicle devices, wearable devices (also known as wearable smart devices).

In the embodiment of the present application, the description is made by taking the terminal device as a vehicle as an example.

FIG. 1 is a functional block diagram of a vehicle 100 provided by an embodiment of the present application, and the vehicle 100 may be a smart vehicle. In one embodiment, the vehicle 100 determines a first lighting mode and a first acquisition mode according to the first feature information, provides illumination according to the first lighting mode, and collects a first polarized image of at least one target according to the first acquisition mode, The vehicle 100 processes the first polarization image according to the first image processing manner, and determines the polarization information of at least one target. The vehicle 100 determines the surrounding environment of the vehicle according to the polarization information of at least one target, so as to provide a basis for the automatic driving of the vehicle.

Vehicle 100 may include various subsystems such as travel system 110 , sensor system 120 , control system 130 , one or more peripheral devices 140 as well as power supply 150 , computer system 160 and user interface 170 . Alternatively, vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple elements. In addition, each subsystem and element of the vehicle 100 may be interconnected by wire or wirelessly.

Propulsion system 110 may include components that provide powered motion for vehicle 100 . In one embodiment, propulsion system 110 may include an engine 111 , a transmission 112 , an energy source 113 and wheels 114 . The engine 111 may be an internal combustion engine, an electric motor, an air compression engine or other types of engine combinations, such as a hybrid engine composed of a gasoline engine and an electric motor, or a hybrid engine composed of an internal combustion engine and an air compression engine. The engine 111 converts the energy source 113 into mechanical energy.

Examples of energy source 113 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electrical power. Energy source 113 may also provide energy to other systems of vehicle 100.

Transmission 112 may transmit mechanical power from engine 111 to wheels 114 . Transmission 112 may include a gearbox, a differential, and a drive shaft. In one embodiment, the transmission 112 may also include other devices, such as clutches. Among other things, drive shafts may include one or more axles that may be coupled to one or more wheels 114 .

The sensor system 120 may include several sensors that sense information about the environment around the vehicle 100 . For example, the sensor system 120 may include a positioning system 121 (the positioning system may be a global positioning system (global positioning system, GPS), or a Beidou system or other positioning systems), an inertial measurement unit (inertial measurement unit, IMU) 122, a radar 123 , a laser radar 124 and a camera 125 . The sensor system 120 may also include sensors that monitor the interior systems of the vehicle 100 (eg, an interior air quality monitor, fuel gauge, oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding properties (position, shape, orientation, velocity, etc.). Such detection and identification is a critical function for the safe operation of autonomous driving of the vehicle 100 .

The positioning system 121 may be used to estimate the geographic location of the vehicle 100 . The IMU 122 is used to sense changes in position and orientation of the vehicle 100 based on inertial acceleration. In one embodiment, IMU 122 may be a combination accelerometer and gyroscope.

The radar 123 may utilize radio signals to sense objects within the surrounding environment of the vehicle 100 . In some embodiments, in addition to sensing objects, radar 123 may be used to sense the velocity and/or heading of an object.

Lidar 124 may utilize laser light to sense objects in the environment in which vehicle 100 is located. In some embodiments, lidar 124 may include one or more laser sources, a laser scanner, and one or more detectors, among other system components.

Camera 125 may be used to capture multiple images of the surrounding environment of vehicle 100 , as well as multiple images within the vehicle cabin. Camera 125 may be a still camera or a video camera. In the embodiment of the present application, the camera 123 may be a polarization camera, which can acquire a polarization image of the target.

The control system 130 may control the operation of the vehicle 100 and its components. Control system 130 may include various elements including steering system 131 , accelerator 132 , braking unit 133 , computer vision system 134 , route control system 135 , and obstacle avoidance system 136 .

The steering system 131 is operable to adjust the heading of the vehicle 100 . For example in one embodiment it could be a steering wheel system.

The throttle 132 is used to control the operating speed of the engine 111 and thus the speed of the vehicle 100 .

The braking unit 133 is used to control the deceleration of the vehicle 100 . The braking unit 133 may use friction to slow the wheels 114 . In other embodiments, the brake unit 133 can convert the kinetic energy of the wheel 114 into electric current. The braking unit 133 may also take other forms to slow down the wheels 114 to control the speed of the vehicle 100 .

Computer vision system 134 is operable to process and analyze images captured by camera 125 to identify objects and/or features in the environment surrounding vehicle 100 as well as physical and facial features of a driver within the vehicle cabin. Objects and/or features may include traffic signals, road conditions, and obstacles, and driver's physical features and facial features include driver's behavior, gaze, expression, etc. The computer vision system 134 may use object recognition algorithms, structure from motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system 134 may be used to map the environment, track objects, estimate the speed of objects, determine driver behavior, face recognition, and the like.

The route control system 135 is used to determine the travel route of the vehicle 100 . In some embodiments, route control system 135 may combine data from sensors, positioning system 121 , and one or more predetermined maps to determine a travel route for vehicle 100 .

The obstacle avoidance system 136 is used to identify, evaluate and avoid or otherwise overcome potential obstacles in the environment of the vehicle 100 .

Of course, in an example, the control system 130 may add some components not shown above; or use other components to replace some of the components shown above; or may also reduce some of the components shown above.

Vehicle 100 interacts with external sensors, other vehicles, other computer systems, or users through peripherals 140 . Peripherals 140 may include wireless communication system 141 , on-board computer 142 , microphone 143 and/or speaker 144 .

In some embodiments, peripheral device 140 provides a means for a user of vehicle 100 to interact with user interface 170 . For example, on-board computer 142 may provide information to a user of vehicle 100 . The user interface 170 may also operate the on-board computer 142 to receive user input. The on-board computer 142 can be operated through a touch screen. In other cases, peripheral device 140 may provide a means for vehicle 100 to communicate with other devices located within the vehicle. For example, microphone 143 may receive audio (eg, voice commands or other audio input) from a user of vehicle 100 . Similarly, speaker 144 may output audio to a user of vehicle 100 .

The wireless communication system 141 may communicate wirelessly with one or more devices, either directly or via a communication network. For example, the wireless communication system 141 may use 3G cellular communications, such as CDMA, EVDO, GSM/GPRS, or 4G cellular communications, such as LTE, or 5G cellular communications. The wireless communication system 141 can use WiFi to communicate with a wireless local area network (wireless local area network, WLAN). In some embodiments, the wireless communication system 141 may communicate directly with the device using an infrared link, Bluetooth, or ZigBee. Wireless communication system 141 may also utilize other wireless protocols to communicate with devices. For example, various vehicle communication systems. The wireless communication system 141 may include one or more dedicated short range communications (DSRC) devices.

The power supply 150 may provide power to various components of the vehicle 100 . In one embodiment, the power source 150 may be a rechargeable lithium ion or lead acid battery. One or more packs of such batteries may be configured as a power source to provide power to various components of the vehicle 100 . In some embodiments, the power source 150 and the energy source 113 can be implemented together, such as a pure electric vehicle or a gasoline-electric hybrid vehicle among new energy vehicles.

Some or all functions of the vehicle 100 are controlled by the computer system 160 . Computer system 160 may include at least one processor 161 executing instructions 1621 stored in a non-transitory computer-readable medium such as data storage device 162 . Computer system 160 may also be a plurality of computing devices that control individual components or subsystems of vehicle 100 in a distributed manner.

Processor 161 may be any conventional processor, such as a commercially available central processing unit (central processing unit, CPU). Alternatively, the processor may be a dedicated device such as an application specific integrated circuit (ASIC) or other hardware-based processor. Although FIG. 1 functionally illustrates a processor, memory, and other elements in the same physical enclosure, those of ordinary skill in the art will appreciate that the processor, computer system, or memory may actually include Multiple processors, computer systems, or memories within a physical enclosure, or including multiple processors, computer systems, or memories that may not be stored within the same physical enclosure. For example, memory may be a hard drive, or other storage medium located in a different physical enclosure. Accordingly, a reference to a processor or a computer system will be understood to include references to a collection of processors or computer systems or memories that may operate in parallel, or a collection of processors or computer systems or memory that may not operate in parallel. Rather than using a single processor to perform the steps described herein, some components, such as the steering and deceleration components, may each have their own processor that only performs calculations related to component-specific functions.

In various aspects described herein, the processor may be located in a device remote from the vehicle and in wireless communication with the vehicle. In other aspects, some of the processes described herein are executed on a processor disposed within the vehicle while others are executed by a remote processor, including taking the necessary steps to perform a single maneuver.

In some embodiments, the data storage device 162 may contain instructions 1621 (eg, program logic) executable by the processor 161 to perform various functions of the vehicle 100 , including all or part of the functions described above. Data storage 162 may also contain additional instructions, including sending data to, receiving data from, interacting with, and/or performing operations on, one or more of travel system 110 , sensor system 120 , control system 130 , and peripherals 140 . control instructions.

In addition to instructions 1621, data storage device 162 may also store data such as road maps, route information, the vehicle's position, direction, speed, and other such vehicle data, among other information. Such information may be used by the vehicle 100 and the computer system 160 during operation of the vehicle 100 in autonomous, semi-autonomous, and/or manual modes.

For example, in a possible embodiment, the data storage device 162 can acquire obstacle information in the surrounding environment obtained by the vehicle based on the sensors in the sensor system 120, such as other vehicles, road edges, and obstacles such as green belts. Information such as the position, the distance between the obstacle and the vehicle, and the distance between obstacles. The data storage device 162 can also obtain environmental information from the sensor system 120 or other components of the vehicle 100. The environmental information can be, for example, whether there are green belts, lanes, pedestrians, etc. near the environment where the vehicle is currently located, or the vehicle calculates the current location through a machine learning algorithm. Whether there are green belts, pedestrians, etc. near the environment. In addition to the above content, the data storage device 162 can also store the state information of the vehicle itself and the state information of other vehicles interacting with the vehicle, wherein the state information of the vehicle includes but not limited to the position, speed, acceleration, heading angle etc. In this way, the processor 161 can obtain these information from the data storage device 162, and determine the passable area of the vehicle based on the environmental information of the environment where the vehicle is located, the state information of the vehicle itself, the state information of other vehicles, etc., and based on the passable area A final driving strategy is determined to control the automatic driving of the vehicle 100 .

A user interface 170 for providing information to or receiving information from a user of the vehicle 100 . Optionally, user interface 170 may interact with one or more input/output devices within set of peripheral devices 140 , such as one or more of wireless communication system 141 , onboard computer 142 , microphone 143 , and speaker 144 .

Computer system 160 may control vehicle 100 based on information obtained from various subsystems (eg, travel system 110 , sensor system 120 , and control system 130 ) and information received from user interface 170 . For example, computer system 160 may, based on information from control system 130 , control steering system 131 to change the vehicle's heading to avoid obstacles detected by sensor system 120 and obstacle avoidance system 136 . In some embodiments, computer system 160 may control many aspects of vehicle 100 and its subsystems.

Optionally, one or more of these components described above may be installed separately from or associated with the vehicle 100 . For example, data storage device 162 may exist partially or completely separate from vehicle 100 . The above components may be coupled together in a wired and/or wireless manner for communication.

Optionally, the above-mentioned components are just an example. In practical applications, components in the above-mentioned modules may be added or deleted according to actual needs. FIG. 1 should not be construed as limiting the embodiment of the present application.

An autonomous vehicle traveling on a road, such as the vehicle 100 above, can determine an adjustment instruction for the current speed according to other vehicles in its surrounding environment. Wherein, the objects in the surrounding environment of the vehicle 100 may be other types of objects such as traffic control equipment or green belts. In some examples, each object within the surrounding environment may be considered independently and the speed adjustment command for vehicle 100 may be determined based on the object's respective characteristics, such as its current speed, acceleration, distance to the vehicle, and the like.

Optionally, the vehicle 100 as an autonomous vehicle or its associated computer equipment (such as the computer system 160, computer vision system 134, and data storage device 162 of FIG. 1 ) can obtain the state of the surrounding environment based on the identified measurement data (for example, traffic, rain, ice on the road, etc.), and determine the relative position of the obstacle and the vehicle in the surrounding environment at the current moment. Optionally, the boundary of the passable area formed by each obstacle depends on each other. Therefore, all the acquired measurement data can also be used to determine the boundary of the passable area of the vehicle together, and the actual impassable area in the passable area is removed. passable area. The vehicle 100 is able to adjust its driving strategy based on the detected traversable area of the vehicle. In other words, the self-driving car can determine what steady state the vehicle needs to adjust to (eg, accelerate, decelerate, turn, or stop, etc.) based on the detected traversable area of the vehicle. During this process, other factors may also be considered to determine the speed adjustment command of the vehicle 100 , such as the lateral position of the vehicle 100 on the driving road, the curvature of the road, the proximity of static and dynamic objects, and so on.

In addition to providing instructions to adjust the speed of the self-driving car, the computing device may also provide instructions to modify the steering angle of the vehicle 100 such that the self-driving car follows a given trajectory and/or maintains a distance between the self-driving car and nearby objects (e.g. Cars in adjacent lanes) safe lateral and longitudinal distances.

The above-mentioned vehicles 100 may be cars, trucks, motorcycles, buses, boats, airplanes, helicopters, lawn mowers, recreational vehicles, playground vehicles, construction equipment, trams, golf carts, trains, and trolleys, etc. The application examples are not particularly limited.

In other embodiments of the present application, the self-driving vehicle may also include a hardware structure and/or a software module, and realize the above-mentioned functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether one of the above-mentioned functions is executed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application and design constraints of the technical solution.

In yet another possible implementation, as shown in FIG. 2, the image processing method provided in the embodiment of the present application can be applied to the image processing system 100 shown in FIG. 2. As shown in FIG. 2, the image processing system 100 It includes: an image processing device 10 , an illumination device 20 , and an image acquisition device 30 .

Wherein, the image processing device 10 is configured to control and manage the image processing method provided in the embodiment of the present application, and process the acquired polarization image to obtain polarization information of the polarization image.

The illuminating device 20 is used for providing polarized light illumination for the environment. The illuminating device 20 includes one or more polarized light sources, and different polarized light sources provide polarized light with the same or different polarization states. In the embodiment of the present application, the illuminating device 20 includes at least two polarized light sources, and the at least two polarized light sources can provide polarized light of more than two polarization states, so as to meet the requirement of the image processing device to acquire quantitative polarization information of the target.

As an example, as shown in FIG. 3 , the illuminating device 20 includes four polarized light sources, and the four polarized light sources respectively provide polarized light of four different polarization states.

The light source of the lighting device may be a light emitting diode (light emitting diode, LED), or a laser.

The method for providing polarized light by the lighting device includes: using a polarized light source to directly emit polarized light, using a polarizer to filter a non-polarized light source to obtain polarized light, using a wave plate to filter a non-polarized light source to obtain polarized light, using a polarizer and a wave The plates jointly filter the unpolarized light source to obtain polarized light.

Specifically, the above four polarized light sources are: 0° polarized light source 201 , 45° polarized light source 202 , 90° polarized light source 203 , and 135° polarized light source 204 .

It should be pointed out that the polarized light of different polarization states provided by the above-mentioned polarized light source is only an exemplary illustration. In a specific implementation, the lighting device 20 may include any number of polarized light sources, and each polarized light source may provide polarized light of any polarized state. This application does not limit this. For example, the four polarized light sources in the illuminating device 20 may also be: 0° polarized light, 45° polarized light, 90° polarized light, and left-handed polarized light (or right-handed polarized light).

The image acquisition device 30 is configured to acquire polarization images. The image acquisition device 30 may be an image acquisition device including the above-mentioned pixel-level coated polarization sensor. In the embodiment of the present application, the image acquisition device 30 can acquire polarization images of more than three polarization states of the target. For example, the image acquisition device 30 acquires polarization images of four polarization states of the target.

It should be pointed out that in the embodiment of the present application, the image processing device 10 , the image acquisition device 20 , and the lighting device 30 can be integrated into the same device, for example, integrated into mobile phones, vehicles and other devices. If integrated, the three may share at least one processor to perform corresponding processing operations, or may also set respective processors to perform corresponding processing operations. As an example, in an intelligent driving scenario, the image acquisition device 30 may include the camera 125 described in the above-mentioned vehicle 100 . The lighting device 20 may include a lighting device added to the vehicle 100 . The image processing device 10 is the computer system 160 described in the above-mentioned vehicle 100 ; or, more specifically, the image processing device 10 may be the processor 161 described in the above-mentioned vehicle 100 .

In addition, the image processing device 10, the image acquisition device 20, and the lighting device 30 may be separately arranged in different devices. For example, in an intelligent driving scenario, the image acquisition device 30 is the camera 125 described in the above-mentioned vehicle 100 . The lighting device 20 is a new lighting device in the vehicle 100 . The image processing device 10 is a remote processing device, for example, a road side unit (RSU) set on the roadside; another example is a mobile edge computing (MEC) device set at the far end . In the case that the image processing device 10 , the image acquisition device 20 , and the lighting device 30 are arranged separately, respective processors may be provided among the three devices to perform corresponding processing operations. At this time, the image acquisition device 20 may be a smart camera, or a similar image acquisition device with a processing function. The lighting device 30 may be an intelligent lighting device, or a similar lighting device with a processing function.

In the embodiment of the present application, the image processing device 10 is an MEC disposed at a remote end, and the lighting device and the image acquisition device are disposed in a vehicle as an example for illustration.

In the current autonomous driving scenario, the vehicle needs to use the visible light camera to detect the light intensity, shape, color and other information of each target in the surrounding environment of the vehicle. In autonomous driving scenarios, objects refer to various objects that exist around the vehicle. For example, other vehicles, obstacles, lane markings, road signs, signal lights and other objects that require vehicle recognition to assist in automatic driving.

However, there are many limitations in the application of visible light cameras in autonomous driving scenarios. For example, in a scene with too much ambient light, the target is prone to glare, making it difficult to distinguish the target from other surrounding objects.

In a scene where the ambient light is too weak, the reflected light intensity of the target and surrounding objects is weak (dark light), which may also cause the target to be difficult to distinguish from other surrounding objects.

In addition, the target and its surrounding objects may also have the problem of different objects having the same spectrum. Even if the ambient light intensity is normal, the reflective characteristics of the target and its surrounding objects are similar, making it difficult to distinguish the target from other surrounding objects.

In order to solve the problems of glare, dark light, and foreign matter in the same spectrum that exist in visible light cameras. Currently, a polarized light camera is proposed. The polarized light camera can not only obtain the grayscale information and color information of the image collected by the visible light camera, but also collect the polarization information of the target. Because different objects usually have different polarization information. Therefore, the polarized light camera can well solve the problem that the target is difficult to distinguish from other surrounding objects in the scene of glare, dark light and foreign objects with the same spectrum.

For example, by determining the polarization information of the target object when the intensity of visible light is strong or weak, the target can be identified under strong light and low light conditions. When the color of the target is similar to that of other objects, the polarized light that characterizes the material of the object can be used to distinguish the target with similar color and different material.

Currently, in the field of autonomous driving, the method of using polarization cameras to determine objects around the vehicle is mainly the method of passive polarization imaging. That is to say, the polarization camera collects images of the surrounding environment under the current ambient light and performs polarization processing on these images to generate a polarization image, and the polarization camera sends the polarization image to the image processing device. After receiving the image, the image processing device processes the polarization image to extract polarization information of the polarization image. Generally speaking, the image processing device can use a trained neural network model to process the polarization image to obtain the polarization information of the polarization image; or, it can also use an image fusion method to determine the polarization information of the polarization image.

However, when using a polarization camera to determine the polarization information of a target, there are still the following problems:

1. The polarization image has a variety of polarization information, such as polarization modulated polarization image, Stokes vector, polarization angle, polarization degree, Mueller matrix. Different polarization information can represent different characteristics of the target, and the processing methods of different polarization information obtained from polarization images are different. When currently determining the polarization information of the polarization image, the image processing device directly obtains all the polarization information that can be obtained from the polarization image, which will increase the calculation amount of the image processing device.

2. The polarization camera can only collect the polarization image of the target under ambient light. Based on this, the image processing device can only determine the qualitative polarization information of the target according to the polarization image. Qualitative polarization information can only be used to distinguish an object from other objects in the scene it belongs to, not to determine the object's material.

3. The polarization image collected by the polarization camera has energy loss after being filtered by the polarizer, and the imaging effect is poor in dark light scenes.

In order to solve the above-mentioned technical problems, an embodiment of the present application provides an image processing method. The image processing device determines the first feature information that can characterize the scene characteristics and obtain the polarization information indicating the target. The image processing device determines the target according to the first feature information. The scene in which it is located provides polarized illumination, acquires the polarization image of the target, and processes the polarization image of the target to determine the polarization information of the target.

Based on this, the image processing device can select a corresponding image processing method according to the acquisition of the polarization information of the target, and obtain the polarization information of the target, avoiding the problem of large calculation amount of the image processing device caused by obtaining the full amount of polarization information of the target.

In addition, the image processing device can determine the Mueller matrix of the target according to the polarized light provided for the target and the collected polarized light reflected by the target, and then determine the quantitative polarization information of the target. In addition, the image processing device can also provide illumination for the target, which improves the imaging clarity of the target in dark and light scenes.

It should be pointed out that the image processing method provided by the embodiment of the present application can be applied not only to the automatic driving scene, but also to other scenes that need to obtain the polarization image of the target, such as various scenes applied to image recognition (such as human Face recognition, telemedicine, etc.), this application does not limit this.

The image processing method provided in the embodiment of the present application may be applied in scenarios such as an automatic driving scene, an intelligent driving scene, or an advanced driving assistance system (advanced driving assistance system, ADAS), which is not limited in this application.

As shown in FIG. 4 , the image processing method provided by the embodiment of the present application includes the following S400-S403. The following is a specific description of performing S400-S403:

S400. The image processing apparatus acquires first feature information.

The first feature information is used to characterize scene features and to indicate acquisition of polarization information of at least one target. The first characteristic information corresponds to the first illumination mode, the first collection mode and the first image processing mode.

In a possible implementation manner, the scene feature is a scene feature of an environment where at least one target is located. For example, the scene feature may be the ambient light intensity of the scene where the at least one target is located.

Acquisition of the polarization information of at least one target refers to which polarization information of the plurality of polarization information of the target is the polarization information of the at least one target to be acquired.

It should be noted that the first characteristic information may be the characteristic information generated after the image processing device acquires the scene characteristics and the polarization information of at least one target; it may also be the first characteristic information generated by other devices and sent to the image processing device feature information, which is not limited in this application.

S401. The image processing device determines a first illumination mode, a first collection mode, and a first image processing mode according to the first feature information.

Wherein, the first illumination mode is used to characterize: an illumination mode that provides polarized light illumination for at least one target.

The first acquisition mode is used to characterize: the acquisition mode of collecting the polarization image of at least one target illuminated according to the first illumination mode.

The first image processing manner is used to characterize: an image processing manner for processing the polarization image of at least one target collected according to the first acquisition manner.

In a possible implementation manner, the image processing device is pre-configured with correspondences between the first feature information and the first illumination manner, the first collection manner, and the first image processing manner. After the image processing device acquires the first feature information, the image processing device determines the corresponding first lighting mode, first collection mode, and first image processing mode according to the corresponding relationship.

S402. The image processing device controls polarized illumination according to the first illumination mode, and, according to the first collection mode, controls to collect a first polarization image of at least one target under the first illumination mode.

It should be noted that the image processing device controlling the polarized illumination may be that the image processing device controls itself to perform illumination, or that the image processing device controls the illumination device to perform illumination, which is not limited in this application.

Similarly, the image processing device controlling the acquisition of the first polarization image may be that the image acquisition control itself acquires the first polarization image of at least one target under the first illumination mode, or the image processing device controls the image acquisition device to acquire at least one target in the first illumination mode. A first polarized image under an illumination mode. This application does not limit this.

S403. The image processing device processes the first polarization image according to the first image processing manner, and acquires polarization information of at least one target.

Based on the above technical solution, in the image processing method provided by the embodiment of the present application, the image processing device can determine the corresponding image processing method according to the acquisition of the polarization information of the target, and acquire the polarization information of the target, avoiding the problem caused by the acquisition of the full amount of polarization information of the target. The image processing device has a large amount of calculation.

In the following, the first characteristic information, the first illumination mode, the first acquisition mode, and the first image processing mode involved in the embodiment of the present application are exemplarily described:

a) The scene features in the first feature information include light intensity information of the environment. The polarization information in the first feature information includes at least one of the following: a second polarization image, qualitative polarization information, or quantitative polarization information.

b) The first lighting mode belongs to a set of polarized lighting modes, and the set of polarized lighting modes includes at least one of the following: synchronous lighting with multiple polarized light sources, asynchronous lighting with multiple polarized light sources, or no lighting.

Wherein, the multi-polarization light source synchronous illumination is: providing polarized illumination through multiple polarized light sources with different polarization states simultaneously. The asynchronous illumination of multi-polarized light sources is: sequentially provide polarized illumination through multiple polarized light sources with different polarization states. No lighting is: No lighting is provided.

It should be pointed out that, in the embodiment of the present application, synchronous illumination of multiple polarized light sources may specifically be implemented as: all of the multiple polarized light sources provide illumination at the same time, or some of the multiple polarized light sources provide illumination at the same time. This application does not limit this.

c) The first collection method belongs to a collection of polarization collection methods, and the collection of polarization collection methods includes: a synchronous collection operation of a single-frame polarization image, and an asynchronous collection operation of a multi-frame polarization image.

The synchronous acquisition operation of a single-frame polarization image includes: acquiring a plurality of polarization images of different polarization states from a single image by acquiring a single image of at least one target.

The asynchronous acquisition operation of multiple frames of polarization images includes: acquiring multiple images of at least one target, each of the multiple images including multiple polarization images of different polarization states.

d) The first image processing method belongs to a polarization image processing method set, and the polarization image processing method set includes: a polarization image acquisition operation, a qualitative polarization information acquisition operation, and a quantitative polarization information acquisition operation.

Wherein, the polarization image acquiring operation is used to acquire a second polarization image, and the second polarization image is a polarization image of the first polarization image in a different polarization state.

The operation of acquiring qualitative polarization information is used to acquire at least one item of the following polarization information of at least one target: Stokes vector, degree of polarization, or angle of polarization.

The quantitative polarization information acquisition operation is used to acquire the Mueller matrix of at least one target.

Based on this, in the case of different ambient light intensities and different polarization information acquisition conditions, the image processing device can determine to provide a corresponding illumination mode for the target, and collect a polarization image of the target under the corresponding illumination mode. The image processing device can obtain the polarization information required by the target by processing the polarization images of the target collected under different illumination modes.

In a possible implementation manner, referring to FIG. 4 , as shown in FIG. 5 , the foregoing S400 may specifically be implemented through the following S400a to S400c.

S400a. The image processing device determines the ambient light intensity.

In a possible implementation manner, the ambient light intensity of the environment is detected by the ambient light detection module. When the image processing device needs to determine the first feature information, the image processing device sends an ambient light intensity detection instruction to the ambient light detection module. The ambient light detection module detects the current ambient light intensity of the environment where at least one target is located according to the indication. The ambient light detection module sends the ambient light intensity to the image processing device after determining the ambient light intensity of the environment where the at least one target is currently located. Correspondingly, the image processing device receives the ambient light intensity from the ambient light detection module.

In yet another possible implementation manner, the image information of the environment where at least one target is located is collected by a visible light image collection device. When the image processing device needs to determine the first feature information, the image processing device sends a visible light image collection instruction to the visible light image collection device. The visible light image acquisition device acquires a visible light image of the environment where at least one target is currently located according to the visible light image acquisition instruction. The visible light image acquisition device sends the visible light image of the current environment where at least one target is located to the image processing device. Correspondingly, the image processing device receives the visible light image from the visible light image acquisition device. The image acquisition device processes the visible light image to determine the ambient light intensity.

It should be pointed out that the above is only an example of how the image processing device determines the ambient light intensity. During specific implementation, the image processing device may also obtain the ambient light intensity through other methods, which is not limited in this application.

S400b. The image processing device determines the polarization information of the target to be acquired.

In a possible implementation manner, the image processing device is preset with the polarization information to be acquired corresponding to the target in each scene. After the image processing device determines the scene for acquiring the polarization information, the image processing device may determine the polarization information to be acquired according to the scene.

In the following, the image processing device determines to obtain polarization information in an automatic driving scene as an example (the following scenes may also be intelligent driving scenes, which are not specifically limited in this application, and are only illustrated here):

When the scene is lane line detection or driver detection task, the image processing device determines that the polarization information to be acquired of the target is the second polarization image.

When the scene is detecting surrounding vehicles, the image processing device determines that the polarization information to be acquired of the target is qualitative polarization information.

When the scene is metal manhole cover detection, the image processing device determines that the polarization information to be acquired of the target is quantitative polarization information.

It should be pointed out that the present application does not limit the sequence of S400a and S400b. For example, the image processing apparatus may first execute S400a and then execute S400b; or, the image processing apparatus may first execute S400b and then execute S400a; or, the image processing apparatus may execute S400a and S400b simultaneously. This application does not limit this.

S400c. The image processing device generates first feature information according to the ambient light intensity and the polarization information of the target to be acquired.

Based on this, the image processing device can generate corresponding first feature information according to the acquired ambient light intensity and current scene information.

In a possible implementation manner, referring to FIG. 4 , as shown in FIG. 5 , the above S402 may be specifically implemented through the following S402a-S402g. Next, describe S402a-S402g in detail:

S402a. The image processing apparatus generates a first instruction according to the first lighting mode.

Wherein, the first instruction is used to indicate the first lighting mode. Specifically, the first instruction is used to indicate which lighting method in the polarized light lighting method set the first lighting method is.

For example, the first instruction is used to indicate that the first lighting mode is multi-polarized light source synchronous lighting; or, the first instruction is used to indicate that the first lighting mode is multi-polarized light source asynchronous lighting; or, the first instruction is used to indicate the first The lighting mode is no lighting.

S402b. The image processing device sends a first instruction to the lighting device. Correspondingly, the lighting device receives the first instruction from the image processing device.

S402c. The lighting device controls a plurality of polarized light sources with different polarization states to provide lighting according to the first instruction.

In a possible implementation manner, the lighting device has at least three lighting modes, which are:

Lighting mode 1, the lighting device controls all or part of the light sources in multiple polarized light sources to be turned on at the same time; lighting mode 2, the lighting device controls each polarized light source in the multiple polarized light sources to turn on in turn; lighting mode 3, the lighting device controls multiple polarized light sources All polarized light sources in the system are all turned off.

Wherein, when the first instruction indicates that the first lighting mode is multi-polarized light source synchronous lighting, the lighting device adopts lighting mode 1 to provide lighting for the target.

In the case where the first instruction is used to indicate that the first lighting mode is multi-polarized light source asynchronous lighting, the lighting device uses lighting mode 2 to provide lighting for the target.

When the first instruction is used to indicate that the first lighting mode is no lighting, the lighting device uses lighting mode 3 to provide lighting for the target.

That is to say, when the first instruction is used to indicate that the first lighting mode is synchronous lighting of multiple polarized light sources, the lighting device controls all or some of the multiple polarized light sources to be turned on simultaneously, and all of the multiple polarized light sources Or part of the light source provides illumination for the target at the same time.

When the first instruction is used to indicate that the first lighting mode is multi-polarized light source asynchronous lighting, the lighting device controls each of the multiple polarized light sources to turn on sequentially, and the multiple polarized light sources sequentially provide illumination for the target.

When the first instruction is used to indicate that the first lighting mode is no lighting, the lighting device controls each of the multiple polarized light sources to be turned off, and none of the multiple polarized light sources provides illumination for the target.

As an example, referring to Fig. 3, in lighting mode 1, the lighting device can control four polarized light sources with different polarization states to illuminate simultaneously; or, the lighting device can control two polarized light sources whose polarization degrees differ by 90° to illuminate simultaneously; for example, Control the 0° polarized light source and the 90° polarized light source to illuminate at the same time, or control the 45° polarized light source and the 135° polarized light source to illuminate at the same time.

In lighting mode 2, the lighting device controls the 0° polarized light source to illuminate in the first time period, controls the 45° polarized light source to illuminate in the second time period, and controls the 90° polarized light source to illuminate in the third time period, Control the 135° polarized light source to illuminate in the fourth time period. The second time period is a time period after the first time period, the third time period is a time period after the second time period, and the fourth time period is a time period after the third time period.

S402d. The image processing apparatus generates a second instruction according to the first collection manner.

Wherein, the second instruction is used to indicate the first collection mode. Specifically, the second instruction is used to indicate which image acquisition mode in the polarization acquisition mode set the first acquisition mode is.

For example, the second instruction is used to indicate that the first acquisition mode is a synchronous acquisition operation of a single-frame polarization image; or, the second instruction is used to indicate that the first acquisition mode is an asynchronous acquisition operation of multiple-frame polarization images.

It should be noted that, in the embodiment of the present application, the image processing apparatus executes S402d while executing any one of steps S402a-S402c. Alternatively, the image processing apparatus may execute S402d before or after executing any one of steps S402a-S402c, which is not limited in this application. For example, the image processing apparatus may first execute S400a and then execute S400d; or, the image processing apparatus may first execute S400d and then execute S400a; or, the image processing apparatus may execute S400a and S400d simultaneously. This application does not limit this. For the relationship between S402d and S402a and S402b, reference may be made to the relationship between S400d and S400a, which will not be repeated in this application.

S402e. The image processing device sends a second instruction to the image acquisition device. Correspondingly, the image acquisition device receives the second instruction from the image processing device.

It should be pointed out that S402e is a step performed after S402d, and the relationship between S402e and S402a-S402c can refer to the above-mentioned relationship between S402d and S402a-S402c, which will not be repeated here.

S402f. The image acquisition device acquires the first polarization image according to the second instruction.

In a possible implementation manner, the image acquisition device has at least two image acquisition modes, which are:

Image collection method 1. The image collection device collects a single image of at least one target, and obtains multiple polarization images of different polarization states from the single image. Image collection mode 2. The image collection device collects multiple images of at least one target, and each of the multiple images includes multiple polarization images of different polarization states.

Wherein, when the second instruction is used to indicate that the first acquisition mode is a synchronous acquisition operation of a single-frame polarization image, the image acquisition device adopts image acquisition mode 1 to acquire the polarization image of at least one target.

In the case where the second instruction is used to indicate that the first acquisition mode is an asynchronous acquisition operation of multi-frame polarization images, the image acquisition device adopts image acquisition mode 2 to acquire the polarization image of at least one target.

That is to say, when the second instruction is used to indicate that the first acquisition method is a single-frame polarization image synchronous acquisition operation, the image acquisition device acquires a single image of at least one target; the image acquisition device acquires from the single image Multiple polarization images of different polarization states.

In the case where the second instruction is used to indicate that the first acquisition method is an asynchronous acquisition operation of multiple frames of polarization images, by acquiring multiple images of at least one target, each of the multiple images includes multiple polarizations of different polarization states image.

It should be noted that, in the embodiment of the present application, the image acquisition device acquires a polarization image of at least one target when it is determined that the illumination device provides illumination.

For example, when the illumination device provides illumination, the illumination effect lasts for a period of time, and the image acquisition device acquires a polarized image of at least one target during the illumination duration.

In a possible implementation manner, when the lighting device provides lighting for at least one target using lighting mode 1, the lighting device sends trigger information to the image acquisition device, where the trigger information is used to trigger the image acquisition device to acquire images. In response to the trigger information, the image acquisition device acquires a polarization image of at least one target.

When the lighting device provides illumination for at least one target in the lighting mode 2, and the image acquisition device uses the image acquisition mode 2 to capture the polarization image of the at least one target. Whenever the lighting device uses a polarized light source to provide illumination for at least one target, the lighting device sends trigger information to the image acquisition device to trigger the image acquisition device to collect a polarized image of at least one target under the illumination of the light source. In this way, the polarization image of the target under the illumination of each light source can be sequentially collected by the image acquisition device.

S402g. The image acquisition device sends the first polarization image to the image processing device. Correspondingly, the image processing device receives the first polarization image from the image acquisition device.

Based on the above technical solution, the image processing device can acquire the first polarization image of at least one target according to the first illumination mode and the first acquisition mode.

In a possible implementation manner of S403, in the case that the first image processing manner includes: a polarization image acquisition operation, S403 may be implemented in the manner described in the following manner 1.

In the case that the first image processing manner includes: obtaining qualitative polarization information, S403 may be implemented in the manner described in the following manner 2.

In the case where the first image processing manner includes: the quantitative polarization information acquisition operation, S403 may be implemented in the manner described in the following manner 3.

In the following, mode 1, mode 2 and mode 3 are described in detail respectively:

Mode 1. The image processing device determines the second polarization image from multiple first polarization images of the target.

Specifically, the image processing device determines multiple first polarization images of the target, and the polarization states of the multiple first polarization images are different.

The image processing device determines that a polarization image whose imaging effect satisfies a preset condition among the plurality of first polarization images is the second polarization image.

Wherein, the imaging effect meeting the preset condition may be represented by at least one of the following: the sharpness of the object in the image meets the preset condition, and the contrast between the object in the image and other objects meets the preset condition. Alternatively, the imaging effect meeting the preset condition may also be expressed in other ways, which is not limited in this application.

As an example, in a glare scene, the vehicle needs to determine the traffic marking lines on the road ahead.

As shown in FIG. 6 a , it is the first polarization image of the 0° polarization direction collected by the image acquisition device.

As shown in FIG. 6 b , it is the first polarization image with a polarization direction of 45° collected by the image collection device.

As shown in FIG. 6 c , it is the first polarization image of the 90° polarization direction collected by the image acquisition device.

As shown in FIG. 6d , it is the first polarization image with a polarization direction of 135° collected by the image collection device.

In FIGS. 6a-6d, due to the influence of glare, the polarized image collected by the image collection device may not accurately display the traffic marking lines on the road.

For example, in Fig. 6a-Fig. 6d, the vehicle needs to recognize the traffic marking line of the road ahead, but affected by glare, the left-turn sign located in the frame line as shown in Fig. 6a, Fig. 6b, and Fig. 6d may not be recognized. correctly identified.

After the image processing device receives the first polarization images shown in Figures 6a-6d, after image processing, it determines that the first polarization image with the highest contrast among the four first polarization images is the first polarization image shown in Figure 6c. polarized image. At this time, the image processing device determines that the first polarization image shown in FIG. 6c is the target second polarization image, and the image processing device outputs the second polarization image.

Mode 2. The image processing device determines the qualitative polarization information of the target according to the multiple first polarization images of the target.

Wherein, the qualitative polarization information of the target at least includes: the Stokes vector of the target, the degree of polarization of the target, and the polarization angle of the target.

Specifically, the image processing device acquires the first polarization image of the 0° polarization direction of at least one target, the first polarization image of the 45° polarization direction, the first polarization image of the 90° polarization direction, and the first polarization image of the 135° polarization direction. image.

The Stokes vector of the target determined by the image acquisition device satisfies the following formula 1:

Wherein, S is the Stokes vector of the target, and S ₀ , S ₁ , S ₂ , S ₃ are all values in the Stokes vector S. I ₀ represents the light intensity of polarized light reflected by the target under the condition of 0° polarized light source illumination, I ₄₅ represents the light intensity of polarized light reflected by the target under the condition of 45° polarized light source illumination, I ₉₀ represents the light intensity of polarized light reflected by the target under the condition of 90° polarized light source Intensity of polarized light reflected under lighting conditions, I ₁₃₅ means the light intensity of polarized light reflected by the target under the lighting condition of 135° polarized light source, _Ir means the light intensity of polarized light reflected by the target under the lighting condition of right-handed light , I _l represents the light intensity of the polarized light reflected by the target under left-handed light illumination conditions.

When plane polarized light propagates along the optical axis in some anisotropic media, its vibration direction will continuously rotate as the propagation distance increases. Right-handed light means that the vibration surface rotates clockwise when viewed against the direction of light propagation; left-handed light means that the vibration surface rotates counterclockwise when viewed against the direction of light propagation.

After determining the Stokes vector of the target, the image acquisition device determines that the degree of polarization of the target satisfies the following formula 2.1:

Wherein, DoLP is the degree of polarization of the target, S ₀ , S ₁ , S ₂ , and S ₃ are the values in the Stokes vector determined in Formula 1 above.

It should be pointed out that in the automatic driving scene, the light intensity I _l of the polarized light reflected by the target under the left-handed light illumination condition, and the light intensity I _r of the polarized light reflected by the target under the right-handed light illumination condition are 0, At this time, the value of S ₃ is 0.

In this case, the image acquisition device determines that the degree of polarization of the target satisfies the following formula 2.2:

Degree-of-polarization images are often used to increase the contrast between targets of different reflectivities and the background.

One example, the target is a vehicle in the image. The visible light image of the target acquired by the image processing device is shown in Figure 7a, and the polarization degree image of the target acquired by the image processing device is shown in Figure 7b.

According to the comparison of Fig. 7a and Fig. 7b, it can be seen that the polarization degree image of the target output by the image processing device can greatly improve the contrast between the target and the background, which is more conducive to the image processing device to detect the target in the image.

The polarization angle of the target determined by the image acquisition device satisfies the following formula 3:

Wherein, AoLP is the polarization angle of the target, S ₁ and S ₂ are the values in the Stokes vector determined in the above formula 1.

Polarization images are often used to increase the contrast between targets with different surface orientations and different surface roughness.

An example where the targets are tarmac and dirt road pavement in the image. The visible light image of the target acquired by the image processing device is shown in Figure 8a, and the polarization degree image of the target acquired by the image processing device is shown in Figure 8b.

According to the comparison of Fig. 8a and Fig. 8b, it can be seen that the contrast between the asphalt road and the dirt road in the polarization angle image of the target output by the image processing device is greatly improved, which is more conducive to the material segmentation of the road by the vehicle.

Way 3: The image processing device determines the quantitative polarization information of the target according to the multiple first polarization images of the target.

Wherein, the quantitative polarization information of the target at least includes the Mueller matrix of the target.

The Mueller matrix of the target determined by the image processing device satisfies the following formula 4

S’＝M×S Formula 4

Among them, S' is the input Stokes vector, S is the output Stokes vector, and M is the Muller matrix.

In the embodiment of the present application, S is the Stokes vector of the target determined by the image processing device, that is, the Stokes vector determined according to Formula 1 above.

S' is determined according to the degree of polarization of the polarized light sequentially provided by the lighting device in lighting mode 2.

Expressed in matrix form, Equation 4 can be expressed by the following Equation 5:

In the above formula 5, S ₀ ', S ₁ ', S ₂ ', S ₃ ' in S' satisfy the following formula 6:

Among them, S ₀ ′, S ₁ ′, S ₂ ′, and S ₃ ′ are all values in the Stokes vector S′. I ₀ ' represents the light intensity of polarized light provided by the 0° polarized light source of the lighting device, I ₄₅ ' represents the light intensity of the polarized light provided by the 45° polarized light source of the lighting device, and I ₉₀ ' represents the 90° polarized light source provided by the lighting device The light intensity of the polarized light, I ₁₃₅ ′ represents the light intensity of the polarized light provided by the 135° polarized light source of the lighting device.

In the above formula 5, the Muller matrix M satisfies the following formula 7

Specifically, the process for the image processing device to determine the above formula 5 is:

Based on formula 1 and formula 6, according to the relationship between the incident light of the target (that is, the polarized light provided by the lighting device) and the reflected light (that is, the reflected light of the target collected by the image acquisition device), the image processing device determines S ₀ , S ₁ , S ₂ , S ₃ and S ₀ ', S ₁ ', S ₂ ', S ₃ ' satisfy the following formula 8:

Based on formula 8, formula 8 can be further transformed to get the above formula 5

Based on the above technical solution, the image processing device can process the first polarization image of the target by using different processing methods in different situations, and determine the corresponding polarization parameters of the target. Meet the requirements of polarization parameters that the vehicle needs to obtain in different driving scenarios.

In a possible implementation manner, the first feature information includes the following scenarios 1-6. In the following, the above scenarios 1-6 are described in detail respectively:

Scenario 1. The first feature information indicates that the ambient light intensity is less than a preset value and indicates to obtain a second polarization image.

That is to say, in scene 1, the first characteristic information is used to indicate: in a scene where the ambient light of the target is weak, acquire the second polarization image of at least one target.

The second polarization image refers to a polarization image whose imaging effect (for example, sharpness, contrast) satisfies a preset condition among multiple polarization images of the target.

It should be pointed out that a scene with weak ambient light may also be called a dark light scene.

In one example, the second polarization image may be a polarization image with the highest contrast among multiple polarization images of the target.

Correspondingly, in this scenario, the first lighting operation determined in conjunction with the above S401, the image processing device includes: synchronous lighting of multiple polarized light sources. The first collection method includes: a synchronous collection operation of a single-frame polarization image. The first image processing method includes: a second polarization image acquisition operation.

That is to say, in a scene with weak ambient light, in order to obtain the second polarized image of at least one target, the image acquisition device determines to use multiple polarized light sources to provide illumination for the scene to which the target belongs at the same time, and the image acquisition device , collecting a single image of the scene to which the target belongs, and determining a plurality of first polarization images of the scene to which the target belongs with different polarization states from the single image. The image acquisition device determines a second polarization image from the multiple first polarization images of different polarization states, and outputs the second polarization image.

In a possible implementation manner, in conjunction with the foregoing S402a-S402f, in S402a, the first instruction generated by the image processing device is used to indicate that the first polarization illumination mode is synchronous illumination of multiple polarization light sources.

In S402c, the lighting device uses lighting mode 1 to provide lighting for the target.

In S402d, the second instruction generated by the image processing device is used to indicate that the first acquisition mode is a synchronous acquisition operation of a single-frame polarization image.

In S402f, the image acquisition device adopts the image acquisition mode 1 to acquire the polarization image of at least one target.

In conjunction with the above S403, in S403, the image processing device processes the acquired first polarization image by using a polarization image acquisition operation to determine a second polarization image. The image processing device outputs the second polarization image.

Based on the above technical solution, when the second polarization image of at least one target needs to be obtained in a scene with weak ambient light, the lighting device provides synchronous illumination of multi-polarized light sources for at least one target, which can increase the light intensity of the scene to which at least one target belongs, The problem of poor imaging effect of the polarization image collected by at least one target in a dark light scene is avoided.

The image acquisition device only acquires a single image of at least one target, which can reduce the number of images that the image acquisition device needs to acquire, thereby reducing the calculation amount of the image acquisition device.

The image processing device determines the second polarization image according to the multiple first polarization images of the single image. Likewise, the number of images to be processed by the image processing device can be reduced, thereby reducing the calculation amount of the image processing device.

Scenario 2. The first feature information indicates that the ambient light intensity is less than a preset value and indicates to acquire qualitative polarization information.

That is to say, in scene 2, the first feature information is used to indicate: in a scene where the target is in a weak ambient light, acquire qualitative polarization information of at least one target.

In this scenario, the first illumination operation determined by the image processing device includes: synchronous illumination of multiple polarization light sources, the first collection method includes: synchronous collection operation of a single-frame polarization image, and the first image processing method includes: qualitative polarization information acquisition operation.

That is to say, in the scene where the ambient light of the target is weak, in order to obtain the qualitative polarization information of at least one target, the image acquisition device determines to use multiple polarized light sources to provide illumination for the scene to which the target belongs at the same time, and the image acquisition device determines After illumination, a single image of the scene to which the target belongs is collected, and multiple first polarization images of different polarization states of the scene to which the target belongs are determined from the single image. The image acquisition device performs a qualitative polarization information acquisition operation according to the plurality of first polarization images of different polarization states, and determines the qualitative polarization information of at least one target.

In conjunction with the above S403, in S403, the image processing device processes the acquired first polarization image by using a qualitative polarization information acquisition operation to determine the qualitative polarization information of at least one target. The image processing device outputs qualitative polarization information of at least one object.

The image processing device determines qualitative polarization information of at least one target according to the plurality of first polarization images of the single image. Likewise, the number of images to be processed by the image processing device can be reduced, thereby reducing the calculation amount of the image processing device.

Scenario 3. The first feature information indicates that the ambient light intensity is less than a preset value and indicates to obtain quantitative polarization information.

That is to say, in scene 3, the first feature information is used to indicate: in a scene where the target is in a weak ambient light, obtain quantitative polarization information of at least one target.

Correspondingly, in this scenario, the first lighting operation determined by the image processing device includes: multi-polarized light source asynchronous lighting, the first acquisition method includes: multi-frame polarization image asynchronous acquisition operation, and the first image processing method includes: quantitative polarization Information acquisition operation.

That is to say, in a scene with weak ambient light, in order to obtain quantitative polarization information of at least one target, the image acquisition device determines to use multiple polarized light sources to sequentially provide illumination for the scene to which the target belongs, and the image acquisition device determines that after the illumination , collecting multiple images of the scene to which the target belongs, and determining multiple first polarization images of the scene to which the target belongs with different polarization states from each of the multiple images. The image acquisition device determines the Mueller matrix of at least one target from the plurality of first polarization images of different polarization states, the image acquisition device determines the quantitative polarization information of at least one target according to the Mueller matrix of at least one target, and the image processing device Quantitative polarization information for at least one target is output.

In a possible implementation manner, in conjunction with the foregoing S402a-S402f, in S402a, the first instruction generated by the image processing device is used to indicate that the first polarization illumination mode is asynchronous illumination with multiple polarization light sources.

In S402c, the lighting device uses lighting mode 2 to provide lighting for the target.

In S402d, the second instruction generated by the image processing device is used to indicate that the first acquisition mode is an asynchronous acquisition operation of multi-frame polarization images.

In S402f, the image acquisition device adopts the image acquisition mode 2 to acquire the polarization image of at least one target.

In conjunction with the above S403, in S403, the image processing device processes the acquired first polarization image by using a quantitative polarization information acquisition operation to determine quantitative polarization information of at least one target. The image processing device outputs quantitative polarization information for at least one object.

Based on the above technical solution, when the quantitative polarization information of at least one target needs to be acquired in a scene with weak ambient light, the lighting device provides asynchronous illumination of multi-polarized light sources for at least one target, so that the image acquisition device can capture at least one target The polarized image formed by reflection under different polarized light sources, based on the polarized image formed by the reflection of at least one target under different polarized light sources, and the polarized light information provided by different polarized light sources, the image processing device can determine the Mueller matrix of at least one target, and then The image processing device may determine quantitative polarization information of at least one object based on the Mueller matrix of at least one object.

Scenario 4. The first characteristic information indicates that the ambient light intensity is greater than or equal to a preset value and indicates to acquire a second polarization image.

That is to say, in scene 4, the first feature information is used to indicate that: in a scene where the target is in a strong ambient light, acquire a second polarization image of at least one target.

Correspondingly, in this scenario, the first illumination operation determined by the image processing device includes: no illumination, the first acquisition method includes: synchronous acquisition operation of a single-frame polarization image, and the first image processing method includes: second polarization image acquisition operation.

That is to say, when the second polarization image of at least one target is acquired in a scene with strong ambient light, the image acquisition device determines that there is no need to use a light source to provide illumination for the scene to which the target belongs, and the image acquisition device directly collects the image of the scene to which the target belongs. a single image, and determine multiple first polarization images of different polarization states of the scene to which the target belongs from the single image. The image acquisition device determines a second polarization image from the multiple first polarization images of different polarization states, and outputs the second polarization image.

In a possible implementation manner, in conjunction with the foregoing S402a-S402f, in S402a, the first instruction generated by the image processing device is used to indicate that the first polarization illumination mode is no illumination.

In S402c, the lighting device uses lighting mode 3 to provide lighting for the target.

Based on the above technical solution, when the second polarization image of at least one target needs to be acquired in a scene with strong ambient light, the lighting device does not need to provide illumination for at least one target. Therefore, the power consumption of the lighting device is reduced.

Scenario 5. The first feature information indicates that the ambient light intensity is greater than or equal to a preset value and indicates to acquire qualitative polarization information.

That is to say, in scene 5, the first characteristic information is used to indicate: in a scene where the target is in a strong ambient light, to acquire qualitative polarization information of at least one target.

Correspondingly, in this scenario, the first illumination operation determined by the image processing device includes: no illumination, the first collection method includes: synchronous acquisition operation of a single-frame polarization image, and the first image processing method includes: qualitative polarization information acquisition operation.

That is to say, when the qualitative polarization information of at least one target is obtained in a scene with strong ambient light, the image acquisition device determines that there is no need to use a light source to provide illumination for the scene to which the target belongs, and the image acquisition device directly collects the unit of the scene to which the target belongs. From this single image, multiple first polarization images of different polarization states of the scene to which the target belongs are determined. The image acquisition device determines a second polarization image from the multiple first polarization images of different polarization states, and outputs the second polarization image.

Scenario 6. The first characteristic information indicates that the ambient light intensity is greater than or equal to a preset value and indicates to obtain quantitative polarization information.

That is to say, in scene 6, the first feature information is used to indicate: in a scene where the target is in a strong ambient light, to acquire quantitative polarization information of at least one target.

That is to say, in a scene with strong ambient light, in order to obtain quantitative polarization information of at least one target, the image acquisition device determines to use multiple polarized light sources to sequentially provide illumination for the scene to which the target belongs, and the image acquisition device determines that after the illumination , collecting multiple images of the scene to which the target belongs, and determining multiple first polarization images of the scene to which the target belongs with different polarization states from each of the multiple images. The image acquisition device determines the Mueller matrix of at least one target from the plurality of first polarization images of different polarization states, the image acquisition device determines the quantitative polarization information of at least one target according to the Mueller matrix of at least one target, and the image processing device Quantitative polarization information for at least one target is output.

In a possible implementation manner, in conjunction with the above S402a-S402f, in S402a, the first instruction generated by the image processing device is used to indicate that the first polarization illumination mode is asynchronous illumination with multiple polarization light sources.

Based on the above technical solution, when the quantitative polarization information of at least one target needs to be acquired in a scene with strong ambient light, the lighting device provides asynchronous illumination of multi-polarized light sources for at least one target, so that the image acquisition device can capture at least one target The polarized image formed by reflection under different polarized light sources, based on the polarized image formed by the reflection of at least one target under different polarized light sources, and the polarized light information provided by different polarized light sources, the image processing device can determine the Mueller matrix of at least one target, and then The image processing device may determine quantitative polarization information of at least one object based on the Mueller matrix of at least one object.

It should be noted that the embodiment of the present application does not limit the wavelength band of the polarized light source provided by the lighting device and the wavelength band of the target polarized image collected by the image acquisition device.

For example, the lighting device can provide various polarized light sources under the current visible light conditions; or, the lighting device can provide polarized light sources in the infrared (infrared radiation, IR) band; or, the lighting device can also provide polarized light sources in other wave bands. This application does not limit this.

Similarly, when the image acquisition device acquires the polarization image of the target, it can acquire the visible light polarization image of the target. Alternatively, the image acquisition device may also acquire polarization images of the target in the IR band; or, the image acquisition device may also acquire polarization images of the target in other bands. This application does not limit this.

It should be pointed out that when the image acquisition device collects polarization images of different bands, it can be realized by adding channel coatings of corresponding bands to the image acquisition device.

For example, when the image acquisition device collects the polarization image of the target in the IR band, the polarization image of the target in the IR band can be acquired by adding an IR channel coating to the image acquisition device.

It should be noted that, in the above-mentioned embodiments, the image processing method is mainly applied to vehicles. In the specific implementation process, the image processing method can also be applied to other devices with lighting, image acquisition, and image processing capabilities. , such as mobile phones, tablet computers, cameras with image processing capabilities, etc. This application does not limit this.

In addition, in this application, the image processing device, the lighting device, and the image acquisition device are separately provided as an example for illustration. In a specific implementation process, the image processing device, the lighting device, and the image acquisition device can also be integrated into the same device. In this case, the implementation of the image processing method in the embodiment of the present application is similar to the implementation described in the foregoing embodiments, which will not be described in detail in the present application.

The solutions in the above embodiments of the present application can be combined under the premise of no contradiction.

The foregoing mainly introduces the solutions of the embodiments of the present application from the perspectives of internal implementation of devices and interaction between devices. It can be understood that each device, for example, an image processing device, an image acquisition device, and an illumination device includes at least one of corresponding hardware structures and software modules for performing each function in order to realize the above-mentioned functions. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of the examples described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software drives hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

In the embodiment of the present application, the functional units of the image processing device, the image acquisition device and the lighting device can be divided according to the above method examples. For example, each functional unit can be divided corresponding to each function, or two or more functions can be integrated into one in a processing unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic, and is only a logical function division, and there may be another division manner in actual implementation.

In the case of using an integrated unit, FIG. 9 shows a possible structural diagram of a processing device (referred to as a processing device 90) involved in the above-mentioned embodiment, and the processing device 90 includes a processing unit 901 and a communication unit 902. , may also include a storage unit 903 . The structural diagram shown in FIG. 9 can be used to illustrate the structures of the image processing device, the image acquisition device and the lighting device involved in the above embodiments.

When the structural diagram shown in FIG. 9 is used to illustrate the structure of the image processing device involved in the above-mentioned embodiments, the processing unit 901 is used to control and manage the actions of the image processing device, for example, to control the image processing device to execute the S400 to S403, S400a to S400c, S401, S402a, S402b, S402d, S402e, S402g, S403 in FIG. 5, and/or actions performed by the image processing apparatus in other processes described in the embodiments of this application. The processing unit 901 can communicate with other device entities through the communication unit 902 , for example, communicate with the image acquisition device and the lighting device shown in FIG. 2 . The storage unit 903 is used to store program codes and data of the image processing device.

When the structural diagram shown in FIG. 9 is used to illustrate the structure of the image processing device involved in the above embodiments, the processing device 90 may be an image processing device or a chip in the image processing device.

When the structural schematic diagram shown in FIG. 9 is used to illustrate the structure of the image acquisition device involved in the above embodiment, the processing unit 901 is used to control and manage the actions of the image acquisition device, for example, to control the image acquisition device to execute the S402e to S402g, and/or actions performed by the image acquisition device in other processes described in the embodiments of this application. The processing unit 901 can communicate with other device entities through the communication unit 902 , for example, communicate with the image processing device and the lighting device shown in FIG. 2 . The storage unit 903 is used for storing program codes and data of the image acquisition device.

When the structural diagram shown in FIG. 9 is used to illustrate the structure of the image acquisition device involved in the above embodiments, the processing device 90 may be an image acquisition device, or a chip in the image acquisition device.

When the structural schematic diagram shown in FIG. 9 is used to illustrate the structure of the lighting device involved in the above embodiment, the processing unit 901 is used to control and manage the actions of the lighting device, for example, to control the lighting device to execute S402b and S402c, and/or actions performed by the lighting device in other processes described in the embodiments of this application. The processing unit 901 can communicate with other device entities through the communication unit 902 , for example, communicate with the image processing device and the image acquisition device shown in FIG. 2 . The storage unit 903 is used for storing program codes and data of the lighting device.

When the structural schematic diagram shown in FIG. 9 is used to illustrate the structure of the lighting device involved in the above embodiment, the processing device 90 may be a lighting device, or a chip in the lighting device.

Wherein, when the processing device 90 is an image processing device, an image acquisition device or a lighting device, the processing unit 901 may be a processor or a controller, and the communication unit 902 may be a communication interface, a transceiver, a transceiver, a transceiver circuit, a transceiver device, etc. . Wherein, the communication interface is a general term, and may include one or more interfaces. The storage unit 903 may be a memory. When the processing device 90 is a chip in an image processing device, an image acquisition device or a lighting device, the processing unit 901 may be a processor or a controller, and the communication unit 902 may be an input interface and/or output interface, a pin or a circuit, etc. The storage unit 903 may be a storage unit (for example, a register, a cache, etc.) in the chip, or a storage unit outside the chip (for example, a read-only memory (read-only memory) -only memory, referred to as ROM), random access memory (random access memory, referred to as RAM), etc.).

Wherein, the communication unit may also be referred to as a transceiver unit. The antenna and control circuit with transceiver function in the processing device 90 can be regarded as the communication unit 902 of the processing device 90 , and the processor with processing function can be regarded as the processing unit 901 of the processing device 90 . Optionally, the device in the communication unit 902 for implementing the receiving function may be regarded as a receiving unit, and the receiving unit is used to perform the receiving step in the embodiment of the present application, and the receiving unit may be a receiver, a receiver, a receiving circuit, and the like.

If the integrated units in FIG. 9 are realized in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage In the medium, several instructions are included to make a computer device (which may be a personal computer, server, or network device, etc.) or a processor (processor) execute all or part of the steps of the methods described in the various embodiments of the present application. Storage media for storing computer software products include: U disk, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk, and other media that can store program codes.

Units in FIG. 9 may also be referred to as modules, for example, a processing unit may be referred to as a processing module.

The embodiment of the present application also provides a schematic diagram of the hardware structure of a processing device (referred to as the processing device 100), referring to Figure 10 or Figure 11, the processing device 100 includes a processor 1001, and optionally, also includes memory 1002. The processor 1001 includes at least one processor.

In a first possible implementation manner, referring to FIG. 10 , the processing device 100 further includes a communication interface 1003 . The processor 1001, the memory 1002 and the communication interface 1003 are connected through a bus. The communication interface 1003 is used for communicating with other devices or communication networks. Optionally, the communication interface 1003 may include a transmitter and a receiver. The device in the communication interface 1003 for implementing the receiving function may be regarded as a receiver, and the receiver is configured to perform the receiving step in the embodiment of the present application. The device in the communication interface 1003 for implementing the sending function may be regarded as a transmitter, and the transmitter is used to perform the sending step in the embodiment of the present application.

Based on the first possible implementation manner, the schematic structural diagram shown in FIG. 10 may be used to illustrate the structure of the image processing device, the image acquisition device or the lighting device involved in the above embodiments.

When the structural schematic diagram shown in FIG. 10 is used to illustrate the structure of the image processing device involved in the above embodiments, the processor 1001 is used to control and manage the actions of the image processing device, for example, the processor 1001 is used to support image processing The device executes S400 to S403 in FIG. 4, S400a to S400c, S401, S402a, S402b, S402d, S402e, S402g, S403 in FIG. 5, and/or an image processing device in other processes described in the embodiments of this application The action performed. The processor 1001 can communicate with other device entities through the communication interface 1003 , for example, communicate with the image acquisition device and the lighting device shown in FIG. 2 . The memory 1002 is used to store program codes and data of the image processing device.

When the schematic structural diagram shown in FIG. 10 is used to illustrate the structure of the image acquisition device involved in the above embodiments, the processor 1001 is used to control and manage the actions of the image acquisition device, for example, the processor 1001 is used to support image acquisition The device executes S402e to S402g in FIG. 5 , and/or actions performed by the image acquisition device in other processes described in the embodiments of this application. The processor 1001 can communicate with other device entities through the communication interface 1003 , for example, communicate with the image processing device and the lighting device shown in FIG. 2 . The memory 1002 is used to store program codes and data of the image acquisition device. It should be noted that when the processing device 100 is used to illustrate the structure of the image acquisition device involved in the above embodiments, the processing device 100 also includes a polarization image collector for collecting a polarization image of the target.

When the structural diagram shown in FIG. 10 is used to illustrate the structure of the lighting device involved in the above-mentioned embodiments, the processor 1001 is used to control and manage the actions of the lighting device, for example, the processor 1001 is used to support the lighting device to execute the diagram S402b and S402c in 5, and/or actions performed by the lighting device in other processes described in the embodiments of this application. The processor 1001 can communicate with other device entities through the communication interface 1003 , for example, communicate with the image processing device and the image acquisition device shown in FIG. 2 . The memory 1002 is used to store program codes and data of the lighting device. It should be noted that when the processing device 100 is used to illustrate the structure of the illumination device involved in the above-mentioned embodiments, the processing device 100 also includes a plurality of light sources in polarization states, which are used to provide polarized illumination for the target.

In a second possible implementation manner, the processor 1001 includes a logic circuit and at least one of an input interface and an output interface. Wherein, the output interface is used to perform the sending action in the corresponding method, and the input interface is used to perform the receiving action in the corresponding method.

Based on the second possible implementation manner, referring to FIG. 11 , the schematic structural diagram shown in FIG. 11 may be used to illustrate the structure of the image processing device, image acquisition device or lighting device involved in the above embodiments.

When the structural diagram shown in FIG. 11 is used to illustrate the structure of the image processing device involved in the above-mentioned embodiments, the processor 1001 is used to control and manage the actions of the image processing device, for example, the processor 1001 is used to support image processing The device executes S400 to S403 in FIG. 4, S400a to S400c, S401, S402a, S402b, S402d, S402e, S402g, S403 in FIG. 5, and/or an image processing device in other processes described in the embodiments of this application The action performed. The processor 1001 may communicate with other device entities through at least one of an input interface and an output interface, for example, communicate with an image acquisition device and a lighting device shown in FIG. 2 . The memory 1002 is used to store program codes and data of the image processing device.

When the schematic structural diagram shown in FIG. 11 is used to illustrate the structure of the image acquisition device involved in the above-mentioned embodiments, the processor 1001 is used to control and manage the actions of the image acquisition device, for example, the processor 1001 is used to support image acquisition The device executes S402e to S402g in FIG. 5 , and/or actions performed by the image acquisition device in other processes described in the embodiments of this application. The processor 1001 may communicate with other device entities through at least one of an input interface and an output interface, for example, communicate with an image processing device and a lighting device shown in FIG. 2 . The memory 1002 is used to store program codes and data of the image acquisition device.

When the structural diagram shown in FIG. 11 is used to illustrate the structure of the lighting device involved in the above-mentioned embodiments, the processor 1001 is used to control and manage the actions of the lighting device, for example, the processor 1001 is used to support the lighting device to execute the diagram S402b and S402c in 5, and/or actions performed by the lighting device in other processes described in the embodiments of this application. The processor 1001 may communicate with other device entities through at least one of an input interface and an output interface, for example, communicate with an image processing device and an image acquisition device shown in FIG. 2 . The memory 1002 is used to store program codes and data of the lighting device.

Wherein, FIG. 10 and FIG. 11 may also schematically illustrate the system chip in the image processing device. In this case, the actions performed by the above-mentioned image processing device may be implemented by the system chip, and the specific actions performed may refer to the above, and will not be repeated here. Fig. 10 and Fig. 11 may also schematically illustrate the system chip in the image acquisition device. In this case, the above-mentioned actions performed by the image acquisition device may be implemented by the system chip, and the specific actions performed may refer to the above, and will not be repeated here. Fig. 10 and Fig. 11 may also illustrate a system-on-a-chip in a lighting device. In this case, the actions performed by the above lighting device may be implemented by the system chip, and the specific actions performed may refer to the above, and will not be repeated here.

In the implementation process, each step in the method provided by this embodiment may be implemented by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the methods disclosed in connection with the embodiments of the present application may be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

The processor in the present application may include but not limited to at least one of the following: a central processing unit (central processing unit, CPU), a microprocessor, a digital signal processor (DSP), a microcontroller (microcontroller unit, MCU), or Various types of computing devices that run software, such as artificial intelligence processors, each of which may include one or more cores for executing software instructions for calculation or processing. The processor can be a separate semiconductor chip, or can be integrated with other circuits into a semiconductor chip, for example, can form a SoC (on-chip chip) with other circuits (such as codec circuits, hardware acceleration circuits, or various bus and interface circuits). system), or may also be integrated in the ASIC as a built-in processor of the ASIC, and the ASIC integrated with the processor may be packaged separately or together with other circuits. In addition to including a core for executing software instructions for calculation or processing, the processor can further include necessary hardware accelerators, such as field programmable gate array (field programmable gate array, FPGA), PLD (programmable logic device) , or a logic circuit that implements a dedicated logic operation.

The memory in the embodiment of the present application may include at least one of the following types: read-only memory (read-only memory, ROM) or other types of static storage devices that can store static information and instructions, random access memory (random access memory) , RAM) or other types of dynamic storage devices that can store information and instructions, or electrically erasable programmable read-only memory (Electrically erasable programmable read-only memory, EEPROM). In some scenarios, the memory can also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.) , disk storage medium or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

An embodiment of the present application also provides a computer-readable storage medium, including instructions, which, when run on a computer, cause the computer to execute any one of the above methods.

The embodiment of the present application also provides a computer program product containing instructions, which, when run on a computer, causes the computer to execute any one of the above methods.

An embodiment of the present application also provides an image processing system, including: the above-mentioned image processing device, an image acquisition device, and a lighting device.

The embodiment of the present application also provides a chip, the chip includes a processor and an interface circuit, the interface circuit is coupled to the processor, the processor is used to run computer programs or instructions to implement the above method, and the interface circuit is used to communicate with other modules outside the chip to communicate.

In the above-mentioned embodiments, it may be fully or partially implemented by software, hardware, firmware or any combination thereof. When implemented using a software program, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the embodiments of the present application will be generated in whole or in part. A computer can be a general purpose computer, special purpose computer, computer network, or other programmable device. Computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, e.g. Coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that can be accessed by a computer, or may contain one or more data storage devices such as servers and data centers that can be integrated with the medium. The available medium may be a magnetic medium (for example, a floppy disk, a hard disk, or a magnetic tape), an optical medium (for example, DVD), or a semiconductor medium (for example, a solid state disk (solid state disk, SSD for short)) and the like.

Finally, it should be noted that: the above is only a specific implementation of the application, but the scope of protection of the application is not limited thereto, and any changes or replacements within the technical scope disclosed in the application shall be covered by this application. within the scope of the application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims

An image processing method, characterized in that, comprising:

Acquiring first feature information; the first feature information is used to characterize scene features and indicate the acquisition of polarization information of at least one target, the first feature information corresponds to the first lighting mode, the first acquisition mode and the first image processing Way;

Control polarized light illumination according to the first illumination mode, and, according to the first acquisition mode, control and collect a first polarized image of the at least one target under the first illumination mode;

According to the first image processing manner, the first polarization image is processed to acquire polarization information of the at least one target.
The method according to claim 1, characterized in that, after said obtaining the first characteristic information, said method further comprises:

According to the first characteristic information, the first lighting mode, the first collection mode and the first image processing mode are determined.
The method according to claim 1 or 2, characterized in that the scene features include light intensity information of the environment.
The method according to any one of claims 1-3, wherein the polarization information includes at least one of the following: a second polarization image, qualitative polarization information, or quantitative polarization information.
The method according to any one of claims 1-4, wherein the first lighting method belongs to a set of polarized light lighting methods, and the set of polarized light lighting methods includes at least one of the following: synchronous lighting of multiple polarized light sources, Non-synchronous illumination of multi-polarized light sources, or no illumination;

Wherein, the multi-polarized light source synchronous lighting is: providing polarized lighting simultaneously through a plurality of polarized light sources with different polarization states;

The asynchronous illumination of the multi-polarized light sources is: sequentially providing polarized illumination through a plurality of polarized light sources with different polarization states;

The no lighting means: no lighting is provided.
The method according to any one of claims 1-5, wherein the first acquisition mode belongs to a set of polarization acquisition modes, and the set of polarization acquisition modes includes: synchronous acquisition operation of a single-frame polarization image, multi-frame polarization image Asynchronous acquisition operation;

The synchronous acquisition operation of a single-frame polarization image includes: acquiring a plurality of polarization images of different polarization states from the single image by acquiring a single image of the at least one target;

The asynchronous acquisition operation of multiple frames of polarization images includes: acquiring multiple images of the at least one target, each of the multiple images including multiple polarization images of different polarization states.
The method according to any one of claims 1-6, wherein the first image processing method belongs to a polarization image processing method set, and the polarization image processing method set includes: polarization image acquisition operation, qualitative polarization information acquisition Operation, quantitative polarization information acquisition operation;

Wherein, the polarization image acquisition operation is used to acquire the second polarization image, and the second polarization image is a polarization image of the first polarization image in a different polarization state;

The qualitative polarization information acquisition operation is used to acquire at least one of the following polarization information of the at least one target: Stokes vector, polarization degree, or polarization angle;

The quantitative polarization information obtaining operation is used to obtain a Mueller matrix of the at least one target.
The method according to any one of claims 1-7, wherein the first feature information indicates that the ambient light intensity is less than a preset value and indicates to obtain the second polarization image, and the first lighting operation includes: Multiple polarization light sources are illuminated synchronously, the polarization image acquisition method includes: a single frame polarization image synchronous acquisition operation, and the polarization image processing operation includes: polarization image acquisition operation.
The method according to any one of claims 1-7, wherein the first characteristic information indicates that the ambient light intensity is less than a preset value and indicates to obtain qualitative polarization information, and the polarized light illumination operation includes: a multi-polarized light source For synchronous lighting, the polarization image acquisition method includes: a single-frame polarization image synchronous acquisition operation, and the polarization image processing operation includes: qualitative polarization information acquisition operation.
The method according to any one of claims 1-7, wherein the first characteristic information indicates that the ambient light intensity is less than a preset value and indicates to obtain qualitative polarization information, and the polarized light illumination operation includes: a multi-polarized light source For asynchronous illumination, the polarization image acquisition method includes: asynchronous acquisition operation of multi-frame polarization images, and the polarization image processing operation includes: quantitative polarization information acquisition operation.
The method according to any one of claims 1-7, wherein the first feature information indicates that the ambient light intensity is greater than or equal to a preset value and indicates that the second polarized image is acquired, and the polarized light illumination operation Including: no illumination, the polarization image acquisition method includes: a single-frame polarization image synchronous acquisition operation, and the polarization image processing operation includes: polarization image acquisition operation.
The method according to any one of claims 1-7, wherein the first characteristic information indicates that the ambient light intensity is greater than or equal to a preset value and indicates to obtain qualitative polarization information, and the polarized light illumination operation includes: none For illumination, the polarized image acquisition method includes: a single-frame polarized image synchronous acquisition operation, and the polarized image processing operation includes: qualitative polarization information acquisition operation.
The method according to any one of claims 1-7, wherein the first feature information indicates that the ambient light intensity is greater than or equal to a preset value and indicates to obtain quantitative polarization information, and the polarized light illumination operation includes: The polarized light source is asynchronously illuminated, and the polarization image acquisition method includes: asynchronous acquisition operation of multi-frame polarization images, and the polarization image processing operation includes: quantitative polarization information acquisition operation.
An image processing device, characterized by comprising: a processing unit and an acquisition unit;

The acquiring unit is configured to acquire first feature information; the first feature information is used to characterize scene features and indicate the acquisition of polarization information of at least one target, the first feature information corresponds to the first lighting mode, the first A collection method and a first image processing method;

The processing unit is configured to control polarized light illumination according to the first illumination mode, and, according to the first acquisition mode, control to collect a first polarized image of the at least one target under the first illumination mode;

The processing unit is further configured to process the first polarization image according to the first image processing manner to obtain polarization information of the at least one target.
The device according to claim 14, wherein the processing unit is further configured to:

According to the first characteristic information, the first lighting mode, the first collection mode and the first image processing mode are determined.
The device according to claim 15, wherein the processing unit is specifically used for:

generating a first instruction and a second instruction; the first instruction is used to indicate the first lighting mode; the second instruction is used to indicate the first collection mode;

Instructing the acquisition unit to send the first instruction to the lighting device, and to send the second instruction to the image acquisition device.
The device according to claim 16, wherein the processing unit is further configured to:

Instructing the acquisition unit to receive the first polarization image from the image acquisition device.
The device according to any one of claims 14-17, wherein the scene feature includes light intensity information of the environment.
The device according to any one of claims 14-18, wherein the polarization information includes at least one of the following: a second polarization image, qualitative polarization information, or quantitative polarization information.
The device according to any one of claims 14-19, wherein the first lighting method belongs to a set of polarized light lighting methods, and the set of polarized light lighting methods includes at least one of the following: synchronous lighting of multiple polarized light sources, Non-synchronous illumination of multi-polarized light sources, or no illumination;

Wherein, the multi-polarized light source synchronous lighting is: providing polarized lighting simultaneously through a plurality of polarized light sources with different polarization states;

The asynchronous illumination of the multi-polarized light sources is: sequentially providing polarized illumination through a plurality of polarized light sources with different polarization states;

The no lighting means: no lighting is provided.
The device according to any one of claims 14-20, wherein the first acquisition mode belongs to a set of polarization acquisition modes, and the set of polarization acquisition modes includes: synchronous acquisition operation of a single-frame polarization image, multi-frame polarization image Asynchronous acquisition operation;

The synchronous acquisition operation of a single-frame polarization image includes: acquiring a plurality of polarization images of different polarization states from the single image by acquiring a single image of the at least one target;

The operation of asynchronously collecting multiple frames of polarization images includes: collecting multiple images of the at least one target, and each of the multiple images includes multiple polarization images of different polarization states.
The device according to any one of claims 14-21, wherein the first image processing method belongs to a set of polarization image processing methods, and the set of polarization image processing methods includes: polarization image acquisition operation, qualitative polarization information acquisition Operation, quantitative polarization information acquisition operation;

Wherein, the polarization image acquisition operation is used to acquire the second polarization image, and the second polarization image is a polarization image of the first polarization image in a different polarization state;

The qualitative polarization information acquisition operation is used to acquire at least one of the following polarization information of the at least one target: Stokes vector, polarization degree, or polarization angle;

The quantitative polarization information obtaining operation is used to obtain a Mueller matrix of the at least one target.
The device according to any one of claims 14-22, wherein the first feature information indicates that the ambient light intensity is less than a preset value and indicates to obtain the second polarization image, and the first lighting operation includes: Multiple polarization light sources are illuminated synchronously, the polarization image acquisition method includes: a single frame polarization image synchronous acquisition operation, and the polarization image processing operation includes: polarization image acquisition operation.
The device according to any one of claims 14-22, wherein the first characteristic information indicates that the ambient light intensity is less than a preset value and indicates to obtain qualitative polarization information, and the polarized light illumination operation includes: a multi-polarized light source For synchronous lighting, the polarization image acquisition method includes: a single-frame polarization image synchronous acquisition operation, and the polarization image processing operation includes: qualitative polarization information acquisition operation.
The device according to any one of claims 14-22, wherein the first characteristic information indicates that the ambient light intensity is less than a preset value and indicates to obtain qualitative polarization information, and the polarized light illumination operation includes: a multi-polarized light source For asynchronous illumination, the polarization image acquisition method includes: asynchronous acquisition operation of multi-frame polarization images, and the polarization image processing operation includes: quantitative polarization information acquisition operation.
The device according to any one of claims 14-22, wherein the first feature information indicates that the ambient light intensity is greater than or equal to a preset value and indicates that the second polarized image is acquired, and the polarized light illumination operation Including: no illumination, the polarization image acquisition method includes: a single-frame polarization image synchronous acquisition operation, and the polarization image processing operation includes: polarization image acquisition operation.
The device according to any one of claims 14-22, wherein the first feature information indicates that the ambient light intensity is greater than or equal to a preset value and indicates to obtain qualitative polarization information, and the polarized light illumination operation includes: none For illumination, the polarized image acquisition method includes: a single-frame polarized image synchronous acquisition operation, and the polarized image processing operation includes: qualitative polarization information acquisition operation.
The device according to any one of claims 14-22, wherein the first feature information indicates that the ambient light intensity is greater than or equal to a preset value and indicates to obtain quantitative polarization information, and the polarized light illumination operation includes: The polarized light source is asynchronously illuminated, and the polarization image acquisition method includes: asynchronous acquisition operation of multi-frame polarization images, and the polarization image processing operation includes: quantitative polarization information acquisition operation.
A lighting device, characterized by comprising: a plurality of polarized light sources with different polarization states, at least one processor, and a communication interface;

The communication interface is used to receive a first instruction from the image processing device, the first instruction is used to indicate a first lighting mode; the first lighting mode is used to represent the polarized light sources using the plurality of different polarization states the manner in which lighting is provided;

The at least one processor is configured to control the plurality of polarized light sources with different polarization states to provide illumination according to the first instruction.
The device according to claim 29, wherein the first lighting method belongs to a set of polarized light lighting methods, and the set of polarized light lighting methods includes at least one of the following: multi-polarized light source synchronous lighting, multi-polarized light source asynchronous lighting, or no lighting;

Wherein, the multi-polarized light source synchronous lighting is: providing polarized lighting simultaneously through a plurality of polarized light sources with different polarization states;

The asynchronous illumination of the multi-polarized light sources is: sequentially providing polarized illumination through a plurality of polarized light sources with different polarization states;

The no lighting means: no lighting is provided.
The device according to claim 30, wherein the first lighting method comprises: synchronous lighting of the multi-polarized light sources;

The processor is specifically configured to: control one or more polarized light sources in the plurality of polarized light sources with different polarization states to provide polarized illumination.
The device according to claim 30, wherein the first lighting method comprises: non-synchronous lighting of multiple polarized light sources;

The processor is specifically configured to: control each polarized light source among the plurality of polarized light sources with different polarization states to sequentially provide polarized illumination.
The device according to claim 30, wherein the first lighting mode comprises: no lighting;

The processor is specifically configured to: control none of the plurality of polarized light sources with different polarization states to provide illumination.
An image acquisition device, characterized in that it includes: a polarized image acquisition device, at least one processor and a communication interface;

The communication interface is used to receive a second instruction from the image processing device, and the second instruction is used to indicate a first acquisition mode; the first acquisition mode is used to represent a manner of acquiring a polarization image;

The at least one processor is configured to control the polarization image collector to collect polarization images according to the second instruction.
The device according to claim 34, wherein the first acquisition mode belongs to a set of polarization acquisition modes, and the set of polarization acquisition modes includes: a synchronous acquisition operation of a single-frame polarization image, and an asynchronous acquisition operation of a multi-frame polarization image;

The synchronous acquisition operation of the single-frame polarization image includes: acquiring a single image of at least one target, and obtaining multiple polarization images of different polarization states from the single image;

The operation of asynchronously collecting multiple frames of polarization images includes: collecting multiple images of the at least one target, and each of the multiple images includes multiple polarization images of different polarization states.
The device according to claim 35, wherein the first acquisition method comprises: a synchronous acquisition operation of a single-frame polarization image;

The processor is specifically configured to control the polarization image collector to acquire a single image of the at least one target according to the second instruction, and acquire multiple polarization images of different polarization states from the single image .
The device according to claim 35, wherein the first collection method includes: non-synchronous collection operation of multi-frame polarization images;

The processor is specifically configured to, according to the second instruction, control the polarization image collector to acquire images of the at least one target multiple times, determine multiple images of the at least one target, and obtain images from the multiple images respectively. Multiple polarization images with different polarization states are obtained from each of the two images.
A computer-readable storage medium, characterized in that the computer-readable storage medium includes a computer program or an instruction, and when the computer program or instruction is run on a computer, the computer executes the computer program described in claims 1-13. any one of the methods described.
A terminal device, comprising one or more of the following: the image processing device according to any one of the above claims 14-28, the lighting device according to any one of the above claims 29-33, or the above claim 34 - The image capture device according to any one of 37.