CN118612555A - Environmental perception method of self-mobile device, self-mobile device, medium, device - Google Patents

Abstract

The present disclosure relates to the field of artificial intelligence technology, and provides an environment perception method for a self-mobile device, a self-mobile device, a computer storage medium, and an electronic device, wherein the self-mobile device is provided with a main depth camera, and the environment perception method for the self-mobile device includes: during the movement of the self-mobile device, each frame of depth image data is collected by the main depth camera; the main depth camera is configured to: in each data collection cycle, alternately collect M frames of depth image data according to preset M exposure parameters; decode each frame of depth image data to obtain each frame of point cloud corresponding to each frame of depth image data; fuse the M frames of point cloud corresponding to each data collection cycle to obtain a first fused point cloud, and use the first fused point cloud as the point cloud of each frame again, so as to perform environment perception based on the point cloud of each frame. The present disclosure can improve the environmental perception accuracy of the self-mobile device and enhance the accuracy of its navigation, obstacle avoidance and other functions.

Description

Environmental perception method of self-mobile device, self-mobile device, medium, device

Technical Field

The present disclosure relates to the field of artificial intelligence technology, and in particular to an environment perception method for a self-mobile device, a self-mobile device, a computer storage medium, and an electronic device.

Background Art

With the development of science and technology, the application of self-moving devices such as sweeping robots is becoming more and more widespread. By setting a depth camera on the self-moving device, the self-moving device can be helped to better perceive the environment.

However, when there are multiple objects at different distances in the field of view, the depth camera in the related art cannot see both nearby objects and distant objects at the same time, thereby affecting the environmental perception accuracy of the self-mobile device.

In view of this, there is an urgent need in the art to develop a new method and device for environmental perception of self-mobile equipment.

It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure.

Summary of the invention

The purpose of the present disclosure is to provide an environment perception method for a self-mobile device, a self-mobile device, a computer storage medium and an electronic device, thereby at least to a certain extent overcoming the technical problem of insufficient environment perception accuracy of the self-mobile device due to the limitations of related technologies.

Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

According to a first aspect of the present disclosure, there is provided an environment perception method for a self-mobile device, wherein the self-mobile device is provided with a main depth camera, and the method comprises:

During the movement of the self-moving device, each frame of depth image data is collected by the main depth camera; the main depth camera is configured to: in each data collection cycle, alternately collect M frames of depth image data according to preset M exposure parameters, and the M exposure parameters show a monotonically increasing or monotonically decreasing trend; M is an integer greater than 1;

Decoding each frame of the depth image data to obtain each frame of point cloud corresponding to each frame of the depth image data;

The M frames of point clouds corresponding to each data acquisition cycle are fused to obtain a first fused point cloud, and the first fused point cloud is used as the point cloud of each frame again to perform environmental perception based on the point cloud of each frame.

In an exemplary embodiment of the present disclosure, before collecting each frame of depth image data by the main depth camera, the method further includes:

Collecting preview depth image data by the main depth camera according to default exposure parameters;

When the preview point cloud is obtained by decoding the preview depth image data, each frame of the depth image data is collected by the main depth camera.

In an exemplary embodiment of the present disclosure, collecting each frame of depth image data by the main depth camera includes:

Collecting, by the main depth camera, a first exposure parameter among the M exposure parameters, a first frame of depth image data in each data collection cycle;

When the first frame of depth image data is decoded to obtain the first frame of point cloud, the second frame of depth image data in each data acquisition cycle is collected by the main depth camera according to the second exposure parameter of the M exposure parameters;

Until the M-1th frame of depth image data is decoded to obtain the M-1th frame of point cloud, the M-th frame of depth image data in each data acquisition cycle is collected by the main depth camera according to the M-th exposure parameter among the M exposure parameters.

In an exemplary embodiment of the present disclosure, fusing the M frames of point clouds corresponding to each data acquisition cycle to obtain a first fused point cloud includes:

Determine a first point cloud and a second point cloud to be fused from the M frame point clouds;

Determine a first time range according to a first timestamp corresponding to the first point cloud and a second timestamp corresponding to the second point cloud, and obtain motion data of the self-mobile device within the first time range; the motion data includes inertial measurement unit data and wheel speed meter data;

Based on the motion data within the first time range, the first point cloud and the second point cloud are fused to obtain the first fused point cloud.

In an exemplary embodiment of the present disclosure, determining the first point cloud and the second point cloud to be fused from the M frame point clouds includes:

When each current frame point cloud in the M frame point clouds is acquired, the type of the current frame point cloud is determined according to the type label corresponding to the current frame point cloud; the type of the current frame point cloud is determined according to the exposure parameters used when collecting the current frame depth image data corresponding to the current frame point cloud, and the type of the current frame point cloud includes a first type and a second type, and the exposure parameter associated with the first type is less than the exposure parameter associated with the second type;

If the current frame point cloud belongs to the first type, temporarily storing the current frame point cloud in a data queue;

If the current frame point cloud belongs to the second type, determining the current frame point cloud as the first point cloud;

The second point cloud is filtered from the data queue according to the first timestamp corresponding to the first point cloud.

In an exemplary embodiment of the present disclosure, the filtering of the second point cloud from the data queue according to the first timestamp corresponding to the first point cloud includes:

Eliminating invalid point clouds temporarily stored in the data queue and having the same timestamp as the first timestamp;

Filter out target remaining point clouds whose time interval with the first timestamp is less than a preset interval threshold from the remaining point clouds in the data queue as the second point clouds.

In an exemplary embodiment of the present disclosure, the step of fusing the first point cloud and the second point cloud based on the motion data within the first time range to obtain the first fused point cloud includes:

Determine the position transformation data of the self-moving device within the first time range by using a differential model based on the motion data within the first time range;

Performing posture transformation on the second point cloud according to the posture transformation data within the first time range to obtain a first transformed point cloud;

The first transformed point cloud is merged with the first point cloud to obtain the first fused point cloud.

In an exemplary embodiment of the present disclosure, the main depth camera collects the depth image data of each frame in a first working mode, and the first working mode includes a dot matrix mode.

In an exemplary embodiment of the present disclosure, the self-mobile device is also provided with an auxiliary depth camera, the main depth camera and the auxiliary depth camera are arranged at different positions of the self-mobile device, and there is an intersecting and non-overlapping field of view angle interval between the auxiliary depth camera and the main depth camera.

In an exemplary embodiment of the present disclosure, the auxiliary depth camera has the first working mode and the second working mode, and the second working mode includes an area array mode;

After the main depth camera collects each frame of depth image data, the method further includes:

The auxiliary depth camera collects each frame of target depth image data according to the first working mode; the auxiliary depth camera is configured to: in each data collection cycle, alternately collect N frames of target depth image data according to preset N exposure parameters, and the N exposure parameters show a monotonically increasing or monotonically decreasing trend; N is an integer greater than 1;

Decoding each frame of target depth image data to obtain each frame of target point cloud corresponding to each frame of target depth image data;

Fusing the N frames of target point clouds corresponding to each data acquisition cycle to obtain a second fused point cloud;

The first fused point cloud and the second fused point cloud are re-fused to obtain a third fused point cloud, and the third fused point cloud is used as the final point cloud of each frame again, so as to perform environment perception based on the final point cloud of each frame.

In an exemplary embodiment of the present disclosure, collecting each frame of target depth image data by the auxiliary depth camera according to the first working mode includes:

The trigger signal of the auxiliary depth camera is set to a valid level through the main depth camera, so that each frame of target depth image data is collected through the auxiliary depth camera according to the first working mode.

In an exemplary embodiment of the present disclosure, the step of re-fusing the first fused point cloud and the second fused point cloud to obtain a third fused point cloud includes:

Obtaining a third timestamp corresponding to the first fused point cloud, and obtaining a fourth timestamp corresponding to the second fused point cloud;

Determine a second time range according to the third timestamp and the fourth timestamp, and obtain motion data of the mobile device within the second time range;

Based on the motion data within the second time range, the first fused point cloud and the second fused point cloud are re-fused to obtain the third fused point cloud.

In an exemplary embodiment of the present disclosure, the step of re-fusing the first fused point cloud and the second fused point cloud to obtain the third fused point cloud based on the motion data within the second time range includes:

Determine the position transformation data of the self-mobile device within the second time range by using a differential model based on the motion data within the second time range;

Performing posture transformation on the first fused point cloud according to the posture transformation data within the second time range to obtain a second transformed point cloud;

The second transformed point cloud and the second fused point cloud are merged to obtain the third fused point cloud.

In an exemplary embodiment of the present disclosure, after collecting each frame of target depth image data by the auxiliary depth camera according to the first working mode, the method further includes:

After a specified time interval, collecting each frame of reference depth image data according to the second working mode and the automatic exposure parameters by the auxiliary depth camera;

Decoding each frame of the reference depth image data to obtain each frame of the reference point cloud corresponding to the each frame of the reference depth image data, so as to perform environmental perception based on the each frame of the reference point cloud; the automatic exposure parameter depends on the ambient brightness;

There is a preset correlation between the specified time length and the frame rate of the auxiliary depth camera.

In an exemplary embodiment of the present disclosure, the frame rate of the auxiliary depth camera is greater than the frame rate of the main depth camera, and the frame rate of the auxiliary depth camera and the frame rate of the main depth camera satisfy a preset numerical relationship.

In an exemplary embodiment of the present disclosure, the self-mobile device is further provided with an RGB camera, the RGB camera is provided in combination with the auxiliary depth camera, and the frame rate of the RGB camera is the same as the frame rate of the main depth camera;

After the auxiliary depth camera collects each frame of target depth image data according to the first working mode, the method further includes:

Each frame of image data of the self-mobile device during its movement is collected by the RGB camera, so as to perform environmental perception based on each frame of image data.

In an exemplary embodiment of the present disclosure, collecting each frame of image data of the self-moving device during movement by the RGB camera includes:

The trigger signal of the RGB camera is set to a valid level through the main depth camera, so as to collect each frame of image data of the self-moving device during the moving process through the RGB camera.

According to a second aspect of the present disclosure, a self-mobile device is provided, wherein the self-mobile device is provided with a main depth camera, and the self-mobile device comprises:

The acquisition module is configured to collect each frame of depth image data through the main depth camera during the movement of the self-mobile device; the main depth camera is configured to: in each data collection cycle, alternately collect M frames of depth image data according to preset M exposure parameters, and the M exposure parameters show a monotonically increasing or monotonically decreasing trend; M is an integer greater than 1;

A decoding module, configured to decode each frame of the depth image data to obtain each frame of point cloud corresponding to each frame of the depth image data;

The environment perception module is configured to fuse the M frames of point clouds corresponding to each data acquisition cycle to obtain a first fused point cloud, and use the first fused point cloud as the point cloud of each frame to perform environment perception based on the point cloud of each frame.

According to a third aspect of the present disclosure, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for environmental perception of a mobile device described in the first aspect is implemented.

According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the environment perception method of the self-mobile device described in the first aspect above by executing the executable instructions.

It can be seen from the above technical solutions that the environment perception method of the self-mobile device, the self-mobile device, the computer storage medium and the electronic device in the exemplary embodiments of the present disclosure have at least the following advantages and positive effects:

In the technical solutions provided by some embodiments of the present disclosure, on the one hand, the present disclosure can realize real-time environmental perception by acquiring and processing each frame of depth image data in real time during the movement of the self-mobile device, and provide reliability guarantee for the navigation, obstacle avoidance, positioning and other functions of the self-mobile device; further, by configuring the main depth camera to alternately collect M frames of depth image data according to the preset M exposure parameters (the M exposure parameters are monotonically increasing or monotonically decreasing) in each data acquisition cycle, it can be achieved that when the exposure parameter is small, the self-mobile device can see close objects clearly, and when the exposure parameter is large, the self-mobile device can see distant objects clearly; on the other hand, by fusing the M frames of point clouds corresponding to each data acquisition cycle to obtain a first fused point cloud, and reusing the first fused point cloud as the point cloud of each frame, the point cloud of the close-range object can be seen clearly and the point cloud of the distant object can be seen clearly. The problem in the related art that when there are multiple objects at different distances in the field of view, the self-mobile device cannot see distant objects and near objects at the same time is solved, and the environmental perception accuracy of the self-mobile device is improved, thereby enhancing the accuracy of its navigation, obstacle avoidance, positioning and other functions.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

FIG1 is a schematic diagram showing a flow chart of a method for environmental perception of a mobile device in an embodiment of the present disclosure;

FIG2 is a schematic diagram showing a flow chart of how the main depth camera collects each frame of depth image data in M frames of depth image data in each data collection cycle in an embodiment of the present disclosure;

FIG3 is a schematic diagram showing a flow chart of how to fuse M frames of point clouds corresponding to each data acquisition cycle to obtain a first fused point cloud in an embodiment of the present disclosure;

FIG4 is a schematic diagram showing a flow chart of how to determine the first point cloud and the second point cloud to be fused from M frame point clouds in an embodiment of the present disclosure;

FIG5 is a schematic diagram showing a flow chart of how to fuse a first point cloud and a second point cloud to obtain a first fused point cloud based on motion data within a first time range in an embodiment of the present disclosure;

FIG6 is a schematic diagram showing a flow chart of how an auxiliary depth camera collects each frame of target depth image data in an embodiment of the present disclosure;

FIG7 is a schematic diagram showing a process of re-fusing the first fused point cloud and the second fused point cloud to obtain a third fused point cloud in an embodiment of the present disclosure;

FIG8 shows a timing diagram of image output of a main depth camera, an auxiliary depth camera, and an RGB camera in an embodiment of the present disclosure;

FIG. 9 is a timing diagram showing how to set exposure parameters for a main depth camera and an auxiliary depth camera in an embodiment of the present disclosure;

FIG10 is a schematic diagram showing a flow chart of how to obtain each frame of point cloud corresponding to a depth camera in an embodiment of the present disclosure;

FIG11 is a schematic diagram showing a flow chart of how to obtain a fused point cloud for each frame for a single ToF camera in an embodiment of the present disclosure;

FIG12 is a schematic diagram showing an environment perception method of a mobile device in an embodiment of the present disclosure;

FIG13 is a schematic diagram showing the structure of a self-moving device in an exemplary embodiment of the present disclosure;

FIG. 14 is a schematic structural diagram of an electronic device in an exemplary embodiment of the present disclosure.

DETAILED DESCRIPTION

Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

The terms "a", "an", "the" and "said" used in this specification are used to indicate the presence of one or more elements/components/etc.; the terms "including" and "having" are used to express an open-ended inclusion and mean that additional elements/components/etc. may exist in addition to the listed elements/components/etc.; the terms "first" and "second" etc. are used only as labels and are not intended to limit the quantity of their objects.

In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings represent the same or similar parts, and their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

In the related art, when there are objects at different distances in the field of view, a single-exposure depth camera cannot see both nearby and distant objects at the same time. Specifically, when the exposure parameter is large, nearby objects will be overexposed and invisible. When the exposure parameter is small, the distance of distant objects cannot be measured due to weak light intensity.

In the embodiments of the present disclosure, firstly, a method for environment perception of a self-moving device is provided, which at least to some extent overcomes the defect of insufficient environment perception accuracy of the self-moving device in the related art.

FIG1 is a schematic flow chart of an environment perception method for a self-mobile device in an embodiment of the present disclosure. The execution subject of the environment perception method for a self-mobile device may be the self-mobile device.

Referring to FIG. 1 , a method for environment perception of a mobile device according to an embodiment of the present disclosure includes the following steps:

Step S110, during the movement of the self-mobile device, each frame of depth image data is collected by the main depth camera; the main depth camera is configured to: in each data collection cycle, alternately collect M frames of depth image data according to preset M exposure parameters, and the M exposure parameters show a monotonically increasing or monotonically decreasing trend; M is an integer greater than 1;

Step S120, decoding each frame of depth image data to obtain each frame of point cloud corresponding to each frame of depth image data;

Step S130, fusing the M frames of point clouds corresponding to each data collection cycle to obtain a first fused point cloud, and reusing the first fused point cloud as the point cloud of each frame to perform environmental perception based on the point cloud of each frame.

In the technical solution provided by the embodiment shown in FIG1 , on the one hand, the present disclosure can realize real-time environmental perception by acquiring and processing each frame of depth image data in real time during the movement of the self-mobile device, thereby providing reliability guarantee for the navigation, obstacle avoidance, positioning and other functions of the self-mobile device; further, by configuring the main depth camera to alternately acquire M frames of depth image data according to the preset M exposure parameters (the M exposure parameters are monotonically increasing or monotonically decreasing) in each data acquisition cycle, it is possible to enable the self-mobile device to see close objects clearly when the exposure parameter is small, and to enable the self-mobile device to see distant objects clearly when the exposure parameter is large; on the other hand, by fusing the M frames of point clouds corresponding to each data acquisition cycle to obtain a first fused point cloud, and reusing the first fused point cloud as the point cloud of each frame, the point cloud for seeing close objects clearly and the point cloud for seeing distant objects clearly can be fused together, thereby solving the problem in the related art that when there are multiple objects at different distances in the field of view, the self-mobile device cannot see distant objects and near objects at the same time, thereby improving the environmental perception accuracy of the self-mobile device, thereby enhancing the accuracy of its navigation, obstacle avoidance, positioning and other functions.

The specific implementation process of each step in Figure 1 is described in detail below:

First of all, it should be noted that the depth cameras (main depth camera and auxiliary depth camera) in the present disclosure may be ToF cameras.

The full name of ToF camera is Time Of Flight Camera, which is a depth imaging camera. It works by using the camera to send light to the target object, and then measuring the time required for the light to propagate back and forth between the lens and the object. This time difference can be used to calculate the distance of the object. In this way, the depth camera can capture depth information and create a 3D (3Dimensions, three-dimensional space) stereo depth sensing map. The application of this technology can provide accurate distance measurement in a variety of occasions, such as obstacle avoidance in autonomous vehicles, and applications in mobile phone AR (Augmented Reality)/VR (Virtual Reality).

In step S110 , during the movement of the self-mobile device, each frame of depth image data is collected by a main depth camera.

In this step, the main depth camera can be a depth camera set on the left side of the mobile device, which can be represented by ToF1 in the following embodiments. The frame rate of the main depth camera can be represented by fs1. The frame rate of the camera, also known as FPS (Frames Per Second), refers to the number of images that the camera can capture and output in one second.

The main depth camera may collect each frame of depth image data in a first working mode, wherein the first working mode includes a dot matrix mode. The interval between two frames of depth image data collected by the auxiliary depth camera in the dot matrix mode may be 1000/fs1.

Dot matrix refers to the independent light point emitters used in ToF sensors, usually arranged in the form of a matrix. Each light point emitter can be controlled independently to emit light to the target object and measure the return time to calculate the distance and position information of the object. The advantage of dot matrix is high accuracy and can measure smaller objects and details.

The main depth camera in the present disclosure is configured to: in each data acquisition cycle, alternately collect M frames of depth image data according to preset M exposure parameters (which may be exposure duration, exposure duration, also known as exposure time, refers to the time during which light is irradiated to the film or photoreceptor from the opening to the closing of the camera shutter), the M exposure parameters may show a monotonically increasing or monotonically decreasing trend, and M is an integer greater than 1.

When the exposure parameter is in the first parameter interval (for example, 0-t11 interval), it can be called a short exposure parameter; when the exposure parameter is in the second parameter interval (the two endpoint values of the second parameter interval are greater than the two endpoint values of the first parameter interval), it can be called a medium exposure parameter; when the exposure parameter is in the third parameter interval (for example, t22-t33 interval, the two endpoint values of the third parameter interval are greater than the two endpoint values of the second parameter interval), it can be called a long exposure parameter. Assume that M is 2, then the above-mentioned M (2) exposure parameters in the present disclosure can be set to "short exposure parameter-long exposure parameter", or, set to "long exposure parameter-short exposure parameter". Assume that M is 3, then the above-mentioned M (3) exposure parameters in the present disclosure can be set to "short exposure parameter-medium exposure parameter-long exposure parameter", or, set to "long exposure parameter-medium exposure parameter-short exposure parameter". It can be set according to actual conditions, and the present disclosure does not make any special limitation on this.

By setting the M exposure parameters to show a monotonically increasing or monotonically decreasing trend, the self-mobile device can, according to the M exposure parameters with different values, see clearly nearby objects when the exposure parameter is small, and see clearly distant objects when the exposure parameter is large.

Preferably, the parameter difference between the first exposure parameter and the Mth exposure parameter in the M exposure parameters may be greater than a preset parameter difference threshold, so as to ensure that the first exposure parameter and the Mth exposure parameter are of different types, for example, the first exposure parameter is a short exposure parameter, while the Mth exposure parameter is a long exposure parameter, or the first exposure parameter is a long exposure parameter, while the Mth exposure parameter is a short exposure parameter. By setting the M exposure parameters to include both short exposure parameters and long exposure parameters, the numerical difference between different exposure parameters can be increased, so that when there are multiple objects with large distance differences within the field of view (for example, two objects at a very close distance and a very far distance), different exposure parameters can be used to capture and clearly present the multiple objects.

Preferably, in view of the fact that a large compensation error may be caused if the M exposure parameters are set to show a monotonically decreasing trend, in order to avoid the above problem, the present invention can set the above M exposure parameters to show a monotonically increasing trend, that is, the above M exposure parameters can be configured with reference to the above "short exposure parameter-long exposure parameter" or the above "short exposure parameter-medium exposure parameter-long exposure parameter" change trend.

It should be noted that during the movement of the self-mobile device, the main depth camera can first collect preview depth image data according to the default exposure parameters. After obtaining the above preview depth image data, the self-mobile device can decode the above preview depth image. When the preview point cloud after decoding is obtained, each frame of depth image data can be collected by the above main depth camera.

The preview depth image data refers to the initial depth image data collected by the main depth camera without configuring the exposure parameters, for example, the initial depth image data collected at the beginning of the work.

It should be noted that the depth image data collected by the camera is the original digital image data converted from the light source signal from the lens, that is, raw image data. Raw is an unprocessed and uncompressed format, which can be figuratively called a "digital negative". Therefore, it needs to be decoded.

The following is a specific implementation of how the main depth camera collects M frames of depth image data in each data collection cycle:

Referring to FIG. 2 , FIG. 2 shows a schematic flow chart of how the main depth camera collects each frame of depth image data in M frames of depth image data in each data collection cycle in an embodiment of the present disclosure, including steps S201 to S203:

In step S201, the first frame of depth image data in each data collection cycle is collected by a main depth camera according to a first exposure parameter among M exposure parameters.

In this step, an example is taken in which M is 2 and M exposure parameters are set to "short exposure parameter-long exposure parameter" for explanation. If the short exposure parameter is represented by expo1_l and the long exposure parameter is represented by expo1_h, the M exposure parameters can form a sequence: expo1_l-expo1_h, where expo1_l is the first exposure parameter among the M exposure parameters, and expo1_h is the second exposure parameter among the M exposure parameters, which is also the Mth exposure parameter.

Thus, in each data acquisition cycle, the first frame of depth image data in each data acquisition cycle can be acquired by the main depth camera according to the first exposure parameter of the M exposure parameters, that is, the first frame of depth image data is acquired by the main depth camera according to expo1_l.

In step S202, when the first frame of depth image data is decoded to obtain the first frame of point cloud, the second frame of depth image data in each data acquisition cycle is collected by the main depth camera according to the second exposure parameter of the M exposure parameters.

In this step, after obtaining the above-mentioned first frame depth image data, the above-mentioned first frame depth image data can be decoded to obtain the first frame point cloud. When the decoding of the first frame depth image data is completed, the second exposure parameter of the above-mentioned M exposure parameters, namely expo1_h, can be obtained, and the second frame depth image data is collected according to expo1_h through the above-mentioned main depth camera.

In step S203, until the M-1th frame of depth image data is decoded to obtain the M-1th frame of point cloud, the main depth camera is controlled to collect the Mth frame of depth image data in each data collection cycle according to the Mth exposure parameter among the M exposure parameters.

In this step, considering that when M is 2, the M-1th frame depth image data is the first frame depth image data, and thus, this step corresponds to the above step S202: when the decoding of the first frame depth image data is completed, the second exposure parameter of the above M exposure parameters, that is, expo1_h, can be obtained, and the second frame depth image data is collected according to expo1_h through the above main depth camera.

When M is greater than 2, assuming that M is 3, this step corresponds to collecting the third frame of depth image data in each data collection cycle through the main depth camera according to the third exposure parameter of the M exposure parameters when the second frame of depth image data is decoded to obtain the second frame of point cloud data.

By cyclically collecting M frames of depth image data according to the above-mentioned M exposure parameters in each data collection cycle, M frames of depth image data corresponding to each data collection cycle can be obtained, which are alternately collected according to "short exposure parameters-long exposure parameters" or "short exposure parameters-medium exposure parameters-long exposure parameters".

Next, referring to FIG. 1 , in step S120 , each frame of depth image data is decoded to obtain each frame of point cloud corresponding to each frame of depth image data.

In this step, after each frame of depth image data is obtained, each frame of depth image data may be decoded to obtain each frame of point cloud corresponding to each frame of depth image data.

Point cloud is a data structure used to represent the shape of objects in three-dimensional space. It is used to dynamically store a collection of data points usually from a lidar system or other measuring instruments (such as a 3D coordinate measuring machine, a 3D laser scanner, or a photographic scanner). Each point has a corresponding coordinate value, which represents the data structure of the object shape at that point in three-dimensional space. These points contain rich information, such as three-dimensional coordinates (X, Y, Z), color, classification value, intensity value, time, etc. Point cloud is a massive collection of points that express the spatial distribution and surface characteristics of the target in the same spatial reference system. Point cloud can represent three-dimensional shapes or objects, and the real world can be restored through high-precision point cloud.

It should be noted that when each frame of depth image data is decoded to obtain each frame of point cloud data, the timestamp of obtaining each frame of point cloud data can be recorded.

In step S130, the M frames of point clouds corresponding to each data acquisition cycle are fused to obtain a first fused point cloud, and the first fused point cloud is used again as the point cloud of each frame to perform environmental perception based on the point cloud of each frame.

In this step, the M frames of point clouds corresponding to each data collection cycle can be fused to obtain a first fused point cloud, and the first fused point cloud can be used as the point cloud of each frame again, so that the self-mobile device can perceive the environment based on the point cloud of each frame.

Referring to FIG. 3 , FIG. 3 shows a schematic flow chart of how to fuse M frames of point clouds corresponding to each data acquisition cycle to obtain a first fused point cloud in an embodiment of the present disclosure, including steps S301 to S303:

In step S301, a first point cloud and a second point cloud to be fused are determined from M frames of point clouds.

In this step, since the main depth camera collects depth image data frame by frame, the M frame point clouds can be fused according to the type of the current frame point cloud obtained each time in the M frame point clouds.

Specifically, reference may be made to FIG. 4 , which shows a schematic diagram of a process of determining a first point cloud and a second point cloud to be fused from M frame point clouds in an embodiment of the present disclosure, including steps S401 to S404:

In step S401, when each current frame point cloud in the M frame point clouds is acquired, the type of the current frame point cloud is determined according to the type label corresponding to the current frame point cloud.

In this step, when each current frame point cloud is obtained, the type of the current frame point cloud can be determined. Exemplarily, the type of the current frame point cloud is determined according to the exposure parameters used when collecting the current frame depth image data corresponding to the current frame point cloud. The type of the current frame point cloud may include a first type and a second type. The exposure parameter associated with the first type is less than the exposure parameter associated with the second type. Exemplarily, the first type may be a point cloud corresponding to the depth image data collected using the above-mentioned short exposure parameters (which may be referred to as a short exposure point cloud), and the second type may be a point cloud corresponding to the depth image data collected using the above-mentioned long exposure parameters (which may be referred to as a long exposure point cloud).

In step S402, if the current frame point cloud belongs to the first type, the current frame point cloud is temporarily stored in the data queue.

In this step, if the current frame point cloud belongs to the first type, that is, the short exposure point cloud, the current frame point cloud may be temporarily stored in the data queue.

In step S403, if the current frame point cloud belongs to the second type, the current frame point cloud is determined as the first point cloud.

In this step, if the current frame point cloud belongs to the second type, that is, the long exposure point cloud, the current frame point cloud may be determined as the first point cloud.

In step S404, the second point cloud is filtered from the data queue according to the first timestamp corresponding to the first point cloud.

In this step, after determining the first point cloud, the second point cloud can be screened from the data queue according to the first timestamp corresponding to the first point cloud. Specifically, in order to avoid the situation where a frame of depth image data is collected using both short exposure parameters and long exposure parameters at the same location, the invalid point cloud temporarily stored in the data queue with the same first timestamp can be removed first, and then the target remaining point clouds whose time interval with the first timestamp is less than the preset interval threshold can be screened from the remaining point clouds in the data queue (the short exposure point clouds of the second type temporarily stored in the data queue are all short exposure point clouds of the second type) as the above-mentioned second point cloud.

After determining the first point cloud and the second point cloud to be fused, you can then refer to Figure 3. In step S302, determine the first time range according to the first timestamp corresponding to the first point cloud and the second timestamp corresponding to the second point cloud, and obtain motion data from the mobile device within the first time range.

In this step, exemplarily, the first point cloud can be represented by Pts_l, and the first timestamp corresponding to the first point cloud can be tl; the second point cloud can be represented by Pts_s, and the second timestamp corresponding to the second point cloud can be ts.

Thus, the motion data of the self-mobile device within the first time range (from time ts to time tl) can be obtained. The above motion data may include inertial measurement unit data (i.e., IMU data) and wheel speed meter data (i.e., ODOM data, which is an important part of the navigation and positioning of the self-mobile device, and can help the self-mobile device to perform autonomous movement and position identification in indoor or outdoor environments).

Among them, IMU stands for Inertial Measurement Unit, and its Chinese name is Inertial Measurement Unit. It is a camera mainly used to detect and measure acceleration and rotational motion (for example: speed, tilt, rotation and even angle). IMU is usually composed of a variety of cameras, including but not limited to inclinometers, accelerometers, gyroscopes, magnetometers, barometers, etc. These cameras work together to obtain information such as the object's motion, heading, attitude (roll angle, pitch angle and yaw angle) through data processing. Specifically, the inclinometer can be used to measure the vertical inclination of the object relative to the ground, and the accelerometer can measure the acceleration related to inertia, including rotation, gravity and other forms of linear acceleration. The gyroscope can measure the angular velocity of three axes, namely pitch, roll and yaw. The magnetometer is used to measure the magnetic field and can be used to correct the posture. In addition, IMU can also include a barometer to correct the height information. Through the combination of these cameras, accurate measurement and positioning of the object's motion can be achieved.

In step S303, based on the motion data within the first time range, the first point cloud and the second point cloud are fused to obtain a first fused point cloud.

In this step, reference may be made to FIG5 , which shows a schematic diagram of a process of fusing a first point cloud and a second point cloud to obtain a first fused point cloud based on motion data within a first time range in an embodiment of the present disclosure, including steps S501 to S503:

In step S501, a differential model is used to determine position transformation data of a mobile device within a first time range based on motion data within a first time range.

In this step, the differential model is used to estimate the movement of the robot in a given direction. It is based on the robot's speed and direction changes, as well as the distance the wheels have turned during this period. Using IMU data and ODOM data, the change in the robot's posture (position and attitude) from ts to tl can be estimated.

Exemplarily, the above-mentioned differential model can be based on the above-mentioned imu data and odom data to infer the posture transformation data of the mobile device from time ts to time tl. The posture transformation data can be a posture transformation matrix, represented by Ts_l.

In step S502, the second point cloud is transformed in posture according to the posture transformation data within the first time range to obtain a first transformed point cloud.

In this step, the second point cloud Pts_s can be transformed based on the posture transformation data within the first time range to convert it into the first transformed point cloud Pts_s' corresponding to time tl. Specifically, Pts_s' = (Ts_l) ^-1 × Pts_s, where (Ts_l) ^-1 represents the inverse matrix of the posture transformation matrix Ts_l.

In step S503, the first transformed point cloud is merged with the first point cloud to obtain a first fused point cloud.

In this step, the first transformed point cloud Pts_s’ and the first point cloud Pts_l can be merged to obtain the first fused point cloud pts_1.

Similarly, for the M frames of depth image data contained in each data acquisition cycle, the corresponding first fused point cloud can be obtained based on the above process. After obtaining each first fused point cloud, the above first fused point cloud can be used as the point cloud of each frame again, so that the self-mobile device can perform environmental perception based on the point cloud of each frame.

The present disclosure can ensure that the first fused point cloud obtained has no data distortion by performing motion compensation based on IMU and ODOM data when generating the first fused point cloud. By reusing the first fused point cloud as the point cloud of each frame, the self-mobile device can see objects at different distances within the field of view based on the point cloud of each frame, thereby improving the environmental perception accuracy of the self-mobile device.

In an optional implementation, the self-mobile device in the present disclosure may also be provided with an auxiliary depth camera (denoted by ToF2), and the frame rate of the auxiliary depth camera may be fs2, and fs2=2fs1+1.

The main depth camera and the auxiliary depth camera are arranged at different positions of the self-mobile device. Exemplarily, the auxiliary depth camera can be a depth camera arranged in front of the self-mobile device, which can be represented by ToF2 in the following embodiments. Thus, there is an intersecting and non-overlapping field of view angle interval between the auxiliary depth camera and the main depth camera.

The field of view of a camera refers to the range that the camera can observe. The larger the field of view, the larger the observation range. By setting the field of view interval between the auxiliary depth camera and the main depth camera to have an intersection and non-overlapping, the main depth camera and the auxiliary depth camera can simultaneously collect depth image data in the environment.

It should be noted that the auxiliary depth camera has two working modes, namely the first working mode and the second working mode, and the second working mode includes an area array mode.

Among them, area array refers to a larger planar emitter used in ToF cameras, which usually covers the entire sensor field of view. The area array emitter can emit a whole surface of light to the target object and use the entire surface as the light path to measure the distance and position information of the object. The advantage of the area array is its fast processing speed and the ability to measure the distance and position information of multiple target objects at the same time. The area array camera is an imaging tool that uses "surfaces" as units to capture images. It can be exposed in a short time and obtain a complete target image at one time. It has the advantage of intuitive image measurement, and it implements pixel matrix shooting.

The auxiliary depth camera is controlled by the main depth camera, works in slave mode, and works in conjunction with the main depth camera. The following is an explanation of the process of data collection based on the dot matrix mode by the auxiliary depth camera. Referring to Figure 6, Figure 6 shows a flow chart of how the auxiliary depth camera collects each frame of target depth image data in an embodiment of the present disclosure, including steps S601-S604:

In step S601, after the main depth camera collects each frame of depth image data, the auxiliary depth camera collects each frame of target depth image data according to the first working mode.

In this step, after the main depth camera collects each frame of depth image data, the main depth camera ToF1 can set the trigger signal of the auxiliary depth camera to a valid level (for example, a high level) to collect each frame of target depth image data through the auxiliary depth camera in a dot matrix mode. The interval between the auxiliary depth camera collecting two frames of target depth image data in a dot matrix mode can be 1000/fs1.

Among them, when the auxiliary depth camera adopts the dot matrix mode to collect each frame of target depth image data, the mechanism adopted is the same as the mechanism adopted by the above-mentioned main depth camera when collecting each frame of depth image data. Specifically, the auxiliary depth camera is configured to: in each data collection cycle, according to the preset N (N is an integer greater than 1, N can be equal to M, or not equal to M, can be set according to actual conditions, and the present disclosure does not make special restrictions on this) exposure parameters, alternately collect N frames of target depth image data, and the N exposure parameters here also show a monotonically increasing or monotonically decreasing trend. The specific settings of the N exposure parameters are similar to the specific settings of the above-mentioned M exposure parameters, and will not be repeated here.

Preferably, referring to the relevant explanation of the above step S110, in order to avoid the problem of large compensation error caused by setting the N exposure parameters to show a monotonically decreasing trend, the present disclosure can set the above N exposure parameters to show a monotonically increasing trend.

Preferably, referring to the relevant explanation of the above step S110, the parameter difference between the first exposure parameter and the Nth exposure parameter among the above N exposure parameters may also be set to be greater than a preset parameter difference threshold.

Exemplarily, taking N as 2, and N exposure parameters being set as "short exposure parameter-long exposure parameter" as an example for explanation, if the above-mentioned short exposure parameter is represented by expo2_l, and the above-mentioned long exposure parameter is represented by expo2_h, then the sequence composed of the above-mentioned N exposure parameters can be expressed as: expo2_l-expo2_h, expo2_l is the first exposure parameter among the above-mentioned N exposure parameters, and expo2_h is the second exposure parameter among the above-mentioned N exposure parameters, which is also the Nth exposure parameter.

Thus, after the main depth camera ToF1 sets the trigger signal of the auxiliary depth camera to a valid level (high level), the auxiliary depth camera ToF2 can first collect preview depth image data according to the default exposure parameters. When the preview depth image data is decoded to obtain the preview point cloud, the auxiliary depth camera can collect each frame of target depth image data based on the dot matrix mode.

Specifically, when M is 2, the auxiliary depth camera ToF2 can first collect the first frame of target depth image data in each data collection cycle according to the first exposure parameter (i.e., expo2_l) among the N exposure parameters. When the first frame of depth image data is decoded to obtain the first frame of point cloud, the auxiliary depth camera ToF2 can collect the second frame of target depth image data in each data collection cycle according to the second exposure parameter (i.e., expo2_h) among the N exposure parameters to obtain two frames of target depth image data in each data collection cycle. The specific process of the auxiliary depth camera ToF2 collecting each frame of target depth image data can refer to the relevant explanation of the above step S110, which will not be repeated here.

In step S602, each frame of target depth image data is decoded to obtain each frame of target point cloud corresponding to each frame of target depth image data.

In this step, the relevant explanation of the above step S120 can be referred to, and each frame of target depth image data is decoded to obtain each frame of target point cloud corresponding to each frame of target depth image data.

In step S603, N frames of target point clouds corresponding to each data acquisition cycle are fused to obtain a second fused point cloud.

In this step, the relevant explanation of the above step S130 may be referred to, and the N frames of target point clouds corresponding to each data acquisition cycle may be fused to obtain a second fused point cloud.

Specifically, you can refer to the relevant explanations in Figure 3 above, first determine the third point cloud (long exposure point cloud) and the fourth point cloud (short exposure point cloud) to be fused from the N frames of target point cloud, then determine the third time range according to the fifth timestamp corresponding to the third point cloud and the sixth timestamp corresponding to the fourth point cloud, and obtain the motion data (i.e., imu data and odom data) of the self-mobile device within the third time range, further, you can use the differential model to determine the posture transformation data of the self-mobile device within the third time range based on the motion data within the third time range, then, you can perform posture transformation on the fourth point cloud according to the posture transformation data within the third time range to obtain a third transformed point cloud, finally, you can merge the third transformed point cloud with the third point cloud to obtain the second fused point cloud pts_2.

In step S604, the first fused point cloud and the second fused point cloud are re-fused to obtain a third fused point cloud, and the third fused point cloud is used as the final point cloud of each frame again to perform environment perception based on the final point cloud of each frame.

In this step, after obtaining the first fused point cloud pts_1 of each frame corresponding to ToF1 and the second fused point cloud pts_2 of each frame corresponding to the dot pattern of ToF2, the first fused point cloud and the second fused point cloud can be re-fused to obtain a third fused point cloud.

Referring to FIG. 7 , FIG. 7 shows a schematic diagram of a process of re-fusing the first fused point cloud and the second fused point cloud to obtain a third fused point cloud in an embodiment of the present disclosure, including steps S701 to S703:

In step S701, a third timestamp corresponding to the first fused point cloud is obtained, and a fourth timestamp corresponding to the second fused point cloud is obtained.

In this step, the third timestamp t4 corresponding to the first fused point cloud pts_1 may be obtained, and the fourth timestamp t5 corresponding to the second fused point cloud pts_2 may be obtained.

In step S702, a second time range is determined according to the third timestamp and the fourth timestamp, and motion data within the second time range is acquired from the mobile device.

In this step, the second time range (i.e., time t4 to t5) can be determined based on the third timestamp and the fourth timestamp, thereby obtaining the motion data (i.e., imu data and odom data) of the mobile device from time t4 to t5.

In step S703, based on the motion data within the second time range, the first fused point cloud and the second fused point cloud are re-fused to obtain a third fused point cloud.

In this step, exemplarily, the differential model can be used to determine the posture transformation data of the mobile device within the second time range based on the motion data within the second time range, for example: posture transformation matrix T4_5, and then, the first fused point cloud pts_1 is subjected to posture transformation according to the posture transformation data within the second time range to obtain the second transformed point cloud pts_1', specifically, pts_1' = (T4_5) ^-1 *pts_1, wherein (T4_5) ^-1 represents the inverse matrix of the posture transformation matrix within the second time range. Further, the second transformed point cloud pts_1' can be merged with the second fused point cloud pts_2 to obtain the third fused point cloud.

After obtaining the third fused point cloud, the third fused point cloud can be used as the final point cloud of each frame again, so that the self-mobile device can perform environmental perception based on the final point cloud of each frame, such as mapping, positioning and obstacle avoidance.

The present invention obtains the final point cloud of each frame by re-fusing the first fused point cloud obtained by the main depth camera using the dot matrix mode and the second fused point cloud obtained by the auxiliary depth camera using the dot matrix mode, which can make the collected data more comprehensive and reduce motion distortion, thereby improving the environmental perception accuracy of the mobile device.

In an optional embodiment, after the auxiliary depth camera ToF2 collects each frame of target depth image data in the first working mode, the auxiliary depth camera can collect each frame of reference depth image data in the area array mode according to the automatic exposure parameters at intervals of a specified duration (the interval between the auxiliary depth camera collecting two frames of reference depth image data in the area array mode can be 1000/fs1). Exemplarily, the automatic exposure parameters can depend on the ambient brightness. There is a preset correlation between the above-mentioned specified duration and the frame rate of the auxiliary depth camera, which can be 1000/fs2 for example.

Specifically, after the above-specified time interval (1000/fs2), ToF2 can first collect preview depth image data based on the area array mode. When the preview depth image data is decoded to obtain the preview point cloud, the automatic exposure parameter expo3_1 can be obtained. Based on the automatic exposure parameter expo3_1, the first frame of reference depth image data is collected. When the first frame of reference depth image data is decoded to obtain the first frame of point cloud, the automatic exposure parameter expo3_2 can be obtained. Based on the automatic exposure parameter expo3_2, the second frame of reference depth image data is collected. Similarly, multiple frames of reference depth image data are collected based on the area array mode. Therefore, the self-mobile device can perform environmental perception based on each frame of reference point cloud corresponding to each frame of reference depth image data, for example, perform obstacle avoidance.

The present disclosure sets ToF2 to two working modes combining dot matrix and planar matrix modes, so that the target depth image data collected in the dot matrix mode can be fused with the depth image data collected by the main depth camera, making the observation range more comprehensive, which not only ensures the ranging accuracy of distant point clouds, but also reduces various problems caused by multipath (that is, the signal undergoes multiple reflections before reaching the target). At the same time, the reference depth image data collected in the planar matrix mode is closer than that in the dot matrix mode, so that the number of nearby point clouds can be guaranteed, so that the self-moving device can more accurately perceive and respond to changes in the surrounding environment, thereby ensuring the obstacle avoidance effect of the self-moving device.

In an optional implementation, the self-mobile device in the present disclosure may also be provided with an RGB camera, and the RGB camera may be provided in combination with the auxiliary depth camera.

The frame rate of the RGB camera can be fs1, which is the same as the frame rate of the main depth camera. Thus, the interval between two frames of image data collected by the RGB camera can be 1000/fs1. The RGB camera is also controlled by the main depth camera. Specifically, after the auxiliary depth camera ToF2 collects each frame of target depth image data in a dot matrix mode, the main depth camera can set the trigger signal of the RGB camera to an effective level (high level) to collect each frame of image data from the mobile device during movement through the RGB camera, so that the mobile device can perform environmental perception based on each frame of image data. Thus, the present disclosure can avoid the problem of the light spot generated when the RGB camera collects images due to the long exposure time of the RGB camera, thereby ensuring that the image data collected by the RGB camera is clear and visible.

The present disclosure adopts a combination of an RGB camera and a ToF camera. The RGB camera can capture visible light images and provide color or grayscale image data, which are very important for identifying the color, texture and details of objects. The ToF camera can provide depth information in the scene, so the two can provide complementary information to achieve more accurate object recognition, tracking and positioning in a variety of application scenarios.

Referring to FIG8 , FIG8 shows a timing diagram of the output of the main depth camera, the auxiliary depth camera, and the RGB camera in an embodiment of the present disclosure (ie, at what time the main depth camera, the auxiliary depth camera, and the RGB camera start collecting data, respectively), as shown in FIG8 :

The main depth camera is represented by ToF1, and the frame rate of ToF1 is recorded as fs1. The auxiliary depth camera is represented by ToF2, and the frame rate of ToF2 is recorded as fs2, fs2=2fs1+1; the frame rate of the RGB camera is recorded as fs1;

ToF1 has only one working mode, i.e. dot matrix mode. ToF1 starts working first, and after time t1, ToF1 completes the acquisition of one frame of depth image data. The interval between two frames of depth image data output by ToF1 is 1000/fs1.

ToF2 includes two working modes, namely, dot matrix mode and plane matrix mode. After ToF1 completes the acquisition of one frame of depth image data, that is, at time t1, ToF1 sets the trigger signal of ToF2 to a high level, and ToF2 starts to work based on the dot matrix mode, that is, ToF2 starts to acquire each frame of target depth image data based on the dot matrix mode. After the interval t2, ToF2 completes the acquisition of one frame of target depth image data, and the interval between ToF2 outputting two frames of target depth image data is 1000/fs1.

After ToF2 collects each frame of target depth image data based on the dot array mode, at an interval of 1000/fs2, ToF2 starts working based on the area array mode, that is, ToF2 collects each frame of reference depth image data based on the area array mode;

When ToF2 completes the acquisition of a frame of target depth image data based on the dot matrix mode, that is, at time t2, ToF1 sets the trigger signal of the RGB camera to a high level, and the RGB camera starts working, that is, the RGB camera starts to collect each frame of image data, and the interval between two frames of image data output by the RGB camera is 1000/fs1.

Referring to FIG. 9 , FIG. 9 shows a timing diagram of how to set exposure parameters for the main depth camera (ToF1) and the auxiliary depth camera (ToF2) in an embodiment of the present disclosure, as shown in FIG. 9 :

If both M and N are 2, then the M exposure parameters are two exposure parameters (taking the first exposure parameter as a short exposure parameter and the second exposure parameter as a long exposure parameter as an example), and the N exposure parameters are also two exposure parameters (taking the first exposure parameter as a short exposure parameter and the second exposure parameter as a long exposure parameter as an example), then the time when two consecutive frames of ToF1 are decoded and completed can be recorded as t4_1 and t4_2, the time when two consecutive frames of ToF2 are decoded and completed in dot matrix mode can be recorded as t5_1 and t5_2, and the time when two consecutive frames of ToF2 are decoded and completed in area matrix mode can be recorded as t6_1 and t6_2;

Specifically, the short exposure parameter of ToF1 is recorded as expo1_l, and the long exposure parameter is recorded as expo1_h; when the preview depth image data of ToF1 is decoded at time t4_1, the short exposure parameter expo1_l is obtained, and the first frame of depth image data is collected according to the short exposure parameter expo1_l. When the first frame of depth image data is decoded at time t4_2, the long exposure parameter expo1_h is obtained, and the second frame of depth image data is collected according to the long exposure parameter expo1_h. When each frame of depth image data is subsequently collected, the "short exposure parameter-long exposure parameter" cycle setting can be followed;

The Tof2 dot matrix short exposure parameter is recorded as expo2_l, and the long exposure parameter is recorded as expo2_h; Tof2 obtains the short exposure parameter expo2_l when the preview depth image data collected based on the dot matrix mode is decoded at time t5_1, and the first frame of target depth image data is collected according to the short exposure parameter expo2_l. When the first frame of target depth image data is decoded at time t5_2, the long exposure parameter expo2_h is obtained, and the second frame of target depth image data is collected according to the long exposure parameter expo2_h. When each frame of target depth image data is subsequently collected, the "short exposure parameter-long exposure parameter" cycle setting can be followed;

When Tof2 is in area array mode, the automatic exposure parameters of two consecutive frames are recorded as expo3_1 and expo3_2; Tof2 obtains the automatic exposure parameter expo3_1 when the preview depth image data collected in area array mode is decoded at time t6_1, and the first frame of reference depth image data is collected according to the automatic exposure parameter expo3_1. When the decoding of the first frame of reference depth image data is completed at time t6_2, the automatic exposure parameter expo3_2 is obtained, and the second frame of reference depth image data is collected according to the automatic exposure parameter expo3_2. When each frame of reference depth image data is subsequently collected, it is set in sequence according to the calculated automatic exposure parameters.

Referring to FIG. 10 , FIG. 10 shows a schematic flow chart of how to obtain each frame of point cloud corresponding to a depth camera (a main depth camera or an auxiliary depth camera in a dot matrix mode) in an embodiment of the present disclosure, including steps S1001 to S1004:

In step S1001, turn on the ToF camera;

In step S1002, a ToF raw image is obtained;

In step S1003, the ToF raw image is decoded and each frame of point cloud is output;

In step S1004, M frames of point clouds are fused, and the fused point cloud is used as each frame of point cloud again.

Referring to FIG. 11 , FIG. 11 shows a schematic flow chart of how a single ToF camera obtains a fused point cloud for each frame (i.e., step S1004 in FIG. 10 above) in an embodiment of the present disclosure, including steps S1101 to S1108:

In step S1101, start;

In step S1102, the point cloud of the current frame is obtained;

In step S1103, it is determined whether it is the first type (long exposure point cloud) or the second type (short exposure point cloud);

If it is the second type, then proceed to step S1104 and store it in the data queue;

If it is the first type, then proceed to step S1105 to determine whether there is a temporarily stored point cloud in the data queue, and whether the time interval between the timestamp of the temporarily stored point cloud and the timestamp of the current frame point cloud is less than a preset interval threshold;

If not, proceed to step S1106 to clear the queue;

If yes, then go to step S1107 to combine the imu + wheel speed meter data to fuse the point cloud;

In step S1108, the fused point cloud is restored as the point cloud of each frame.

Referring to FIG. 12 , FIG. 12 is a schematic diagram showing an environment perception method of a mobile device in an embodiment of the present disclosure, as shown in FIG. 12 :

For the main depth camera (dot matrix mode) disposed on the left side, the present disclosure can fuse its short exposure point cloud with the long exposure point cloud to obtain the first fused point cloud of each frame;

For the auxiliary depth camera (dot matrix mode) disposed on the front side, the present disclosure can fuse its short exposure point cloud with the long exposure point cloud to obtain a second fused point cloud per frame;

Afterwards, the first fused point cloud of each frame (from the main depth camera) and the second fused point cloud of each frame (from the auxiliary depth camera) can be re-fused, and the obtained third fused point cloud can be used as the final point cloud of each frame. Thus, the self-mobile device can perceive the environment based on the final point cloud of each frame.

The present disclosure also provides a self-mobile device, which is provided with a main depth camera. FIG13 shows a schematic diagram of the structure of the self-mobile device in an exemplary embodiment of the present disclosure; as shown in FIG13 , the self-mobile device 1300 may include an acquisition module 1310, a decoding module 1320 and an environment perception module 1330. Among them:

The acquisition module 1310 is configured to collect each frame of depth image data through the main depth camera during the movement of the self-mobile device; the main depth camera is configured to: in each data collection cycle, alternately collect M frames of depth image data according to preset M exposure parameters, and the M exposure parameters show a monotonically increasing or monotonically decreasing trend; M is an integer greater than 1;

The decoding module 1320 is configured to decode each frame of the depth image data to obtain each frame of point cloud corresponding to each frame of the depth image data;

The environment perception module 1330 is configured to fuse the M frames of point clouds corresponding to each data acquisition cycle to obtain a first fused point cloud, and use the first fused point cloud as the point cloud of each frame to perform environment perception based on the point cloud of each frame.

In an exemplary embodiment of the present disclosure, before collecting each frame of depth image data through the main depth camera, the acquisition module 1310 is configured to:

Collecting preview depth image data by the main depth camera according to default exposure parameters;

When the preview point cloud is obtained by decoding the preview depth image data, each frame of the depth image data is collected by the main depth camera.

In an exemplary embodiment of the present disclosure, the acquisition module 1310 collects the depth image data of each frame through the main depth camera, including:

Collecting, by the main depth camera, a first exposure parameter among the M exposure parameters, a first frame of depth image data in each data collection cycle;

When the first frame of depth image data is decoded to obtain the first frame of point cloud, the second frame of depth image data in each data acquisition cycle is collected by the main depth camera according to the second exposure parameter of the M exposure parameters;

Until the M-1th frame of depth image data is decoded to obtain the M-1th frame of point cloud, the M-th frame of depth image data in each data acquisition cycle is collected by the main depth camera according to the M-th exposure parameter among the M exposure parameters.

In an exemplary embodiment of the present disclosure, the environment perception module 1330 fuses the M frames of point clouds corresponding to each data collection period to obtain a first fused point cloud, including:

Determine a first point cloud and a second point cloud to be fused from the M frame point clouds;

Determine a first time range according to a first timestamp corresponding to the first point cloud and a second timestamp corresponding to the second point cloud, and obtain motion data of the self-mobile device within the first time range; the motion data includes inertial measurement unit data and wheel speed meter data;

Based on the motion data within the first time range, the first point cloud and the second point cloud are fused to obtain the first fused point cloud.

In an exemplary embodiment of the present disclosure, the environment perception module 1330 determines the first point cloud and the second point cloud to be fused from the M frame point clouds, including:

When each current frame point cloud in the M frame point clouds is acquired, the type of the current frame point cloud is determined according to the type label corresponding to the current frame point cloud; the type of the current frame point cloud is determined according to the exposure parameters used when collecting the current frame depth image data corresponding to the current frame point cloud, and the type of the current frame point cloud includes a first type and a second type, and the exposure parameter associated with the first type is less than the exposure parameter associated with the second type;

If the current frame point cloud belongs to the first type, temporarily storing the current frame point cloud in a data queue;

If the current frame point cloud belongs to the second type, determining the current frame point cloud as the first point cloud;

The second point cloud is filtered from the data queue according to the first timestamp corresponding to the first point cloud.

In an exemplary embodiment of the present disclosure, the environment perception module 1330 selects the second point cloud from the data queue according to the first timestamp corresponding to the first point cloud, including:

Eliminating invalid point clouds temporarily stored in the data queue and having the same timestamp as the first timestamp;

Filter out target remaining point clouds whose time interval with the first timestamp is less than a preset interval threshold from the remaining point clouds in the data queue as the second point clouds.

In an exemplary embodiment of the present disclosure, the environment perception module 1330 fuses the first point cloud and the second point cloud to obtain the first fused point cloud based on the motion data within the first time range, including:

Determine the position transformation data of the self-moving device within the first time range by using a differential model based on the motion data within the first time range;

Performing posture transformation on the second point cloud according to the posture transformation data within the first time range to obtain a first transformed point cloud;

The first transformed point cloud is merged with the first point cloud to obtain the first fused point cloud.

In an exemplary embodiment of the present disclosure, the main depth camera collects the depth image data of each frame in a first working mode, and the first working mode includes a dot matrix mode.

In an exemplary embodiment of the present disclosure, the self-mobile device is also provided with an auxiliary depth camera, the main depth camera and the auxiliary depth camera are arranged at different positions of the self-mobile device, and there is an intersecting and non-overlapping field of view angle interval between the auxiliary depth camera and the main depth camera.

In an exemplary embodiment of the present disclosure, the auxiliary depth camera has the first working mode and the second working mode, and the second working mode includes an area array mode;

After the main depth camera collects each frame of depth image data, the environment perception module 1330 is configured to:

The auxiliary depth camera collects each frame of target depth image data according to the first working mode; the auxiliary depth camera is configured to: in each data collection cycle, alternately collect N frames of target depth image data according to preset N exposure parameters, and the N exposure parameters show a monotonically increasing or monotonically decreasing trend; N is an integer greater than 1;

Decoding each frame of target depth image data to obtain each frame of target point cloud corresponding to each frame of target depth image data;

Fusing the N frames of target point clouds corresponding to each data acquisition cycle to obtain a second fused point cloud;

The first fused point cloud and the second fused point cloud are re-fused to obtain a third fused point cloud, and the third fused point cloud is used as the final point cloud of each frame again, so as to perform environment perception based on the final point cloud of each frame.

In an exemplary embodiment of the present disclosure, the environment perception module 1330 collects each frame of target depth image data through the auxiliary depth camera according to the first working mode, including:

The trigger signal of the auxiliary depth camera is set to a valid level through the main depth camera, so that each frame of target depth image data is collected through the auxiliary depth camera according to the first working mode.

In an exemplary embodiment of the present disclosure, the environment perception module 1330 re-integrates the first fused point cloud and the second fused point cloud to obtain a third fused point cloud, including:

Obtaining a third timestamp corresponding to the first fused point cloud, and obtaining a fourth timestamp corresponding to the second fused point cloud;

Determine a second time range according to the third timestamp and the fourth timestamp, and obtain motion data of the mobile device within the second time range;

Based on the motion data within the second time range, the first fused point cloud and the second fused point cloud are re-fused to obtain the third fused point cloud.

In an exemplary embodiment of the present disclosure, the environment perception module 1330 re-fuses the first fused point cloud and the second fused point cloud based on the motion data within the second time range to obtain the third fused point cloud, including:

Determine the position transformation data of the self-mobile device within the second time range by using a differential model based on the motion data within the second time range;

Performing posture transformation on the first fused point cloud according to the posture transformation data within the second time range to obtain a second transformed point cloud;

The second transformed point cloud and the second fused point cloud are merged to obtain the third fused point cloud.

In an exemplary embodiment of the present disclosure, after collecting each frame of target depth image data through the auxiliary depth camera according to the first working mode, the environment perception module 1330 is configured as follows:

After a specified time interval, collecting each frame of reference depth image data according to the second working mode and the automatic exposure parameters by the auxiliary depth camera;

Decoding each frame of the reference depth image data to obtain each frame of the reference point cloud corresponding to the each frame of the reference depth image data, so as to perform environmental perception based on the each frame of the reference point cloud; the automatic exposure parameter depends on the ambient brightness;

There is a preset correlation between the specified time length and the frame rate of the auxiliary depth camera.

In an exemplary embodiment of the present disclosure, the frame rate of the auxiliary depth camera is greater than the frame rate of the main depth camera, and the frame rate of the auxiliary depth camera and the frame rate of the main depth camera satisfy a preset numerical relationship.

In an exemplary embodiment of the present disclosure, the self-mobile device is further provided with an RGB camera, the RGB camera is provided in combination with the auxiliary depth camera, and the frame rate of the RGB camera is the same as the frame rate of the main depth camera;

After the auxiliary depth camera collects each frame of target depth image data according to the first working mode, the environment perception module 1330 is configured as follows:

Each frame of image data of the self-mobile device during its movement is collected by the RGB camera, so as to perform environmental perception based on each frame of image data.

In an exemplary embodiment of the present disclosure, the environment perception module 1330 collects each frame of image data of the self-mobile device during its movement through the RGB camera, including:

The trigger signal of the RGB camera is set to a valid level through the main depth camera, so as to collect each frame of image data of the self-moving device during the moving process through the RGB camera.

The specific details of each module in the above self-mobile device have been described in detail in the corresponding environment perception method of the self-mobile device, so they will not be repeated here.

It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and/or one step may be decomposed into multiple steps, etc.

Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation methods of the present disclosure.

The present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device.

Computer-readable storage media may be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present disclosure, computer-readable storage media may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

Computer-readable storage media can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiments.

In addition, an electronic device capable of implementing the above method is also provided in an embodiment of the present disclosure.

Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "systems".

The electronic device 1400 according to this embodiment of the present disclosure is described below with reference to Fig. 14. The electronic device 1400 shown in Fig. 14 is only an example and should not bring any limitation to the functions and scope of use of the embodiment of the present disclosure.

As shown in FIG14 , the electronic device 1400 is presented in the form of a general-purpose computing device. The components of the electronic device 1400 may include, but are not limited to: the at least one processing unit 1410, the at least one storage unit 1420, a bus 1430 connecting different system components (including the storage unit 1420 and the processing unit 1410), and a display unit 1440.

The storage unit stores a program code, and the program code can be executed by the processing unit 1410, so that the processing unit 1410 executes the steps described in the "Exemplary Method" section of the present specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 1410 can execute as shown in FIG1: Step S110, during the movement of the self-mobile device, each frame of depth image data is collected by the main depth camera; the main depth camera is configured to: in each data collection cycle, alternately collect M frames of depth image data according to preset M exposure parameters, and the M exposure parameters show a monotonically increasing or monotonically decreasing trend; M is an integer greater than 1; Step S120, decode each frame of depth image data to obtain each frame of point cloud corresponding to each frame of depth image data; Step S130, fuse the M frames of point cloud corresponding to each data collection cycle to obtain a first fused point cloud, and use the first fused point cloud as the point cloud of each frame again, so as to perform environmental perception based on the point cloud of each frame.

The storage unit 1420 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 14201 and/or a cache storage unit 14202 , and may further include a read-only storage unit (ROM) 14203 .

The storage unit 1420 may also include a program/utility 14204 having a set (at least one) of program modules 14205, such program modules 14205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

Bus 1430 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

The electronic device 1400 may also communicate with one or more external devices 1500 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1400, and/or communicate with any device that enables the electronic device 1400 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input/output (I/O) interface 1450. In addition, the electronic device 1400 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and/or public networks, such as the Internet) via a network adapter 1460. As shown, the network adapter 1460 communicates with other modules of the electronic device 1400 via a bus 1430. It should be understood that, although not shown in the figure, other hardware and/or software modules may be used in conjunction with the electronic device 1400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims (10)

1. A method for environmental perception of a self-moving device, characterized in that the self-moving device is provided with a main depth camera, and the method comprises: During the movement of the self-moving device, each frame of depth image data is collected by the main depth camera; the main depth camera is configured to: in each data collection cycle, alternately collect M frames of depth image data according to preset M exposure parameters, and the M exposure parameters show a monotonically increasing or monotonically decreasing trend; M is an integer greater than 1; Decoding each frame of the depth image data to obtain each frame of point cloud corresponding to each frame of the depth image data; The M frames of point clouds corresponding to each data acquisition cycle are fused to obtain a first fused point cloud, and the first fused point cloud is used as the point cloud of each frame again to perform environmental perception based on the point cloud of each frame. 2. The method according to claim 1, characterized in that before collecting each frame of depth image data through the main depth camera, the method further comprises: Collecting preview depth image data by the main depth camera according to default exposure parameters; When the preview point cloud is obtained by decoding the preview depth image data, each frame of the depth image data is collected by the main depth camera. 3. The method according to claim 2, characterized in that the collecting of each frame of depth image data by the main depth camera comprises: Collecting, by the main depth camera, a first exposure parameter among the M exposure parameters, a first frame of depth image data in each data collection cycle; When the first frame of depth image data is decoded to obtain the first frame of point cloud, the second frame of depth image data in each data acquisition cycle is collected by the main depth camera according to the second exposure parameter of the M exposure parameters; Until the M-1th frame of depth image data is decoded to obtain the M-1th frame of point cloud, the M-th frame of depth image data in each data acquisition cycle is collected by the main depth camera according to the M-th exposure parameter among the M exposure parameters. 4. The method according to claim 3, characterized in that fusing the M frames of point clouds corresponding to each data acquisition cycle to obtain a first fused point cloud comprises: Determine a first point cloud and a second point cloud to be fused from the M frame point clouds; Determine a first time range according to a first timestamp corresponding to the first point cloud and a second timestamp corresponding to the second point cloud, and obtain motion data of the self-mobile device within the first time range; the motion data includes inertial measurement unit data and wheel speed meter data; Based on the motion data within the first time range, the first point cloud and the second point cloud are fused to obtain the first fused point cloud. 5. The method according to claim 4, characterized in that the step of determining the first point cloud and the second point cloud to be fused from the M frame point clouds comprises: When each current frame point cloud in the M frame point clouds is acquired, the type of the current frame point cloud is determined according to the type label corresponding to the current frame point cloud; the type of the current frame point cloud is determined according to the exposure parameters used when collecting the current frame depth image data corresponding to the current frame point cloud, and the type of the current frame point cloud includes a first type and a second type, and the exposure parameter associated with the first type is less than the exposure parameter associated with the second type; If the current frame point cloud belongs to the first type, temporarily storing the current frame point cloud in a data queue; If the current frame point cloud belongs to the second type, determining the current frame point cloud as the first point cloud; The second point cloud is filtered from the data queue according to the first timestamp corresponding to the first point cloud. 6. The method according to claim 5, characterized in that the step of selecting the second point cloud from the data queue according to the first timestamp corresponding to the first point cloud comprises: Eliminating invalid point clouds temporarily stored in the data queue and having the same timestamp as the first timestamp; Filter out target remaining point clouds whose time interval with the first timestamp is less than a preset interval threshold from the remaining point clouds in the data queue as the second point clouds. 7. The method according to claim 4, characterized in that the fusing the first point cloud and the second point cloud based on the motion data within the first time range to obtain the first fused point cloud comprises: Determine the position transformation data of the self-moving device within the first time range by using a differential model based on the motion data within the first time range; Performing posture transformation on the second point cloud according to the posture transformation data within the first time range to obtain a first transformed point cloud; The first transformed point cloud is merged with the first point cloud to obtain the first fused point cloud. 8. A self-moving device, characterized in that the self-moving device is provided with a main depth camera, and the self-moving device comprises: The acquisition module is configured to collect each frame of depth image data through the main depth camera during the movement of the self-mobile device; the main depth camera is configured to: in each data collection cycle, alternately collect M frames of depth image data according to preset M exposure parameters, and the M exposure parameters show a monotonically increasing or monotonically decreasing trend; M is an integer greater than 1; A decoding module, configured to decode each frame of the depth image data to obtain each frame of point cloud corresponding to each frame of the depth image data; The environment perception module is configured to fuse the M frames of point clouds corresponding to each data acquisition cycle to obtain a first fused point cloud, and use the first fused point cloud as the point cloud of each frame to perform environment perception based on the point cloud of each frame. 9. A computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for environmental perception of a mobile device as claimed in any one of claims 1 to 7 is implemented. 10. An electronic device, comprising: Processor; and A memory, configured to store executable instructions of the processor; The processor is configured to execute the environment perception method for a mobile device as described in any one of claims 1 to 7 by executing the executable instructions.