KR102783986B1 - Method for estimating absolute depth based on relative depth map and optical flow - Google Patents

Abstract

The present disclosure discloses a method for estimating the absolute depth of a pixel included in an image so that an autonomous vehicle or an autonomous robot can drive while avoiding obstacles with low manufacturing cost and computational cost. Specifically, according to the present disclosure, a computing device generates optical flow information based on a first image and a second image captured by one camera, generates a relative depth map for the first image, and estimates the absolute depth for at least one pixel included in the first image based on the optical flow information and the relative depth map.

Description

{METHOD FOR ESTIMATING ABSOLUTE DEPTH BASED ON RELATIVE DEPTH MAP AND OPTICAL FLOW}

The present disclosure relates to a method for estimating the absolute depth of a pixel included in an image, and more particularly, to a method for estimating the absolute depth for at least one pixel included in an image by utilizing a relative depth map and optical flow based on a plurality of images captured by a single camera mounted on an autonomous vehicle or robot.

Autonomous vehicles or robots that move while avoiding obstacles generate driving paths by considering the distances to objects in the surrounding environment using rules or machine learning models. Therefore, the process of estimating information about the distances to objects in the surroundings from information obtained from input devices is an essential element for vehicles or robots to successfully avoid obstacles and drive.

Recently, attempts to implement obstacle avoidance driving by building an artificial intelligence model that can interpret and understand the world that can be observed visually, that is, computer vision, have increased. However, conventional approaches utilizing artificial intelligence models still require images captured from multiple different cameras or LiDAR (Light Detection And Ranging) systems to calculate the distance to objects in the external environment. However, increasing the number of mounted cameras leads to an increase in the manufacturing cost of the vehicle or robot, and the equipment for the LiDAR system requires a considerable volume for operation, which limits the miniaturization of the vehicle or robot.

Therefore, there is a demand in the art for a method of estimating information about the distance to surrounding objects, i.e., an absolute depth estimation method, which can be applied to small vehicles or robots at low manufacturing cost.

Korean registered patent KR 2566417 B1 discloses a method for calculating the length of a work path of a work performing robot.

The present disclosure is made in response to the aforementioned background technology, and aims to estimate absolute depth for at least one pixel included in an image by utilizing a relative depth map and optical flow based on a plurality of images captured by a single camera.

Meanwhile, the technical problems to be achieved by the present disclosure are not limited to the technical problems mentioned above, and various technical problems may be included within a scope obvious to those skilled in the art from the contents to be described below.

According to one embodiment of the present disclosure for achieving the above-described task, a method performed by a computing device for estimating an absolute depth of a pixel included in an image is disclosed. The method may include a step of generating optical flow information based on a first image and a second image captured by a single camera; a step of generating a relative depth map with respect to the first image; and a step of estimating an absolute depth with respect to at least one pixel included in the first image based on the optical flow information and the relative depth map.

In one embodiment, the step of estimating an absolute depth for at least one pixel included in the first image based on the optical flow information and the relative depth map may include: the step of detecting a dynamic object included in the first image based on the optical flow information; the step of calculating a transformation coefficient based on the dynamic object; and the step of estimating the absolute depth for at least one pixel included in the first image based on the transformation coefficient and the relative depth map.

In one embodiment, the step of detecting a dynamic object included in the first image based on the optical flow information may include: the step of detecting a first dynamic object included in the first image based further on position and motion information of the camera; the step of detecting a second dynamic object in an area other than the first dynamic object in the first image based on information of the first dynamic object; and the step of determining the first dynamic object and the second dynamic object as the dynamic objects.

In one embodiment, the step of calculating transformation coefficients based on the detected dynamic object may include: calculating one or more candidate transformation coefficients for each of one or more regions of the first image excluding the dynamic object; and determining the transformation coefficients based on the one or more candidate transformation coefficients.

In one embodiment, the step of determining the transform coefficient based on the one or more candidate transform coefficients may include the step of determining the transform coefficient based on a distribution of the one or more candidate transform coefficients.

In one embodiment, the method may further comprise generating a 3D map for the first image based on the estimated absolute depth for the at least one pixel.

In one embodiment, the step of generating a 3D map for the first image may include: generating a 3D map for the first image based on the estimated absolute depth and the relative depth map for the at least one pixel.

In one embodiment, the step of generating a 3D map for the first image may include: generating a 3D map for the first image further based on the optical flow information.

In one embodiment, the step of generating a 3D map for the first image further based on the optical flow information may include: the step of detecting a dynamic object included in the first image based on the optical flow; the step of generating a 3D map for the dynamic object based on the absolute depth and relative depth maps; and the step of generating a 3D map for an area of the first image excluding the dynamic object based on the optical flow information.

According to one embodiment of the present disclosure for achieving the above-described task, a computer program stored in a computer-readable storage medium is disclosed which causes a computing device to perform operations for estimating an absolute depth of a pixel included in an image. The operations may include: an operation for generating optical flow information based on a first image and a second image captured by a single camera; an operation for generating a relative depth map with respect to the first image; and an operation for estimating an absolute depth with respect to at least one pixel included in the first image based on the optical flow information and the relative depth map.

According to one embodiment of the present disclosure for achieving the above-described task, a computing device for estimating an absolute depth of a pixel included in an image is disclosed. The computing device includes one or more processors and a memory, and the one or more processors are capable of generating optical flow information based on a first image and a second image captured by one camera, generating a relative depth map for the first image, and estimating an absolute depth for at least one pixel included in the first image based on the optical flow information and the relative depth map.

FIG. 1 is a block diagram of a computing device for estimating the absolute depth of a pixel included in an image according to one embodiment of the present disclosure.
FIG. 2 is a schematic diagram illustrating a network function according to one embodiment of the present disclosure.
FIG. 3 is a flowchart illustrating a process for estimating the absolute depth of a pixel included in an image according to one embodiment of the present disclosure.
FIG. 4 is a conceptual diagram for explaining a process of estimating the absolute depth of a pixel included in an image and driving a vehicle or robot according to one embodiment of the present disclosure.
FIG. 5 is a conceptual diagram showing a detailed configuration of a preprocessing module according to one embodiment of the present disclosure.
FIG. 6 is a conceptual diagram illustrating a process of detecting a dynamic object from a first image according to one embodiment of the present disclosure.
FIG. 7 is a conceptual diagram illustrating a process of determining a transform coefficient based on one or more candidate transform coefficients in a transform coefficient calculation module according to one embodiment of the present disclosure.
FIG. 8 is a conceptual diagram illustrating a 3D map for an image according to one embodiment of the present disclosure.
FIG. 9 is a simplified, general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

The present disclosure discloses a method for estimating absolute depth for at least one pixel included in an image by utilizing a relative depth map and optical flow based on multiple images captured by a single camera.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are set forth to provide an understanding of the present disclosure. However, it will be apparent that these embodiments may be practiced without these specific descriptions.

The terms "component," "module," "system," and the like, as used herein, refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or an execution of software. For example, a component may be, but is not limited to, a procedure running on a processor, a processor, an object, a thread of execution, a program, and/or a computer. For example, an application running on a computing device and the computing device may both be components. One or more components may reside within a processor and/or a thread of execution. A component may be localized within a single computer. A component may be distributed between two or more computers. Furthermore, such components may execute from various computer-readable media having various data structures stored therein. The components may communicate via local and/or remote processes, for example, by a signal comprising one or more data packets (e.g., data from one component interacting with another component in a local system, a distributed system, and/or data transmitted via a network such as the Internet to another system via the signal).

Additionally, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X employs A or B" is intended to mean either of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, "X employs A or B" can apply to any of these cases. Furthermore, the term "and/or" as used herein should be understood to refer to and include all possible combinations of one or more of the associated items listed.

Also, the terms "comprises" and/or "comprising" should be understood to mean the presence of the features and/or components. However, it should be understood that the terms "comprises" and/or "comprising" do not exclude the presence or addition of one or more other features, components, and/or groups thereof. Also, unless otherwise specified or clear from the context to refer to the singular form, the singular form as used in the specification and claims should generally be construed to mean "one or more."

And, the term “at least one of A or B” should be interpreted to mean “if it includes only A,” “if it includes only B,” or “if it is combined in the composition of A and B.”

Those skilled in the art should additionally recognize that the various illustrative logical blocks, configurations, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as combinations of electronic hardware, computer software, or both. To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

The description of the disclosed embodiments is provided to enable a person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be apparent to a person skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not limited to the embodiments set forth herein. The present disclosure is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

In the present disclosure, optical flow may refer to a movement pattern of an object, which is a pixel or a collection of pixels, included in an image, for a plurality of images. In this case, the plurality of images may be a plurality of still pictures captured by a camera at regular time intervals, or may be consecutive frames of a video.

In the present disclosure, a relative depth map is a concept used in the fields of computer vision and image processing, and may mean information indicating the relative depth between objects included in a 2D image expressing a three-dimensional object. In general, a relative depth map may be information that relatively expresses how far an object is from a camera, etc. For example, a relative depth map may be information that expresses the relative depth of each pixel of a 2D image as a real number value between 0 and 1, and may be set such that the closer an object is to the camera, the closer the value is to 1, and the farther the object is from the camera, the closer the value is to 0.

In the present disclosure, absolute depth may mean information indicating the depth of a pixel or object included in an image. For example, for a pixel included in an image captured by a camera, the absolute depth of the pixel may be expressed as a value such as '80 cm' or '1 m'.

FIG. 1 is a block diagram of a computing device for estimating the absolute depth of a pixel included in an image according to one embodiment of the present disclosure.

The configuration of the computing device (100) illustrated in FIG. 1 is only a simplified example. In one embodiment of the present disclosure, the computing device (100) may include other configurations for performing a computing environment of the computing device (100), and only some of the disclosed configurations may constitute the computing device (100).

A computing device (100) may include a processor (110), a memory (130), and a network unit (150).

The processor (110) may be composed of one or more cores, and may include a processor for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. The processor (110) may read a computer program stored in the memory (130) and perform data processing for machine learning according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the processor (110) may perform operations for learning a neural network. The processor (110) may perform calculations for learning a neural network, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating weights of a neural network using backpropagation.

At least one of the CPU, GPGPU, and TPU of the processor (110) can process learning of a network function. For example, the CPU and the GPGPU can process learning of a network function and data classification using the network function together. In addition, in one embodiment of the present disclosure, processors of a plurality of computing devices can be used together to process learning of a network function and data classification using the network function. In addition, a computer program executed in a computing device according to one embodiment of the present disclosure can be a CPU, GPGPU, or TPU executable program.

The processor (110) can obtain the first image and the second image. In the present disclosure, the first image and the second image may be images captured by one camera. For example, in the present disclosure, one frame extracted from an image captured continuously by one camera in a moving state may be the first image, and the frame immediately before it may be the second image. Alternatively, it will be apparent to those skilled in the art that the method of the present disclosure may be implemented by having one frame be the first image and the frame immediately after it be the second image.

In the present disclosure, the processor (110) and the camera are mounted on the same device so that the processor (110) can directly obtain an image captured by the camera. As another example, the camera and the processor (110) exist in physically separate spaces, and the processor (110) can obtain an image by receiving an image captured by the camera through the network unit (150) of the computing device (100).

The processor (110) can generate optical flow information based on the first image and the second image. When the first image and the second image are continuous images captured by one camera, the optical flow information can include information on the movement pattern of pixels included in the first image and the second image.

The processor (110) can generate a relative depth map for the first image. At this time, the relative depth map of the first image can be information including the relative positional relationship between each pixel included in the first image. For example, the relative depth map can be information expressing the relative positional relationship of each pixel as a normalized value between 0 and 1.

In one embodiment of the present disclosure, the processor (110) may generate a relative depth map of the first image by utilizing a deep learning model. In this case, the deep learning model may be a pre-learned artificial neural network model that receives an image as input and adds information including a relative positional relationship for each pixel included in the image. However, the method of generating a relative depth map in the present disclosure is not limited to the method of utilizing the deep learning model as described above, and it is clear that various types of relative depth map generation algorithms known in the related art may be used.

For example, a relative depth map may be a set of information that expresses the relative positional relationship of each pixel in a first image as a real number between 0 and 1. At this time, the closer each pixel is, the closer the real number value corresponding to the pixel is to 1, and the farther away each pixel is, the closer the real number value corresponding to the pixel is to 0.

The processor (110) can estimate an absolute depth for at least one pixel included in the first image based on the optical flow information and the relative depth map. To estimate the absolute depth, the processor (110) can detect a dynamic object included in the first image based on the position and movement information of the camera that captured the first image and the second image and the optical flow information. In this case, the dynamic object may mean a pixel or a set of pixels corresponding to an object moving in a coordinate system based on the ground surface.

The processor (110) can calculate a transformation coefficient based on a dynamic object included in the first image. In the present disclosure, the transformation coefficient may be a coefficient for converting information related to relative depth, calculated based on optical flow, into absolute depth.

The processor (110) can estimate the absolute depth for at least one pixel included in the first image based on the conversion coefficient and the relative depth map. For example, if the relative depth map expresses the relative positional relationship between pixels as a real value between 0 and 1, and the closer the pixel is to the camera, the closer the real value corresponding to the pixel is to 1, and the farther the pixel is to 0, then the estimated absolute depth for one pixel included in the first image can be determined as the value of the conversion coefficient of the pixel divided by the information related to the relative depth of the pixel. Specifically, if the unit of the absolute depth is set to meters, and the relative depth of one pixel included in the first image is expressed as 0.8 and the calculated conversion coefficient is 0.5, the absolute depth of the pixel, that is, the distance between the camera and the pixel, can be estimated as 0.5/0.8=0.625m.

The processor (110) can generate a 3D map for the first image based on the estimated absolute depth for at least one pixel. In the present disclosure, the 3D map may mean information related to the spatial coordinate system position of each pixel generated for pixels included in a 2D image. An exemplary representation of the 3D map for the first image is illustrated in FIG. 8.

Through the present disclosure, it is possible to generate an absolute depth of pixels included in an image and a 3D map for the image based on two images captured by one camera. Since a method like the present disclosure does not require additional information acquired from a LiDAR system, it can be used in small autonomous vehicles or robots. In addition, since the present disclosure utilizes two images captured by one camera, the absolute depth can be estimated at a lower cost compared to a method of estimating the absolute depth using two cameras. In addition, since the absolute depth of pixels included in an image is estimated by utilizing both a relative depth map and optical flow information, the effect of high accuracy occurs compared to a method of generating 3D information of an image using only a conventional relative depth map.

According to one embodiment of the present disclosure, the memory (130) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, an SD or XD memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read-Only Memory (ROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Programmable Read-Only Memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. The computing device (100) may also operate in relation to web storage that performs the storage function of the memory (130) on the internet. The description of the above-described memory is merely an example, and the present disclosure is not limited thereto.

The network unit (150) according to one embodiment of the present disclosure can use various wired communication systems such as a public switched telephone network (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and a local area network (LAN).

In addition, the network unit (150) presented in this specification can use various wireless communication systems such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier-FDMA) and other systems.

In the present disclosure, the network unit (150) can use any type of wired or wireless communication system.

The techniques described in this specification can be used in other networks as well as the networks mentioned above.

FIG. 2 is a schematic diagram illustrating a network function according to one embodiment of the present disclosure.

Throughout this specification, the terms computational model, neural network, network function, and neural network may be used interchangeably. A neural network may be composed of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is composed of at least one node. The nodes (or neurons) constituting the neural networks may be interconnected by one or more links.

In a neural network, one or more nodes connected through a link can form a relationship of input node and output node relatively. The concept of input node and output node is relative, and any node in an output node relationship to one node can be in an input node relationship in a relationship to another node, and vice versa. As described above, the input node-to-output node relationship can be created based on a link. One or more output nodes can be connected to one input node through a link, and vice versa.

In a relationship between input nodes and output nodes connected through a single link, the data of the output node can have its value determined based on the data input to the input node. Here, the link interconnecting the input nodes and the output nodes can have a weight. The weight can be variable and can be varied by a user or an algorithm so that the neural network can perform a desired function. For example, when one or more input nodes are interconnected to one output node through each link, the output node can determine the output node value based on the values input to the input nodes connected to the output node and the weight set for the link corresponding to each input node.

As described above, a neural network is a network in which one or more nodes are interconnected through one or more links to form input node and output node relationships within the neural network. Depending on the number of nodes and links within the neural network, the relationship between the nodes and links, and the weight values assigned to each link, the characteristics of the neural network can be determined. For example, if there are two neural networks with the same number of nodes and links and different weight values for the links, the two neural networks can be recognized as being different from each other.

A neural network can be composed of a set of one or more nodes. A subset of the nodes constituting the neural network can form a layer. Some of the nodes constituting the neural network can form a layer based on distances from the initial input node. For example, a set of nodes that are n in distance from the initial input node can form an n layer. The distance from the initial input node can be defined by the minimum number of links that must be passed through to reach the node from the initial input node. However, this definition of a layer is arbitrary for the purpose of explanation, and the order of a layer in a neural network can be defined in a different way than described above. For example, a layer of nodes can be defined by the distance from the final output node.

The initial input node may refer to one or more nodes to which data is directly input without going through a link in a relationship with other nodes in the neural network. Or, in a neural network, in a relationship between nodes based on a link, it may refer to nodes that do not have other input nodes connected by a link. Similarly, the final output node may refer to one or more nodes that do not have an output node in a relationship with other nodes in the neural network. In addition, a hidden node may refer to nodes that constitute the neural network other than the initial input node and the final output node.

According to one embodiment of the present disclosure, a neural network may be a neural network in which the number of nodes in an input layer may be the same as the number of nodes in an output layer, and the number of nodes decreases and then increases as it progresses from the input layer to the hidden layer. In addition, according to another embodiment of the present disclosure, a neural network may be a neural network in which the number of nodes in an input layer may be less than the number of nodes in an output layer, and the number of nodes decreases as it progresses from the input layer to the hidden layer. In addition, according to another embodiment of the present disclosure, a neural network in which the number of nodes in an input layer may be greater than the number of nodes in an output layer, and the number of nodes increases as it progresses from the input layer to the hidden layer. In accordance with another embodiment of the present disclosure, a neural network may be a neural network in a combined form of the neural networks described above.

A deep neural network (DNN) can refer to a neural network that includes multiple hidden layers in addition to the input and output layers. Using a deep neural network, you can identify latent structures of data. That is, you can identify latent structures of photos, text, videos, voices, and music (for example, what objects are in the photo, what the content and emotion of the text are, what the content and emotion of the voice are, etc.). A deep neural network can include a convolutional neural network (CNN), a recurrent neural network (RNN), an auto encoder, a generative adversarial network (GAN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a Q network, a U network, a Siamese network, and a generative adversarial network (GAN). The description of the deep neural network described above is only an example and the present disclosure is not limited thereto.

In one embodiment of the present disclosure, the network function may include an autoencoder. The autoencoder may be a type of artificial neural network for outputting output data similar to input data. The autoencoder may include at least one hidden layer, and an odd number of hidden layers may be arranged between input and output layers. The number of nodes in each layer may be reduced from the number of nodes in the input layer to an intermediate layer called a bottleneck layer (encoding), and then may be expanded symmetrically from the bottleneck layer to the output layer (symmetrical to the input layer). The autoencoder may perform nonlinear dimensionality reduction. The number of input layers and output layers may correspond to the dimension after preprocessing of the input data. In the autoencoder structure, the number of nodes in the hidden layer included in the encoder may have a structure in which the number of nodes decreases as it gets farther away from the input layer. The number of nodes in the bottleneck layer (the layer with the fewest nodes between the encoder and decoder) may be kept above a certain number (e.g., more than half of the input layer), as too few nodes may not transmit enough information.

A neural network can learn in at least one of supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Learning of a neural network can be a process of applying knowledge to the neural network to perform a specific action.

A neural network can be trained in the direction of minimizing the error of the output. In the training of a neural network, training data is repeatedly input into the neural network, the output of the neural network for the training data and the target error are calculated, and the error of the neural network is backpropagated from the output layer of the neural network to the input layer in the direction of reducing the error, thereby updating the weights of each node of the neural network. In the case of supervised learning, training data in which the correct answer is labeled for each training data is used (i.e., labeled training data), and in the case of unsupervised learning, the correct answer may not be labeled for each training data. That is, for example, in the case of supervised learning for data classification, the training data may be data in which categories are labeled for each training data. Labeled training data is input into the neural network, and the error can be calculated by comparing the output (category) of the neural network with the label of the training data. As another example, in the case of unsupervised learning for data classification, the error can be calculated by comparing the input training data with the output of the neural network. The calculated error is backpropagated in the backward direction (i.e., from the output layer to the input layer) in the neural network, and the connection weights of each node in each layer of the neural network can be updated according to the backpropagation. The amount of change in the connection weights of each node to be updated can be determined according to the learning rate. The calculation of the neural network for the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of repetitions of the learning cycle of the neural network. For example, a high learning rate can be used in the early stage of learning of the neural network so that the neural network can quickly acquire a certain level of performance, thereby increasing efficiency, and a low learning rate can be used in the later stage of learning to increase accuracy.

In learning neural networks, learning data can generally be a subset of real data (i.e., data to be processed using the learned neural network), and therefore, there may be a learning cycle in which errors for the learning data decrease but errors for the real data increase. Overfitting is a phenomenon in which excessive learning on the learning data increases errors for the real data. For example, a neural network that learned cats by showing a yellow cat may not recognize cats when it sees cats other than yellow, which may be a type of overfitting. Overfitting can act as a cause of increasing errors in machine learning algorithms. Various optimization methods can be used to prevent overfitting. To prevent overfitting, methods such as increasing the learning data, regularization, dropout that deactivates some nodes of the network during the learning process, and utilization of batch normalization layers can be applied.

FIG. 3 is a flowchart illustrating a process for estimating the absolute depth of a pixel included in an image according to one embodiment of the present disclosure.

According to the present disclosure, a process for estimating an absolute depth of a pixel included in an image may include a step (S110) of generating optical flow information based on a first image and a second image captured by one camera, a step (S130) of generating a relative depth map for the first image, and a step (S150) of estimating an absolute depth for at least one pixel included in the first image based on the optical flow information and the relative depth map.

In step S110, the processor (110) can generate optical flow information based on the first image and the second image captured by one camera.

In step S130, the processor (110) can generate a relative depth map for the first image. An exemplary method of generating a relative depth map has been described above with reference to FIG. 1.

In step S150, the processor (110) can estimate an absolute depth for at least one pixel included in the first image based on the optical flow information and the relative depth map. To estimate the absolute depth, the processor (110) can detect a dynamic object included in the first image based on the position and movement information of the camera that captured the first image and the second image and the optical flow information.

The processor (110) can calculate the transformation coefficient based on the dynamic object included in the first image. A specific method of calculating the transformation coefficient is described below with reference to FIG. 7.

The processor (110) can estimate an absolute depth for at least one pixel included in the first image based on the transformation coefficient and the relative depth map.

The processor (110) can generate a 3D map for the first image based on the estimated absolute depth for at least one pixel.

In a first embodiment of the present disclosure, the processor (110) can estimate the absolute depth for all pixels included in the first image by applying a transformation coefficient used to estimate the absolute depth of at least one pixel included in the first image to all pixels of the first image. Thereafter, the processor (110) can generate a 3D map for the first image based on the estimated absolute depth for all pixels.

In a second embodiment of the present disclosure, the processor (110) may apply a transformation coefficient used to estimate the absolute depth of at least one pixel included in the first image to a pixel corresponding to a dynamic object in the first image to estimate the absolute depth of the pixel corresponding to the dynamic object. Thereafter, the processor (110) may generate a 3D map for the dynamic object based on the estimated absolute depth for the dynamic object. Meanwhile, the processor (110) may generate a 3D map for an area excluding the dynamic object in the first image using only optical flow information. Thereafter, the processor (110) may generate a 3D map for the entire first image by integrating the 3D map for the dynamic object and the 3D map for the area excluding the dynamic object in the first image. In the second embodiment, if a dynamic object is included in the image, the relative depth map may be further utilized only for the dynamic object, and the 3D map may be generated for an area other than the dynamic object using only optical flow information. Therefore, in the case of the second embodiment, the effect of reducing the amount of computation required to create a 3D map of an image occurs.

FIG. 4 is a conceptual diagram for explaining a process of estimating the absolute depth of a pixel included in an image and driving a vehicle or robot according to one embodiment of the present disclosure.

In the present disclosure, a method for estimating the absolute depth of a pixel included in an image can be linked to a driving system of an autonomous vehicle or robot (hereinafter referred to as 'robot').

The camera (411) of the robot can obtain the first image and the second image. At this time, the camera (411) is attached to the robot and can rotate around the central axis by the driving module (421), and can move along a certain path by the operation of the moving robot by the driving module (421).

The preprocessing module (421) can receive the first image and the second image captured by the camera (411) and generate a relative depth map and optical flow information. The detailed configuration of the preprocessing module (421) is described later with reference to FIG. 5.

The classification module (422) can model the relationship between each variable based on the relative depth map, optical flow information, and information related to the movement and position of the camera (411) generated by the driving module (421). In addition, the classification module (422) can detect a dynamic object from the first image based on the optical flow information.

The transformation coefficient calculation module (423) can calculate transformation coefficients for calculating absolute depth from a relative depth map based on optical flow information. In one embodiment of the present disclosure, the transformation coefficient calculation module (423) can calculate candidate transformation coefficients for one or more pixels constituting an image and determine transformation coefficients based on the candidate transformation coefficients. A specific method for calculating transformation coefficients will be described below with reference to FIG. 7.

The depth estimation module (424) can calculate an absolute depth for at least one pixel included in a first image acquired from the camera (411) based on the relative depth map, optical flow information, and transformation coefficients. In addition, the depth estimation module (424) can generate a 3D map for the entire first image. In the present disclosure, the transformation coefficients and the 3D map can be continuously updated at specific time intervals during operation of the robot.

In the present disclosure, candidate transformation coefficients corresponding to each pixel constituting an image are calculated, and the most appropriate value among each candidate transformation coefficient can be determined as the transformation coefficient. When the robot is stationary, the depth estimation module (424) can calculate an absolute depth for a relative depth map in the stationary state based on the transformation coefficients determined by the transformation coefficient calculation module (423) and the coordinate information of the pixel corresponding to the transformation coefficients (information on the coordinates of the pixel included in the image and the absolute depth of the corresponding pixel). In addition to the transformation coefficients, by additionally using the coordinate information of the pixel corresponding to the transformation coefficients (information on the coordinates of the pixel included in the image and the absolute depth of the corresponding pixel), when the robot is stationary but the pixel corresponding to the transformation coefficients includes a dynamic object (i.e., when the dynamic object passes by the corresponding location), an appropriate transformation coefficient can be determined again. For example, if the transformation coefficient determined at time t is 0.5 and the coordinate of the pixel with the transformation coefficient of 0.5 is (142, 322), then after the driving module (421) moves the robot at the time when the pixel with the coordinate (142, 322) is classified as a dynamic object, a new transformation coefficient can be determined from a new image obtained from the moving robot.

The driving module (421) can determine the driving path of the robot based on the 3D map for the first image from the depth estimation module (424). For example, the driving module (421) can recognize static objects and dynamic objects currently existing around the robot based on the 3D map for the first image, and determine a driving path that does not collide with each object by calculating the expected movement path of the dynamic object.

FIG. 5 is a conceptual diagram showing a detailed configuration of a preprocessing module according to one embodiment of the present disclosure.

In the present disclosure, the preprocessing module (500) may include an image receiving submodule (510), a relative depth estimation submodule (520), and an optical flow operation submodule (530).

FIG. 6 is a conceptual diagram illustrating a process of detecting a dynamic object from a first image according to one embodiment of the present disclosure.

In the present disclosure, the processor (110) can detect a dynamic object by classifying pixels included in the first image (610) into pixels (620) corresponding to dynamic objects and pixels (630) corresponding to static objects based on optical flow information and position and movement information of the camera. For example, the processor (110) can detect a dynamic object in the first image (610) by using angular velocity information of the camera, information related to an angle between the camera and the robot, and information related to the speed of the robot as position and movement information of the camera.

In one embodiment of the present disclosure, the processor (110) can determine whether a pixel or a set of pixels included in a first image is included in a dynamic object by compensating for the movement pattern of pixels included in the optical flow information and the position and movement information of the camera. For example, if the optical flow information is generated from the first image and the second image captured while the camera is stationary, the processor (110) can determine that a pixel or a set of pixels showing a moving pattern is included in a dynamic object by utilizing the information of 'stationary'.

As another example, if optical flow information is generated from the first image and the second image captured while the camera is moving at a constant speed with the robot, the processor (110) may use information about the 'robot speed and camera position' to correct the movement of each pixel included in the optical flow information, and then determine that a pixel or a set of pixels showing a moving pattern are included in a dynamic object. It is obvious to a person skilled in the art that in the process of reflecting information about the robot speed and the camera position, mutual conversion may occur between the coordinate system of the pixels included in the image, the coordinate system based on the camera, and the coordinate system based on the ground surface.

Through the above embodiment, the processor (110) can not only detect a dynamic object when the camera is stationary, but also detect an object with a relative speed of 0 with respect to a moving camera as a dynamic object.

In order to more accurately detect a dynamic object in the first image, the processor (110) may detect a first dynamic object in the first image, and additionally detect a second dynamic object in an area other than the first dynamic object in the first image based on information about the first dynamic object. For example, the processor (110) may initially detect the first dynamic object, and then assign a label to all objects (pixels or a set of pixels) included in the first image including the first dynamic object. Thereafter, the processor (110) may determine an object that exhibits a similar movement to the movement of the first dynamic object among each labeled object as the second dynamic object. That is, the processor (110) may detect a second dynamic object in an area other than the first dynamic object in the first image based on information about the first dynamic object. Thereafter, the processor (110) may determine the first dynamic object and the second dynamic object as a dynamic object.

FIG. 7 is a conceptual diagram illustrating a process of determining a transform coefficient based on one or more candidate transform coefficients in a transform coefficient calculation module according to one embodiment of the present disclosure.

In the present disclosure, one transformation coefficient (722) can be calculated for the first image (710), and candidate transformation coefficients (721) corresponding to each of a plurality of regions generated by dividing the first image (710) are calculated, and then the transformation coefficients (722) can be determined based on the candidate transformation coefficients (721).

For example, the processor (110) may divide the first image (710) into nine regions excluding dynamic objects, and generate nine candidate transformation coefficients (721) corresponding to each of the nine regions (712). Thereafter, the processor (110) may determine transformation coefficients based on the nine candidate transformation coefficients (721).

In one embodiment, the processor (110) may determine a smallest value among a plurality of candidate transformation coefficients as the transformation coefficient. The smaller the candidate transformation coefficient, the closer the absolute depth estimation value with respect to the pixel, that is, the distance between the camera and the object corresponding to the pixel is estimated. Therefore, when a small value is determined as the transformation coefficient, the robot can avoid obstacles more safely. However, the method of determining the transformation coefficient based on a plurality of candidate transformation coefficients in the present disclosure is not limited to the above example, and various methods may be used, such as selecting an appropriate value from the candidate transformation coefficients using a separate algorithm such as the RANSAC (RANdom SAmple Consensus) algorithm, or determining an average of the candidate transformation coefficients within a certain range as the transformation coefficient.

In the present disclosure, when a camera for acquiring the first image and the second image is mounted on a moving robot, the processor (110) can store the transformation coefficients calculated at each point in time in the memory (130) to generate a set of transformation coefficients. For example, the processor (110) can acquire the first image and the second image captured again by the camera at each point in time during the driving process of the robot, and calculate the transformation coefficients for each point in time. Thereafter, the processor (110) can update the set of transformation coefficients by comparing it with the sets of previously calculated transformation coefficients. By continuously calculating and updating the transformation coefficients for each point in time in this way, the performance of the robot that drives to avoid obstacles based on the absolute depth estimated through the present disclosure can be improved.

FIG. 8 is a conceptual diagram illustrating a 3D map for an image according to one embodiment of the present disclosure.

As illustrated in Fig. 8, the processor (110) can generate a 3D map on the right based on the first image on the left. A robot equipped with a camera that captures the first image and the second image can generate a driving path for obstacle avoidance based on the generated 3D map.

Meanwhile, a computer-readable medium storing a data structure according to one embodiment of the present disclosure is disclosed.

A data structure can refer to the organization, management, and storage of data that enables efficient access and modification of data. A data structure can refer to the organization of data to solve a specific problem (e.g., data retrieval in the shortest time, data storage, data modification). A data structure can also be defined as a physical or logical relationship between data elements designed to support a specific data processing function. A logical relationship between data elements can include a connection relationship between user-defined data elements. A physical relationship between data elements can include an actual relationship between data elements that are physically stored in a computer-readable storage medium (e.g., a persistent storage device). A data structure can specifically include a collection of data, a relationship between data, and a function or command that can be applied to the data. An effectively designed data structure enables a computing device to perform operations while using minimal resources of the computing device. Specifically, a computing device can increase the efficiency of operations, reading, insertion, deletion, comparison, exchange, and searching through an effectively designed data structure.

Data structures can be divided into linear data structures and nonlinear data structures depending on the form of the data structure. A linear data structure can be a structure in which only one piece of data is connected after another piece of data. Linear data structures can include lists, stacks, queues, and deques. A list can mean a series of data sets that have an internal order. A list can include a linked list. A linked list can be a data structure in which data is connected in a way that each piece of data has a pointer and is connected in a single line. In a linked list, a pointer can include information on the connection to the next or previous piece of data. A linked list can be expressed as a singly linked list, a doubly linked list, or a circular linked list depending on its form. A stack can be a data listing structure that allows limited access to data. A stack can be a linear data structure in which data can be processed (e.g., inserted or deleted) only at one end of the data structure. Data stored in a stack can be a data structure in which the later it enters, the sooner it comes out (LIFO - Last in First Out). A queue is a data list structure that allows limited access to data, and unlike a stack, it can be a data structure that comes out later (FIFO - First in First Out) as data is stored later. A deck can be a data structure that can process data at both ends of the data structure.

A nonlinear data structure can be a structure in which multiple data are connected behind one data. A nonlinear data structure can include a graph data structure. A graph data structure can be defined by vertices and edges, and an edge can include a line connecting two different vertices. A graph data structure can include a tree data structure. A tree data structure can be a data structure in which there is only one path connecting two different vertices among multiple vertices included in the tree. In other words, it can be a data structure that does not form a loop in a graph data structure.

Throughout this specification, the terms computational model, neural network, network function, and neural network may be used interchangeably. Hereinafter, they are collectively described as neural networks. The data structure may include a neural network. And the data structure including the neural network may be stored in a computer-readable medium. The data structure including the neural network may also include preprocessed data for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, loss functions for learning the neural network, etc. The data structure including the neural network may include any of the components among the above-described configurations. That is, the data structure including the neural network may be configured to include all or any combination of preprocessed data for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, loss functions for learning the neural network, etc. In addition to the above-described configurations, the data structure including the neural network may include any other information that determines the characteristics of the neural network. In addition, the data structure may include all forms of data used or generated in the computational process of the neural network, and is not limited to the above-described matters. The computer-readable medium may include a computer-readable recording medium and/or a computer-readable transmission medium. The neural network may be composed of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. The neural network is composed of at least one node.

The data structure may include data input to the neural network. The data structure including the data input to the neural network may be stored in a computer-readable medium. The data input to the neural network may include training data input during the neural network training process and/or input data input to the neural network after training is completed. The data input to the neural network may include data that has undergone preprocessing and/or data that is the target of preprocessing. The preprocessing may include a data processing process for inputting data to the neural network. Accordingly, the data structure may include data that is the target of preprocessing and data generated by the preprocessing. The above-described data structure is merely an example, and the present disclosure is not limited thereto.

The data structure may include weights of the neural network. (In this specification, the terms weight and parameter may be used interchangeably.) And the data structure including the weights of the neural network may be stored in a computer-readable medium. The neural network may include a plurality of weights. The weights may be variable and may be variable by a user or an algorithm so that the neural network may perform a desired function. For example, when one or more input nodes are interconnected to one output node by respective links, the output node may determine a data value output from the output node based on values input to the input nodes connected to the output node and weights set for links corresponding to each of the input nodes. The above-described data structure is merely an example, and the present disclosure is not limited thereto.

By way of example and not limitation, the weights may include weights that are variable during the neural network learning process and/or weights that have completed neural network learning. The weights that are variable during the neural network learning process may include weights at the start of a learning cycle and/or weights that are variable during a learning cycle. The weights that have completed neural network learning may include weights that have completed a learning cycle. Accordingly, the data structure including the weights of the neural network may include the data structure including the weights that are variable during the neural network learning process and/or weights that have completed neural network learning. Therefore, the above-described weights and/or combinations of each weight are included in the data structure including the weights of the neural network. The above-described data structures are merely examples and the present disclosure is not limited thereto.

The data structure including the weights of the neural network can be stored in a computer-readable storage medium (e.g., memory, hard disk) after going through a serialization process. Serialization can be a process of converting the data structure into a form that can be stored in the same or different computing devices and reconstructed and used later. The computing device can serialize the data structure to transmit and receive data over a network. The data structure including the weights of the serialized neural network can be reconstructed in the same computing device or another computing device through deserialization. The data structure including the weights of the neural network is not limited to serialization. Furthermore, the data structure including the weights of the neural network can include a data structure (e.g., B-Tree, Trie, m-way search tree, AVL tree, Red-Black Tree in nonlinear data structures) for increasing the efficiency of computation while minimizing the use of computing device resources. The above are examples and the present disclosure is not limited thereto.

The data structure may include hyper-parameters of the neural network. And the data structure including the hyper-parameters of the neural network may be stored in a computer-readable medium. The hyper-parameters may be variables that are changed by the user. The hyper-parameters may include, for example, a learning rate, a cost function, a number of repetitions of a learning cycle, weight initialization (e.g., setting a range of weight values to be weight initialization targets), and the number of Hidden Units (e.g., the number of hidden layers, the number of nodes in the hidden layer). The above-described data structure is only an example, and the present disclosure is not limited thereto.

FIG. 9 is a simplified, general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

Although the present disclosure has been described above as being generally implemented by a computing device, those skilled in the art will appreciate that the present disclosure may also be implemented in combination with computer-executable instructions and/or other program modules that may be executed on one or more computers and/or as a combination of hardware and software.

In general, program modules include routines, programs, components, data structures, and the like that perform particular tasks or implement particular abstract data types. Furthermore, those skilled in the art will appreciate that the methods of the present disclosure can be practiced with other computer system configurations, including single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which may be operatively connected to one or more associated devices.

The described embodiments of the present disclosure can also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Computers typically include a variety of computer-readable media. Any media that can be accessed by a computer can be a computer-readable media, and such computer-readable media includes both volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media. By way of example, and not limitation, computer-readable media can include computer-readable storage media and computer-readable transmission media. Computer-readable storage media includes both volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be accessed by a computer and used to store the desired information.

Computer-readable transmission media typically includes any information delivery media that embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism. The term modulated data signal means a signal that has one or more of its characteristics set or changed so as to encode information in the signal. By way of example, and not limitation, computer-readable transmission media includes wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above are also intended to be included within the scope of computer-readable transmission media.

An exemplary environment (1100) implementing various aspects of the present disclosure is illustrated, including a computer (1102) including a processing unit (1104), a system memory (1106), and a system bus (1108). The system bus (1108) couples system components, including but not limited to the system memory (1106), to the processing unit (1104). The processing unit (1104) may be any of a variety of commercially available processors. Dual processors and other multiprocessor architectures may also be utilized as the processing unit (1104).

The system bus (1108) may be any of several types of bus structures that may additionally be interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. The system memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). A basic input/output system (BIOS) is stored in nonvolatile memory (1110), such as ROM, EPROM, EEPROM, and the BIOS contains basic routines that help transfer information between components within the computer (1102), such as during start-up. The RAM (1112) may also include high-speed RAM, such as static RAM, for caching data.

The computer (1102) also includes an internal hard disk drive (HDD) (1114) (e.g., EIDE, SATA)—which may also be configured for external use within a suitable chassis (not shown), a magnetic floppy disk drive (FDD) (1116) (e.g., for reading from or writing to a removable diskette (1118)), and an optical disk drive (1120) (e.g., for reading from or writing to a CD-ROM disk (1122) or other high capacity optical media such as a DVD). The hard disk drive (1114), the magnetic disk drive (1116), and the optical disk drive (1120) may be connected to the system bus (1108) by a hard disk drive interface (1124), a magnetic disk drive interface (1126), and an optical drive interface (1128), respectively. An interface (1124) for implementing an external drive includes at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

These drives and their associated computer-readable media provide nonvolatile storage of data, data structures, computer-executable instructions, and the like. In the case of a computer (1102), the drives and media correspond to storing any data in a suitable digital format. While the description of computer-readable media above has referred to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those of ordinary skill in the art will appreciate that other types of media readable by a computer, such as zip drives, magnetic cassettes, flash memory cards, cartridges, and the like, may also be used in the exemplary operating environment, and that any such media may contain computer-executable instructions for performing the methods of the present disclosure.

A number of program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), may be stored in the drive and RAM (1112). All or portions of the operating system, applications, modules, and/or data may also be cached in RAM (1112). It will be appreciated that the present disclosure may be implemented in a variety of commercially available operating systems or combinations of operating systems.

A user may enter commands and information into the computer (1102) via one or more wired/wireless input devices, such as a keyboard (1138) and a pointing device such as a mouse (1140). Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, and the like. These and other input devices are often connected to the processing unit (1104) via an input device interface (1142) that is coupled to the system bus (1108), but may be connected by other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, and the like.

A monitor (1144) or other type of display device is also connected to the system bus (1108) via an interface, such as a video adapter (1146). In addition to the monitor (1144), the computer typically includes other peripheral output devices (not shown), such as speakers, a printer, and so on.

The computer (1102) may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) (1148), via wired and/or wireless communications. The remote computer(s) (1148) may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other conventional network node, and may generally include many or all of the components described for the computer (1102), but for simplicity, only the memory storage device (1150) is illustrated. The logical connections illustrated include wired/wireless connections to a local area network (LAN) (1152) and/or a larger network, such as a wide area network (WAN) (1154). Such LAN and WAN networking environments are common in offices and businesses, and facilitate enterprise-wide computer networks, such as intranets, all of which may be connected to a worldwide computer network, such as the Internet.

When used in a LAN networking environment, the computer (1102) is connected to the local network (1152) via a wired and/or wireless communications network interface or adapter (1156). The adapter (1156) may facilitate wired or wireless communications to the LAN (1152), which may also include a wireless access point installed therein for communicating with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) may include a modem (1158), be connected to a communications computing device on the WAN (1154), or have other means for establishing communications over the WAN (1154), such as via the Internet. The modem (1158), which may be internal or external and wired or wireless, is connected to the system bus (1108) via a serial port interface (1142). In a networked environment, program modules described for the computer (1102) or portions thereof may be stored in a remote memory/storage device (1150). It will be appreciated that the network connections depicted are exemplary and other means of establishing a communications link between the computers may be used.

The computer (1102) is configured to communicate with any wireless device or object that is configured and operates in wireless communication, such as a printer, a scanner, a desktop and/or portable computer, a portable data assistant (PDA), a communication satellite, any equipment or location associated with a wireless detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or may simply be an ad hoc communication between at least two devices.

Wi-Fi (Wireless Fidelity) allows you to connect to the Internet and other things without wires. Wi-Fi is a wireless technology that allows devices, such as computers, to send and receive data, indoors and outdoors, anywhere within the coverage area of a base station, similar to a cell phone. Wi-Fi networks use a wireless technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in the unlicensed 2.4 and 5 GHz radio bands, at data rates of, for example, 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual band).

Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

Those skilled in the art will appreciate that the various illustrative logical blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, various forms of programs or design code (referred to herein for convenience as software), or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

The various embodiments presented herein can be implemented as a method, an apparatus, or an article of manufacture using standard programming and/or engineering techniques. The term article of manufacture includes a computer program, a carrier, or a medium accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, the various storage media presented herein include one or more devices and/or other machine-readable media for storing information.

It is to be understood that the specific order or hierarchy of steps in the processes presented is an example of exemplary approaches. It is to be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of the present disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

The description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the scope of the disclosure. Thus, the disclosure is not intended to be limited to the embodiments disclosed herein, but is to be construed in the widest scope consistent with the principles and novel features disclosed herein.

Claims (11)

A method performed by a computing device to estimate the absolute depth of a pixel contained in an image,
A step of generating optical flow information based on first and second images captured by a single camera mounted on an autonomous vehicle;
For the first image, a step of generating a relative depth map;
A step of estimating an absolute depth for at least one pixel included in the first image based on the optical flow information and the relative depth map; and
A step of calculating a driving path of the autonomous vehicle based on the absolute depth;
Including,
A step of estimating an absolute depth for at least one pixel included in the first image based on the optical flow information and the relative depth map comprises:
A step of detecting a dynamic object included in the first image based on the optical flow information;
A step of calculating a conversion coefficient based on the above dynamic object; and
A step of estimating the absolute depth for at least one pixel included in the first image based on the transformation coefficient and the relative depth map;
Including,
Based on the above dynamic object, the steps for calculating the conversion coefficient are:
A step of calculating one or more candidate transformation coefficients for each of one or more regions of the first image excluding the dynamic object; and
A step of determining the transformation coefficient based on the one or more candidate transformation coefficients;
Including,
method. delete In paragraph 1,
Based on the optical flow information, the step of detecting a dynamic object included in the first image is:
A step of detecting a first dynamic object included in the first image further based on the position and movement information of the camera;
A step of detecting a second dynamic object in an area other than the first dynamic object in the first image based on information of the first dynamic object; and
A step of determining the first dynamic object and the second dynamic object as the dynamic objects;
Including,
method.
delete In paragraph 1,
Based on the one or more candidate transformation coefficients, the step of determining the transformation coefficients comprises:
A step of determining the transformation coefficient based on a distribution of the one or more candidate transformation coefficients;
Including,
method. In paragraph 1,
A step of generating a 3D map for the first image based on the estimated absolute depth for at least one pixel;
Including more,
method.
In paragraph 6,
The steps for generating a 3D map for the first image above are:
A step of generating a 3D map for the first image based on the estimated absolute depth and the relative depth map for at least one pixel;
Including,
method.
In paragraph 7,
The steps for generating a 3D map for the first image above are:
A step of generating a 3D map for the first image further based on the optical flow information;
Including,
method.
In Article 8,
Further based on the optical flow information, the step of generating a 3D map for the first image comprises:
A step of detecting a dynamic object included in the first image based on the optical flow;
A step of generating a 3D map for the dynamic object based on the absolute depth and relative depth maps; and
A step of generating a 3D map for an area of the first image excluding the dynamic object based on the optical flow information;
Including,
method.
A computer program stored on a computer-readable storage medium that causes a computing device to perform operations for estimating the absolute depth of a pixel included in an image, said operations comprising:
An operation of generating optical flow information based on first and second images captured by a single camera mounted on an autonomous vehicle;
For the first image, an operation of generating a relative depth map;
An operation of estimating an absolute depth for at least one pixel included in the first image based on the optical flow information and the relative depth map; and
An operation of calculating a driving path of the autonomous vehicle based on the absolute depth;
Including,
An operation of estimating an absolute depth for at least one pixel included in the first image based on the optical flow information and the relative depth map comprises:
An operation of detecting a dynamic object included in the first image based on the optical flow information;
An operation for calculating a conversion coefficient based on the above dynamic object; and
An operation of estimating the absolute depth for at least one pixel included in the first image based on the transformation coefficient and the relative depth map;
Including,
Based on the above dynamic object, the operation to calculate the conversion coefficient is:
An operation of calculating one or more candidate transformation coefficients for each of one or more regions of the first image excluding the dynamic object; and
An operation of determining the transform coefficient based on the one or more candidate transform coefficients;
Including,
A computer program stored on a computer-readable storage medium. A computing device for estimating the absolute depth of a pixel contained in an image,
one or more processors; and
memory;
Including,
One or more of the above processors,
Generate optical flow information based on the first image and the second image captured by a single camera mounted on an autonomous vehicle,
For the first image above, generate a relative depth map,
Based on the optical flow information and the relative depth map, an absolute depth is estimated for at least one pixel included in the first image, and
Based on the above absolute depth, the driving path of the autonomous vehicle is calculated,
Based on the optical flow information and the relative depth map, estimating an absolute depth for at least one pixel included in the first image:
Detecting a dynamic object included in the first image based on the optical flow information;
Calculating the conversion coefficient based on the above dynamic object; and
Estimating the absolute depth for at least one pixel included in the first image based on the transformation coefficient and the relative depth map;
Including,
Based on the above dynamic object, calculating the conversion coefficient is:
Calculating one or more candidate transformation coefficients for each of one or more regions of the first image excluding the dynamic object; and
Determining the transform coefficient based on the one or more candidate transform coefficients;
Including,
Computing device.

Images