KR102890784B1 - Method and System for Realizing Digital Twin for Warehouse Using 2D Semantic Map and Object Modeling - Google Patents

Abstract

The present invention relates to a method and system for implementing a digital twin for a logistics warehouse using a 2D semantic map and object modeling, and more specifically, to a method and system for implementing a digital twin for a logistics warehouse using a 2D semantic map and object modeling, in which a service server receives 2D lidar data and image data from a logistics transport robot in a logistics warehouse, constructs a 2D semantic map including static objects, dynamic objects, and semantic information, places object modeling on objects of the 2D semantic map to implement a digital twin, and places virtual objects on the digital twin to generate robot learning data for the logistics transport robot.

Description

Method and System for Realizing a Digital Twin for a Warehouse Using 2D Semantic Map and Object Modeling

The present invention relates to a method and system for implementing a digital twin for a logistics warehouse using a 2D semantic map and object modeling, and more specifically, to a method and system for implementing a digital twin for a logistics warehouse using a 2D semantic map and object modeling, in which a service server receives 2D lidar data and image data from a logistics transport robot in a logistics warehouse, constructs a 2D semantic map including static objects, dynamic objects, and semantic information, places object modeling on objects of the 2D semantic map to implement a digital twin, and places virtual objects on the digital twin to generate robot learning data for the logistics transport robot.

With the expansion of online distribution, logistics warehouses are growing in size. Furthermore, rising minimum wages are driving the development of robotic systems for logistics automation. Previously, logistics automation required separate facilities like guidelines and QR codes.

For logistics automation, it is crucial to create a map of the warehouse, where various logistics, logistics transport robots, and workers are intertwined, and furthermore, to implement a digital twin of the warehouse.

SLAM can be used to build a map of a warehouse without adding additional facilities. SLAM technology can be used to recognize objects within the warehouse and create a map. To achieve this, an artificial neural network-based inference model is used, and images, lidar data, and other data are input into the artificial neural network to recognize objects.

LiDAR, short for 'Light Detection And Ranging' or 'Laser Imaging, Detection and Ranging', is a remote sensing technology based on laser technology. LiDAR is a technology that detects distance, direction, speed, temperature, material distribution, and concentration characteristics by measuring two travel times of emitted laser pulses to determine the distance between the sensor and the surface.

LiDAR comes in two-dimensional and three-dimensional varieties. While 3D LiDAR can be used to map warehouse interiors, it is expensive, and processing the data measured by 3D LiDAR requires complex algorithms and a long time.

Accordingly, there is a need for an invention that is cheaper than a 3D lidar, utilizes a data 2D lidar and image camera to recognize objects, create a map within a logistics warehouse, and determine the location of logistics transport robots within the warehouse, ultimately laying the foundation for logistics automation.

Meanwhile, a digital twin refers to a technology that replicates a space or object in 3D space and implements that space or object in 3D space. To implement a digital twin, the location and shape of each space and object, identical to their real-world counterparts, are required in 3D.

To automate logistics, logistics transport robots must be taught how to identify and respond to potential problems that may arise in a warehouse. Training logistics transport robots in a digital twin space will allow them to be trained more economically and efficiently in a real-world environment.

Therefore, there is a need for technology that can implement a digital twin that corresponds to an actual logistics warehouse and transform it into an optimal space for training the digital twin.

The present invention relates to a method and system for implementing a digital twin for a logistics warehouse using a 2D semantic map and object modeling, and more specifically, to a method and system for implementing a digital twin for a logistics warehouse using a 2D semantic map and object modeling, in which a service server receives 2D lidar data and image data from a logistics transport robot in a logistics warehouse, constructs a 2D semantic map including static objects, dynamic objects, and semantic information, places object modeling on objects of the 2D semantic map to implement a digital twin, and places virtual objects on the digital twin to generate robot learning data for the logistics transport robot.

In order to solve the above problem, one embodiment of the present invention is a method for implementing a digital twin for a logistics warehouse performed by a service server that performs communication with a logistics transport robot, wherein the logistics transport robot includes a 2D lidar camera capable of collecting 2D lidar data and an image camera capable of collecting image data, and receives the 2D lidar data and the image data from the logistics transport robot, and detects static objects and dynamic objects based on the 2D lidar data and the image data; a 2D semantic map construction step that constructs a 2D semantic map including semantic information for a planar geometry for the logistics warehouse and objects including the static objects and dynamic objects located within the planar geometry based on the static objects, or the static objects and the dynamic objects detected in the information detection step, located inside the logistics warehouse; A method for implementing a digital twin for a logistics warehouse is provided, including: a digital twin implementation step of displaying an object modeling corresponding to each semantic information among a plurality of object models pre-stored in the service server at the location of the corresponding object on the 2D semantic map based on the semantic information of each object located on the 2D semantic map to implement a digital twin for a logistics warehouse; and a robot learning data generation step of implementing a learning digital twin by adding a virtual object displayed as an object modeling based on the digital twin and forming an environment and conditions different from the digital twin, and deploying a virtual logistics transport robot operating in the learning digital twin to generate robot learning data for the virtual logistics transport robot interacting with the object included in the learning digital twin.

In one embodiment of the present invention, the method for implementing a digital twin for the logistics warehouse may further include an object modeling generation step, and the object modeling generation step may include a modeling reception step for receiving, from a user terminal, an object modeling that is a 3D modeling corresponding to the actual shape of an object corresponding to semantic information of the static object and the dynamic object; and a modeling storage step for matching and storing the object modeling with the corresponding semantic information.

In one embodiment of the present invention, the learning digital twin includes a first learning twin and a second learning twin, and the first learning twin is implemented by adding the virtual object to the digital twin and adding object modeling corresponding to the virtual object, and the second learning twin is implemented by changing part or all of an existing object included in the digital twin and including changed object modeling for the changed object, and adding the virtual object to the digital twin and adding object modeling corresponding to the virtual object.

In one embodiment of the present invention, the robot learning data generation step may include a scenario implementation step of implementing a learning digital twin by adding a virtual object based on a preset scenario including the environment and conditions; a scenario execution step of placing the virtual logistics transport robot on the learning digital twin and receiving, from the virtual logistics transport robot, interaction data in which the virtual logistics transport robot performs the corresponding scenario and interacts with an object included in the learning digital twin; and a robot learning data generation step of generating robot learning data based on the interaction data, thereby implementing a digital twin for a logistics warehouse.

In one embodiment of the present invention, the service server further performs a location information derivation step, and the location information derivation step detects at least one of a static object and a dynamic object near the mobile robot from real-time 2D lidar data and image data received from the mobile robot, and searches the 2D semantic map based on the detected information, thereby deriving location information of the mobile robot on the 2D semantic map, and the service server detects the static object and the dynamic object based on the 2D lidar data and the image data according to a preset rule, and the preset rule may correspond to a rule for detecting an object from the 2D lidar data and the image data, searching the object in the 2D semantic map, and determining whether the object is a static object or a dynamic object.

In order to solve the above-described problem, one embodiment of the present invention is a system for implementing a digital twin for a logistics warehouse, including a logistics transport robot and a service server that performs communication with the logistics transport robot, wherein the logistics transport robot includes a 2D lidar camera capable of collecting 2D lidar data and an image camera capable of collecting image data, and the service server includes: an information detection unit that receives the 2D lidar data and the image data from the logistics transport robot, and detects static objects and dynamic objects based on the 2D lidar data and the image data; a 2D semantic map construction unit that constructs a 2D semantic map including semantic information for a planar geometry of the logistics warehouse and objects including the static objects and dynamic objects located within the planar geometry based on the static objects, or the static objects and the dynamic objects, located within the planar geometry, detected by the information detection unit; A high-speed SLAM positioning system is provided, comprising: a digital twin implementation unit that displays an object modeling corresponding to each semantic information among a plurality of object models pre-stored in the service server at the location of the corresponding object on the 2D semantic map based on the semantic information of each object located on the 2D semantic map to implement a digital twin for a logistics warehouse; and a learning unit that implements a learning digital twin by adding a virtual object displayed as an object modeling based on the digital twin and forming an environment and conditions different from the digital twin, and deploying a virtual logistics transport robot operating in the learning digital twin to generate robot learning data for the virtual logistics transport robot interacting with the object included in the learning digital twin.

According to one embodiment of the present invention, a 2D semantic map including static objects, dynamic objects, semantic information and location information of each is constructed, thereby providing an automated logistics transport robot that operates based on information included in the 2D semantic map, and ultimately providing an effect of laying the foundation for achieving logistics automation.

According to one embodiment of the present invention, since a dynamic object is derived based on 2D lidar data and image data collected by a logistics transport robot, location information for the dynamic object can be derived, thereby achieving the effect of being able to identify the location of a moving object without adding a separate device (GPS, Marker, etc.).

According to one embodiment of the present invention, since data is collected and processed by an image camera and a 2D lidar, the amount of data to be processed is small, no complex algorithm is required, so the data processing speed is fast and a 2D semantic map can be constructed more cheaply.

According to one embodiment of the present invention, a 2D semantic map includes static objects, dynamic objects, and semantic information and location information of each, and thus can provide an effect of providing a more lightweight 2D map.

According to one embodiment of the present invention, since a logistics transport robot can detect static objects and dynamic objects using real-time data received, it can have the effect of providing information that enables the logistics transport robot to avoid objects.

According to one embodiment of the present invention, since static objects and dynamic objects are derived based on a 2D semantic map, errors in object recognition in a logistics warehouse where static objects and dynamic objects are mixed can be reduced, and the effect of being able to determine the exact location of an object can be achieved.

According to one embodiment of the present invention, the service server stores object modeling for semantic information and places the object modeling at a corresponding location based on the semantic information on a 2D semantic map, thereby reducing the computational load for implementing a digital twin.

According to one embodiment of the present invention, a digital twin is implemented based on a 2D semantic map that is updated in real time, thereby providing an effect of providing a digital twin that is synchronized in real time with the corresponding logistics warehouse.

According to one embodiment of the present invention, robot learning data for an actual logistics transport robot can be derived and trained from the operations of a virtual logistics transport robot interacting with objects on a digital twin, thereby achieving the effect of achieving logistics automation.

According to one embodiment of the present invention, a user terminal can operate a logistics transport robot on a digital twin and an actual logistics transport robot, thereby achieving the effect of generating learning data by remotely controlling the logistics transport robot.

According to one embodiment of the present invention, learning data for a logistics transport robot can be generated on a digital twin, thereby reducing the operating cost for learning of the logistics transport robot and enabling efficient use of time.

FIG. 1 schematically illustrates a configuration for performing a method for implementing a digital twin for a logistics warehouse according to one embodiment of the present invention.
Figure 2 schematically illustrates detailed steps of a method for implementing a digital twin for a logistics warehouse according to one embodiment of the present invention.
Figure 3 schematically illustrates a first inference model and a second inference model according to one embodiment of the present invention.
Figure 4 schematically illustrates a 2D semantic map construction step according to one embodiment of the present invention.
Figure 5 schematically illustrates a preset rule according to one embodiment of the present invention.
Figure 6 schematically illustrates a location information derivation step according to one embodiment of the present invention.
Figure 7 schematically illustrates detailed steps of a location information derivation step according to one embodiment of the present invention.
Figure 8 schematically illustrates detailed steps of an image camera and location information derivation step according to one embodiment of the present invention.
Figure 9 schematically illustrates an object modeling generation step according to one embodiment of the present invention.
Figure 10 schematically illustrates a digital twin implementation step according to one embodiment of the present invention.
Figure 11 schematically illustrates a robot learning data generation step according to one embodiment of the present invention.
Figure 12 schematically illustrates a learning digital twin according to one embodiment of the present invention.
Figure 13 schematically illustrates detailed steps of a robot learning data generation step according to one embodiment of the present invention.
Figure 14 schematically illustrates the internal configuration of a computing device according to one embodiment of the present invention.

Hereinafter, various embodiments and/or aspects are now disclosed with reference to the drawings. In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of one or more aspects. However, it will be apparent to one skilled in the art that such aspects may be practiced without these specific details. The following description and the attached drawings detail specific exemplary aspects of one or more aspects. However, these aspects are exemplary, and it is to be understood that any of the various methods within the principles of the various aspects may be utilized, and the description is intended to encompass all such aspects and their equivalents.

Additionally, various aspects and features will be presented by systems that may include a number of devices, components, and/or modules. It is also to be understood and appreciated that various systems may include additional devices, components, and/or modules, and/or may not include all of the devices, components, and modules discussed in connection with the drawings.

The terms "embodiment," "example," "aspect," and "example" used herein may not be construed as implying that any aspect or design described is better or advantageous than other aspects or designs. The terms "part," "component," "module," "system," and "interface" used below generally refer to computer-related entities, and may refer to, for example, hardware, a combination of hardware and software, or software.

Additionally, it should be understood that the terms "comprises" and/or "comprising" imply the presence of the features and/or components, but do not preclude the presence or addition of one or more other features, components and/or groups thereof.

Additionally, terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. The term and/or includes a combination of a plurality of related described items or any of a plurality of related described items.

Additionally, in the embodiments of the present invention, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in the embodiments of the present invention.

FIG. 1 schematically illustrates a configuration for performing a method for implementing a digital twin for a logistics warehouse according to one embodiment of the present invention.

As illustrated in FIG. 1, the logistics transport robot (2000) may include a 2D lidar camera (2100) capable of collecting 2D lidar data and an image camera (2200) capable of collecting image data.

A logistics transport robot (2000) is a robot that moves within a logistics warehouse and performs tasks within the logistics warehouse, and may include a 2D lidar camera (2100), an image camera (2200), and a communication module.

The logistics transport robot (2000) in the present invention can be understood as a concept that includes various types of logistics transport means such as forklifts, carts, and robots, and can be operated autonomously, by people, and by user terminals.

The above 2D lidar camera (2100) can acquire 2D lidar data, and the 2D lidar data can include 2D point cloud data.

The image camera (2200) may include a camera capable of acquiring images, such as a stereo camera, an RGB camera, or an RGB-D camera, and may acquire image data. The image data may include an image containing three-dimensional information about objects based on pixels within the acquired image.

The above 2D lidar camera (2100) and image camera (2200) can be installed on the logistics transport robot (2000) to obtain information on various directions, such as the front, rear, and side of the logistics transport robot (2000).

The above communication module can perform communication with the service server (1000) and the user terminal (3000), transmit 2D lidar data and image data to the service server (1000), and receive judgment information, location information, commands, control signals, etc. from the service server (1000) or the user terminal (3000). That is, the logistics transport robot (2000) can be operated by a person (manager, user) through autonomous driving and the user terminal.

In the present invention, a 2D lidar camera (2100) rather than a 3D lidar camera is used to acquire and process 2D lidar data, thereby achieving the effect of improving the computation speed.

The service server (1000) is a computing device including one or more processors and one or more memories, and has a computer program in which commands for implementing a method of implementing a digital twin for a logistics warehouse are recorded, and the method can be performed by executing the computer program on the user terminal (3000).

Specifically, the service server (1000) can communicate with multiple logistics transport robots (2000) within a logistics warehouse and receive 2D lidar data and image data from the logistics transport robots (2000).

The above service server (1000) can construct a grid map based on the 2D lidar data and image data, detect static and dynamic objects, derive semantic information, construct a 2D semantic map, and implement a digital twin. Details will be described later.

In addition, the service server (1000) communicates with a user terminal (3000) to derive learning data from a high-speed SLAM positioning method and a digital twin of a logistics warehouse, thereby providing the user terminal (3000) with the grid map, 2D semantic map, location information of multiple logistics transport robots (2000), and a digital twin for the logistics warehouse.

The above user terminal (3000) may include a computing device including one or more processors and one or more memories, such as a desktop, laptop, or smartphone.

The grid map, 2D semantic map, location information of multiple logistics transport robots (2000), and digital twin provided by the service server (1000) can be displayed through a web browser, computer program, application, etc. provided in the user terminal (3000), and commands, control signals, etc. for operation within the logistics warehouse and within the digital twin can be transmitted to the logistics transport robots (2000) through the service server (1000).

In the present invention, the user terminal can operate a logistics transport robot on a digital twin, a logistics transport robot deployed in an actual logistics warehouse, and a logistics transport robot deployed in an actual logistics warehouse visualized on a digital twin.

Figure 2 schematically illustrates detailed steps of a method for implementing a digital twin for a logistics warehouse according to one embodiment of the present invention.

As illustrated in FIG. 2, a high-speed SLAM positioning method performed by a service server (1000) that performs communication with a logistics transport robot (2000) comprises: an information detection step (S100) of receiving 2D lidar data and image data from the logistics transport robot (2000), and detecting static objects and dynamic objects based on the 2D lidar data and image data; a 2D semantic map construction step (S200) of constructing a 2D semantic map including semantic information on objects including the static objects and dynamic objects located within the planar geometry of the logistics warehouse based on the static objects, or the static objects and dynamic objects, detected in the information detection step (S100); And a location information derivation step (S300) of detecting at least one static object and dynamic object near the logistics transport robot (2000) from real-time 2D lidar data and image data received from the logistics transport robot (2000), and deriving location information of the logistics transport robot (2000) on the 2D semantic map by searching the 2D semantic map based on the detected information; a digital twin implementation step (S400) of implementing a digital twin for the logistics warehouse by displaying an object modeling corresponding to each semantic information among a plurality of object models pre-stored in the service server (1000) at the location of the object on the 2D semantic map based on the semantic information of each object located on the 2D semantic map; And, based on the digital twin, a learning digital twin is implemented by adding a virtual object displayed by object modeling, forming an environment and conditions different from the digital twin, and a virtual logistics transport robot (2000) operating in the learning digital twin is placed, thereby generating robot learning data for the virtual logistics transport robot (2000) that interacts with the object included in the learning digital twin (S500);

The information detection step (S100), the 2D semantic map construction step (S200), the location information derivation step (S300), the digital twin implementation step (S400), and the robot learning data generation step (S500) can be performed by the service server (1000).

The above logistics transport robot (2000) can obtain 2D lidar data with the 2D lidar camera (2100) included in the logistics transport robot (2000) and obtain image data with the image camera (2200) while moving within the logistics warehouse.

In the information detection step (S100), the service server (1000) can receive 2D lidar data and image data from the logistics transport robot (2000), and can detect static objects and dynamic objects based on the 2D lidar data and image data.

The above 2D lidar data and image data can be received at preset time intervals according to the speed of the logistics transport robot (2000), thereby reducing the data load by not receiving unnecessary data.

Specifically, the service server (1000) may include a first inference model and a second inference model including one or more artificial intelligence networks, and may input the 2D lidar data and image data into the first inference model and the second inference model to detect static objects and dynamic objects. In addition, the service server (1000) may detect static objects and dynamic objects from the 2D lidar data and image data according to preset rules.

Objects classified as static objects may include non-moving structures such as pillars, shelves, and walls within a logistics warehouse, and objects classified as dynamic objects may include moving objects such as workers within a logistics warehouse and logistics transport robots (2000). In other words, the static objects and dynamic objects may correspond to a distinction as to whether the objects are non-moving structures, moving objects, or people.

In the 2D semantic map construction step (S200), the service server (1000) can construct a 2D semantic map based on static objects and dynamic objects for the entire received logistics warehouse.

Specifically, the service server (1000) can derive semantic information including the meaning (information) of each object based on one or more of the static objects and dynamic objects.

The above semantic information may include information (meaning) regarding the identity of the static or dynamic object. For example, the semantic information may correspond to a pillar, shelf, wall, or worker logistics transport robot (2000).

The above service server (1000) can construct a 2D semantic map containing semantic information. The 2D semantic map can include a first semantic map for static objects and a second semantic map for static and dynamic objects. Details will be described later.

In the location information derivation step (S300), the service server (1000) can receive 2D lidar data and image data in real time from the logistics transport robot (2000), and can detect at least one of static objects and dynamic objects from the real-time 2D lidar data and real-time image data. Based on the detected information, at least one of the static objects and dynamic objects can be queried in the 2D semantic map, and location information on the 2D semantic map for the location of the logistics transport robot (2000) can be derived.

In the digital twin implementation step (S400), the service server (1000) can implement a digital twin for the logistics warehouse using semantic information for each object, including static and dynamic objects, a 2D semantic map, and object models pre-stored for each of the semantic information. Further details will be described later.

In the robot learning data generation step (S500), the service server (1000) can implement a learning digital twin by adding a virtual object to the digital twin, and can generate robot learning data from interactions between a virtual logistics transport robot (2000) operating on the learning digital twin and objects operating on the learning digital twin. Details will be described later.

1. High-speed SLAM positioning method and system using 2D lidar data and image data

SLAM technology can be utilized to build a map of a logistics warehouse. While conventional SLAM technology utilizes 3D LiDAR, the present invention utilizes 2D LiDAR and an image camera (2200), resulting in faster data processing and a more cost-effective system than 3D LiDAR-based SLAM.

Figure 3 schematically illustrates a first inference model and a second inference model according to one embodiment of the present invention.

As illustrated in FIG. 3, the service server (1000) includes a first inference model and a second inference model, and the first inference model can detect static objects and dynamic objects from one image data and derive semantic information about the static objects and dynamic objects, and the second inference model can detect static objects and dynamic objects from a plurality of image data and derive semantic information about the static objects and dynamic objects.

The service server (1000) may include a first inference model and a second inference model including one or more learned artificial neural networks such as CNN, DNN, and CapsNet.

In the above information detection step (S100) and location information derivation step (S300), static object and dynamic object detection and semantic information derivation based on the 2D lidar data and image data can be performed by the first inference model and the second inference model.

That is, by inputting the above 2D lidar data or image data into the first and second inference models, static and dynamic objects can be detected and semantic information can be derived.

In Fig. 3, image data is shown as being input to the first inference model and the second inference model, but the 2D lidar data or image data may be input to the first inference model and the second inference model.

In Fig. 3, image data input is described as an example. That is, the description of Fig. 3 can also be applied to inputting 2D lidar data.

Figure 3 (A) corresponds to a drawing illustrating the first inference model.

A single image data can be input into the first inference model. The image data described below in (A) of Fig. 3 refers to a single image data.

The first inference model can detect static or dynamic objects from the objects contained in the image data and derive semantic information for each static or dynamic object.

Specifically, the first inference model can detect objects in the image data, determine whether each object is a static object or a dynamic object, and assign a static object or a dynamic object to the detected object.

Next, the first inference model can derive semantic information for each detected static object and dynamic object.

That is, when one image data is input into the first inference model, a static object or dynamic object and corresponding semantic information for each object contained in the image data can be output as output values.

For example, as in (A) of Fig. 3, the image data may include object A and a person, and the image data may be input into the first inference model.

The above first inference model detects object A for object A from the image data, detects object B for a person, determines whether object A and object B are static or dynamic objects, determines that object A is a static object that does not move, and determines that person is a dynamic object that moves, and can derive semantic information about what object A and object B are. At this time, the semantic information may correspond to object A and person.

The first inference model can be trained with images of static objects and images of dynamic objects, and can be trained with data that matches semantic information to each of the static and dynamic objects.

Figure 3 (B) corresponds to a drawing illustrating the second inference model.

The first inference model can be input with multiple image data. The image data described below in (B) of Fig. 3 refers to multiple image data.

The second inference model can detect static or dynamic objects from the objects contained in the image data and derive semantic information for each static or dynamic object.

Specifically, the second inference model can detect objects in the image data, determine whether each object is a static object or a dynamic object, and assign a static object or a dynamic object to the detected object.

Next, the second inference model can derive semantic information for each detected static object and dynamic object.

That is, when multiple image data are input into the second inference model, static objects or dynamic objects and corresponding semantic information for each object contained in the image data can be output as output values.

For example, as in (B) of Fig. 3, the image data may include object A and a person, and the image data may be input into the second inference model.

The second inference model detects object A for object A from the image data, detects object B for a person, determines whether object A and object B are static or dynamic objects, determines that object A is a static object that does not move, and determines that person is a dynamic object that moves, and can derive semantic information about what object A and object B are. At this time, the semantic information may correspond to object A and person.

The second inference model can learn by detecting changes in objects in multiple image data. That is, by comparing multiple image data, it can determine if an object is moving as a dynamic object. Furthermore, it can learn using data that matches semantic information to each static and dynamic object.

According to one embodiment of the present invention, the data input to the first inference model and the second inference model may preferably correspond to image data, and 2D lidar data may be utilized as an auxiliary means. That is, the static object and the dynamic object may include 2D lidar data. That is, by determining whether the object is the static object or the dynamic object from the image data, the static object or the dynamic object including the 2D lidar data at the time when the image data was acquired may be detected.

According to one embodiment of the present invention, the first inference model detects information from one data (2D lidar data or image data), and the second inference model detects information from multiple data, so the accuracy of the first inference model and the second inference model is relatively high in the second inference model, and the operation speed of the first inference model can be relatively high, so that the administrator can have the effect of setting the processing speed and accuracy as needed.

Figure 4 schematically illustrates a 2D semantic map construction step (S200) according to one embodiment of the present invention.

As illustrated in FIG. 4, in the 2D semantic map construction step (S200), a 2D semantic map including semantic information on objects including the static objects and dynamic objects located within the logistics warehouse and the planar geometry of the logistics warehouse can be constructed based on the static objects, or the static objects and dynamic objects located within the planar geometry, located within the logistics warehouse, detected in the information detection step (S100).

The above semantic map construction step may include a grid map derivation step for deriving a grid map, which is a planar geometry of the logistics warehouse, based on static objects and dynamic objects detected from 2D lidar data and image data for the entire interior of the logistics warehouse in the information detection step (S100); a semantic information derivation step for deriving semantic information for each of the static objects and dynamic objects detected from 2D lidar data and image data for the entire interior of the logistics warehouse; and a 2D semantic map synthesis step for constructing a 2D semantic map including semantic information corresponding to the static object or dynamic object of an object including the static object and dynamic object located in the grid map.

Specifically, the service server (1000) can receive 2D lidar data and image data for the entire interior of the logistics warehouse from one or more logistics transport robots (2000) within the logistics warehouse.

According to one embodiment of the present invention, a logistics transport robot (2000) can scan the entire logistics warehouse to collect 2D lidar data and image data for the entire logistics warehouse in order to build a 2D semantic map, and transmit the same to a service server (1000).

The service server (1000) that receives 2D lidar data and image data for the entire interior of the logistics warehouse can input the 2D lidar data and image data for the entire interior of the logistics warehouse into the first or second inference model described above to detect static objects and dynamic objects for the corresponding 2D lidar data and image data, and derive semantic information for the corresponding static objects or dynamic objects.

The above static object may include data (2D lidar data or image data) that detects the static object, and the above dynamic object may include data (2D lidar data or image data) that detects the dynamic object.

The above service server (1000) can derive a grid map, which is a planar geometry of a logistics warehouse, based on the detected static and dynamic objects.

Specifically, since the static object and dynamic object include the corresponding 2D lidar data or image data, the grid map can be derived based on at least one of the static object, 2D lidar data and image data, or at least one of the static object, dynamic object and 2D lidar data and image data.

More specifically, since the above 2D lidar data may include coordinate values for the corresponding logistics transport robot (2000) and the corresponding object, a grid map can be constructed by continuously outputting static objects and dynamic objects to the corresponding positions from the starting point by continuously acquiring static objects and dynamic objects.

That is, the above grid map may display objects including static objects and dynamic objects at their respective locations.

According to one embodiment of the present invention, the grid map may include a first grid map including only static objects, and a second grid map including both static and dynamic objects. Grid map F illustrated in FIG. 4 may correspond to the second grid map.

Afterwards, the service server (1000) can build a 2D semantic map including semantic information corresponding to static objects and dynamic objects on the grid map.

That is, the above 2D semantic map may include a static object, semantic information corresponding to the static object, a dynamic object, and semantic information corresponding to the dynamic object.

According to one embodiment of the present invention, the 2D semantic map may include a first semantic map including a static object and semantic information of the static object, and a second semantic map including a static object, semantic information of the static object, a dynamic object, and semantic information of the dynamic object.

In Fig. 4, objects corresponding to A, C, D, and E may correspond to static objects, object corresponding to B may correspond to dynamic objects, and A, B, C, D, and E may correspond to semantic information. For example, A may correspond to a pillar, B may correspond to a moving person, C and D may correspond to a shelf, and E may correspond to a wall.

According to one embodiment of the present invention, a 2D semantic map may display different information depending on one or more of the semantic information and the object.

Figure 5 schematically illustrates a preset rule according to one embodiment of the present invention.

As illustrated in FIG. 5, the service server (1000) includes a static semantic inference model and a dynamic semantic inference model, and detects static objects and dynamic objects based on the 2D lidar data and image data according to preset rules, the static semantic inference model derives semantic information about the static object from the static object, the dynamic semantic inference model derives semantic information about the dynamic object from the dynamic object, and the preset rules may correspond to rules for detecting object information from the 2D lidar data and image data, searching for the object information in the 2D semantic map, and determining whether the object information is a static object or a dynamic object.

The service server (1000) can detect static and dynamic objects by applying preset rules to 2D lidar data and image data.

Specifically, the service server (1000) can detect object information about an object from the 2D lidar data and image data. According to one embodiment of the present invention, the object information can be detected by inputting the 2D lidar data and image data into an inference model capable of deriving object information.

The above object information can be searched in a 2D semantic map that includes at least one of a static object and a dynamic object, and if there is a matching static object or dynamic object, the object information can be determined as a static object or a dynamic object based on the result.

According to one embodiment of the present invention, as illustrated in FIG. 5, the 2D semantic map may correspond to a first semantic map that includes only static objects, so that when the object information is searched, if the corresponding object information is not found in the first semantic map, it may be determined to be a dynamic object, and if the corresponding object information is found, it may be determined to be a static object.

Afterwards, the service server (1000) can input static objects judged by preset rules into a static semantic model to derive semantic information about the corresponding static objects, and can input dynamic objects judged by preset rules into a dynamic semantic model to derive semantic information about the corresponding dynamic objects.

Each of the above static semantic model and the above dynamic semantic model can be learned by semantic information matched to each static object or dynamic object.

According to one embodiment of the present invention, when a 2D semantic map is constructed, the service server (1000) can detect static objects and dynamic objects according to preset rules, thereby reducing the computational load by the first or second inference model.

Figure 6 schematically illustrates a location information derivation step (S300) according to one embodiment of the present invention.

As illustrated in FIG. 6, in the location information derivation step (S300), at least one of static objects and dynamic objects near the logistics transport robot (2000) is detected from real-time 2D lidar data and image data received from the logistics transport robot (2000), and based on the detected information, the 2D semantic map is searched, thereby deriving the location information of the logistics transport robot (2000) on the 2D semantic map.

The service server (1000) can derive location information of each of multiple logistics transport robots (2000) within the logistics warehouse and display this on the 2D semantic map.

Specifically, the service server (1000) can receive 2D lidar data and image data in real time from multiple logistics transport robots (2000) within a logistics warehouse.

From the received 2D lidar data and image data, static objects and dynamic objects near the logistics transport robot (2000) can be detected by the first or second inference model described above and preset rules, and each semantic information can be derived by the first or second inference model, the static semantic inference model, and the dynamic semantic inference model.

The service server (1000) can search for detected static objects, dynamic objects, and their respective semantic information in the 2D semantic map, and derive location information based on the search. Details on this will be described later.

According to one embodiment of the present invention, the location information can be displayed on the 2D semantic map, so that the manager can effectively identify the location of the logistics transport robot (2000) through the 2D semantic map displayed on the user terminal (3000).

According to one embodiment of the present invention, the location information of a logistics transport robot (2000) can be derived on a 2D semantic map, and semantic information can be provided on the 2D semantic map, so that the logistics transport robot (2000) can have the effect of providing auxiliary information for performing work within a logistics warehouse.

In Fig. 6, since the 2D lidar data and image data collected by the logistics transport robot (2000) include dynamic objects, the second semantic map including dynamic objects is utilized, but in cases where only static objects are included, the first semantic map can be utilized.

Figure 7 schematically illustrates detailed steps of a location information derivation step (S300) according to one embodiment of the present invention.

As illustrated in FIG. 7, the position derivation step may include a first real-time information detection step (S311) of detecting at least one static object and dynamic object near the mobile robot from the real-time 2D lidar data and image data; a first information search step (S312) of deriving semantic information for at least one of the static objects and dynamic objects detected in the first real-time information detection step (S311) and searching for the semantic information in the 2D semantic map; and a first position information derivation step (S313) of deriving position information of the logistics transport robot (2000) on the 2D semantic map based on the semantic information on the 2D semantic map if there is semantic information on the 2D semantic map that matches the semantic information in the first information search step (S312).

In the first real-time information detection step (S311), a static object, a dynamic object, or both a static object and a dynamic object can be detected from the real-time 2D lidar data and image data. In the first real-time information detection step (S311), at least one of the static object and the dynamic object can be detected by at least one of the first inference model, the second inference model, and the preset rules described above.

In the first information retrieval step (S312), semantic information can be derived from at least one of the static objects and dynamic objects detected in the first real-time information detection step (S311). This can be performed by at least one of the first inference model, the second inference model, the static semantic inference model, and the dynamic semantic inference model described above.

Next, in the first information search step (S312), the corresponding semantic information can be searched in the 2D semantic map.

If the semantic information pertains to a static object, it can be retrieved from either the first or second semantic map. However, it is preferable to retrieve it from the first semantic map to increase computation speed. If the semantic information pertains to a dynamic object, it can be retrieved from the second semantic map.

The first location information derivation step (S313) can be performed when there is semantic information on the 2D semantic map that matches the corresponding semantic information.

In the first location information derivation step (S313), the location information of the corresponding logistics transport robot (2000) on the 2D semantic map can be derived based on the semantic information on the 2D semantic map.

In another embodiment of the present invention, in the first information search step (S312), at least one of the detected static object information and dynamic objects can be searched in the 2D semantic map, and in the first location information derivation step (S313), if at least one of the detected static object information and dynamic objects exists on the 2D semantic map, location information of the corresponding logistics transport robot (2000) on the 2D semantic map can be derived based on at least one of the detected static object information and dynamic objects.

Figure 8 schematically illustrates detailed steps of an image camera (2200) and a location information derivation step (S300) according to one embodiment of the present invention.

As illustrated in FIG. 8, the image camera (2200) includes an RGB-D camera capable of measuring a distance per pixel, and the image data collected by the RGB-D camera includes distance information per pixel, and the position derivation step includes: a second real-time information detection step (S321) of detecting at least one of a static object and a dynamic object near the mobile robot from the real-time 2D lidar data and the image data; a second information inquiry step (S322) of deriving semantic information for at least one of the static objects and dynamic objects detected in the second real-time information detection step (S321) and searching for the semantic information in the 2D semantic map; And in the second information search step (S322), if there is semantic information on the 2D semantic map that matches the semantic information, a second location information derivation step for deriving location information of the corresponding logistics transport robot (2000) on the 2D semantic map based on the distance information of the real-time image data and the semantic information on the 2D semantic map may be included.

Figure 8 (A) corresponds to a drawing depicting an image camera (2200).

The image camera (2200) may correspond to a camera that can include distance information between pixels, such as an RGB-D camera.

The image data acquired by the image camera (2200) may include distance information between a plurality of objects included in the image data, and distance information between the logistics transport robot (2000) on which the image camera (2200) is installed and the plurality of objects.

Figure 8 (B) corresponds to a drawing illustrating the detailed steps of the location information derivation step (S300).

In the second real-time information detection step (S321), static objects, dynamic objects, or both static and dynamic objects can be detected from the real-time 2D lidar data and image data. In the second real-time information detection step (S321), at least one of the static objects and dynamic objects can be detected by at least one of the first inference model, the second inference model, and the preset rules described above.

In the second information retrieval step (S322), semantic information can be derived from at least one of the static objects and dynamic objects detected in the second real-time information detection step (S321). This can be performed by at least one of the first inference model, the second inference model, the static semantic inference model, and the dynamic semantic inference model described above.

Next, in the second information search step (S322), the corresponding semantic information can be searched in the 2D semantic map.

If the semantic information pertains to a static object, it can be retrieved from either the first or second semantic map. However, it is preferable to retrieve it from the first semantic map to increase computation speed. If the semantic information pertains to a dynamic object, it can be retrieved from the second semantic map.

The second location information derivation step can be performed when there is semantic information on the 2D semantic map that matches the corresponding semantic information.

In the second location information derivation step, the location information of the corresponding logistics transport robot (2000) on the 2D semantic map can be derived based on the semantic information on the 2D semantic map and the distance information of the corresponding image data.

The above distance information may correspond to the distance from the corresponding logistics transport robot (2000) to the object, and for example, as shown in FIG. 7, it may correspond to the distance Nm from the corresponding logistics transport robot (2000) to object A and the distance Mm to object B, and may be displayed as N'm and M'm to which the ratio of the corresponding 2D semantic map is applied in the 2D semantic map.

According to one embodiment of the present invention, when a circle is drawn with the object as the center and the distance information between the objects from the logistics transport robot (2000) as the radius, the overlapping portion can be derived as the location information of the logistics transport robot (2000).

In another embodiment of the present invention, in the second information search step (S322), at least one of the detected static object information and dynamic objects can be searched in the 2D semantic map, and in the second location information derivation step, if at least one of the detected static object information and dynamic objects exists on the 2D semantic map, the location information of the corresponding logistics transport robot (2000) on the 2D semantic map can be derived based on at least one of the detected static object information and dynamic objects.

2. Method and system for implementing a digital twin for a logistics warehouse performed by a service server that communicates with a logistics transport robot.

As described above, the service server (1000) of the present invention can construct a 2D semantic map including at least one static object and dynamic object for a logistics warehouse and semantic information of each object. Hereinafter, a method and system for implementing a digital twin for a logistics warehouse performed by the service server (1000) that performs communication with the logistics transport robot (2000) of the present invention will be described.

Figure 9 schematically illustrates an object modeling generation step according to one embodiment of the present invention.

As illustrated in FIG. 9, the method for implementing a digital twin for the logistics warehouse may further include an object modeling generation step, and the object modeling generation step may include a modeling reception step for receiving, from a user terminal (3000), an object modeling that is a 3D modeling corresponding to the actual shape of an object corresponding to the semantic information of the static object and the dynamic object; and a modeling storage step for matching and storing the object modeling with the corresponding semantic information.

The present invention facilitates the implementation of digital twins by placing object models at the locations of each object displayed on a generated 2D semantic map. To this end, the service server (1000) may store object models for each semantic piece of information. The object models may be generated by a user (person) or the service server (1000) and may include the actual shape of the corresponding object.

In the modeling reception step, the service server (1000) can receive object modeling created by the user from the user terminal (3000).

In the modeling storage step, the service server (1000) can match the object modeling received in the modeling reception step to semantic information corresponding to the object modeling.

Specifically, the service server (1000) stores a plurality of static objects and dynamic objects, and semantic information is stored for each static object or dynamic object, and a corresponding object modeling can be matched to each semantic information.

For example, in the service server (1000), a logistics transport robot (2000) as a dynamic object may be stored as semantic information corresponding to the logistics transport robot (2000), and an object modeling including wheels, a body, multiple cameras, etc., which are shapes corresponding to the logistics transport robot (2000) may be matched.

The above object modeling may correspond to 3D modeling created using a 3DCAD program such as Inventor, CATIA, Solidworks, or Rhino, and may be a shape corresponding to the actual shape of an object, person, etc. corresponding to the above semantic information.

According to one embodiment of the present invention, the service server (1000) stores object modeling for semantic information and places the object modeling at a corresponding location based on the semantic information on a 2D semantic map, thereby reducing the computational load for implementing a digital twin.

Figure 10 schematically illustrates a digital twin implementation step (S400) according to one embodiment of the present invention.

As illustrated in FIG. 10, the digital twin implementation step (S400) can implement a digital twin for a logistics warehouse by displaying an object modeling corresponding to each semantic information among a plurality of object models pre-stored in the service server (1000) at the location of the corresponding object on the 2D semantic map based on the semantic information of each object located on the 2D semantic map.

In the present invention, a digital twin can be implemented by utilizing a pre-generated 2D semantic map, thereby achieving the effect of enabling easier implementation of a digital twin without a complicated process.

Specifically, the 2D semantic map may include objects including static objects and dynamic objects, the objects may be displayed at corresponding locations, and the service server (1000) may store semantic information for each object according to the corresponding location, and object modeling matching the semantic information.

The service server (1000) can implement a digital twin by placing object modeling of the corresponding semantic information at the location of each object in the 2D semantic map.

As illustrated in Fig. 10, the second semantic map including both static objects and dynamic objects can include objects corresponding to A to E, and object modeling matching the corresponding semantic information of each object can be placed at the location of the corresponding object.

In Fig. 10, in the second semantic map, a digital twin can be implemented by placing an object modeling corresponding to a column, which is semantic information, at position A, an object modeling corresponding to a worker, which is semantic information, at position B, an object modeling corresponding to a shelf and box, which is semantic information, at position C, an object modeling corresponding to a pallet, which is semantic information, at position D, and an object modeling corresponding to a wall, which is semantic information, at position E.

That is, the digital twin can be synchronized with an actual logistics warehouse and object modeling on the digital twin can be performed according to a 2D semantic map that is updated in real time based on 2D lidar data and image data received from a logistics transport robot (2000). Accordingly, a real-time digital twin can be displayed on the user terminal (3000).

Figure 11 schematically illustrates a robot learning data generation step (S500) according to one embodiment of the present invention.

As illustrated in FIG. 11, the robot learning data generation step (S500) implements a learning digital twin that forms an environment and conditions different from the digital twin by adding a virtual object displayed by object modeling based on the digital twin, and deploys a virtual logistics transport robot (2000) that operates in the learning digital twin, thereby generating robot learning data for the virtual logistics transport robot (2000) that interacts with an object included in the learning digital twin, and the virtual object added in the robot learning data generation step (S500) includes at least one of a static object and a dynamic object, and the dynamic object corresponding to the virtual object can be operated by a preset operation rule or a learned artificial neural network-based object learning model.

Figure 11 (A) is a drawing illustrating the details of the robot learning data generation step (S500).

In the present invention, learning data can be derived by adding a virtual object to a digital twin corresponding to a logistics warehouse.

Specifically, a virtual object that does not exist in the logistics warehouse corresponding to the digital twin can be added, and a learning digital twin can be implemented by adding an environment and conditions different from the digital twin.

The above virtual objects can be positioned and placed by the user. Accordingly, the learning digital twin can change its environment and conditions.

Additionally, in order to change the above environment and conditions, the speed of each dynamic object, the size and position of each object, etc. can be changed.

The service server (1000) can place a logistics transport robot in the learning digital twin by visualizing it through object modeling, and the logistics transport robot placed in the learning digital twin can be operated by the user terminal (3000) or the robot learning model.

According to one embodiment of the present invention, as illustrated in FIG. 10, the learning digital twin can be displayed (A) on the user terminal (3000) as a first-person viewpoint of a logistics transport robot placed in the learning digital twin, and the logistics transport robot placed in the learning digital twin can be operated within the learning digital twin by the user terminal (3000).

The logistics transport robot deployed in the above learning digital twin can interact with objects included in the above learning digital twin, and accordingly, robot learning data on the operations of the logistics transport robot can be generated.

Figure 11 (B) corresponds to a drawing depicting the details of a virtual object.

The above virtual object may include static objects and dynamic objects. That is, the virtual object added to the learning digital twin may include one or more of a static object and a dynamic object.

The added dynamic object can operate according to preset operation rules or object learning models, and thus can exert the effect of providing an environment and conditions to the learning digital twin.

The above preset motion rules may include the speed, motion path, etc. of the dynamic object, and the dynamic object that operates by the object learning model may correspond to autonomous driving.

The above object learning model may include an inference model including one or more learned artificial neural networks such as CNN, DNN, and CapsNet, and may be learned by the robot learning data.

The above object learning model can learn by extracting motion information from object modeling on a digital twin that implements the motion of objects within a logistics warehouse in real time according to a 2D semantic map that is updated in real time. In other words, motion information of actual objects can be extracted and taught to the object learning model.

Figure 12 schematically illustrates a learning digital twin according to one embodiment of the present invention.

As illustrated in FIG. 12, the learning digital twin includes a first learning twin and a second learning twin, and the first learning twin is implemented by adding a virtual object to the digital twin and adding object modeling corresponding to the virtual object, and the second learning twin is implemented by changing some or all of an existing object included in the digital twin and including changed object modeling for the changed object, and adding the virtual object to the digital twin and adding object modeling corresponding to the virtual object.

A learning digital twin may include a first learning twin and a second learning twin.

The above first learning twin may correspond to a virtual object added to the digital twin implemented in the digital twin implementation step (S400).

The above second learning twin may correspond to a digital twin created by changing an object included in the digital twin implemented in the digital twin implementation step (S400) and adding a virtual object to the changed digital twin created by changing the object included in the digital twin.

The above-mentioned modified digital twin may include a modified object that has changed the size, speed, motion path, number, location, semantic information (type), etc. of the object included in the existing digital twin. A second learning twin can be implemented by placing a modified object model corresponding to the modified object at the location of the modified object.

Figure 13 schematically illustrates detailed steps of a robot learning data generation step (S500) according to one embodiment of the present invention.

As illustrated in FIG. 13, the robot learning data generation step (S500) may include a scenario implementation step (S510) of implementing a learning digital twin by adding a virtual object based on a preset scenario including the environment and conditions; a scenario execution step (S520) of placing the virtual logistics transport robot (2000) on the learning digital twin and receiving interaction data from the virtual logistics transport robot (2000) in which the virtual logistics transport robot (2000) performs the corresponding scenario and interacts with an object included in the learning digital twin; and a learning data generation step (S530) of generating robot learning data based on the interaction data.

The robot learning data generation step (S500) may include a scenario implementation step (S510), a scenario execution step (S520), and a learning data generation step. Since the robot learning data generation step (S500) is performed by the service server (1000), the scenario implementation step (S510), the scenario execution step (S520), and the learning data generation step (S530) may be performed by the service server (1000).

Specifically, in the scenario implementation step (S510), a virtual object can be added based on the scenario.

The above scenario may include the environment and conditions for a logistics warehouse and may include tasks to be performed by a logistics transport robot (2000).

The above environment and conditions may include the type, speed, size, and number of objects, and the above task may correspond to a task that the logistics transport robot (2000) must perform within a logistics warehouse. For example, the task may involve the logistics transport robot (2000) retrieving a specific item from a specific area under conditions of an environment that is twice as dense and 1.5 times as fast as usual.

In one embodiment of the present invention, in the scenario implementation step (S510), necessary virtual objects can be added according to the scenario. That is, the first learning twin can be implemented.

In another embodiment of the present invention, in the scenario implementation step (S510), necessary virtual objects can be added according to the scenario, and objects of an existing digital twin can be modified to create modified objects. In other words, the second learning twin can be implemented.

Next, in the scenario execution step (S520), a virtual logistics transport robot (2000) can be placed in the learning digital twin including the first learning twin or the second learning twin, and the virtual logistics transport robot (2000) performs the task included in the scenario on the learning digital twin, and when interaction data is generated by interacting with objects included in the learning digital twin, the service server (1000) can receive the interaction data.

The above virtual logistics transport robot (2000) can be operated by the operation of a user terminal (3000) or a robot learning model. That is, the virtual logistics transport robot (2000) operated by the above robot learning model can correspond to autonomous driving.

The above robot learning model may include an inference model including one or more learned artificial neural networks such as CNN, DNN, and CapsNet, and may be learned by the robot learning data and the motion information.

The above interaction data may correspond to the action data that the virtual logistics transport robot (2000) reacts to and responds to objects surrounding the virtual logistics transport robot (2000). For example, this may correspond to the action taken when a person suddenly appears in front of the robot.

In the learning data generation step (S530), the service server (1000) can generate robot learning data based on the interaction data. The robot learning data may correspond to data capable of training a learned artificial neural network inference model, including a DNN that operates a logistics transport robot (2000), such as the robot learning model.

The above robot learning data can be generated by including data surrounding the virtual logistics transport robot (2000), such as situational data, conditional data, and environmental data for the learning digital twin, in the interaction data. In other words, the robot learning data can correspond to data that enables autonomous driving, such as how to react in certain situations.

According to one embodiment of the present invention, a learning digital twin can be implemented by adding a virtual object that does not exist in the logistics warehouse to a digital twin corresponding to the logistics warehouse, thereby implementing a scenario with various environments and conditions, thereby generating robot learning data according to various situations and conditions.

Figure 14 schematically illustrates the internal configuration of a computing device according to one embodiment of the present invention.

The service server (1000) and user terminal (3000) illustrated in the above-described FIG. 1 may include components of the computing device (11000) illustrated in the above-described FIG. 14.

As illustrated in FIG. 14, the computing device (11000) may include at least one processor (11100), memory (11200), peripheral interface (11300), input/output subsystem (I/Osubsystem) (11400), power circuit (11500), and communication circuit (11600). In this case, the computing device (11000) may correspond to the service server (1000) and user terminal (3000) illustrated in FIG. 1.

The memory (11200) may include, for example, high-speed random access memory, a magnetic disk, SRAM, DRAM, ROM, flash memory, or non-volatile memory. The memory (11200) may include software modules, instruction sets, or other various data necessary for the operation of the computing device (11000).

At this time, access to the memory (11200) from other components such as the processor (11100) or peripheral interface (11300) may be controlled by the processor (11100).

The peripheral interface (11300) may couple input and/or output peripherals of the computing device (11000) to the processor (11100) and memory (11200). The processor (11100) may execute software modules or instruction sets stored in the memory (11200) to perform various functions for the computing device (11000) and process data.

The input/output subsystem can couple various input/output peripherals to the peripheral interface (11300). For example, the input/output subsystem can include a controller for coupling peripherals such as a monitor, a keyboard, a mouse, a printer, or, as needed, a touchscreen or sensor to the peripheral interface (11300). In another aspect, the input/output peripherals can be coupled to the peripheral interface (11300) without going through the input/output subsystem.

The power circuit (11500) may supply power to all or part of the components of the terminal. For example, the power circuit (11500) may include a power management system, one or more power sources such as a battery or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or any other components for power generation, management, and distribution.

The communication circuit (11600) may enable communication with another computing device using at least one external port.

Alternatively, as described above, the communication circuit (11600) may enable communication with other computing devices by transmitting and receiving RF signals, also known as electromagnetic signals, including RF circuits, as needed.

This embodiment of FIG. 14 is only an example of a computing device (11000), and the computing device (11000) may have some components illustrated in FIG. 14 omitted, may further include additional components not illustrated in FIG. 14, or may have a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may further include a touch screen or a sensor, in addition to the components illustrated in FIG. 14, and may include a circuit for RF communication of various communication methods (WiFi, 3G, LTE, Bluetooth, NFC, Zigbee, etc.) in the communication circuit (11600). Components that can be included in the computing device (11000) may be implemented as hardware including one or more signal processing or application-specific integrated circuits, software, or a combination of both hardware and software.

Methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded on a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application exclusively for mobile terminals. The application to which the present invention is applied may be installed on a service server (1000) or a user terminal (3000) through a file provided by a file distribution system. For example, the file distribution system may include a file transmission unit (not shown) that transmits the file according to a request from the service server (1000) or the user terminal (3000).

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and/or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to perform a desired operation or, independently or collectively, command the processing device. The software and/or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed across network-connected computing devices and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

According to one embodiment of the present invention, a 2D semantic map including static objects, dynamic objects, semantic information and location information of each is constructed, thereby providing an automated logistics transport robot that operates based on information included in the 2D semantic map, and ultimately providing an effect of laying the foundation for achieving logistics automation.

According to one embodiment of the present invention, since a dynamic object is derived based on 2D lidar data and image data collected by a logistics transport robot, location information for the dynamic object can be derived, thereby achieving the effect of being able to identify the location of a moving object without adding a separate device (GPS, Marker, etc.).

According to one embodiment of the present invention, since data is collected and processed by an image camera and a 2D lidar, the amount of data to be processed is small, no complex algorithm is required, so the data processing speed is fast and a 2D semantic map can be constructed more cheaply.

According to one embodiment of the present invention, a 2D semantic map includes static objects, dynamic objects, and semantic information and location information of each, and thus can provide an effect of providing a more lightweight 2D map.

According to one embodiment of the present invention, since a logistics transport robot can detect static objects and dynamic objects using real-time data received, it can have the effect of providing information that enables the logistics transport robot to avoid objects.

According to one embodiment of the present invention, since static objects and dynamic objects are derived based on a 2D semantic map, errors in object recognition in a logistics warehouse where static objects and dynamic objects are mixed can be reduced, and the effect of being able to determine the exact location of an object can be achieved.

According to one embodiment of the present invention, the service server stores object modeling for semantic information and places the object modeling at a corresponding location based on the semantic information on a 2D semantic map, thereby reducing the computational load for implementing a digital twin.

According to one embodiment of the present invention, a digital twin is implemented based on a 2D semantic map that is updated in real time, thereby providing an effect of providing a digital twin that is synchronized in real time with the corresponding logistics warehouse.

According to one embodiment of the present invention, robot learning data for an actual logistics transport robot can be derived and trained from the operations of a virtual logistics transport robot interacting with objects on a digital twin, thereby achieving the effect of achieving logistics automation.

According to one embodiment of the present invention, a user terminal can operate a logistics transport robot on a digital twin and an actual logistics transport robot, thereby achieving the effect of generating learning data by remotely controlling the logistics transport robot.

According to one embodiment of the present invention, learning data for a logistics transport robot can be generated on a digital twin, thereby reducing the operating cost for learning of the logistics transport robot and enabling efficient use of time.

Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and/or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims (6)

A method for implementing a digital twin for a logistics warehouse performed by a service server that communicates with a logistics transport robot.
The above logistics transport robot is,
It includes a 2D lidar camera capable of collecting 2D lidar data and an image camera capable of collecting image data.
An information detection step of receiving the 2D lidar data and image data from a logistics transport robot and detecting static objects and dynamic objects based on the 2D lidar data and image data;
A 2D semantic map construction step for constructing a 2D semantic map including semantic information on objects including the static objects and dynamic objects located within the logistics warehouse and the planar geometry of the logistics warehouse based on the static objects, or the static objects and dynamic objects located within the planar geometry, detected in the above information detection step;
A digital twin implementation step for implementing a digital twin for a logistics warehouse in real time by displaying an object modeling that is a 3D modeling corresponding to the class and actual shape of an object corresponding to each semantic information among a plurality of object models pre-stored in the service server based on the semantic information of each object located in the 2D semantic map; and
A robot learning data generation step is included, which creates a learning digital twin in real time by adding a virtual object displayed by object modeling based on the digital twin, forming an environment and conditions different from the digital twin, and deploying a virtual logistics transport robot operating in the learning digital twin to generate robot learning data for the virtual logistics transport robot interacting with the object included in the learning digital twin.
The above robot learning data includes situation data, condition data, environmental data, and interaction data for the learning digital twin.
The above interaction data corresponds to motion data including the motion in which the virtual logistics transport robot reacts and responds to the motion of the object included in the learning digital twin based on a pre-set scenario according to the situation data, condition data, and environment data.
The above learning digital twin includes a first learning twin and a second learning twin,
The above first learning twin is,
The virtual object is added to the digital twin and object modeling corresponding to the virtual object is added and implemented.
The above second learning twin is,
According to the above-described scenario, some or all of the existing objects included in the digital twin are changed, and the changed object modeling for the changed objects is included, and the virtual object is added to the digital twin, and the object modeling corresponding to the virtual object is added and implemented.
The above 2D semantic map construction step is:
A grid map derivation step for deriving a grid map, which is a planar geometry for the logistics warehouse, by sequentially outputting static objects and dynamic objects to the corresponding positions from a starting point based on the static objects and dynamic objects detected for the 2D lidar data and image data for the entire interior of the logistics warehouse in the above information detection step;
A semantic information derivation step for deriving semantic information for each static object and dynamic object detected from 2D lidar data and image data for the entire interior of the logistics warehouse; and
A method for implementing a digital twin for a logistics warehouse, comprising a 2D semantic map synthesis step of constructing a 2D semantic map including semantic information corresponding to the static object or dynamic object of an object including static objects and dynamic objects located within the above grid map.
In claim 1,
The method for implementing a digital twin for the above logistics warehouse further includes an object modeling generation step,
The above object modeling creation step is,
A modeling reception step for receiving, from a user terminal, an object modeling corresponding to the actual shape of an object corresponding to the semantic information of the static object and dynamic object, which is a 3D modeling; and
A method for implementing a digital twin for a logistics warehouse, comprising a modeling storage step of matching the above object modeling with the corresponding semantic information and storing it.
delete In claim 1,
The above robot learning data generation step is:
A scenario implementation step for implementing a learning digital twin by adding a virtual object based on a preset scenario including the above environment and conditions;
A scenario execution step of placing the virtual logistics transport robot on the learning digital twin and receiving interaction data from the virtual logistics transport robot in which the virtual logistics transport robot performs the scenario and interacts with an object included in the learning digital twin; and
A method for implementing a digital twin for a logistics warehouse, comprising a learning data generation step of generating robot learning data based on the above interaction data.
In claim 1,
The above service server further performs the location information derivation step,
The above location information derivation step is,
Detecting at least one static object and dynamic object near the mobile robot from real-time 2D lidar data and image data received from the mobile robot, and searching the 2D semantic map based on the detected information to derive location information of the mobile robot on the 2D semantic map,
The above service server detects static and dynamic objects based on 2D lidar data and image data according to preset rules.
The above preset rules are:
A method for implementing a digital twin for a logistics warehouse, which detects an object from the above 2D lidar data and image data, searches for the object in the above 2D semantic map, and determines whether the object is a static object or a dynamic object based on a rule.
A system for implementing a digital twin for a logistics warehouse, including a logistics transport robot and a service server that communicates with the logistics transport robot.
The above logistics transport robot is,
It includes a 2D lidar camera capable of collecting 2D lidar data and an image camera capable of collecting image data.
The above service server,
An information detection unit that receives the 2D lidar data and image data from a logistics transport robot and detects static objects and dynamic objects based on the 2D lidar data and image data;
A 2D semantic map construction unit that constructs a 2D semantic map including semantic information on objects including the static objects and dynamic objects located within the logistics warehouse and the planar geometry of the logistics warehouse based on the static objects, or the static objects and dynamic objects located within the logistics warehouse, detected by the information detection unit;
A digital twin implementation unit that implements a digital twin for a logistics warehouse in real time by displaying an object modeling that is a 3D modeling corresponding to the class and actual shape of an object corresponding to each semantic information among a plurality of object models pre-stored in the service server based on the semantic information of each object located in the 2D semantic map; and
A robot learning data generation unit is provided, which creates a learning digital twin in real time by adding a virtual object displayed by object modeling based on the digital twin, forming an environment and conditions different from the digital twin, and deploying a virtual logistics transport robot operating in the learning digital twin to generate robot learning data for the virtual logistics transport robot interacting with the object included in the learning digital twin.
The above robot learning data includes situation data, condition data, environmental data, and interaction data for the learning digital twin.
The above interaction data corresponds to motion data including the motion in which the virtual logistics transport robot reacts and responds to the motion of the object included in the learning digital twin based on a pre-set scenario according to the situation data, condition data, and environment data.
The above learning digital twin includes a first learning twin and a second learning twin,
The above first learning twin is,
The virtual object is added to the digital twin and object modeling corresponding to the virtual object is added and implemented.
The above second learning twin is,
According to the above-described scenario, some or all of the existing objects included in the digital twin are changed, and the changed object modeling for the changed objects is included, and the virtual object is added to the digital twin, and the object modeling corresponding to the virtual object is added and implemented.
The above 2D semantic map construction unit,
A grid map derivation step for deriving a grid map, which is a planar geometry for the logistics warehouse, by outputting static objects and dynamic objects to the corresponding positions from a starting point based on static objects and dynamic objects detected from 2D lidar data and image data for the entire interior of the logistics warehouse in the above information detection unit;
A semantic information derivation step for deriving semantic information for each static object and dynamic object detected from 2D lidar data and image data for the entire interior of the logistics warehouse; and
A system for implementing a digital twin for a logistics warehouse, which performs a 2D semantic map synthesis step of constructing a 2D semantic map including semantic information corresponding to the static object or dynamic object of an object including static objects and dynamic objects located within the above grid map.