JP7577608B2 - Location determination device, location determination method, and location determination system - Google Patents

Description

The present disclosure relates to a location determination device, a location determination method, and a location determination system.

In recent years, with the advancement of IT, numerous sensors have been installed in society and an extremely large amount of data has been accumulated. In this context, various methods for utilizing accumulated video data are being considered. In particular, as the amount of video content such as photographs, videos, and images increases, there is a demand for a means to generate an environmental map from this video content and accurately identify the position of a moving object moving within that environment.

Conventionally, so-called Simultaneous Localization and Mapping (SLAM) has been known as a method for simultaneously performing self-location estimation and creating an environmental map. In SLAM, a mobile object equipped with a sensor such as Lidar senses the surrounding environment while traveling, thereby creating a two-dimensional or three-dimensional environmental map.
SLAM allows a mobile object to create an environmental map in an unknown environment, and then use the map information to avoid obstacles and complete a specific task.

However, in SLAM, only static objects in the image (e.g., non-moving objects such as buildings, furniture, and walls) are used to create the environmental map; when dynamic objects (e.g., moving objects such as humans, animals, and automobiles) are present in the image, the static objects and background necessary for map creation are partially hidden, reducing the accuracy of the map.
Therefore, in order to generate a highly accurate environmental map using SLAM, it is important to identify dynamic objects in the image and exclude them from the SLAM process.

Several methods have been proposed for identifying dynamic objects in video and excluding them from SLAM processing.
For example, Chinese Patent Publication No. 111950561 (Patent Document 1) describes a technique that "The present invention discloses a method for removing semantic SLAM dynamic points based on semantic segmentation, including the following steps: 1) perform semantic segmentation on image frames using a PSPNet semantic segmentation network, and perform classification according to the pixel segmentation result. Extract ORB features of the new frame I, calculate the descriptor, and then match the new frame I with F. 3) Set a threshold M. 4) Classify the feature points according to the motion e of the dynamic points and static points. 5) Associate semantic labels with the classified static point set to construct a local semantic map. 6) Repeat steps 1) to 5) to update the local semantic map until all local maps are created and a global map is obtained. In this method, dynamic feature points in the environment that have a significant impact on the accuracy of the system can be eliminated, and static feature points can be used to build an octomap that can be interpreted with semantic information with high accuracy."

Chinese Patent Publication No. 111950561

The method described in Patent Document 1 performs semantic segmentation on all frames that make up the target image, thereby identifying and eliminating dynamic objects. In this way, after eliminating the dynamic objects, a map of the environment can be generated by performing SLAM processing using the static objects.

However, in the method described in Patent Document 1, semantic segmentation is performed on all frames of the input video, which requires a huge amount of computing resources. For this reason, it is difficult to introduce the method described in Patent Document 1 when the available computing resources of a smart device, for example, are limited. In addition, in many cases, dynamic objects are present only in some frames of the input video, so performing semantic segmentation on frames in which no dynamic objects exist is wasteful.
Furthermore, in the method described in Patent Document 1, the determination of whether an object is dynamic or static is made based only on the amount of movement of the object. For this reason, when a movable object such as a human, animal, or automobile (hereinafter, a "dynamic class object") temporarily stops moving, the object may be mistaken for a static object. If an environment map is then generated in SLAM processing using a dynamic class object that has been mistaken for a static object, the accuracy of the environment map is significantly impaired, resulting in a decrease in precision.

Therefore, the present disclosure aims to provide a position identification means capable of generating a highly accurate environmental map and identifying the position of a moving object while reducing the amount of required computing resources by predicting frames (hereinafter referred to as "dynamic frames") that are likely to contain dynamic content in an input video and performing dynamic object segmentation processing only on these dynamic frames.

In order to solve the above problem, one representative positioning device disclosed herein is a positioning device that identifies the position of a moving object moving in an arbitrary environment, and includes an image feature extraction unit that analyzes an input video showing the environment and extracts features for each of a plurality of frames included in the input video; a selective division determination unit that predicts a dynamic frame including dynamic content from the input video based on any one of the amount of movement between the plurality of frames included in the input video, the movement speed of the imaging unit that captured the input video, and the ratio of pixels of an object corresponding to a dynamic class in the frame; a dynamic object division unit that performs a dynamic object division process on the dynamic frame and generates image semantic information indicating whether the pixel belongs to an object corresponding to a dynamic class or an object corresponding to a static class for each pixel in the dynamic frame; a feature classification unit that classifies each feature as a dynamic feature or a static feature based on the semantic information of the image and the features; and a SLAM unit that generates a three-dimensional map showing the environment in a three-dimensional manner and first moving object position information indicating the position and orientation of the moving object based on the static features and the semantic information of the image.

According to the present disclosure, by predicting dynamic frames in an input video that are likely to contain dynamic content and performing dynamic object segmentation processing only on these dynamic frames, it is possible to provide a position identification means that can generate a highly accurate environmental map and identify the position of a moving object while reducing the amount of computing resources required.
Other objects, configurations and effects will become apparent from the following description of the preferred embodiment of the invention.

FIG. 1 illustrates a computer system for implementing an embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of a hardware configuration of a position identifying device according to the first embodiment of the present disclosure. FIG. 3 is a block diagram illustrating a flow of a map generation process in the position identification device according to the first embodiment of the present disclosure. FIG. 4 is a diagram illustrating a flow of a position identification process in the position identification device according to the first embodiment of the present disclosure. FIG. 5 is a block diagram illustrating a flow of a dynamic content determination process in the position identifying device according to the first embodiment of the present disclosure. FIG. 6 is a block diagram illustrating a flow of a feature classification process in the position identifying device according to the first embodiment of the present disclosure. FIG. 7 is a diagram illustrating an example of a hardware configuration of a position identifying device according to a second embodiment of the present disclosure. FIG. 8 is a block diagram illustrating a flow of a moving object position information correction process in a position specifying device according to the second embodiment of the present disclosure. FIG. 9 is a block diagram illustrating a flow of a GPS association process and a map reading process in a position identifying device according to the second embodiment of the present disclosure. FIG. 10 is a diagram illustrating an example of a hardware configuration of a position identifying device according to a third embodiment of the present disclosure. FIG. 11 is a block diagram illustrating a flow of a risk management process in a position identifying device according to a third embodiment of the present disclosure. FIG. 12 is a block diagram illustrating the functions of an interface unit according to an embodiment of the present disclosure. FIG. 13 is a diagram illustrating an example of the use of skip mask parameters according to an embodiment of the present disclosure.

Below, an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are indicated by the same reference numerals.

First, referring to FIG. 1, a computer system 100 for implementing an embodiment of the present disclosure will be described. The mechanisms and devices of the various embodiments disclosed herein may be applied to any suitable computing system. The main components of the computer system 100 include one or more processors 102, memory 104, a terminal interface 112, a storage interface 113, an I/O (input/output) device interface 114, and a network interface 115. These components may be interconnected via a memory bus 106, an I/O bus 108, a bus interface unit 109, and an I/O bus interface unit 110.

Computer system 100 may include one or more general purpose programmable central processing units (CPUs) 102A and 102B, collectively referred to as processors 102. In some embodiments, computer system 100 may include multiple processors, and in other embodiments, computer system 100 may be a single CPU system. Each processor 102 executes instructions stored in memory 104 and may include an on-board cache.

In some embodiments, memory 104 may include random access semiconductor memory, storage devices, or storage media (either volatile or non-volatile) for storing data and programs. Memory 104 may store all or a portion of the programs, modules, and data structures that implement the functions described herein. For example, memory 104 may store a location application 150. In some embodiments, location application 150 may include instructions or descriptions that execute on processor 102 the functions described below.

In some embodiments, the location application 150 may be implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and/or other physical hardware devices instead of or in addition to a processor-based system. In some embodiments, the location application 150 may include data other than instructions or descriptions. In some embodiments, cameras, sensors, or other data input devices (not shown) may be provided to communicate directly with the bus interface unit 109, the processor 102, or other hardware of the computer system 100.

Computer system 100 may include a bus interface unit 109 that provides communication between processor 102, memory 104, display system 124, and I/O bus interface unit 110. I/O bus interface unit 110 may be coupled to an I/O bus 108 for transferring data to and from various I/O units. I/O bus interface unit 110 may communicate via I/O bus 108 with multiple I/O interface units 112, 113, 114, and 115, also known as I/O processors (IOPs) or I/O adapters (IOAs).

The display system 124 may include a display controller, a display memory, or both. The display controller may provide video, audio, or both data to the display device 126. The computer system 100 may also include one or more sensors or other devices configured to collect data and provide the data to the processor 102.

For example, computer system 100 may include biometric sensors to collect data such as heart rate data or stress level data, environmental sensors to collect data such as humidity data, temperature data, or pressure data, and motion sensors to collect acceleration data, movement data, etc. Other types of sensors may also be used. Display system 124 may be connected to a display device 126, such as a standalone display screen, a television, a tablet, or a handheld device.

The I/O interface unit provides the ability to communicate with various storage or I/O devices. For example, the terminal interface unit 112 can be fitted with user I/O devices 116, such as user output devices, such as a video display, a television with speakers, and user input devices, such as a keyboard, a mouse, a keypad, a touchpad, a trackball, buttons, a light pen, or other pointing devices. A user may use a user interface to input input data or instructions to the user I/O devices 116 and the computer system 100 and receive output data from the computer system 100 by manipulating the user input devices. The user interface may be displayed on a display, played through speakers, or printed via a printer, for example, via the user I/O devices 116.

The storage interface 113 allows for the attachment of one or more disk drives or direct access storage devices 117 (usually magnetic disk drive storage devices, but may also be an array of disk drives or other storage devices configured to appear as a single disk drive). In some embodiments, the storage device 117 may be implemented as any secondary storage device. The contents of the memory 104 may be stored in the storage device 117 and retrieved from the storage device 117 as needed. The I/O device interface 114 may provide an interface to other I/O devices such as printers, fax machines, etc. The network interface 115 may provide a communications path so that the computer system 100 and other devices can communicate with each other. This communications path may be, for example, a network 130.

In some embodiments, computer system 100 may be a device that receives requests from other computer systems (clients) without a direct user interface, such as a multi-user mainframe computer system, a single-user system, or a server computer. In other embodiments, computer system 100 may be a desktop computer, a portable computer, a laptop, a tablet computer, a pocket computer, a telephone, a smartphone, or any other suitable electronic device.

Next, the hardware configuration of the location identification device according to the first embodiment of the present disclosure will be described with reference to FIG. 2.

FIG. 2 is a diagram illustrating an example of a hardware configuration of the position identifying device 200 according to the first embodiment of the present disclosure.
The location determination device 200 shown in FIG. 2 is a device configured to execute the location determination means according to an embodiment of the present disclosure, and may be realized, for example, as a smartphone, tablet, laptop, dedicated circuit, etc., but is not limited thereto.
In addition, the positioning device 200 according to the embodiment of the present disclosure is provided in a moving object 275 and is assumed to move in an environment together with the moving object 275. The moving object 275 is not particularly limited as long as it is capable of autonomous movement, such as a human being, an animal, or a robot with an automatic driving function. Furthermore, the form of the positioning device 200 may be determined according to the moving object 275. For example, when the moving object 275 is a human being, the positioning device 200 may be realized as a personal terminal such as a smartphone, a tablet, or a notebook computer, and when the moving object 275 is a robot, the positioning device 200 may be a dedicated circuit included in the robot.

In addition, in this disclosure, "environment" refers to the environment that is the subject of map generation and location identification of a moving object 275, and may include any environment, such as a city, a park, a factory, an office building, a station, an airport, etc.
In addition, in this disclosure, an "environmental map" refers to a map that describes information about the surrounding environment, such as buildings, structures, fixed objects, elevation differences, and the presence or absence of obstacles that affect movement.

As shown in FIG. 2, the position identification device 200 includes a photographing unit 205, a processor 210, a memory 215, an image feature extraction unit 220, a selective division determination unit 225, a dynamic object division unit 230, a feature classification unit 235, a SLAM unit 240, a position identification unit 245, and an interface unit 250.

The photographing unit 205 is a functional unit for acquiring an image showing the environment (hereinafter referred to as an "input image"). For example, the photographing unit 205 may be a laser range scanner (range sensor, Lidar), a camera (RGB camera, infrared camera, 360-degree camera), an encoder, or the like. As an example, when the position identification device 200 is realized using a smartphone, the photographing unit 205 may be a camera built into the smartphone. In addition, it is desirable that the photographing unit 205 is disposed so that the position identification device 200 can photograph an image of the environment while moving.
The input video may be data including a number of frames, such as a moving image showing an environment, where a frame is each still image that constitutes the input video.
2 shows an example of a configuration in which the image capturing unit 205 is mounted on the position identifying device 200, but the present disclosure is not limited to this. For example, the position identifying device 200 itself may be installed as a remote server that is geographically separated from the environment, and the image capturing unit 205 may be provided on a mobile object 275 that moves within the environment.

2 shows an example of a configuration of the position identification device 200 including a single image capture unit 205, but the present disclosure is not limited thereto. For example, the position identification device 200 may include multiple image capture units 205 arranged to capture images at different angles, or multiple different types of image capture units 205 (such as an RGB camera and Lidar). As an example, the position identification device 200 may be configured to include a first image capture unit that captures the front of the moving body 275 and a second image capture unit that captures the rear, or a 360-degree camera that can capture 360 degrees around the moving body 275 and Lidar. In this way, by using a camera that captures not only the front of the moving body 275 but also the surroundings, it is possible to generate a more accurate 3D map.

The processor 210 is a computing device that executes instructions to realize the functions of each functional unit of the position determination device 200, and is substantially similar to the processor 102 shown in FIG. 1, for example, so a description thereof will be omitted here.

Memory 215 is a random access semiconductor memory, storage device, or storage medium (either volatile or non-volatile) that stores data and program instructions used by each functional unit of location identification device 200, and is substantially similar to memory 104 shown in FIG. 1, for example, so its description is omitted here.

The image feature extraction unit 220 is a functional unit for analyzing the input video acquired by the image capture unit 205 and extracting features for each of the multiple frames included in the input video. The features here are information (vectors, etc.) that represent the features of an object in a frame, and may include, for example, color, size, shape, etc. The image feature extraction unit 220 here may use any existing means, such as SIFT (Scale Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF).

The selective division determination unit 225 is a functional unit that predicts, as a dynamic frame to be subjected to dynamic object division processing (so-called semantic segmentation), a frame that is likely to contain dynamic content from among a plurality of frames included in the input video acquired by the imaging unit 205. More specifically, the selective division determination unit 225 predicts a dynamic frame that contains dynamic content from the input video based on any one of the amount of movement between a plurality of frames included in the input video, the movement speed of the imaging unit 205 that acquired the input video, and the ratio of pixels of an object corresponding to a dynamic class in the frame.
In this context, dynamic content refers to image information corresponding to dynamic objects that can be moved (e.g., humans, animals, automobiles, etc.), whereas static content refers to image information corresponding to static objects that cannot be moved (e.g., buildings, furniture, walls, etc.).
The details of the process of the selective division determination unit 225 will be described later, and therefore will not be described here.

The dynamic object splitter 230 is a functional unit that performs dynamic object splitting processing on the dynamic frame predicted by the selective split determination unit 225, and generates image semantic information for each pixel in the dynamic frame that indicates whether the pixel belongs to an object corresponding to a dynamic class or an object corresponding to a static class.
The dynamic object segmentation unit 230 here may use any existing means, such as Mask R-CNN (Mask Region-Based Convolutional Neural Network), PSPNet (Pyramid Scene Parsing Network), and BiSeNet (Bilateral Segmentation Network).

The feature classification unit 235 is a functional unit that classifies each feature of a frame of an input video as a dynamic feature (i.e., a feature belonging to a dynamic object) or a static feature (i.e., a feature belonging to a static object) based on the semantic information of the image generated by the dynamic object division unit 230 and the features extracted by the image feature extraction unit 220.
In some embodiments, the feature classifier 235 may classify a feature as a static feature or a dynamic feature based on the amount of movement of the feature between frames, where the amount of movement between frames refers to the difference between multiple frames (e.g., the difference in pixel information) that occurs due to the movement of the moving object 275.

The SLAM unit 240 is a functional unit that generates a three-dimensional map that shows the environment in a stereoscopic manner based on the static features classified by the feature classification unit 235 and the semantic information of the images generated by the dynamic object division unit 230.
The SLAM unit 240 here may use existing Simultaneous Localization and Mapping means, and is not particularly limited in this disclosure.

The position identification unit 245 is a functional unit for identifying the position of the moving body 275 in the environment based on the three-dimensional map of the environment generated by the SLAM unit 240, the static features classified by the feature classification unit 235, and the semantic information of the image generated by the dynamic object segmentation unit 230.

The interface unit 250 is a functional unit that provides various information to the user of the position identification device 200 and accepts input from the user. As an example, the interface unit 250 may be a touch screen. The interface unit 250 may display a three-dimensional map generated by the SLAM unit 240, display the position of the mobile object 275 on the three-dimensional map, accept input of the mobile object's destination, and display a recommended route to the destination.

The position identification device 200 configured as described above can predict dynamic frames in the input video that are likely to contain dynamic content, and perform dynamic object segmentation processing only on these dynamic frames, thereby providing a position identification means that can generate a highly accurate environmental map and identify the position of a moving object while reducing the amount of computing resources required.

Next, the flow of the map generation process in the position identification device according to the first embodiment of the present disclosure will be described with reference to FIG.

FIG. 3 is a block diagram showing the flow of a map generation process 300 in the position identification device 200 according to the first embodiment of the present disclosure. The map generation process 300 shown in FIG. 3 is a process for generating a three-dimensional map of an environment in which a mobile body equipped with the position identification device 200 (e.g., the mobile body 275 shown in FIG. 2) moves, and is executed by each functional unit of the position identification device 200.

First, an image capturing unit (for example, the image capturing unit 205 in the position identification device 200 shown in FIG. 2, not shown in FIG. 3) acquires an input video 305. As described above, this input video 305 is an image showing an environment in which the mobile body moves, and is composed of a plurality of frames. As an example, here, the image capturing unit may be disposed on the mobile body in a state in which it is possible to capture the environment, and the input video 305 may be acquired by capturing images of the environment while the mobile body is moving.
An input image 305 acquired by the imaging unit is input to the image feature extraction unit 220 and the selective division determination unit 225 .

The image feature extraction unit 220 receives the input video 305 captured by the image capture unit, analyzes the input video 305, and extracts features of each frame. As described above, the features here are information (e.g., vectors) that represent the features of an object in a frame, and may include, for example, color, size, shape, and the like.
The features extracted by the image feature extraction unit 220 are input to the selective division determination unit 225 .

The selective division determination unit 225 predicts, from among a plurality of frames included in the input video 305 acquired by the imaging unit, a frame that is likely to include dynamic content as a dynamic frame to be subjected to dynamic object division processing. More specifically, the selective division determination unit 225 inputs the input video 305, the features of each frame extracted by the image feature extraction unit 220, and moving object position history information 310 (described later) (note that, when the map generation processing 300 is performed for the first time, the moving object position history information 310 does not exist), and predicts a dynamic frame including dynamic content from the input video 305 based on any one of the movement amount between the plurality of frames included in the input video 305, the movement speed of the imaging unit that acquired the input video 305, and the ratio of pixels of an object corresponding to a dynamic class in the frame.
Dynamic frames that are predicted to contain dynamic content are input to a dynamic object segmenter 230 , and non-dynamic frames that are predicted to not contain dynamic content are input to a feature classifier 235 .
The details of the process of the selective division determination unit 225 will be described later, and therefore will not be described here.

The dynamic object division unit 230 performs dynamic object division processing on the dynamic frame predicted by the selective division determination unit 225, and generates, for each pixel in the dynamic frame, image semantic information 320 indicating whether the pixel belongs to an object corresponding to a dynamic class or an object corresponding to a static class. Here, the dynamic object division unit 230 may determine whether the pixel belongs to an object corresponding to a dynamic class or an object corresponding to a static class based on dynamic class information 315 indicating information of an object considered to be a dynamic class.
The image semantic information 320 generated by the dynamic object segmentation unit 230 is input to the SLAM unit 240 and the feature classifier unit 235.

The feature classifier 235 classifies each feature of the frames of the input video 305 as a dynamic feature (i.e., a feature belonging to a dynamic object) or a static feature (i.e., a feature belonging to a static object) based on the image semantic information 320 generated by the dynamic object segmenter 230 and the features extracted by the image feature extractor 220. In one embodiment, the feature classifier 235 may classify a feature as a static feature or a dynamic feature based on the amount of movement of the feature between frames.
Features classified as static features are input to the SLAM unit 240, while features classified as dynamic features are treated as outliers and excluded from the SLAM unit 240 processing.

The SLAM unit 240 generates a three-dimensional map 325 and first moving object position information 330 based on the static features classified by the feature classification unit 235 and the semantic information 320 of the image generated by the dynamic object division unit 230.
The 3D map 325 here is a 3D map corresponding to the environment shown in the input image 305. By performing the map generation process 300 shown in Figure 3 based on input images showing different parts of the same environment and combining each of the generated 3D maps, a 3D map showing the entire environment can be generated.

The first moving object position information 330 here is information indicating the position (e.g., geographic coordinates, etc.) and orientation (north, east, west, south, etc.) of the moving object. In reality, the first moving object position information 330 is information indicating the position and orientation of the imaging unit derived from the input video acquired by the imaging unit, but since the imaging unit is provided in the moving object, it is also information indicating the position and orientation of the moving object.
Furthermore, it is possible to generate moving object position history information 310 indicating the movement of the moving object/image capture unit in a time series from the change in the position of the moving object/image capture unit in a plurality of different frames, which is indicated in this first moving object position information 330. As will be described later, this moving object position history information 310 may be used when predicting dynamic frames.
The three-dimensional map 325 generated by the SLAM unit 240 and the first mobile object position information 330 are input to the interface unit 250.

The interface unit 250 outputs the three-dimensional map 325 and the first mobile object position information 330 generated by the SLAM unit 240. As an example, the interface unit 250 may display the three-dimensional map 325 and the position of the mobile object on the three-dimensional map 325. In addition, in one embodiment, the interface unit 250 may receive input of a destination of the mobile object and then display a recommended route to the destination.

According to the map generation process 300 described above, dynamic frames that are likely to contain dynamic content are predicted, and dynamic object segmentation processing is performed only on these dynamic frames, making it possible to construct a highly accurate environmental map while reducing the amount of computing resources required.

Next, the flow of the location identification process in the location identification device according to the first embodiment of the present disclosure will be described with reference to FIG.

Fig. 4 is a diagram showing a flow of a position identification process 400 in the position identification device 200 according to the first embodiment of the present disclosure. The position identification process 400 shown in Fig. 4 is a process for identifying a position of a moving body (human, robot, etc.) equipped with the position identification device 200 in an arbitrary environment by using a three-dimensional map 325 generated in advance by the map generation process 300 described with reference to Fig. 3, and is executed by each functional unit of the position identification device 200.
Note that the basic process flow of the position identification process 400 shown in FIG. 4 is similar to that of the map generation process 300 described with reference to FIG. 3, except that, instead of the SLAM unit that generates the 3D map 325, a position identification unit 245 that identifies the position of a moving object based on a pre-created 3D map 325 is used. Therefore, repeated explanations will be omitted.

The position identification unit 245 inputs and analyzes the static features classified by the feature classification unit 235, the semantic information 320 of the image generated by the dynamic object division unit 230, and the three-dimensional map 325 generated by the map generation process 300 described with reference to Figure 3 and stored in the memory 215, to generate first moving object position information 330 indicating the position (e.g., geographical coordinates, etc.) and direction (north, east, west, south, etc.) of the moving object on the three-dimensional map 325.
This first mobile object position information 330 is transferred to the interface section 250 and output by the interface section 250 .

According to the position identification process 400 described above, dynamic frames that are likely to contain dynamic content are predicted, and dynamic object segmentation processing is performed only on these dynamic frames, thereby making it possible to accurately identify the position of a moving object in an environment while reducing the amount of computing resources required.

Next, the flow of the dynamic content determination process in the location identification device according to the first embodiment of the present disclosure will be described with reference to FIG.

As described above, in the present disclosure, from among the multiple frames that make up a video frame, dynamic object segmentation processing is performed only on dynamic frames that are predicted to contain dynamic content, and non-dynamic frames that are predicted not to contain dynamic content are excluded from the dynamic object segmentation processing, thereby making it possible to accurately identify the position of a moving object in an environment while reducing the amount of computing resources required.
More specifically, the determination of dynamic content according to the embodiment of the present disclosure is based on a skip mask parameter 550, which is a parameter that specifies the number of frames in the input video to be excluded from the dynamic object segmentation process. The skip mask parameter 550 can be set and updated based on the amount of movement between multiple frames included in the input video, the movement speed of the camera that captured the input video, and the ratio of pixels of objects corresponding to dynamic classes in the frames.
By using this skip mask parameter 550, frames which are unlikely to contain dynamic content are skipped (i.e., excluded from the dynamic object segmentation process), and only frames which are likely to contain dynamic content are subjected to the dynamic object segmentation process, thereby significantly saving computing resources compared to performing dynamic object segmentation on all frames of the input video.

FIG. 5 is a block diagram showing the flow of a dynamic content determination process 500 in a localization device according to a first embodiment of the present disclosure. The dynamic content determination process 500 shown in FIG. 5 is a process for predicting dynamic frames that are likely to contain dynamic content using skip mask parameters 550, and is mainly executed by an image feature extraction unit, a selective split determination unit, and a dynamic object splitter.

First, the input video 305 is input to the selective division determination unit 225 and the image feature extraction unit 220.

As described above, the image feature extraction unit 220 inputs the input video 305 acquired by the image capture unit, analyzes the input video 305, and extracts features of each frame. These features are used when estimating the amount of movement, which will be described later.

The selective division determination unit 225 performs a modulo operation for each frame constituting the input video 305 using an image index indicating the order of the frame and the skip mask parameter 550, and if the remainder is "0" (i.e., if the remainder of the division of the image index by the skip mask parameter 550 is 0), the frame is treated as a dynamic frame and inputs it to the dynamic object division unit 230. On the other hand, if the remainder of the division operation is not "0", a movement amount estimation described below is performed on the frame.
Here, by using this division operation, a frame is input to the dynamic object division unit for each number of frames set in the skip mask parameters, and for the remaining frames, the movement amount estimation described below is performed.
Note that since the skip mask parameter 550 has not yet been set for the first frame in the input video, this first frame is directly input to the dynamic object divider 230 .

Next, for frames where the remainder of the division operation is "0", the selective split determination unit 225 performs a movement amount estimation based on the features extracted by the image feature extraction unit 220 and information on the previous frame 510 in the input video (hereinafter referred to as the "previous frame") of the target frame.
Here, the selective division determination unit 225 estimates the inter-frame movement amount e by comparing the characteristics of the target frame with the previous frame 510. As described above, the inter-frame movement amount e here means the difference between multiple frames (e.g., the difference in pixel information) that occurs due to the movement of a moving object. If the estimated movement amount e exceeds a predetermined movement amount standard M, the target frame is considered to be a dynamic frame and is input to the dynamic object division unit 230.
On the other hand, if the estimated movement amount e is less than or equal to the predetermined movement amount reference M, the target frame is regarded as a non-dynamic frame and is input to a feature classifier (not shown in FIG. 5).
In this way, by performing motion estimation on non-dynamic frames, even if dynamic content is present among frames that have been excluded by setting the skip mask parameters, the dynamic content can be detected and the frames containing this dynamic content can be reclassified as dynamic frames. In this way, frames containing dynamic content can be excluded from the SLAM unit with high accuracy, making it possible to generate a highly accurate 3D map.

As described above, the dynamic object segmentation unit 230 uses the dynamic class information 315 to perform dynamic object segmentation processing on the dynamic frame predicted by the selective segmentation determination unit 225, and generates image semantic information 320 for each pixel in the dynamic frame that indicates whether the pixel belongs to an object corresponding to a dynamic class or an object corresponding to a static class.

Next, the selective split determination unit 225 sets the skip mask parameters 550 based on the ratio of dynamic content in the frame using the semantic information 320 of the image generated by the dynamic object splitter 230. More specifically, the selective split determination unit 225 calculates the ratio of the number of pixels of the object corresponding to the dynamic class to the total number of pixels of the frame, and determines whether the calculated ratio satisfies a predetermined criterion regarding the ratio of dynamic content.
If the calculated ratio does not meet the predetermined dynamic content percentage criteria, the skip mask parameter 550 is not set/updated, whereas if the calculated ratio meets the predetermined dynamic content percentage criteria, the skip mask parameter 550 is set to "1".

In addition, the selective division determination unit 225 may set the skip mask parameter 550 using the moving object position history information 310. As described above, this moving object position history information 310 is information indicating the position of the moving object/photographing unit in a plurality of different frames. The selective division determination unit 225 estimates the moving speed of the moving object/photographing unit based on the change in the position of the moving object/photographing unit indicated in this moving object position history information 310. Thereafter, the selective division determination unit 225 may set/update the skip mask parameter 550 based on the estimated moving speed. Here, as a general rule, it is desirable for the selective division determination unit 225 to set the skip mask parameter 550 to a larger value the faster the moving speed of the moving object/photographing unit is, and to set the skip mask parameter 550 to a smaller value the slower the moving speed of the moving object/photographing unit is.
For example, in one embodiment, the selective division determination unit 225 may classify the calculated moving speed of the moving object/photographing unit into one of categories such as "Fast", "Medium", "Slow", etc. based on a preset threshold value, and set the skip mask parameter 550 according to the category. As an example, when the selective division determination unit 225 determines that the moving speed of the moving object/photographing unit is "Fast", it may set the skip mask parameter 550 to "2", when the selective division determination unit 225 determines that the moving speed of the moving object/photographing unit is "Medium", it may set the skip mask parameter 550 to "5", and when the selective division determination unit determines that the moving speed of the moving object/photographing unit is "Slow", it may set the skip mask parameter 550 to "10".

According to the dynamic content determination process 500 described above, the skip mask parameters 550 can be set and updated based on the amount of movement between multiple frames included in the input video, the movement speed of the image capture unit that captured the input video, and the ratio of pixels of objects corresponding to dynamic classes in the frames. In addition, by processing the input video using the skip mask parameters 550, it is possible to skip non-dynamic frames that do not contain dynamic content (e.g., frames with little movement between frames, frames with a slow movement speed of the moving body/image capture unit, frames with few pixels corresponding to dynamic objects, etc.), while determining dynamic frames that are likely to contain dynamic content (e.g., frames with a lot of movement between frames, frames with a fast movement speed of the moving body/image capture unit, frames with many pixels corresponding to dynamic objects, etc.).

Next, the flow of the feature classification process in the location identification device according to the first embodiment of the present disclosure will be described with reference to FIG.

Figure 6 is a block diagram showing the flow of feature classification processing 600 in the position identification device according to the first embodiment of the present disclosure. The feature classification processing 600 shown in Figure 6 is processing for classifying the features of an input video as static features or dynamic features, and is mainly executed by the image feature extraction unit 220, the selective division determination unit 225, and the feature classification unit 235.

First, the input video 305 is input to the selective division determination unit 225 and the image feature extraction unit 220.

As described above, the image feature extraction unit 220 inputs the input video 305 acquired by the image capture unit, analyzes the input video 305, and extracts features 615 for each frame. These features 615 are used for estimating the amount of movement and for feature classification, which will be described later.

The selective division determination unit 225 performs a modulo operation for each frame constituting the input video 305 using an image index indicating the order of the frame and the skip mask parameter 550, and if the remainder is "0" (i.e., if the remainder of the division of the image index by the skip mask parameter 550 is 0), it sets the frame as a dynamic frame and inputs it to the dynamic object division unit 230. On the other hand, if the remainder of the division operation is not "0", feature classification is performed on the features of the frame.
In addition, in the case of the first frame in the input video, the skip mask parameter 550 has not yet been set, so this first frame is directly input to the dynamic object divider 230 .

As described above, the dynamic object splitter 230 uses the pre-created dynamic class information 315 to perform dynamic object splitting processing on the dynamic frame predicted by the selective split determination unit 225, and generates image semantic information 320 for each pixel in the dynamic frame indicating whether the pixel belongs to an object corresponding to a dynamic class or an object corresponding to a static class.
This image semantic information 320 is input to the feature classifier 235 .

The feature classification unit 235 classifies each feature of the input video frame as a dynamic feature 622 or a candidate static feature 624 based on the image semantic information 320 generated by the dynamic object segmentation unit 230 and the features 615 extracted by the image feature extraction unit 220. Here, features classified as dynamic features 622 are treated as outliers and excluded from the processing of the SLAM unit 240.
On the other hand, for features classified as static feature candidates 624, the feature classification unit 235 may perform further feature classification based on the above-mentioned inter-frame movement amount e. More specifically, the feature classification unit 235 compares the movement amount e estimated based on the comparison of features between the target frame and the previous frame 510 with a predetermined movement amount reference M, and classifies the feature as a dynamic feature 622 if the movement amount e exceeds the predetermined movement amount reference M, and classifies the feature as a static feature 632 if the movement amount e is equal to or less than the predetermined movement amount reference M.

According to the feature classification process 600 described above, the features of the input image can be classified into either static features 631 or dynamic features 622. Furthermore, as described above, by performing feature classification twice based on the image semantic information 320 and the amount of movement between frames, it is possible to identify dynamic content present in non-dynamic frames that have been skipped by setting the skip mask parameters, and to correctly classify objects that have been misidentified based on the image semantic information 320 (objects that do not move even though they correspond to a dynamic class, or objects that have moved even though they correspond to a static class).

Next, the hardware configuration of the location identification device according to the second embodiment of the present disclosure will be described with reference to FIG. 7.

FIG. 7 is a diagram illustrating an example of a hardware configuration of a position identifying device 700 according to the second embodiment of the present disclosure.
As described above, the SLAM unit 240 and the position identification unit 245 according to the first embodiment of the present disclosure generate first moving object position information indicating the position and orientation of the moving object 275 in a three-dimensional map of an arbitrary environment. The position identification device 700 according to the second embodiment of the present disclosure can generate corrected position information indicating the position and orientation of the moving object 275 more accurately based on the first moving object position information and the second movement position information derived from the movement sensor data related to the movement of the moving object.
In addition, the positioning device 700 according to the second embodiment of the present disclosure has substantially the same configuration as the positioning device 200 according to the first embodiment, except that in addition to the configuration of the positioning device 200 according to the first embodiment described above, the positioning device 700 includes an inertial measurement unit 705, a GPS unit 710, a filter unit 715, and a position information correction unit 720. Therefore, repeated explanations will be omitted below.

The inertial measurement unit 705 is a functional unit for detecting the angles (or angular velocities) and acceleration of the three axes governing the motion as moving sensor data. The inertial measurement unit 705 can calculate the three-dimensional angular velocity and acceleration of the moving body 275 using the acquired three-axis gyro and three-directional accelerometer.

The GPS (Global Positioning System) unit 710 is a functional unit for acquiring geographical position information (coordinates, etc.) of the mobile unit 275 based on signals from GPS satellites.

The filter unit 715 is a functional unit that applies a predetermined filter process to the movement sensor data acquired by the inertial measurement unit 705 to remove noise from the movement sensor data.

The position information correction unit 720 is a functional unit for generating corrected moving body position information that more accurately indicates the position and orientation of the moving body 275 based on the first moving body position information generated by the above-mentioned SLAM unit 240 or the position identification unit 245 and the second moving body position information derived from the moving sensor data acquired by the inertial measurement unit 705.

The position determining device 700 according to the second embodiment of the present disclosure, configured as described above, can determine the position of the moving body 275 more accurately than the position determining device 200 according to the first embodiment, and can link the 3D map generated by the SLAM unit 240 to GPS data.

Next, the flow of the mobile object position information correction process in the position identification device according to the second embodiment of the present disclosure will be described with reference to FIG.

FIG. 8 is a block diagram showing the flow of a moving object position information correction process 800 in a position identification device according to a second embodiment of the present disclosure. The moving object position information correction process 800 shown in FIG. 8 is a process for generating corrected moving object position information that more accurately indicates the position and orientation of the moving object 275, based on the first moving object position information generated by the SLAM unit 240 or the position identification unit 245 and the second moving object position information generated by the inertial measurement unit. The moving object position information correction process 800 is executed by the SLAM unit 240, the position identification unit 245, the inertial measurement unit 705, the filter unit 715, and the position information correction unit 720.

First, the input image 305 acquired by the imaging unit is input to the SLAM unit 240 or the position identification unit 245 of the position identification device. The SLAM unit 240 or the position identification unit 245 performs the map generation process 300 shown in FIG. 3 or the position identification process 400 shown in FIG. 4 to generate first mobile object position information 330 indicating the position (e.g., geographic coordinates, etc.) and direction (north, east, west, south, etc.) of the mobile object.

The inertial measurement unit 705 also acquires moving sensor data 805 indicating the three-dimensional angular velocity and acceleration of the moving body equipped with a position identification device including the inertial measurement unit 705 while the moving body is moving.

Next, the filter unit 715 performs a predetermined filtering process on the movement sensor data 805 acquired by the inertial measurement unit 705 to remove noise from the sensor data and generate filtered data 810 .
Here, any existing noise removal means may be used to remove noise from the mobile sensor data 805, and is not particularly limited in this disclosure.

Next, the position information correction unit 720 generates second moving object position information 830 indicating the position and orientation of the moving object based on the filtered data 810 generated by the filter unit 715. Note that, in reality, this second moving object position information 830 is information indicating the position and orientation of the inertial measurement unit 705 derived from the mobile sensor data acquired by the inertial measurement unit 705. And, since the inertial measurement unit 705 is provided in the moving object, this second moving object position information 830 is also information indicating the position and orientation of the moving object.
Here, in order to derive the position and orientation of the moving object from the filtered data 810, the position information correction unit 720 may use any existing means, and is not particularly limited in this disclosure.

Next, the position information correction unit 720 generates corrected moving body position information 850 based on the above-mentioned first moving body position information 330 and second moving body position information 830, which can indicate the position and orientation of the moving body more accurately than the first moving body position information 330 or the second moving body position information 830.
The corrected mobile object position information 850 generated here may be output, for example, via an interface unit of the position specifying device (for example, the interface unit 250 of the position specifying device 700 shown in FIG. 7).

As described above, in the position determination device according to the second embodiment of the present disclosure, corrected moving body position information that more accurately indicates the position and orientation of the moving body can be generated based on the first moving body position information 330 generated by the SLAM unit 240 or the position determination unit 245 and the second moving body position information 830 generated by the inertial measurement unit 705.

Next, the flow of the GPS association process and the map reading process in the position identification device according to the second embodiment of the present disclosure will be described with reference to FIG.

As described above, the positioning device 700 according to the second embodiment of the present disclosure includes a GPS unit 710. By using the GPS unit 710 to associate GPS data with the three-dimensional map generated by the SLAM unit 240, it becomes possible to determine not only the position of a mobile body in a specific environment, but also the global position of the mobile body.

Figure 9 is a block diagram showing the flow of the GPS association process 900 and the map reading process 950 in the position determination device according to the second embodiment of the present disclosure.

First, the GPS association process 900 will be described. The GPS association process 900 is a process for associating GPS data with a three-dimensional map generated by the SLAM unit 240 and storing the data in the map database 925.

First, the SLAM unit 240 generates a 3D map 325 corresponding to the input image 305, for example, according to the map generation process 300 shown in FIG. 3.

After the three-dimensional map 325 is created, the GPS unit 710 acquires GPS data 905 indicating the global position of the mobile object. Then, the GPS unit 710 associates the acquired GPS data 905 with the three-dimensional map 325. Here, as a means for associating the GPS data 905 with the three-dimensional map 325, the GPS unit 710 may add a geotag indicating the geographical coordinates (latitude, longitude) of a specific point (such as a park, an office building, or a station) in the three-dimensional map 325.
Thereafter, the GPS unit 710 stores the three-dimensional map 325 associated with the GPS data 905 in a map database (hereinafter, "map DB") 925 managed by a server device (such as a cloud server) installed in a geographically distant location via a communication network 915 such as the Internet.

Next, the map loading process 950 will be described. The map loading process 950 is a process for loading a three-dimensional map associated with GPS data. This map loading process 950 may be performed automatically when a mobile object enters an environment in which a three-dimensional map 325 associated with GPS data 905 has already been created and stored in the map DB 925.

The GPS unit 710 of the mobile body transmits a map request including geographical information (latitude, longitude, etc.) 955 of the position identification device 700 (i.e., the mobile body) to the map DB 925 via the communication network 915. Note that in this disclosure, a "map request" is information requesting a pre-created environmental map (e.g., a three-dimensional map stored in the map DB 925 by the above-mentioned GPS association process 900) that corresponds to the current geographical information of the mobile body.
Thereafter, the map DB 925 searches for a three-dimensional map 325 corresponding to the geographical information 955 included in the map request received from the position identification device 700, and transmits the searched three-dimensional map 325 via the communication network 915. After that, the position identification device 700 that has received the three-dimensional map 325 uses the three-dimensional map 325 to perform the position identification process 400 described with reference to Fig. 4, for example, to identify the position of the mobile object equipped with the position identification device 700 on the entire earth.

According to the GPS association process 900 and map reading process 950 described above, by storing GPS data in association with a three-dimensional map of the environment generated by the SLAM unit 240, a moving object that later enters the environment can obtain and use the previously generated three-dimensional map from the map DB 925, eliminating the need to generate a three-dimensional map from scratch and enabling the location of the moving object on the entire globe to be identified.

Next, the hardware configuration of the location identification device according to the third embodiment of the present disclosure will be described with reference to FIG. 10.

FIG. 10 is a diagram illustrating an example of a hardware configuration of a position identifying device 1000 according to a third embodiment of the present disclosure.
As described above, the positioning devices according to the first and second embodiments of the present disclosure can generate mobile object position information indicating the position and orientation of the mobile object 275 in a three-dimensional map of an arbitrary environment. The positioning device 1000 according to the third embodiment of the present disclosure can notify the mobile object 275 of a risk that may affect the mobile object 275 based on the mobile object position information and risk position information indicating a risk in the environment.
In addition, the location identification device 1000 according to the third embodiment of the present disclosure has a configuration substantially similar to that of the location identification device 700 according to the second embodiment, except that in addition to the configuration of the location identification device 700 according to the second embodiment described above, the location identification device 1000 includes a risk management unit 1010 and a notification unit 1020. Therefore, repeated explanations will be omitted below.

The risk management unit 1010 is a functional unit for determining risks that may affect the mobile unit 275, based on corrected mobile unit position information indicating the position and orientation of the mobile unit 275 in the environment, and risk position information indicating the positions of risk candidates. Here, risk means, for example, things that have the potential to prevent the movement of the mobile unit 275 or cause damage or obstruction to the mobile unit 275, and may include, for example, large machinery, hazardous substances, areas with poor security, etc.

The notification unit 1020 is a functional unit for generating and outputting a risk notification to inform the mobile unit 275 of the risk determined by the risk management unit 1010. In one embodiment, the notification may include the location (e.g., geographic coordinates) of the risk determined to have the potential to affect the mobile unit 275, instructions for avoiding the risk, etc.

As described above, the positioning device 1000 according to the third embodiment of the present disclosure can determine the risks present in the environment in which the mobile object 275 moves and can notify the mobile object 275 of the risks. This can improve the safety of the mobile object 275 in the environment.

Next, the flow of risk management processing in the location identification device according to the third embodiment of the present disclosure will be described with reference to FIG. 11.

FIG. 11 is a block diagram showing the flow of a risk management process 1100 in a position identification device according to a third embodiment of the present disclosure. The risk management process 1100 is a process for determining risks that may affect a moving body and outputting a notification of the risk, and is executed by the risk management unit 1010 and the notification unit 1020 in the position identification device 1000 shown in FIG. 10.

First, the risk management unit 1010 acquires the 3D map 325 generated by the SLAM unit 240, the corrected mobile object position information 850 generated by the position information correction unit 720, and risk position information 1115 indicating the position of the risk in the environment. The risk position information 1115 here may be stored in a server device accessible via a communication network such as the Internet. As an example, when the target environment is a factory, the risk position information 1115 may be stored in a server in the factory.
In addition, although a case where the corrected moving body position information 850 is used will be described as an example here, the present disclosure is not limited thereto, and the first moving body position information or the second moving body position information may be used instead of the corrected moving body position information 850. However, since the corrected moving body position information 850 has higher accuracy than the first moving body position information and the second moving body position information, it is desirable to use the corrected moving body position information 850.

Next, the risk management unit 1010 determines risks that may affect the mobile body based on the acquired three-dimensional map 325, the corrected mobile body position information 850, and the risk position information 1115. Here, the risk management unit 1010 may determine, for example, from among the risk candidates shown in the risk position information 1115, risk candidates whose distance to the current position of the mobile body is less than a predetermined distance standard, as risks that may affect the mobile body.

Thereafter, the notification unit 1020 generates and outputs a risk notification to inform the mobile unit of the risk determined by the risk management unit 1010. As described above, this notification may include the location (e.g., geographic coordinates) of the risk determined to have the potential to affect the mobile unit, instructions for avoiding the risk, and the like.
In one embodiment, the notification unit 1020 may output the created risk notification to an interface unit (e.g., the interface unit 250 in the position identification device 1000 shown in FIG. 3) and display it to the user via a touch screen or the like.

The risk management process 1100 described above can determine the risks present in the environment in which the mobile body moves and can notify the mobile body of the risks. This can improve the safety of the mobile body in the environment.

Next, the function of the interface unit according to the embodiment of the present disclosure will be described with reference to FIG.

As described above, the interface unit according to the embodiment of the present disclosure is a functional unit for providing various information to a user (e.g., in the case where the mobile object is a human user having a personal terminal equipped with a position identification device) and accepting input from the user. As an example, the interface unit 250 may be a touch screen. The interface unit 250 may, for example, display a three-dimensional map generated by the SLAM unit, display the position of the mobile object on the three-dimensional map (i.e., the user's position), accept input of a destination of the mobile object, and display a recommended route to the destination.
12 is a block diagram showing functions of an interface unit 250 according to an embodiment of the present disclosure. As shown in FIG. 12, the interface unit 250 includes a three-dimensional map display function 1205, a mobile object position display function 1210, a destination setting function 1215, a route guidance function 1220, a map search function 1225, a map reading function 1230, and a risk display function 1235.

The three-dimensional map display function 1205 is a function for displaying a three-dimensional map generated by the map generation process 300 described with reference to FIG. 3, or a three-dimensional map obtained from a map database. Here, the interface unit 250 may display the three-dimensional map to the user via, for example, a touch screen.

The mobile object position display function 1210 is a function for displaying the position of a mobile object on a three-dimensional map. Here, the mobile object position display function 1210 may display the position of a mobile object on a three-dimensional map according to any of the above-mentioned first mobile object position information, second mobile object position information, and corrected mobile object position information. However, since the corrected mobile object position information has higher accuracy than the first mobile object position information and the second mobile object position information, it is preferable to use the corrected mobile object position information.

The destination setting function 1215 is a function for setting the destination of a mobile object (i.e., the user) based on input from a user (e.g., when the mobile object is a human user having a personal terminal equipped with a positioning device). For example, here, the human user may set the destination by selecting the desired destination with a finger or the like on a three-dimensional map displayed on a touch screen or the like by the interface unit 250.

The route guidance function 1220 is a function for determining and displaying a recommended route for guiding a mobile object to a destination set by the destination setting function 1215. Here, the route guidance 1220 may determine a route according to, for example, an existing automated driving function.

The map search function 1225 is a function for obtaining a created 3D map corresponding to the geographical information of a moving object obtained by the above-mentioned GPS unit or the like from a map DB (e.g., map DB 925 shown in FIG. 9) stored in the server based on the geographical information of the moving object.

The map loading function 1230 is a function for loading a created 3D map obtained from the map DB by the map search function 1225 and displaying it via the interface unit.

The risk display function 1235 is a function for displaying the risks determined and notified by the risk management process 1100 described with reference to FIG. 11. Here, information on risks that may affect the mobile object may be displayed on a three-dimensional map displayed via the interface unit.

The interface unit described above allows a human user moving around within the environment to easily check the various pieces of information output to the positioning device.

Next, referring to FIG. 13, we will explain an example of how to use the skip mask parameters according to an embodiment of the present disclosure.

As described above, the dynamic content determination process according to the embodiment of the present disclosure (e.g., the dynamic content determination process 500 shown in FIG. 5) is performed based on a skip mask parameter. The skip mask parameter is a parameter that specifies the number of frames in the input video to be excluded from the dynamic object segmentation process. By using the skip mask parameter, frames that are unlikely to contain dynamic content are skipped (i.e., excluded from the dynamic object segmentation process), and only frames that are likely to contain dynamic content are subject to the dynamic object segmentation process, which can significantly save computing resources compared to performing the dynamic object segmentation process on all frames of the input video.
The skip mask parameters are set and updated based on the amount of movement between multiple frames included in the input video, the movement speed of the image capture unit that captured the input video, and the ratio of pixels of objects corresponding to the dynamic class in the frames.
13 is a diagram showing an example of using skip mask parameters according to an embodiment of the present disclosure. For convenience of explanation, the following example of use describes a case where the skip mask parameters are set based on the ratio of dynamic content of a frame, but as described above, the present disclosure is not limited thereto, and the skip mask parameters may be set according to not only the ratio of dynamic content but also the moving speed of the moving body/image capture unit, the movement between frames, etc.

First, an input video 1300 including at least a number of frames, such as frame 1301, frame 1307, frame 1308, frame 1309, and frame 1310, is acquired, input to a positioning device according to an embodiment of the present disclosure, and is subjected to dynamic content determination processing. As described above, when the dynamic content determination processing is performed for the first time, the skip mask parameters have not yet been set, so the selective split determination unit directly inputs frame 1301, which is the first frame, to the dynamic object splitter.

As a result of analyzing frame 1301 by the dynamic object segmentation unit, it is determined that there are no dynamic objects such as a human being, and therefore the dynamic content ratio is low and does not satisfy the dynamic content ratio criterion. For this reason, the skip mask parameter is set to a predetermined value such as "5 frames." Here, the value of the skip mask parameter may be set by, for example, an administrator of the position identification device, or may be determined based on input video used in the past to generate a map.

Next, the selective split determination unit skips five frames from frame 1301 (i.e., frames 1302 to 1306, not shown in FIG. 13) in the input video 1300 in accordance with the skip mask parameters, and inputs frame 1307 to the dynamic object splitter.

Frame 1307 is analyzed by the dynamic object segmentation unit and is therefore determined to have a high dynamic content ratio and to satisfy the dynamic content ratio criterion because it contains a dynamic object (human). For this reason, the skip mask parameter is updated to, for example, "0 frames."

Next, the selective split determination unit inputs the next frame, frame 1308, in the input video 1300 to the dynamic object splitter without skipping any frames from frame 1307 onwards in accordance with the skip mask parameters.

As a result of analyzing frame 1308 by the dynamic object segmentation unit, it is determined that, like frame 1307, a dynamic object (human) is present, so the dynamic content ratio is high and the dynamic content ratio criterion is met. For this reason, the skip mask parameters are not changed and remain at "0 frames."

Frame 1309, like frames 1307 and 1308, is determined to have a high proportion of dynamic content and meet the dynamic content proportion criterion due to the presence of a dynamic object (a human), and therefore the skip mask parameter is not changed and remains at "0 frames".
In this manner, the selective split decision unit inputs each frame to the dynamic object splitter without skipping any frames until the frame no longer meets the dynamic content proportion criterion.

On the other hand, when frame 1310 is input to the dynamic object segmentation unit and analyzed, it is determined that frame 1310 does not contain a dynamic object such as a human, has a low ratio of dynamic content, and does not satisfy the dynamic content ratio criterion. Therefore, the skip mask parameter is set to a predetermined value such as "5 frames", and the selective segmentation determination unit skips 5 frames from frame 1310 in the input video 1300 in accordance with the skip mask parameter, and then inputs the next frame (not shown in FIG. 13) to the dynamic object segmentation unit.

In this way, by using the skip mask parameters, it is possible to determine dynamic frames that are likely to contain dynamic content while skipping non-dynamic frames that do not contain dynamic content. This allows dynamic frames to be excluded from the dynamic object segmentation process while non-dynamic frames are input to the skip unit for map generation, thereby significantly saving computing resources compared to performing dynamic object segmentation on all frames of the input video.

Although the embodiment of the present invention has been described above, the present invention is not limited to the above-mentioned embodiment, and various modifications are possible without departing from the gist of the present invention.

200, 700, 1000 Position identification device 205 Photography unit 210 Processor 215 Memory 220 Image feature extraction unit 225 Selective division judgment unit 230 Dynamic object division unit 235 Feature classification unit 240 SLAM unit 245 Position identification unit 250 Interface unit 705 Inertial measurement unit 710 GPS unit 715 Filter unit 720 Position information correction unit 1010 Risk management unit 1020 Notification unit

Claims (12)

A position determining device for determining the position of a moving object in an arbitrary environment, comprising:
an image feature extraction unit that analyzes an input video showing the environment and extracts features for each of a plurality of frames included in the input video;
a selective division determination unit that predicts a dynamic frame including dynamic content from the input video based on any one of an amount of movement between the plurality of frames included in the input video, a movement speed of a shooting unit that captured the input video, and a ratio of pixels of an object corresponding to a dynamic class in the frame; and
a dynamic object segmentation unit that performs a dynamic object segmentation process on the dynamic frame and generates, for each pixel in the dynamic frame, semantic information of the image that indicates whether the pixel belongs to an object corresponding to a dynamic class or an object corresponding to a static class;
a feature classification unit that classifies each feature as a dynamic feature or a static feature based on semantic information of the image and the features;
a SLAM unit that generates a three-dimensional map that stereoscopically illustrates the environment and first moving object position information that indicates a position and an orientation of the moving object based on the static features and semantic information of the image;
A location determination device comprising: The selective division determination unit is
estimating a moving speed of the imaging unit based on a change in a position of the imaging unit that acquired the input video in the plurality of frames;
setting a skip mask parameter that specifies a number of frames to be excluded from the dynamic object segmentation process in the input video based on a moving speed of the image capture unit;
processing the input video based on the skip mask parameters to skip non-dynamic frames that do not include dynamic content and to predict the dynamic frames;
2. The position determining device according to claim 1 . The selective division determination unit is
determining an amount of motion between the frames by comparing the features extracted from the non-motion frame with features of a frame immediately preceding the non-motion frame in the input video;
If the amount of movement between the frames satisfies a predetermined movement amount criterion, the non-dynamic frame is determined to be a dynamic frame.
3. The position determining device according to claim 2 . The feature classification unit is
determining the amount of movement of the static features between the frames;
inputting, into the SLAM unit, only those static features whose inter-frame movement amount does not satisfy a predetermined movement amount criterion;
4. The position determining device according to claim 3 . The selective division determination unit is
According to the semantic information of the image, if the ratio of pixels of the object corresponding to the dynamic class in the target frame included in the input video meets a predetermined dynamic content ratio criterion;
setting the skip mask parameter to a lower value;
5. The position determining device according to claim 4. The location identification device is
an inertial measurement unit for acquiring movement sensor data relating to the movement of the moving object;
a filter unit that performs a filtering process on the movement sensor data to generate filtered movement sensor data;
generating second mobile object position information indicative of a position and an orientation of the mobile object based on the filtered motion sensor data;
a position information correction unit that generates corrected moving object position information that more accurately indicates a position and an orientation of the moving object based on the first moving object position information and the second moving object position information;
The location determination device of claim 1 further comprising: The location identification device is
Risk Management Department,
A notification unit;
Further comprising:
The risk management department
determining a risk that may affect the moving object based on the corrected moving object position information and risk position information indicating the position of a risk candidate;
generating and outputting a risk notification indicating instructions for avoiding said risk;
7. The position determining device according to claim 6, The location identification device is
acquiring GPS data indicative of the location of said mobile object on the globe;
a GPS unit that stores the GPS data in a map database in association with the three-dimensional map;
8. The position determining device according to claim 7, A method for determining a location of a moving object in an arbitrary environment, comprising:
analyzing an input video showing the environment and extracting features for each of a plurality of frames included in the input video;
setting a skip mask parameter that specifies the number of frames to be excluded from a dynamic object segmentation process based on any one of an amount of movement between the plurality of frames included in the input video, a moving speed of a shooting unit that captured the input video, and a ratio of pixels of an object corresponding to a dynamic class in the frames;
processing the input video based on the skip mask parameters to skip non-dynamic frames that do not contain dynamic content and to predict dynamic frames that contain dynamic content;
performing a dynamic object segmentation process on the dynamic frame to generate, for each pixel in the dynamic frame, image semantic information indicating whether the pixel belongs to an object corresponding to a dynamic class or an object corresponding to a static class;
classifying each feature as a dynamic feature or a static feature based on semantic information of the image and the features;
determining the amount of movement of the static features between the frames;
generating a three-dimensional map showing the environment in a three-dimensional manner and first moving object position information showing a position and an orientation of the moving object based on the static features, among the static features, a feature where the movement amount between the frames does not satisfy a predetermined movement amount criterion and semantic information of the image;
A location determination method comprising: A location determination system for determining the location of a moving object in an arbitrary environment, comprising:
a position determination device installed on the mobile object and configured to generate a three-dimensional map showing the environment;
a map database for storing the three-dimensional map is connected via a communication network;
The location identification device is
an image feature extraction unit that analyzes an input video showing the environment and extracts features for each of a plurality of frames included in the input video;
a selective division determination unit that predicts a dynamic frame including dynamic content from the input video based on any one of an amount of movement between the plurality of frames included in the input video, a movement speed of a shooting unit that captured the input video, and a ratio of pixels of an object corresponding to a dynamic class in the frame; and
a dynamic object segmentation unit that performs a dynamic object segmentation process on the dynamic frame and generates, for each pixel in the dynamic frame, semantic information of the image that indicates whether the pixel belongs to an object corresponding to a dynamic class or an object corresponding to a static class;
a feature classification unit that classifies each feature as a dynamic feature or a static feature based on semantic information of the image and the features;
a SLAM unit that generates the three-dimensional map that stereoscopically shows the environment and first mobile object position information that shows a position and an orientation of the mobile object based on the static features and semantic information of the image, and stores the three-dimensional map in the map database via the communication network;
A location determination system comprising: The location identification device is
acquiring GPS data indicative of the location of said mobile object on the globe;
a GPS unit that stores the GPS data in the map database in association with the three-dimensional map;
11. The location system according to claim 10, The GPS unit includes:
transmitting a map request to said map database via said communications network, said map request including GPS data indicating a global position of said mobile unit;
receiving from the map database a three-dimensional map corresponding to the global location of the moving object;
12. The location system according to claim 11,

Images