Optimal Multiple Drone Placement to Minimize Target Localization Error

This is extension of my PhD research (peripheral-central vision system.)

Multiple host drones have two different cameras (one omnidirectional cameras and one perspective camera) and there are multiple target drones. The multiple host drones should move and cover all the target drones, and the localization error of each target drone should be minimized.

I made a simulation that target drones are randomly generated with random speed and position. The optimal positions and orientation, camera types, and camera orientation of host drones are computed using the interior-point method, and observe the target drones during observation time period. After observing targets during the observation time period, the new optimal values for the next target drone path are re-computed, the host drones go to the next positions. The video below shows the simulation.

Multiple target tracking using the peripheral-central vision system

The peripheral-central vision system in the previous post was tracking only one target and estimating the position of it using only two cameras. The another idea is “what if there are multiple targets?”. Then all the targets should be observed continuously in terms of a specific order.

Travelling Salesman Problem (TSP) solution is a good way to solve this since this problem is about visiting all the cities for a salesman with the shortest path; so I applied this algorithm to look at all the targets with the minimum maneuver of the PTZ camera. For example, if the PTZ camera wants to see a target (A) and there are other targets (B, C) on the way to look at the target A, then why don’t we look at B and C for the efficiency? This is the concept of my TSP algorithm for multi-target tracking.

The video below shows the preliminary results of my system. The omnicam detects multiple red targets and sends a cue according to my TSP algorithm to the PTZ camera through ROS. Then the PTZ camera observes each target and the targets are localized.

Multiple target tracking using PCV system

Aircraft path prediction based on the pose data by learning wind disturbance

Update: This path prediction algorithm is published in Aerospace Science and Technology Journal (https://www.sciencedirect.com/science/article/abs/pii/S1270963820307124?casa_token=OAWdgUmnXUUAAAAA:CWoVpx8LnAbdtq9HOQU7dt0WfN9qnUBvwdIt3YwIeniBeWQ2mTDAPaBNp2yn0kHVBgBtRTYsOw). Check out for more details!

The peripheral-central vision system mentioned in the previous post can estimate the position using the triangulation. Also, one of two cameras is a zoom enabled high-resolution camera which is able to even compute the pose of the target aircraft using 3D pose estimation algorithm. This post talks about how the path prediction can be improved if we have aircraft pose data.

Just to explain easily, if we know the roll angle of a fixed-wing aircraft (e.g. 30 deg roll angle), we then expect the fixed-wing aircraft will go to the right direction.

This will go to the right with this roll angle.

Likewise, the pose data definitely helps the future path prediction. I devised an algorithm to predict aircraft future path using the energy rate and aircraft equations of motion. After future path prediction, the algorithm estimates wind disturbance from the past data and corrects it for the better path prediction. This algorithm is presented in my recent journal paper and this is on the review. I will post detailed equations once my paper is published.

To evaluate my algorithm performance, I used an actual flight data of a small fixed-wing aircraft. I got the actual position and pose data from Pixhawk controller, replayed in MATLAB and applied my path prediction algorithm to the actual path. Let’s see the video below. The first video is when I used only position data. The light blue line represents the entire path, the thick blue line is predicted path during the prediction time horizon and the green line shows the actual path during the prediction time horizon.

When I use only position data for the path prediction

We can see the predicted path is not that good. Then let’s see when I applied my algorithm to the same flight data.

When I use pose data for the path prediction

The path prediction became much better, and this can even predict very maneuvering flight path.

Peripheral-Central Stereo Vision System

This is my main PhD research. I am working on ‘Intelligent Visual Tracking System for UAS tracking using Peripheral-Central Vision’. Conventional stereo vision system uses two normal identical cameras; therefore, this normal stereo vision system has a limited field of view(FOV) so that it cannot see all of the area around itself at the same time.

Peripheral-Central stereo vision system

In order to address this issue, we decided to use an omnidirectional camera and a pan-tilt-zoom(PTZ) camera. The omnidirectional camera has a 360-degree FOV, so that it can see all of the area around it, but the image resolution is low. Thus, the omnidirectional camera is used only for the initial detection cue. Once the target is detected by the omnidirectional camera, the detection cue is then sent to PTZ camera through ROS.

Then PTZ camera pans and tilts toward the target and the target is detected and classified using YOLO (https://pjreddie.com/darknet/yolo/). I posted the implementation video on my blog as well(https://ckrobotics.wordpress.com/2018/12/22/yolov2-object-classification-detection-implementation/). After the target is detected and classified, the PTZ camera zooms on the target to get a more high-resolution image of the target.

Finally the target is detected by two cameras and we can estimate the target location using my target position estimation algorithm.

This video shows how system works. The top-left video is from the omnidirectional camera and the target is detected by background subtraction. This detection cue is then sent to the PTZ camera(top-right), and the PTZ camera stares at the target using the cue and detects and classifies the target (for this example, the target is classified as ‘person’). The position of the target is estimated using the ranging algorithm (bottom-left). In the bottom-right video, we can see the PTZ camera is automatically tracking the target based on the detection cue.

Detailed algorithm and information are in my paper (https://arc.aiaa.org/doi/abs/10.2514/6.2019-2074). If you have any question, please contact me.

How to set up SSH from computer to TX2

SSH(Secure Shell) is a cryptographic network protocol for operating network services securely over an unsecured network(Wikipedia: https://en.wikipedia.org/wiki/Secure_Shell). To simply say, through SSH, you can connect the remote computer(in my case TX2) and control it through antenna. This can be used to send some commands to an onboard computer of a drone like TX2 so that we can control the drone in the air remotely (start a ROS launch file of TX2 or such).

Basically to do this is pretty simple.

  1. Connect two computer (Host – TX2) using ubiquiti radio (or just ithernet cable or wifi will work if you want to)
  2. From TX2, check the ipaddress using ifconfig command.
  3. From the host computer, type
    ssh -X nvidia@IPADDRESS
  4. Then it will require the password of TX2 and type.
  5. You will be connected to TX2.
  6. (optional) If you want to use graphical program(such as gedit) on TX2 remotely, you can do that by using this command.
    export DISPLAY=:10
  7. If you want exit SSH, just use command ‘exit’.

One thing you need to consider is the ipaddress of TX2 is changing whenever you reboot it. Therefore, getting a static IP for TX2 would be comfortable. To do that, you can go to “/etc/network/interfaces” on TX2 and add the last line like below (reference: https://devtalk.nvidia.com/default/topic/1031542/jetson-tx2/how-to-set-manual-ip-address-on-the-tx2/):

# interfaces(5) file used by ifup(8) and ifdown(8)
# Include files from /etc/network/interfaces.d:
source-directory /etc/network/interfaces.d
source interfaces.d/eth0

After that, go to “/etc/network/interfaces.d/eth0” (if there is no file, create one) and adjust IP address you want, your netmask and gateway.

auto eth0
iface eth0 inet static
address 192.168.1.20
netmask 255.255.255.0
gateway 192.168.1.2

If you want to multiple command window through SSH, you can use ‘byobu’ package on Ubuntu.

MBZIRC- truck tracking

I attended MBZIRC 2017 competition (http://www.mbzirc.com/) which is an international robotics competition in Abu Dhabi. I was in charge of one of the challenges of the competition and the mission of the challenge was to make a fully autonomous drone which takes off, finds a moving truck and land on the truck fully automatically. Our team built our own drone, mounted two cameras(fisheye lens camera, normal camera), Piksi RTK-GPS, lidar and Jetson TX1.

Our drones we built. We built multiple hexacopters for other cooperative drone missions.

For the challenge I was in charge, the truck is running on an 8-figure track and the drone takes off, detects the truck and lands on that. We simulated this on ROS-Gazebo.

MBZIRC Challenge 1 Simulation

For this video, I am using only Optical flow for the detection. I made it compute the offset of the truck position from the image center and use it for the control. Once the truck is inside of the center square I specified, I made it go down. I didn’t use PID control at this moment, so the control is a little jerky, but later I added template matching for the detection which was great to detect the figure on the truck and PID control for the drone control. Here, in this video, although I used only simple offset-based control and optical flow, the drone is automatically landing on the truck.

This is our first progress video and we had 4 missions for this competition. The field test of my mission (Challenge 1) is from 4:18.

Caffe installation

Caffe is one of most popular deep learning framework like TensorFlow. Many deep learning codes are based on Caffe, so knowing how to use this would be pretty useful for implementing deep learning programs.

Caffe has good documentation for installation but depends on the environment of the computer, it generates a bunch of errors. I also had so much error to install it, so I just would like to describe here how I installed and what kind of error I got during the installation procedure.

This post is based on the Caffe’s installation guide on their Github repo (https://github.com/BVLC/caffe/wiki/Ubuntu-16.04-or-15.10-Installation-Guide). I followed the instruction exactly, but as I mentioned, you might have some error even though you follow it, so I writing this post based on my experience.

Caffe requires some software installed before its installation.

  • Python
  • OpenCV
  • Cuda Toolkit
  • CuDNN

Actually each of the requirement will take much your time to install if you don’t have it yet, especially I spent so much time to install Cuda at the first time, so I will post this soon as well. Assuming you already have all of the required softwares on your machine, and please make sure your PYTHONPATH includes the OpenCV location you want to use(in case you have multiple version of OpenCVs). PYTHONPATH can be added from .bashrc file.

I am using Ubuntu 16.04, OpenCV 3.4, Cuda 8.0 and 5.1 CuDNN. Once you are ready with all the requirement, then git clone the Caffe repo to where you want and what you have to do after that is just three things.

  • Modify makefile.config
  • Modify makefile
  • Build

Seems pretty simple, but this part gave me a lot of consecutive errors and I finally figured out what I have to do after searching for all the solution for the error. Once you completed the download, copy Makefile.config.example and change the name to Makefile.config. And then, the contents should be modified. Please find each of the lines in the file, and modify them as below.

CUDA_DIR  := /usr/local/cuda-8.0

PYTHON_INCLUDE := /usr/include/python2.7 \
        /usr/local/lib/python2.7/dist-packages/numpy/core/include
WITH_PYTHON_LAYER := 1

PYTHON_LIB := /usr/lib/x86_64-linux-gnu
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/hdf5/serial

After that, as shown in the official instruction, you need to make a symbolic link to libhdf5 using commands below.

cd /usr/lib/x86_64-linux-gnu 
sudo ln -s libhdf5_serial.so.10.1.0 libhdf5.so 
sudo ln -s libhdf5_serial_hl.so.10.0.2 libhdf5_hl.so

Next step is modifying the makefile. If you find “USE_OPENCV” in Makefile, you will see some lines which specifies LIBRARIES. You can add it as below.

LIBRARIES += glog gflags protobuf leveldb snappy \
          lmdb boost_system boost_filesystem hdf5_hl hdf5 m \
          opencv_core opencv_highgui opencv_imgproc opencv_imgcodecs opencv_videoio

Now build it. 

cd /directory/to/Caffe/
mkdir build
cd build
make clean
make all

If you want to build PyCaffe,

cd /directory/to/Caffe/
make pycaffe

The last step is to add the Caffe directory to PYTHONPATH. You can open .bashrc file and add it as below.

gedit ~/.bashrc
export PYTHONPATH=/directory/to/caffe/python:$PYTHONPATH

The installation is done. Enjoy Caffe!

Troubleshoot

  1. NVCC sm20 sm21 related error

Delete lines below in makefile.config. This warning doesn’t seem critical for building, so you don’t need to do though…

-gencode arch=compute_20,code=sm_20 \        
-gencode arch=compute_20,code=sm_21 \

  2. tools/CMakeFiles/upgrade_solver_proto_text.dir/build.make:128: recipe for target ‘tools/upgrade_solver_proto_text

Edit “caffe/cmake/Dependencies.cmake”, at line 28

---  list(APPEND Caffe_LINKER_LIBS ${HDF5_LIBRARIES})
+++  list(APPEND Caffe_LINKER_LIBS ${HDF5_LIBRARIES} ${HDF5_HL_LIBRARIES})

(reference: https://devtalk.nvidia.com/default/topic/1037599/jetson-tx2/installation-of-caffe-error/)

  3. //usr/local/lib/libopencv_core.so.3.4: undefined reference to `cudaEventElapsedTime@libcudart.so.9.0′

In my case, my OpenCV is installed on Cuda 9.0 from the first place, so I reinstalled OpenCV to depend on Cuda 8.0.

Generating COCO style dataset

MS COCO dataset is one of the largest object detection, segmentation, and captioning dataset (http://cocodataset.org/#home). Because of the huge size of the data( 123,287 images, 886,284 instances), COCO dataset is largely used for image neural network training. OpenPose that I posted last time is also used COCO dataset for training since this dataset includes bounding boxes, segmentation of the objects in images and even keypoints of person subject.

For my project, I need the keypoints of the aircraft, but unfortunately, COCO dataset provides keypoints of only person subject. For other 90 categories of objects, it has bounding box data around the objects, segmentation data of objects, category data, etc. You can check sample images on their website(http://cocodataset.org/#explore).

As I mentioned, what I need for my research is the keypoint data of the aircraft, and OpenPose(https://github.com/CMU-Perceptual-Computing-Lab/openpose) I am going to use is also trained based on COCO dataset, so I decided to make my own COCO style dataset.

Keypoint data of COCO (using Visipedia Annotation Tools)

COCO dataset consists of images and annotation file(json format), so what I need to know is how to annotate the images to json format. The dataset format is like below.

{
"info" : info, 
"images" : [image], 
"annotations" : [annotation], 
"licenses" : [license],
}

info{
"year" : int, 
"version" : str, 
"description" : str, 
"contributor" : str, 
"url" : str, 
"date_created" : datetime,
}

image{
"id" : int, 
"width" : int, 
"height" : int, 
"file_name" : str, 
"license" : int, 
"flickr_url" : str, 
"coco_url" : str, 
"date_captured" : datetime,
}

license{
"id" : int, 
"name" : str, 
"url" : str,
}

Therefore, the json format should look like that and I found some good annotation tools to make that json files.

I tried to find a tool that can annotate both segmentation and keypoints of objects, but I couldn’t find one that do both, so I used two tools, Visipedia for keypoints and COCO-Annotator for object segmentation.

Both tools has a very good documentations for installation, so using those tools was not that hard. For my dataset, I firstly made segmentation data of the aircraft using COCO-Annotator.

Segmentation data annotation using COCO-Annotator

After that, I made keypoint data using Visipedia and exported json file and put the segmentation and area data from COCO-Annotator to the json file from the Visipedia.

Aircraft keypoint annotation using Visipedia Annotation Tools

Finally, I got a COCO style json file that includes segmentation data, keypoints data and bounding box data of a fixed-wing aircraft for my test. Next post will be about the deep learning training for OpenPose implementation for the fixed-wing aircraft keypoint detection.

OpenPose real-time body pose estimation implementation

OpenPose(https://github.com/CMU-Perceptual-Computing-Lab/openpose) is a great real-time body pose tracker. This computes 2D confidence maps of body parts location and affinity fields are parsed for the 2D keypoints of people in the image.

I am planning to train aircraft images instead of person images to get the keypoints of aircraft in the image for the pose analysis.