8000 update README.md · pulp-platform/pulp-dronet@0491cb9 · GitHub
[go: up one dir, main page]
Skip to content

Commit 0491cb9

Browse files
DP8DP8
committed
update README.md
1 parent c2d0c2a commit 0491cb9

File tree

1 file changed

+93
-84
lines changed

1 file changed

+93
-84
lines changed

‎README.md‎

Lines changed: 93 additions & 84 deletions
Original file line numberDiff line numberDiff line change
@@ -11,13 +11,13 @@ Copyright (C) 2019 ***ETH Zürich***. All rights reserved.
1111
If you use **PULP-DroNet** in an academic or industrial context, please cite the following publications:
1212

1313
Publications:
14-
* *A 64mW DNN-based Visual Navigation Engine for Autonomous Nano-Drones* [preprint](https://arxiv.org/abs/1805.01831) [IEEE IoT Journal](https://ieeexplore.ieee.org/document/8715489)
15-
* *An Open Source and Open Hardware Deep Learning-powered Visual Navigation Engine for Autonomous Nano-UAVs* [preprint](https://arxiv.org/abs/1905.04166) [IEEE DCOSS]()
14+
* *A 64mW DNN-based Visual Navigation Engine for Autonomous Nano-Drones* [arXiv preprint](https://arxiv.org/abs/1805.01831) -- [IEEE IoT Journal](https://ieeexplore.ieee.org/document/8715489)
15+
* *An Open Source and Open Hardware Deep Learning-powered Visual Navigation Engine for Autonomous Nano-UAVs* [arXiv preprint](https://arxiv.org/abs/1905.04166) -- [IEEE DCOSS](https://ieeexplore.ieee.org/document/8804776)
1616

1717
[PULP Platform Youtube](https://www.youtube.com/c/PULPPlatform) channel (subscribe it!): [Video1](https://youtu.be/57Vy5cSvnaA) [Video2](https://youtu.be/JKY03NV3C2s)
1818

1919
~~~~
20-
@article{palossi2019PULPDroNetV1,
20+
@article{palossi2019pulpdronetIoTJ,
2121
author={D. {Palossi} and A. {Loquercio} and F. {Conti} and E. {Flamand} and D. {Scaramuzza} and L. {Benini}},
2222
title={A 64mW DNN-based Visual Navigation Engine for Autonomous Nano-Drones},
2323
journal={IEEE Internet of Things Journal},
@@ -28,11 +28,16 @@ Publications:
2828
~~~~
2929

3030
~~~~
31-
@article{palossi2019PULPDroNetV2,
32-
author={Palossi, Daniele and Conti, Francesco and Benini, Luca},
33-
title={An Open Source and Open Hardware Deep Learning-powered Visual Navigation Engine for Autonomous Nano-UAVs},
34-
journal={arXiv preprint arXiv:1905.04166},
35-
year={2019}
31+
@inproceedings{palossi2019pulpdronetDCOSS,
32+
author={D. {Palossi} and F. {Conti} and L. {Benini}},
33+
booktitle={2019 15th International Conference on Distributed Computing in Sensor Systems (DCOSS)},
34+
title={An Open Source and Open Hardware Deep Learning-Powered Visual Navigation Engine for Autonomous Nano-UAVs},
35+
pages={604-611},
36+
keywords={autonomous navigation, nano-size UAVs, deep learning, CNN, heterogeneous computing, parallel ultra-low power, bio-inspired},
37+
doi={10.1109/DCOSS.2019.00111},
38+
ISSN={2325-2944},
39+
month={May},
40+
year={2019},
3641
}
3742
~~~~
3843

@@ -45,60 +50,61 @@ This means that all the complex computations are done directly aboard the vehicl
4550
The visual navigation engine is composed of both a software and a hardware part.
4651
The former is based on the previous [DroNet](https://github.com/uzh-rpg/rpg_public_dronet) project developed by the [RPG](http://rpg.ifi.uzh.ch/) from the University of Zürich (UZH).
4752
DroNet is a shallow convolutional neural network (CNN) which has been used to control a standard-size quadrotor in a set of environments via remote computation.
48-
The hardware soul of PULP-DroNet is embodied by the *PULP-Shield* an ultra-low power visual navigation module featuring a Parallel Ultra-Low-Power ([PULP](https://www.pulp-platform.org/)) GAP8 System-on-Chip (SoC) from [GreenWaves Technologies](https://greenwaves-technologies.com/) (GWT), an ultra-low power camera, and off-chip Flash/DRAM memory; the shield is designed as a pluggable PCB for the [Crazyflie 2.0](https://www.bitcraze.io/crazyflie-2/) nano-drone.
53+
The hardware soul of PULP-DroNet is embodied by the *PULP-Shield* an ultra-low power visual navigation module featuring a Parallel Ultra-Low-Power ([PULP](https://www.pulp-platform.org/)) GAP8 System-on-Chip (SoC) from [GreenWaves Technologies](https://greenwaves-technologies.com/) (GWT), an ultra-low power HiMax HBM01 camera, and off-chip Flash/DRAM memory; the shield is designed as a pluggable PCB for the [Crazyflie 2.0](https://www.bitcraze.io/crazyflie-2/)/[2.1](https://www.bitcraze.io/crazyflie-2-1/) nano-drone.
4954

5055
Then, we developed a [general methodology](https://arxiv.org/abs/1805.01831) for deploying state-of-the-art deep learning algorithms on top of ultra-low power embedded computation nodes, like a miniaturized drone.
5156
Our novel methodology allowed us first to deploy DroNet on the PULP-Shield, and then demonstrating how it enables the execution the CNN on board the CrazyFlie 2.0 within only 64-284mW and with a throughput of 6-18 frame-per-second!
5257
Finally, we field-prove our methodology presenting a closed-loop fully working demonstration of vision-driven autonomous navigation relying only on onboard resources, and within an ultra-low power budget.
53-
See the videos on the [PULP Platform Youtube](https://www.youtube.com/channel/UCpad_lwSfoMZkb6X7FdjU0g) channel ([Video1](https://youtu.be/JKY03NV3C2s)).
58+
See the videos on the [PULP Platform Youtube](https://www.youtube.com/channel/UCpad_lwSfoMZkb6X7FdjU0g) channel ([Video](https://youtu.be/JKY03NV3C2s)).
5459

5560
We release here, as open source, all our code, hardware designs, datasets, and trained networks.
5661

5762

5863
## 2. Getting started
5964
The PULP-DroNet project uses the following terminology:
60-
* **PULP-Shield**: it is our pluggable PCB compatible with the Crazyflie 2.0 nano-drone. The shield is designed to host either a PULP GAP8 SoC from GWT or a PULP Mr.Wolf SoC (academic chip).
65+
* **PULP-Shield**: it is our pluggable PCB compatible with the Crazyflie 2.0/2.1 nano-drone. The shield is designed to host either a PULP GAP8 SoC from GWT or a PULP Mr.Wolf SoC (academic chip).
6166
* **PULP Virtual Platform**: it is a software simulation environment running on your machine. It simulates the execution of your application on the selected PULP SoC (e.g., GAP8, Mr.Wolf, etc.).
6267
* **Autotiler**: it is a software tool developed in collaboration with GWT and released as `lib.a`. It produces a set of C files required to compile PULP-DroNet. The default version of the generated files is already included in this release. Thus the tool is not strictly required.
6368
* **GAPuino board**: it is the GWT Arduino-compatible development board that includes a PULP GAP8 SoC ([spec](https://greenwaves-technologies.com/product/gapuino/)).
6469

6570
The project's structure is the following:
6671

6772
```
68-
├── pulp-dronet/
69-
│ ├── bin/
70-
│ │ ├── PULPDroNet_GAPuino
71-
│ │ ├── PULPDroNet_PULPShield
72-
│ ├── dataset/
73-
│ │ ├── Himax_Dataset/
74-
│ │ ├── Udacity_Dataset/
75-
│ │ ├── Zurich_Bicycle_Dataset/
76-
│ ├── imgs/
77-
│ │ ├── PULP_dataset.png
78-
│ │ ├── PULP_drone.png
79-
│ │ ├── PULP_proto.png
80-
│ │ ├── PULP_setup.png
81-
│ ├── PULP-Shield/
82-
│ │ ├── GAP8/
83-
│ │ ├── jtag-convboard/
84-
│ ├── src/
85-
│ │ ├── autotiler/
86-
│ │ ├── config.h
87-
│ │ ├── config.ini
88-
│ │ ├── PULPDronet.c
89-
│ │ ├── PULPDronetGenerator.c
90-
│ │ ├── PULPDronetKernels.c
91-
│ │ ├── PULPDronetKernels.h
92-
│ │ ├── PULPDronetKernelsInit.c
93-
│ │ ├── PULPDronetKernelsInit.h
94-
│ │ ├── Makefile
95-
│ │ ├── run_dataset.sh
96-
│ ├── weights/
97-
│ │ ├── binary/
98-
│ │ ├── WeightsPULPDroNet.raw
99-
│ ├── LICENSE.apache.md
100-
│ ├── LICENSE_README.md
101-
│ ├── README.md
73+
.
74+
└── pulp-dronet/
75+
├── bin/
76+
│ ├── PULPDroNet_GAPuino
77+
│ └── PULPDroNet_PULPShield
78+
├── dataset/
79+
│ ├── Himax_Dataset/
80+
│ ├── Udacity_Dataset/
81+
│ └── Zurich_Bicycle_Dataset/
82+
├── imgs/
83+
│ ├── PULP_dataset.png
84+
│ ├── PULP_drone.png
85+
│ ├── PULP_proto.png
86+
│ └── PULP_setup.png
87+
├── PULP-Shield/
88+
│ ├── GAP8/
89+
│ └── jtag-convboard/
90+
├── src/
91+
│ ├── autotiler/
92+
│ ├── config.h
93+
│ ├── config.ini
94+
│ ├── PULPDronet.c
95+
│ ├── PULPDronetGenerator.c
96+
│ ├── PULPDronetKernels.c
97+
│ ├── PULPDronetKernels.h
98+
│ ├── PULPDronetKernelsInit.c
99+
│ ├── PULPDronetKernelsInit.h
100+
│ ├── Makefile
101+
│ └── run_dataset.sh
102+
├── weights/
103+
│ ├── binary/
104+
│ └── WeightsPULPDroNet.raw
105+
├── LICENSE.apache.md
106+
├── LICENSE_README.md
107+
└── README.md
102108
```
103109

104110
More in detail:
@@ -127,7 +133,7 @@ To start running PULP-DroNet, there are few preliminary steps you have to follow
127133
* **Modify/Produce the PULP-Shield**: you want to modify or reproduce our PULP-Shield, we release all the *Altium* design of it:
128134
* you can import the `PULP-Shield/GAP8` project into your Altium Designer tool and update the schematics or the PCB files;
129135
* we also release the fabrication files. Thus, you are ready to go with the production of our PULP-Shield.
130-
* **Run PULP-DroNet on the CarzyFlie 2.0**: you want to run the PULP-DroNet binary as it is, on a PULP-Shield mounted on a CrazyFlie 2.0 nano-drone. In this case, you need to:
136+
* **Run PULP-DroNet on the CarzyFlie 2.0/2.1**: you want to run the PULP-DroNet binary as it is, on a PULP-Shield mounted on a CrazyFlie 2.0/2.1 nano-drone. In this case, you need to:
131137
* [install the PULP-SDK](#sdk);
132138
* [coming soon...](#pulp_cf)
133139

@@ -177,9 +183,9 @@ If you prefer, you can configure them in your `~/.bashrc` file.
177183
In the rest of this README, we assume the following file-system structure:
178184

179185
```
180-
├── .
181-
│ ├── pulp-dronet/
182-
│ ├── pulp-sdk/
186+
.
187+
├── pulp-dronet/
188+
└── pulp-sdk/
183189
```
184190

185191
If your file-system looks different, please adjust the commands in the next sections accordingly to your folders' locations.
@@ -201,21 +207,22 @@ To double-check that the `basic kernels` and `generators` are correctly download
201207

202208

203209
```
204-
├── autotiler/
205-
│ ├── include/
206-
│ │ ├── AutoTilerLib.h
207-
│ │ ├── AutoTilerLibTypes.h
208-
│ │ ├── CNN_BasicKernels.h
209-
│ │ ├── CNN_Generator.h
210-
│ │ ├── CNN_HwCE.h
211-
│ │ ├── Gap8.h
212-
│ │ ├── HashName.h
213-
│ │ ├── KernelLibStdTypes.h
214-
│ │ ├── StdTypes.h
215-
│ ├── src/
216-
│ │ ├── CNN_BasicKernels.c
217-
│ │ ├── CNN_Generator.c
218-
│ │ ├── CNN_HwCE.c
210+
.
211+
└── autotiler/
212+
├── include/
213+
│ ├── AutoTilerLib.h
214+
│ ├── AutoTilerLibTypes.h
215+
│ ├── CNN_BasicKernels.h
216+
│ ├── CNN_Generator.h
217+
│ ├── CNN_HwCE.h
218+
│ ├── Gap8.h
219+
│ ├── HashName.h
220+
│ ├── KernelLibStdTypes.h
221+
│ └── StdTypes.h
222+
└── src/
223+
├── CNN_BasicKernels.c
224+
├── CNN_Generator.c
225+
└── CNN_HwCE.c
219226
```
220227

221228

@@ -239,10 +246,11 @@ This information will be sent to GWT, and you will automatically receive at the
239246
To double-check the Autotiler library is correctly installed you can look for it under `pulp-dronet/src/autotiler` and see:
240247

241248
```
242-
├── autotiler/
243-
│ ├── lib/
244-
│ │ ├── libtile.a
245-
│ │ ├── libtile.dronet.a
249+
.
250+
└── autotiler/
251+
└── lib/
252+
├── libtile.a
253+
└── libtile.dronet.a
246254
```
247255

248256

@@ -256,7 +264,7 @@ The PULP-DroNet dataset is composed by the following three sub-sets, provided as
256264
<img style="float: right;" src="imgs/PULP_dataset.png" width="100%">
257265

258266
The original [Udacity](https://github.com/udacity/self-driving-car) and [Zurich Bicycle](https://github.com/uzh-rpg/rpg_public_dronet) datasets were previously released in their respective open-source projects.
259-
We redistribute part of them (i.e., only the *testing* part of the datasets) with modified resolution, color-scale, and format to match our Himax ultra-low-power camera configuration.
267+
We redistribute part of them (i.e., only the *testing* part of the datasets) with modified resolution, color-scale, and format to match our Himax HBM01 ultra-low-power camera configuration.
260268
To download all submodules, you can execute the following commands in your terminal:
261269

262270
~~~~shell
@@ -268,18 +276,19 @@ $ git submodule update -- dataset/
268276
Then the final structure of the dataset should look like this:
269277

270278
```
271-
├── pulp-dronet/
272-
│ ├── dataset/
273-
│ │ ├── Himax_Dataset/
274-
│ │ │ ├── test_2/
275-
│ │ │ ├── ...
276-
│ │ │ ├── test_23/
277-
│ │ ├── Udacity_Dataset/
278-
│ │ │ ├── HMB_3/
279-
│ │ ├── Zurich_Bicycle_Dataset/
280-
│ │ │ ├── DSCN2571/
281-
│ │ │ ├── ...
282-
│ │ │ ├── GOPR0386/
279+
.
280+
└── pulp-dronet/
281+
└── dataset/
282+
├── Himax_Dataset/
283+
│ ├── test_2/
284+
│ ├── ...
285+
│ └── test_23/
286+
├── Udacity_Dataset/
287+
│ └── HMB_3/
288+
└── Zurich_Bicycle_Dataset/
289+
├── DSCN2571/
290+
├── ...
291+
└── GOPR0386/
283292
```
284293

285294
Each sub-folder contains a variable number of gray-scale `.pgm` images and one `.txt` or `.csv` file with all the ground-truth labels.
@@ -379,7 +388,7 @@ Once the environment is set, you need first to flash the network's weights/biase
379388

380389
**Programming the PULP-Shield**
381390

382-
In case you want to use our PULP-Shield board (both connected to the CrazyFlie 2.0 or stand-alone) you need to have an Olimex Openocd ARM JTAG debugger [ARM-USB-OCD-H](https://www.olimex.com/Products/ARM/JTAG/ARM-USB-OCD-H/) and a *PULP JTAG converter board*.
391+
In case you want to use our PULP-Shield board (both connected to the CrazyFlie 2.0/2.1 or stand-alone) you need to have an Olimex Openocd ARM JTAG debugger [ARM-USB-OCD-H](https://www.olimex.com/Products/ARM/JTAG/ARM-USB-OCD-H/) and a *PULP JTAG converter board*.
383392

384393
<img style="float: right;" src="imgs/PULP_setup.png" width="100%">
385394

@@ -453,5 +462,5 @@ Note that in case you are using a GAPuino board you need to replace `--cable=ftd
453462

454463

455464
<a name="pulp_cf"></a>
456-
## 6. Run PULP-DroNet on the Crazyflie 2.0
457-
This section describes the use case of running PULPDroNet on the PULP-Shield when it is plugged to the Crazyflie 2.0.
465+
## 6. Run PULP-DroNet on the Crazyflie 2.0/2.1
466+
This section describes the use case of running PULPDroNet on the PULP-Shield when it is plugged to the Crazyflie 2.0/2.1.

0 commit comments

Comments
 (0)
0