8000 esp32: control the python heap size via NVS variables · micropython/micropython@e272f26 · GitHub
[go: up one dir, main page]
Skip to content

Commit e272f26

Browse files
tvetve
committed
esp32: control the python heap size via NVS variables
This commit allows the user to limit the micropython heap size by setting NVS variables. This supports: - ensuring that esp-idf has enough memory for multiple TLS connections, TLS plus BLE, etc. - leaving some SPIRAM memory for native modules, such as camera framebuffers - reducing the SPIRAM memory to limit GC sweep times See the docs changes for more details.
1 parent fd719ad commit e272f26

File tree

4 files changed

+149
-1
lines changed

4 files changed

+149
-1
lines changed

‎docs/esp32/quickref.rst

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -537,3 +537,61 @@ the corresponding functions, or you can use the command-line client
537537

538538
See the MicroPython forum for other community-supported alternatives
539539
to transfer files to an ESP32 board.
540+
541+
Controlling the Python heap size
542+
--------------------------------
543+
544+
_This feature is only supported with esp-idf v4 and above._
545+
546+
By default MicroPython allocates the largest contiguous chunk of memory to the python heap.
547+
On a "simple" esp32 this comes out to around 100KB and on an esp32 with external SPIRAM this
548+
ends up being the full SPIRAM, typically 4MB. This default allocation may not be desirable
549+
and can be reduced for two use-cases by setting one of two variables in the "micropython" NVS
550+
namespace, see :ref:`esp32.NVS <esp32.NVS>` for details about accessing NVS (Non-Volatile
551+
Storage).
552+
553+
Because MicroPython allocates the heap as one of the very first actions it is not possible to run
554+
python code to set the heap size. This is the reason that NVS variables are used and it also means
555+
that a hard reset is necessary after setting the variables before they take effcet.
556+
A typical use is for `main.py` to check heap sizes and, if they're not appropriate,
557+
to set an NVS variable and perform a hard reset.
558+
559+
The first use-case for this feature is to guarantee that ESP-IDF has some minimum amount of memory
560+
to work with. For example, by default without SPIRAM there is around 90KB left for ESP-IDF
561+
right at boot time. This is not enough for two TLS connections and not enough for one
562+
TLS connection and BLE either. (While 90KB might seem like a lot, it disappears quickly once
563+
Wifi is started and sockets are connected.)
564+
565+
To give esp-idf a bit more memory, use something like:
566+
567+
import machine
568+
from esp32 import NVS, idf_heap_info
569+
570+
idf_free = sum([h[2] for h in idf_heap_info(HEAP_DATA)])
571+
print("IDF heap free:", idf_free)
572+
573+
nvs = NVS("micropython")
574+
nvs.set_i32("min_idf_heap", 120000)
575+
nvs.commit()
576+
machine.reset()
577+
578+
Setting the "min_idf_heap" NVS variable to 120000 tells MicroPython to reduce its heap allocation
579+
from the default such that at least 120000 bytes are left for esp-idf.
580+
581+
A second use case is to reduce GC times when using SPIRAM. A GC collection has to read sequentially
582+
though all of RAM during its sweep phase. When using a SPIRAM with the default 4MB allocation this
583+
takes about 90ms (assuming 240Mhz cpu and 80Mhz QIO SPIRAM), which is very impactful in a not so
584+
good way. Often applications only need a few hundred KB and this can be accomplished by setting the
585+
"max_mp_heap" NVS variable to the desired size in bytes.
586+
587+
A similar use-case is with an SPIRAM where it is desired to leave memory to a native module, for
588+
example to allocate a camera framebuffer. The size of the MP heap can be limited to at most 300KB
589+
using something like:
590+
591+
import machine
592+
from esp32 import NVS
593+
594+
nvs = NVS("micropython")
595+
nvs.set_i32("max_mp_heap", 300*1024)
596+
nvs.commit()
597+
machine.reset()

‎ports/esp32/boards/sdkconfig.spiram

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,12 @@
33
CONFIG_ESP32_SPIRAM_SUPPORT=y
44
CONFIG_SPIRAM_CACHE_WORKAROUND=y
55
CONFIG_SPIRAM_IGNORE_NOTFOUND=y
6-
CONFIG_SPIRAM_USE_MEMMAP=y
6+
CONFIG_SPIRAM_USE_MEMMAP=n
7+
CONFIG_SPIRAM_USE_CAPS_ALLOC=y
78

89
# v3.3-only (renamed in 4.0)
910
CONFIG_SPIRAM_SUPPORT=y
11+
12+
# temporary
13+
CONFIG_LOG_DEFAULT_LEVEL_INFO=y
14+
CONFIG_LOG_DEFAULT_LEVEL_WARN=n

‎ports/esp32/main.c

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,45 @@ void mp_task(void *pvParameter) {
7979
uart_init();
8080
machine_init();
8181

82+
#if MICROPY_ESP_IDF_4
83+
// Allocate MicroPython heap. By default we grab the largest contiguous chunk (GC requires
84+
// having a contiguous heap). But this can be customized by setting two NVS variables.
85+
// There's nothing special here about SPIRAM because the SDK config should set
86+
// CONFIG_SPIRAM_USE_CAPS_ALLOC=y which makes any external SPIRAM automatically
87+
// show up here under MALLOC_CAP_8BIT memory.
88+
#define MIN_HEAP_SIZE (20 * 1024) // simple safety to avoid ending up with a non-functional heap
89+
size_t avail_heap_size = heap_caps_get_largest_free_block(MALLOC_CAP_8BIT); // in bytes
90+
size_t total_free = heap_caps_get_free_size(MALLOC_CAP_8BIT);
91+
size_t mp_task_heap_size = avail_heap_size; // proposed MP heap size
92+
nvs_handle_t mp_nvs;
93+
if (nvs_open("micropython", NVS_READONLY, &mp_nvs) == ESP_OK) {
94+
// implement minimum heap left for esp-idf
95+
int32_t min_idf_heap_size = 0; // minimum we leave to esp-idf, in bytes
96+
if (nvs_get_i32(mp_nvs, "min_idf_heap", &min_idf_heap_size) == ESP_OK) {
97+
if (total_free - mp_task_heap_size < min_idf_heap_size) {
98+
// we can't take the largest contig chunk, need to leave more to esp-idf
99+
mp_task_heap_size = total_free - min_idf_heap_size;
100+
if (mp_task_heap_size < MIN_HEAP_SIZE) {
101+
mp_task_heap_size = MIN_HEAP_SIZE;
102+
}
103+
}
104+
}
105+
// implement maximum MP heap size
106+
int32_t max_mp_heap_size = mp_task_heap_size; // max we alllocate to MicroPython, in bytes
107+
if (nvs_get_i32(mp_nvs, "max_mp_heap", &max_mp_heap_size) == ESP_OK) {
108+
if (max_mp_heap_size > MIN_HEAP_SIZE && mp_task_heap_size > max_mp_heap_size) {
109+
// we're about to create too large a heap, be more modest
110+
mp_task_heap_size = max_mp_heap_size;
111+
}
112+
}
113+
nvs_close(mp_nvs);
114+
}
115+
void *mp_task_heap = heap_caps_malloc(mp_task_heap_size, MALLOC_CAP_8BIT);
116+
if (avail_heap_size != mp_task_heap_size) {
117+
printf("Heap limited: avail=%d, actual=%d, left to idf=%d\n",
118+
avail_heap_size, mp_task_heap_size, total_free - mp_task_heap_size);
119+
}
120+
#else
82121
// TODO: CONFIG_SPIRAM_SUPPORT is for 3.3 compatibility, remove after move to 4.0.
83122
#if CONFIG_ESP32_SPIRAM_SUPPORT || CONFIG_SPIRAM_SUPPORT
84123
// Try to use the entire external SPIRAM directly for the heap
@@ -103,6 +142,8 @@ void mp_task(void *pvParameter) {
103142
size_t mp_task_heap_size = heap_caps_get_largest_free_block(MALLOC_CAP_8BIT);
104143
void *mp_task_heap = malloc(mp_task_heap_size);
105144
#endif
145+
#endif
146+
106147

107148
soft_reset:
108149
// initialise the stack pointer for the main thread

‎tests/esp32/limit_heap.py

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# Test MicroPython heap limits on the esp32.
2+
# This test requires resetting the device and thus does not simply run in run-tests.
3+
# You can run this test manually using pyboard and hitting ctrl-c after each reset and rerunning
4+
# the script. It will cycle through the various settings.
5+
import gc
6+
import machine
7+
from esp32 import NVS, idf_heap_info, HEAP_DATA
8+
9+
nvs = NVS("micropython")
10+
try:
11+
min_idf = nvs.get_i32("min_idf_heap")
12+
except OSError:
13+
min_idf = 0
14+
try:
15+
max_mp = nvs.get_i32("max_mp_heap")
16+
except OSError:
17+
max_mp = None
18+
19+
mp_total = gc.mem_alloc() + gc.mem_free()
20+
print("MP heap:", mp_total)
21+
idf_free = sum([h[2] for h in idf_heap_info(HEAP_DATA)])
22+
print("IDF heap free:", idf_free)
23+
24+
if min_idf == 0:
25+
nvs.set_i32("min_idf_heap", 100000)
26+
nvs.commit()
27+
print("IDF MIN heap changed to 100000")
28+
machine.reset()
29+
elif max_mp is None:
30+
nvs.set_i32("max_mp_heap", 50000)
31+
nvs.commit()
32+
print("MAX heap changed to 50000")
33+
machine.reset()
34+
else:
35+
try:
36+
nvs.erase_key("min_idf_heap")
37+
except OSError:
38+
pass
39+
try:
40+
nvs.erase_key("max_mp_heap")
41+
except OSError:
42+
pass
43+
print("Everything reset to default")
44+
machine.reset()

0 commit comments

Comments
 (0)
0