8000 Failure to allocate when free memory is very large - severe fragmentation? · Issue #2057 · micropython/micropython · GitHub
[go: up one dir, main page]

Skip to content

Failure to allocate when free memory is very large - severe fragmentation? #2057

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ryannathans opened this issue May 9, 2016 · 22 comments
Closed

Comments

@ryannathans
Copy link
Contributor

Hi all

We're experiencing the following issue at least once per hour of runtime.

Task Network Router Task took 442480 us, mem free pre gc: 41296 B post gc: 46352 B, freed 5056 B
Task Hardware WatchDog took 103 us, mem free pre gc: 46224 B post gc: 46352 B, freed 128 B
Task NETWORK WDT TASK took 153 us, mem free pre gc: 46192 B post gc: 46352 B, freed 160 B
Task ____ APP TASK took 215 us, mem free pre gc: 46096 B post gc: 46352 B, freed 256 B
Task ____ APP TASK took 221 us, mem free pre gc: 46096 B post gc: 46352 B, freed 256 B
Task ____ Application took 251 us, mem free pre gc: 46096 B post gc: 46352 B, freed 256 B
Task RegularReset took 99 us, mem free pre gc: 46224 B post gc: 46352 B, freed 128 B
Task RESET TASK took 117 us, mem free pre gc: 46224 B post gc: 46352 B, freed 128 B
Task LogKeeper took 101 us, mem free pre gc: 46192 B post gc: 46352 B, freed 160 B
Task NETWORK INTERFACE CONTROL TASK took 142 us, mem free pre gc: 46224 B post gc: 46352 B, freed 128 B
Task ____ BOOT CTRL TASK took 120 us, mem free pre gc: 46224 B post gc: 46352 B, freed 128 B
Task GPS SENDING TASK took 120 us, mem free pre gc: 46224 B post gc: 46352 B, freed 128 B
Task GPS info packet generator took 129 us, mem free pre gc: 46224 B post gc: 46352 B, freed 128 B
Task NMEA composite dictionary generator took 132 us, mem free pre gc: 46224 B post gc: 46352 B, freed 128 B
Task NMEA strings parser took 1509 us, mem free pre gc: 44688 B post gc: 45200 B, freed 512 B
Task File Downloader took 292 us, mem free pre gc: 45008 B post gc: 45200 B, freed 192 B
Task ____ REGISTRATION TASK took 210 us, mem free pre gc: 45072 B post gc: 45200 B, freed 128 B
Task ____ MONITOR TASK took 157 us, mem free pre gc: 45072 B post gc: 45200 B, freed 128 B
Task ___ TASK took 157 us, mem free pre gc: 45072 B post gc: 45200 B, freed 128 B
Task BATTERY TASK took 168 us, mem free pre gc: 45072 B post gc: 45200 B, freed 128 B
PACKET TERMINATOR CLIPPED BY BUFFER, DROPPING
UNHANDLED EXCEPTION: Network Router Task
WARNING: Descheduling task

Traceback (most recent call last):
  File "core/scheduler.py", line 83, in _run_next
  File "softmodules/router.py", line 253, in run
  File "softmodules/router.py", line 155, in rx_network
  File "softmodules/router.py", line 49, in frame_packet
MemoryError: memory allocation failed, allocating 581 bytes
Traceback (most recent call last):
  File "main.py", line 50, in <module>
  File "core/scheduler.py", line 55, in run
  File "core/scheduler.py", line 93, in _run_next
  File "core/scheduler.py", line 91, in _run_next
  File "core/scheduler.py", line 83, in _run_next
  File "softmodules/router.py", line 253, in run
  File "softmodules/router.py", line 155, in rx_network
  File "softmodules/router.py", line 49, in frame_packet
**MemoryError: memory allocation failed, allocating 581 bytes**
SYSTEM RESETTING IN LESS THAN 30 SECONDS

We're failing to allocate 581 bytes of memory when we have over 45k of free memory, what's going on? I can only assume this is some kind of severe fragmentation?

I've done a bit of investigation and uncovered the following discussions. My understanding is that memory fragmentation is a known issue and isn't currently being worked on. Progress towards a solution occurs slowly and in small increments.
#1161
#1168
#1604

http://forum.micropython.org/viewtopic.php?f=3&t=455
https://github.com/micropython/micropython/wiki/Memory-Manager

Do you guys have any suggestions for improving robustness or techniques we can use to avoid this? Is there a way to measure memory fragmentation and restart the device safely when fragmentation approaches a threshold? Are there certain objects that significantly contribute to fragmentation that I should avoid using when writing code?

I like @dpgeorge 's ideas on the forum. Either having a memory-manager that defrags the memory or is able to allocate in fragmented memory using filesystem-like blocks.

@dpgeorge
Copy link
Member

What hardware are you running on?

You can use micropython.mem_info(1) to show a dump of the heap and how it's being used. Please report that.

@ryannathans
Copy link
Contributor Author

@dpgeorge I wasn't aware of that function. Will provide when I'm with the device again, some time in the next 12ish hours, maybe sooner.

Using STM32F439 (64k more ram allocated to heap than f405)

I took the ~100KB of code (tabs not spaces!) and built it in as frozen mpy. I've now got 100k total free ram and the issue doesnt occur frequently enough to be of concern (for now...)

@dpgeorge
Copy link
Member

built it in as frozen mpy

I assume you mean using mpy-cross, the config option MICROPY_MODULE_FROZEN_MPY, and mpy-tool? Yes, doing this is the best way to free up RAM.

@ryannathans
Copy link
Contributor Author
ryannathans commented May 11, 2016

@dpgeorge I'm compiling with the following arguments, that seems to take care of the hard work for me.

make -j12 BOARD=STM32F439 FROZEN_MPY_DIR=scripts

@ryannathans
Copy link
Contributor Author

What do the letters in the dump refer to?

Task NMEA composite dictionary generator took 116 us, mem free pre gc: 48800 B post gc: 48928 B, freed 128 B
Task NMEA strings parser took 1815 us, mem free pre gc: 47232 B post gc: 47744 B, freed 512 B
Task BaseTask took 138 us, mem free pre gc: 47616 B post gc: 47744 B, freed 128 B
Task File Downloader took 271 us, mem free pre gc: 47552 B post gc: 47744 B, freed 192 B
Task ____ REGISTRATION TASK took 202 us, mem free pre gc: 47616 B post gc: 47744 B, freed 128 B
Task __ MONITOR TASK took 153 us, mem free pre gc: 47616 B post gc: 47744 B, freed 128 B
Task IGNITION TASK took 142 us, mem free pre gc: 47616 B post gc: 47744 B, freed 128 B
Task BATTERY TASK took 148 us, mem free pre gc: 47616 B post gc: 47744 B, freed 128 B
PACKET TERMINATOR CLIPPED BY BUFFER, DROPPING
UNHANDLED EXCEPTION: Network Router Task
WARNING: Descheduling task

Traceback (most recent call last):
  File "core/scheduler.py", line 83, in _run_next
  File "softmodules/router.py", line 253, in run
  File "softmodules/router.py", line 155, in rx_network
  File "softmodules/router.py", line 49, in frame_packet
MemoryError: memory allocation failed, allocating 582 bytes
Traceback (most recent call last):
  File "main.py", line 52, in <module>
  File "core/scheduler.py", line 55, in run
  File "core/scheduler.py", line 93, in _run_next
  File "core/scheduler.py", line 91, in _run_next
  File "core/scheduler.py", line 83, in _run_next
  File "softmodules/router.py", line 253, in run
  File "softmodules/router.py", line 155, in rx_network
  File "softmodules/router.py", line 49, in frame_packet
MemoryError: memory allocation failed, allocating 582 bytes
SYSTEM RESETTING IN LESS THAN 30 SECONDS

stack: 336 out of 15360
GC: total: 166400, used: 145840, free: 20560
No. of 1-blocks: 3433, 2-blocks: 434, max blk sz: 257
GC memory layout; from 20003600:
03600: MDhhhhhtttttttttttttttttttttttttttttttttttttththhhhBLLhhhhTLhhth
03a00: hMDhhhhhMDhtthththttDhhhtttttthhhhThLhhhhhhhhhhhhhhhhhhMDDhDDhhh
03e00: htttttttBhhhhtttttttthhhttttttthhtttttthhtttttTTTTThttttttthhttt
04200: tttthBLBhhhhhhThMDBBTBhhttttttttttttttttttttttttttttttttttthtthh
04600: ttttttthttttttthttttttttthhBtBtBhLFhDhhLhBBhttttttttTDhthDhDDhDh
04a00: tDhtDhththtDTthttttttthttttttttttDhtDhtDhtDhDhtDhDhtttLhtLhtttTD
04e00: hTDhTDhTDhTDhTDhTTTLhTThhhhLhhhhttttttththttttDDhhtttttttttttttt
05200: ttttttThthttttthhhhttTthhhTTTTThThtttttttttttttttthtttthLhhhhTTh
05600: ThhTTtTthttttttttttttttttttttttttttttttttttttttttttttttttttttttt
05a00: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
05e00: ttttttThhThhhttthDTTDDDLhDDhhhththhThTthhDLhhthhhhhhTthThTtTTTTh
06200: hThThhTthLhhhhTthhThThLhhtthhthhttttthtttttBhhBththttthhhthttttt
06600: ttttttttMDMThDBBBBBhBThhhttttttthttthLhtttThhhhthhMDDBBBBBBBBhtt
06a00: thtttthDBBThBhtthttthhtttttthThhtthtthtthtthttthtttthttthtththht
06e00: thTThhhThTtTthThhTtTthhhTtThTthTTThtthThttThhhTTtTtTTthTTthTTThT
07200: ThttttttDhhttttttttttttttttttttttthtthtDBhtttttttDTBBBhtttBBBBBB
07600: htttBhDhttttthttthDTDhhtttthttttttthtttThDhDTDLhhthhhhLhhhhhtttt
07a00: tttThhhDLhhthhtttTtTDhhhTDLhhDhtMDThhLhhhhTthhttttthtDDMDDDTthtD
07e00: BBThhBBhttttTThhttttttthttthhThBhthtttttttDhhhThtttttttttttttttt
08200: ttttthtttttttttttttttttttttttttttttttttttttttthThttthttththtttht
08600: ThhtttThtthtttTthTMhtthttThDLDhTDTthDhhLhhtthhhTthhttttDDhhttDTt
08a00: hhhhhBhthTThhLhhttthhhhhttthtthhhLhttttttthhhhThhhtttttthtDDhttT
08e00: ththhtthhtthhthhthhthhtthhthhthhtthhthhthhthhthhthhtttthhtttthhh
09200: LhhhhtthhhhtttttttttththhhhhhhhhhhTThhLhhhTThhttttthttDThhhttttt
09600: ttDDLDhhtDLhhthhttttttthtttttththtttttttMDhttttthTBBDTTtThtttttt
09a00: thtttTBBBBhthThThThLhhhttttttthhhttThtttttthttthhLhhhThttttttttT
09e00: hthhttttttDhhttDhMDhhtthDThhLhhhhhhhhDBTthLhhhhhhhhhhBhhhttTTBhh
0a200: TtDLhhhtttBBBhhttThttthTDhthttttththtttttLhDhtthttttttttttthtttt
0a600: ttthtthLhThThhttthttttthtthttttthttttttttttthttttttttttttttttttt
0aa00: ttttttthhthtthhttthhtthhhhttttthhhthhthtthhtttttttttttttttttthTt
0ae00: hhhttTthtTtMhhtLhhTLhttttttthtthhhhtttttttMDhhhthttDMhDhttThhttt
0b200: tttthtthTthhhthtthhtttttttttthtthTthDTTtTtBBTTTtTthhDTBBhttttttt
0b600: BThttthBBhDhttttttThttttttthttttttttThtthhhTthtthttttttthhttttth
0ba00: hthhtthtthhttttttthhhthttttthhtttttttthhhhhTtTttBBTTtTtTthtttTtB
0be00: htthhhttthttttTDTtTthtttttttTtTthhhBThhhBhTtTtTthttttttttttthttt
0c200: tttttttttttttttTthhttttthtttTtDBhthThttthhDBhtThhtBhThttThttthtt
0c600: TthttThttttttttttthttTthhthtthtthttthtttthhhttthttttththttthhttt
0ca00: tttttttthhhTthhThttttttthtthhDTThBhhtthttthttttttttThttthDhBThhh
0ce00: tthhhtthttttttthhttttttthtttDTBhThttThttthhttthtthhhhhTthhhtthhh
0d200: httTthhMDhttThDhhhhThhthhtttttttttttttttttttttttttttTthhhttBhTht
0d600: thhTthttTTtTthhhThTthhhhttthttthtthhhTththTtTthhhtthhhhhhhttMDht
0da00: tThhTthDBTthttTthttttttthhtthTthhhhhhhTtBBThtthhhhttBthDhDTBhTBh
0de00: hthtthttththttthDTBhththtttDBBBhhBhThDBhhhhtthhhhtthhhtthhhTthhh
0e200: tthhhTthtthhhttTthhthhhtttthhtttttthhhthttttttthhhthhhtthhtttttt
0e600: thhtttttttttttttttttttthtthhhhthhhhhhhhhhhhthhtthttttttttttttttt
0ea00: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
0ee00: ththhhThthtTthttttttthttthttthtttTBhtthhhhhthhhhhhhhhtthhhhttTth
0f200: hBTthhhttttttttttttBThhhhttthttthhhDThhTBTThhhBhhtttDThtttttttth
0f600: tthtttThttthDTBhBTtttBBBBTththhhtthhhhtthhBhBhTDhttttttttTBhTThh
0fa00: tttttttththhtthtttThhhttttttttttthttttttthtthhhttTttthhThhtthTth
0fe00: hhhDTtThhhTthttttttthtthhttttttttttthhhtthhhhhtthhhTthTthhhhtthh
10200: TTthhhhhhhthtthhhhthBhhthttthhDTBhThDTTBBhhttBhTthttttttthtttThD
10600: TTthttttttthhttttttthhhhhttthttthtthhhhhhhhhTttthhhhhhhttttttthh
10a00: hhhhhhhBhThttThttthDTBhTBThhhttthttttthttttDhTBhTBhBBBBBBTLhhttt
10e00: hhthhhhhhhtttDhThhTTthhhhhhBBBBBBBhBhTThhttthttthhhhhttttttthhht
11200: tttttttttttttthhhhhhhhhhhhhhhtttttttttttttthttttttttttthtttttttt
11600: tttttthhhDhhhhhhhhhhhhhhhhhthhhhhhhhhhhhtthhhhTthhhttttttthhhhhh
11a00: hhhhthhhhhttttttthhFhttttttthhhhhhhhTthttthMDhhhhhhhhhhhhhthhhhh
11e00: hhhhhTthhhtttDhThhhhhhhhTthhhDBhttttttthtBhhBtBhhhBBBBBBBBBBhhhh
12200: hhhhhhhhhhhhhhhtthhhhhhthhhhhhhhhhhhhhBhThttthttttthDhDhhhDhtttt
12600: htttttttttttttttttthhhhhhhhhhhhhhttttthttttttttttttttthhhhhhhthh
12a00: hhhhhhhhhhhTthhhhhtttthhhthhhhhhthhhhhhhhhhhhhhhhhhhhhthhhhhhthh
12e00: hhhhTthhhhhhhhhhhhthhhhthhhhthhthhthhththhthhttttttttttttttthhht
13200: httttthhttttttttttttthhttttttttttttttttttttttttttttttttttththhtt
13600: thhttthhtthhhhtthhthttttttttttttttttttttthhhhhhttttttthhhthttttt
13a00: tthhhthttttttthhhtthttttttttttttthhhhhthttttttttttttttttthhhhhth
13e00: ttttttttttthhttttthhhhhhhhttttthttttthhhhhhhhhhhDDhhhttttttthttt
14200: ttttttttttttthhthhhhhhhhhhhhhhhhhhhhthttttttttttthhTthhhhhhhhhhh
14600: hhhhhtthtttttttthhhttttttttttttttttttttttttttttttttttttttttttttt
14a00: ttttttttttttttttttthhhhhttttttthttttttttttthhhhhhttttttthhhhhhhh
14e00: hhhhhhhhDhhhhhhhhhhhhhtttLhhhhthhhhhhhLhhhhhhhhhhhhhhhhttttttFDL
15200: hhthhhtthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhDDDhhhttthtDDhhhhDLhhthhth
15600: hhhhtTthhttttttthhhhhhhhhhhhhhhhttttthhhhhhhttthtDDhhhhhhhLhhhhh
15a00: hDLhhthhhhLhhhhhhhhtttttthDLhhthhhhDLhhthhDLhhthhtDDLhhthDDLhThh
15e00: thhtttLhhtDhDLhhthhhhhhhttttthhtDLhttttttthhthhhhLhhhhhLhDLhhhhh
16200: thhhhhhLhhhTthhhhhhhDLhhthhDLhhhhhththhDLhhhhhhhhhhhhtDDhLhhthtt
16600: htttttDhhhhDLhhthhhDLhhhhhhhhhhhhhhhhhhttttttttttttttttttttttttt
16a00: ttttttttttttttttththhhhhLhhhhhhhhhhhhDhhLhhthhhhhhhhhhLhhhhhhhth
16e00: DhhhDLhhthDDLhhthhhhhhhhhthttttththhttthtDDhhhhhhhLhhhhhhhhhhthh
17200: hhhhhttththhttththDhhhhhDLhhthttthtDhDhhhthhhhhhtthhhttththhhhhh
17600: hhhhhhhhhhhhhhhhhhhthththhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhthhhhhh
17a00: hhhhhhhhhhhhhhtTthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
17e00: hhhhhhhhthhhhhhhhhhhhhhTthhhhthhhhhhhhhhhhhhTthhhhhhhhhhhhhhhhht
18200: thhhhhhhhhhhhhhhhhhhhthhhhhhthhthhtthhthhhhthhthhtththhhhhthhhth
18600: htttttthhttttthhttttttthhttttttttthtttthhthhthhthhthhtthhhhtthht
18a00: hhthhhhthhtthhhhtthhhhtthhhttttttttttththhhhhhthhthhthttttttthtt
18e00: thhhhhhthhthhthhhhhhhhthhhhthhhthhthhtthhtthhttttttttttthtttttth
19200: htthhhhhhthhhhhhhhthhhhthhhhhhthhhhthhhhhhthhhhthhhhhhthhhhthttt
19600: ttttthhhhhtttttttttttthttttttthhhthttttttthhttttttttttttttthhhth
19a00: hhhhtthhtttttttttththhhhhhhhhttttttththhtthhthhhhthhthhtthtthhtt
19e00: tttttthhhthhthtttttthhttttthhhthhthtttttttthhthhtttttttththhhhhh
1a200: hhhhhttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
1a600: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
1aa00: ttttttttttttttttttttttttttttttttttttthhhhhhhhhhhTthhhhhhhhhhhhhh
1ae00: hhhhhhhhhhhhhhhhhhhhhhThttttttthhhhhhhhhhhhhhhhhhthhhhhttttttthh
1b200: hhhhhhhhTthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhthhhhttttttthhh
1b600: hhhhhhhhhhhhhTthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
1ba00: hhhhhhhhthhhhhhhhhhhhhhTthhhhhtttttttttttttttttttttttttttttttttt
1be00: tttttttttttttttttttttttttttttttttttthttttttttttttttttttttttttttt
1c200: tttttttthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhthhhthhhhhhhhhhhhhhTtTthh
1c600: hhhhhhhhhhhhhhhhhhhtttthtttttttttttttttttthtthtthttttttttttttttt
1ca00: tttttththttttttttttttthttttthtthtttttthhhhtthhhttttttthtttthhtth
1ce00: hhtthhhhhhhhhhttttttthtttttthhhhhhhhhhhhhhhthhhhhhhhhhhhthhhhhhh
1d200: hhhhhhhTthhhttttttttttttttttthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
1d600: hhhhhhhhhthhhhhhhhhhhhhhTthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
1da00: hhhhhhhhthhhhhhhthhthhhhhhthhhhhhhthhhhhhhhhhhTthhhhhthhhhhhhhhh
1de00: hhhhhthhhhthhthhhhhttttthhttttttthhhhhhhhttttttttttttttttttttttt
1e200: ttththhhhhhthhthttttthhttttttttttttttttthhhhhhttttttttttttttttth
1e600: tthhhhthhthhthhhhhhthhthhthttttttttttttttttttthhhthhthhthtttthht
1ea00: ttthhhthhhtttttttttttttttttttttttthhhthhthttttttthhhthhthttthhtt
1ee00: hhtthhtttthhtttttttttttthtthhttttthhthtttthhttthhttthhttttthhhth
1f200: hthtttthhhthhthtttttttttthhhthhthttttttthhtttttttttttttthhhththh
1f600: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
1fa00: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
1fe00: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
20200: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
20600: httttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
20a00: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
20e00: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
21200: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
21600: thhhhhhhhhhTthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhthhhhhhhhhhhhhh
21a00: Tthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhthhhhhh
21e00: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
22200: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
22600: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
22a00: ttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttth
22e00: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
23200: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
23600: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
23a00: ttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttth
23e00: hhhhhhhhTthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
24200: hthhhhhhhhhhhhtttttthhhTthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
24600: hhhhhhhhhhhhhhhhthhhhhhhhhhhhhhTthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
24a00: hhhhhhhhhhhhhhhhhhhhhhhhhthhhhhhhhhhhhhhTthhhhhhhhhhhhhhhhhhhhhh
24e00: Ththhhhhhhhhhhhhhhhh...........hht..............Tt..............
25200: ..............hh.........hh............hht..............Tt......
25600: ......................hh.........hh............hht..............
25a00: Tt............................hh.........hh............hht......
25e00: ........Tt...................................hh.........hh......
26200: ......hht..............Tt.........hh.........hh............hht..
26600: ............Tthttttttttttttttttttttttttttttttttttt....hh........
26a00: .hh............hht..............Tt..................hT..........
26e00: .....hh.........hh............hht..............Tt...............
27200: ......hh.........hh............hht..............Tthttttttttttttt
27600: tttttttttttttttttttttt..........................hh.........hh...
27a00: .........hht..............Tthttttttttttttttttttttttttttttttttttt
27e00: httttttttttttttttttttttttttttttttttt............................
28200: hh.........hh............hht..............Tt....................
28600: ........hh.........hh............hht..............Tt............
28a00: ................hhhh.......hh..hh.h......hht.hh.........htTthht.
28e00: ..........Tt........Tt.hh.........hh............hht.............
29200: .Tt.............................hh.........hh............hht....
29600: ..........Tt................hh.........hh............hht........
29a00: ......Tthttttttttttttttttttttttttttttttttttttttt..hh.........hh.
29e00: ...........hht.....hT.......Tt............................hh....
2a200: .....hh............h.hhht.............Tthttttttttttttttttttttttt
2a600: tttttttttttthttttttttttttttttttttttttttttttttttttttttttttttttttt
2aa00: ttttttttttt.hh.........hh............hht..............Tt........
2ae00: ....................hh.........hh............hht..............Tt
2b200: hh.........hh............hht..hh.....h....hTt.....h............h
2b600: ht..............Tt............................hh.........hh.....
2ba00: .......hht..............Tt............................hh........
2be00: .hh............hht.........hh..h

@dhylands
Copy link
Contributor

The letters correspond to objects in the heap. The dots are free memory
(IIRC each dot corresponds to 16 bytes. To allocate 582 bytes, you'd need
37 consecutive dots.

The letters correspond to different types of objects in the heap. See:
https://github.com/micropython/micropython/blob/master/py/gc.c#L732-L742

Each object consists of one head block and 0-n tail blocks ('t')

On Tue, May 10, 2016 at 6:36 PM, Ryan Shaw notifications@github.com wrote:

What do the letters in the dump refer to?

Task NMEA composite dictionary generator took 116 us, mem free pre gc: 48800 B post gc: 48928 B, freed 128 B
Task NMEA strings parser took 1815 us, mem free pre gc: 47232 B post gc: 47744 B, freed 512 B
Task BaseTask took 138 us, mem free pre gc: 47616 B post gc: 47744 B, freed 128 B
Task File Downloader took 271 us, mem free pre gc: 47552 B post gc: 47744 B, freed 192 B
Task ____ REGISTRATION TASK took 202 us, mem free pre gc: 47616 B post gc: 47744 B, freed 128 B
Task __ MONITOR TASK took 153 us, mem free pre gc: 47616 B post gc: 47744 B, freed 128 B
Task IGNITION TASK took 142 us, mem free pre gc: 47616 B post gc: 47744 B, freed 128 B
Task BATTERY TASK took 148 us, mem free pre gc: 47616 B post gc: 47744 B, freed 128 B
PACKET TERMINATOR CLIPPED BY BUFFER, DROPPING
UNHANDLED EXCEPTION: Network Router Task
WARNING: Descheduling task

Traceback (most recent call last):
File "core/scheduler.py", line 83, in _run_next
File "softmodules/router.py", line 253, in run
File "softmodules/router.py", line 155, in rx_network
File "softmodules/router.py", line 49, in frame_packet
MemoryError: memory allocation failed, allocating 582 bytes
Traceback (most recent call last):
File "main.py", line 52, in
File "core/scheduler.py", line 55, in run
File "core/scheduler.py", line 93, in _run_next
File "core/scheduler.py", line 91, in _run_next
File "core/scheduler.py", line 83, in _run_next
File "softmodules/router.py", line 253, in run
File "softmodules/router.py", line 155, in rx_network
File "softmodules/router.py", line 49, in frame_packet
MemoryError: memory allocation failed, allocating 582 bytes
SYSTEM RESETTING IN LESS THAN 30 SECONDS

stack: 336 out of 15360
GC: total: 166400, used: 145840, free: 20560
No. of 1-blocks: 3433, 2-blocks: 434, max blk sz: 257
GC memory layout; from 20003600:
03600: MDhhhhhtttttttttttttttttttttttttttttttttttttththhhhBLLhhhhTLhhth
03a00: hMDhhhhhMDhtthththttDhhhtttttthhhhThLhhhhhhhhhhhhhhhhhhMDDhDDhhh
03e00: htttttttBhhhhtttttttthhhttttttthhtttttthhtttttTTTTThttttttthhttt
04200: tttthBLBhhhhhhThMDBBTBhhttttttttttttttttttttttttttttttttttthtthh
04600: ttttttthttttttthttttttttthhBtBtBhLFhDhhLhBBhttttttttTDhthDhDDhDh
04a00: tDhtDhththtDTthttttttthttttttttttDhtDhtDhtDhDhtDhDhtttLhtLhtttTD
04e00: hTDhTDhTDhTDhTDhTTTLhTThhhhLhhhhttttttththttttDDhhtttttttttttttt
05200: ttttttThthttttthhhhttTthhhTTTTThThtttttttttttttttthtttthLhhhhTTh
05600: ThhTTtTthttttttttttttttttttttttttttttttttttttttttttttttttttttttt
05a00: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
05e00: ttttttThhThhhttthDTTDDDLhDDhhhththhThTthhDLhhthhhhhhTthThTtTTTTh
06200: hThThhTthLhhhhTthhThThLhhtthhthhttttthtttttBhhBththttthhhthttttt
06600: ttttttttMDMThDBBBBBhBThhhttttttthttthLhtttThhhhthhMDDBBBBBBBBhtt
06a00: thtttthDBBThBhtthttthhtttttthThhtthtthtthtthttthtttthttthtththht
06e00: thTThhhThTtTthThhTtTthhhTtThTthTTThtthThttThhhTTtTtTTthTTthTTThT
07200: ThttttttDhhttttttttttttttttttttttthtthtDBhtttttttDTBBBhtttBBBBBB
07600: htttBhDhttttthttthDTDhhtttthttttttthtttThDhDTDLhhthhhhLhhhhhtttt
07a00: tttThhhDLhhthhtttTtTDhhhTDLhhDhtMDThhLhhhhTthhttttthtDDMDDDTthtD
07e00: BBThhBBhttttTThhttttttthttthhThBhthtttttttDhhhThtttttttttttttttt
08200: ttttthtttttttttttttttttttttttttttttttttttttttthThttthttththtttht
08600: ThhtttThtthtttTthTMhtthttThDLDhTDTthDhhLhhtthhhTthhttttDDhhttDTt
08a00: hhhhhBhthTThhLhhttthhhhhttthtthhhLhttttttthhhhThhhtttttthtDDhttT
08e00: ththhtthhtthhthhthhthhtthhthhthhtthhthhthhthhthhthhtttthhtttthhh
09200: LhhhhtthhhhtttttttttththhhhhhhhhhhTThhLhhhTThhttttthttDThhhttttt
09600: ttDDLDhhtDLhhthhttttttthtttttththtttttttMDhttttthTBBDTTtThtttttt
09a00: thtttTBBBBhthThThThLhhhttttttthhhttThtttttthttthhLhhhThttttttttT
09e00: hthhttttttDhhttDhMDhhtthDThhLhhhhhhhhDBTthLhhhhhhhhhhBhhhttTTBhh
0a200: TtDLhhhtttBBBhhttThttthTDhthttttththtttttLhDhtthttttttttttthtttt
0a600: ttthtthLhThThhttthttttthtthttttthttttttttttthttttttttttttttttttt
0aa00: ttttttthhthtthhttthhtthhhhttttthhhthhthtthhtttttttttttttttttthTt
0ae00: hhhttTthtTtMhhtLhhTLhttttttthtthhhhtttttttMDhhhthttDMhDhttThhttt
0b200: tttthtthTthhhthtthhtttttttttthtthTthDTTtTtBBTTTtTthhDTBBhttttttt
0b600: BThttthBBhDhttttttThttttttthttttttttThtthhhTthtthttttttthhttttth
0ba00: hthhtthtthhttttttthhhthttttthhtttttttthhhhhTtTttBBTTtTtTthtttTtB
0be00: htthhhttthttttTDTtTthtttttttTtTthhhBThhhBhTtTtTthttttttttttthttt
0c200: tttttttttttttttTthhttttthtttTtDBhthThttthhDBhtThhtBhThttThttthtt
0c600: TthttThttttttttttthttTthhthtthtthttthtttthhhttthttttththttthhttt
0ca00: tttttttthhhTthhThttttttthtthhDTThBhhtthttthttttttttThttthDhBThhh
0ce00: tthhhtthttttttthhttttttthtttDTBhThttThttthhttthtthhhhhTthhhtthhh
0d200: httTthhMDhttThDhhhhThhthhtttttttttttttttttttttttttttTthhhttBhTht
0d600: thhTthttTTtTthhhThTthhhhttthttthtthhhTththTtTthhhtthhhhhhhttMDht
0da00: tThhTthDBTthttTthttttttthhtthTthhhhhhhTtBBThtthhhhttBthDhDTBhTBh
0de00: hthtthttththttthDTBhththtttDBBBhhBhThDBhhhhtthhhhtthhhtthhhTthhh
0e200: tthhhTthtthhhttTthhthhhtttthhtttttthhhthttttttthhhthhhtthhtttttt
0e600: thhtttttttttttttttttttthtthhhhthhhhhhhhhhhhthhtthttttttttttttttt
0ea00: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
0ee00: ththhhThthtTthttttttthttthttthtttTBhtthhhhhthhhhhhhhhtthhhhttTth
0f200: hBTthhhttttttttttttBThhhhttthttthhhDThhTBTThhhBhhtttDThtttttttth
0f600: tthtttThttthDTBhBTtttBBBBTththhhtthhhhtthhBhBhTDhttttttttTBhTThh
0fa00: tttttttththhtthtttThhhttttttttttthttttttthtthhhttTttthhThhtthTth
0fe00: hhhDTtThhhTthttttttthtthhttttttttttthhhtthhhhhtthhhTthTthhhhtthh
10200: TTthhhhhhhthtthhhhthBhhthttthhDTBhThDTTBBhhttBhTthttttttthtttThD
10600: TTthttttttthhttttttthhhhhttthttthtthhhhhhhhhTttthhhhhhhttttttthh
10a00: hhhhhhhBhThttThttthDTBhTBThhhttthttttthttttDhTBhTBhBBBBBBTLhhttt
10e00: hhthhhhhhhtttDhThhTTthhhhhhBBBBBBBhBhTThhttthttthhhhhttttttthhht
11200: tttttttttttttthhhhhhhhhhhhhhhtttttttttttttthttttttttttthtttttttt
11600: tttttthhhDhhhhhhhhhhhhhhhhhthhhhhhhhhhhhtthhhhTthhhttttttthhhhhh
11a00: hhhhthhhhhttttttthhFhttttttthhhhhhhhTthttthMDhhhhhhhhhhhhhthhhhh
11e00: hhhhhTthhhtttDhThhhhhhhhTthhhDBhttttttthtBhhBtBhhhBBBBBBBBBBhhhh
12200: hhhhhhhhhhhhhhhtthhhhhhthhhhhhhhhhhhhhBhThttthttttthDhDhhhDhtttt
12600: htttttttttttttttttthhhhhhhhhhhhhhttttthttttttttttttttthhhhhhhthh
12a00: hhhhhhhhhhhTthhhhhtttthhhthhhhhhthhhhhhhhhhhhhhhhhhhhhthhhhhhthh
12e00: hhhhTthhhhhhhhhhhhthhhhthhhhthhthhthhththhthhttttttttttttttthhht
13200: httttthhttttttttttttthhttttttttttttttttttttttttttttttttttththhtt
13600: thhttthhtthhhhtthhthttttttttttttttttttttthhhhhhttttttthhhthttttt
13a00: tthhhthttttttthhhtthttttttttttttthhhhhthttttttttttttttttthhhhhth
13e00: ttttttttttthhttttthhhhhhhhttttthttttthhhhhhhhhhhDDhhhttttttthttt
14200: ttttttttttttthhthhhhhhhhhhhhhhhhhhhhthttttttttttthhTthhhhhhhhhhh
14600: hhhhhtthtttttttthhhttttttttttttttttttttttttttttttttttttttttttttt
14a00: ttttttttttttttttttthhhhhttttttthttttttttttthhhhhhttttttthhhhhhhh
14e00: hhhhhhhhDhhhhhhhhhhhhhtttLhhhhthhhhhhhLhhhhhhhhhhhhhhhhttttttFDL
15200: hhthhhtthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhDDDhhhttthtDDhhhhDLhhthhth
15600: hhhhtTthhttttttthhhhhhhhhhhhhhhhttttthhhhhhhttthtDDhhhhhhhLhhhhh
15a00: hDLhhthhhhLhhhhhhhhtttttthDLhhthhhhDLhhthhDLhhthhtDDLhhthDDLhThh
15e00: thhtttLhhtDhDLhhthhhhhhhttttthhtDLhttttttthhthhhhLhhhhhLhDLhhhhh
16200: thhhhhhLhhhTthhhhhhhDLhhthhDLhhhhhththhDLhhhhhhhhhhhhtDDhLhhthtt
16600: htttttDhhhhDLhhthhhDLhhhhhhhhhhhhhhhhhhttttttttttttttttttttttttt
16a00: ttttttttttttttttththhhhhLhhhhhhhhhhhhDhhLhhthhhhhhhhhhLhhhhhhhth
16e00: DhhhDLhhthDDLhhthhhhhhhhhthttttththhttthtDDhhhhhhhLhhhhhhhhhhthh
17200: hhhhhttththhttththDhhhhhDLhhthttthtDhDhhhthhhhhhtthhhttththhhhhh
17600: hhhhhhhhhhhhhhhhhhhthththhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhthhhhhh
17a00: hhhhhhhhhhhhhhtTthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
17e00: hhhhhhhhthhhhhhhhhhhhhhTthhhhthhhhhhhhhhhhhhTthhhhhhhhhhhhhhhhht
18200: thhhhhhhhhhhhhhhhhhhhthhhhhhthhthhtthhthhhhthhthhtththhhhhthhhth
18600: htttttthhttttthhttttttthhttttttttthtttthhthhthhthhthhtthhhhtthht
18a00: hhthhhhthhtthhhhtthhhhtthhhttttttttttththhhhhhthhthhthttttttthtt
18e00: thhhhhhthhthhthhhhhhhhthhhhthhhthhthhtthhtthhttttttttttthtttttth
19200: htthhhhhhthhhhhhhhthhhhthhhhhhthhhhthhhhhhthhhhthhhhhhthhhhthttt
19600: ttttthhhhhtttttttttttthttttttthhhthttttttthhttttttttttttttthhhth
19a00: hhhhtthhtttttttttththhhhhhhhhttttttththhtthhthhhhthhthhtthtthhtt
19e00: tttttthhhthhthtttttthhttttthhhthhthtttttttthhthhtttttttththhhhhh
1a200: hhhhhttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
1a600: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
1aa00: ttttttttttttttttttttttttttttttttttttthhhhhhhhhhhTthhhhhhhhhhhhhh
1ae00: hhhhhhhhhhhhhhhhhhhhhhThttttttthhhhhhhhhhhhhhhhhhthhhhhttttttthh
1b200: hhhhhhhhTthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhthhhhttttttthhh
1b600: hhhhhhhhhhhhhTthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
1ba00: hhhhhhhhthhhhhhhhhhhhhhTthhhhhtttttttttttttttttttttttttttttttttt
1be00: tttttttttttttttttttttttttttttttttttthttttttttttttttttttttttttttt
1c200: tttttttthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhthhhthhhhhhhhhhhhhhTtTthh
1c600: hhhhhhhhhhhhhhhhhhhtttthtttttttttttttttttthtthtthttttttttttttttt
1ca00: tttttththttttttttttttthttttthtthtttttthhhhtthhhttttttthtttthhtth
1ce00: hhtthhhhhhhhhhttttttthtttttthhhhhhhhhhhhhhhthhhhhhhhhhhhthhhhhhh
1d200: hhhhhhhTthhhttttttttttttttttthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
1d600: hhhhhhhhhthhhhhhhhhhhhhhTthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
1da00: hhhhhhhhthhhhhhhthhthhhhhhthhhhhhhthhhhhhhhhhhTthhhhhthhhhhhhhhh
1de00: hhhhhthhhhthhthhhhhttttthhttttttthhhhhhhhttttttttttttttttttttttt
1e200: ttththhhhhhthhthttttthhttttttttttttttttthhhhhhttttttttttttttttth
1e600: tthhhhthhthhthhhhhhthhthhthttttttttttttttttttthhhthhthhthtttthht
1ea00: ttthhhthhhtttttttttttttttttttttttthhhthhthttttttthhhthhthttthhtt
1ee00: hhtthhtttthhtttttttttttthtthhttttthhthtttthhttthhttthhttttthhhth
1f200: hthtttthhhthhthtttttttttthhhthhthttttttthhtttttttttttttthhhththh
1f600: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
1fa00: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
1fe00: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
20200: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
20600: httttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
20a00: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
20e00: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
21200: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
21600: thhhhhhhhhhTthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhthhhhhhhhhhhhhh
21a00: Tthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhthhhhhh
21e00: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
22200: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
22600: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
22a00: ttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttth
22e00: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
23200: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
23600: tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
23a00: ttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttth
23e00: hhhhhhhhTthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
24200: hthhhhhhhhhhhhtttttthhhTthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
24600: hhhhhhhhhhhhhhhhthhhhhhhhhhhhhhTthhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
24a00: hhhhhhhhhhhhhhhhhhhhhhhhhthhhhhhhhhhhhhhTthhhhhhhhhhhhhhhhhhhhhh
24e00: Ththhhhhhhhhhhhhhhhh...........hht..............Tt..............
25200: ..............hh.........hh............hht..............Tt......
25600: ......................hh.........hh............hht..............
25a00: Tt............................hh.........hh............hht......
25e00: ........Tt...................................hh.........hh......
26200: ......hht..............Tt.........hh.........hh............hht..
26600: ............Tthttttttttttttttttttttttttttttttttttt....hh........
26a00: .hh............hht..............Tt..................hT..........
26e00: .....hh.........hh............hht..............Tt...............
27200: ......hh.........hh............hht..............Tthttttttttttttt
27600: tttttttttttttttttttttt..........................hh.........hh...
27a00: .........hht..............Tthttttttttttttttttttttttttttttttttttt
27e00: httttttttttttttttttttttttttttttttttt............................
28200: hh.........hh............hht..............Tt....................
28600: ........hh.........hh............hht..............Tt............
28a00: ................hhhh.......hh..hh.h......hht.hh.........htTthht.
28e00: ..........Tt........Tt.hh.........hh............hht.............
29200: .Tt.............................hh.........hh............hht....
29600: ..........Tt................hh.........hh............hht........
29a00: ......Tthttttttttttttttttttttttttttttttttttttttt..hh.........hh.
29e00: ...........hht.....hT.......Tt............................hh....
2a200: .....hh............h.hhht.............Tthttttttttttttttttttttttt
2a600: tttttttttttthttttttttttttttttttttttttttttttttttttttttttttttttttt
2aa00: ttttttttttt.hh.........hh............hht..............Tt........
2ae00: ....................hh.........hh............hht..............Tt
2b200: hh.........hh............hht..hh.....h....hTt.....h............h
2b600: ht..............Tt............................hh.........hh.....
2ba00: .......hht..............Tt............................hh........
2be00: .hh............hht.........hh..h


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#2057 (comment)

Dave Hylands
Shuswap, BC, Canada
http://www.davehylands.com

@ryannathans
Copy link
Contributor Author

@dhylands thanks for the clarification. Why does the gc mem free function report 47k free but this report 20k?

@dhylands
Copy link
Contributor

I'm not sure. There are 1285 dots = 1285 x 16 = 20560 bytes.

gc.mem_free reports the exact same thing that is used to report the free memory in this line:

GC: total: 166400, used: 145840, free: 20560

i.e. both places call gc_info and report info.free, so I'd have to assume that 27K of memory was allocated for something between the 2 calls.

I do see 4 x 4k blocks, but without knowing what the heap looked like when 47k was free, its hard to saw.

@ryannathans
Copy link
Contributor Author

The router task that I've written generates a lot of immediately collectable objects (read packet from buffer, process metadata, write to uart, repeat many times). The "free memory" printed by mem_info(1) seems to be before a GC - that's probably why it's ~27k less than when I print mem_free (I collect between every "task")

Why doesn't it automatically garbage collect and then attempt the allocation? Or is it doing this and for some reason is unable to free 582 bytes of contiguous memory?

I'll rewrite this code at a later date to pass a memoryview to uart.write and see how that goes.

@dhylands
Copy link
Contributor

My understanding is that if the allocation fails, then it will perform a gc, and only if that fails to free up enough memory will it be considered a memory allocation failure.

@dpgeorge
Copy link
Member

Why doesn't it automatically garbage collect and then attempt the allocation?

It is. As Dave says it does a collection then tries to allocate again, then it'll fail if it can't get enough contiguous memory.

for some reason is unable to free 582 bytes of contiguous memory?

Probably there's just not 582 bytes (37 blocks) of contiguos heap. I would run a gc.collect() and then a micropython.mem_info(1) to make sure that it's really this that's the issue.

Is the above output of mem_info(1) with frozen bytecode?

It sounds like you are doing a lot of allocations and churning through the heap very quickly. And it looks like you operate with around 71%-87% of heap usage which is pretty tight for a program that has a large churn.

I would try rewriting some parts to not allocate so much memory, eg by preallocating buffers and using memoryview when possible.

< 8000 /task-lists>

@dpgeorge
Copy link
Member
dpgeorge commented May 11, 2016

Try applying the following patch to master (using git am < file >):
0001-py-Zero-out-all-heap-memory-on-allocation.txt

It might help to collect more unused heap.

@ryannathans
Copy link
Contributor Author

@dpgeorge the above output is straight after crashing, prior freezing bytecode and changes that freed around 60k of ram. It now runs with 109k free.

Since these changes our device hasn't been crashing (yet? - we are auto rebooting hourly) I also intend to rewrite parts further to allocate even less by using memoryviews, but I'm yet to investigate how well supported they are (eg can I pass it without concerns to a uart.write())

@ryannathans
Copy link
Contributor Author

Will investigate with the patch tomorrow when I'm with the device again. I appreciate your help, keep up the solid work.

@dpgeorge
Copy link
Member

I also intend to rewrite parts further to allocate even less by using memoryviews, but I'm yet to investigate how well supported they are (eg can I pass it without concerns to a uart.write())

memoryview should be at the same level of support as bytes and bytearray (and array) objects. You should be able to pass a memoryview to anything that needs a buffer (readable or writable).

@ryannathans
Copy link
Contributor Author

Excellent. We will be able to reduce memory consumption much further. Is there currently any way to "freeze" boot.py and main.py?

Will report back with more logs and test results within a day.

@dpgeorge
Copy link
Member

Is there currently any way to "freeze" boot.py and main.py?

No. boot.py/main.py are files, not modules (and only modules can be frozen). But in main.py you can just do import mymain; mymain.main(), whene mymain is a frozen module.

@ryannathans
Copy link
Contributor Author

@dpgeorge Tested the patch, after 5 to 10 minutes a device running with the patch has 4-5k more free memory than devices running without the patch. An increase in free memory was only noted after routing a large number of packets, not after idling, which is probably what you'd expect.

8000

@dpgeorge
Copy link
Member

@ryannathans thanks for testing, good that it helped a bit.

@ryannathans
Copy link
Contributor Author

@dpgeorge has that patch made it in to master? Eyeballed the commit list but didn't notice it

@dpgeorge
Copy link
Member

has that patch made it in to master?

No. It would impact performance, so needs to be considered properly.

tannewt pushed a commit to tannewt/circuitpython that referenced this issue Aug 17, 2019
This fixes a problem with USB MIDI messages 0xc and 0xd and
Closes: micropython#2057
@dpgeorge
Copy link
Member

MICROPY_GC_CONSERVATIVE_CLEAR forces the GC to zero out all memory on allocation.

Also try using gc.threshold(5000) to reduce fragmentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants
0