Description
Hi all
We're experiencing the following issue at least once per hour of runtime.
Task Network Router Task took 442480 us, mem free pre gc: 41296 B post gc: 46352 B, freed 5056 B
Task Hardware WatchDog took 103 us, mem free pre gc: 46224 B post gc: 46352 B, freed 128 B
Task NETWORK WDT TASK took 153 us, mem free pre gc: 46192 B post gc: 46352 B, freed 160 B
Task ____ APP TASK took 215 us, mem free pre gc: 46096 B post gc: 46352 B, freed 256 B
Task ____ APP TASK took 221 us, mem free pre gc: 46096 B post gc: 46352 B, freed 256 B
Task ____ Application took 251 us, mem free pre gc: 46096 B post gc: 46352 B, freed 256 B
Task RegularReset took 99 us, mem free pre gc: 46224 B post gc: 46352 B, freed 128 B
Task RESET TASK took 117 us, mem free pre gc: 46224 B post gc: 46352 B, freed 128 B
Task LogKeeper took 101 us, mem free pre gc: 46192 B post gc: 46352 B, freed 160 B
Task NETWORK INTERFACE CONTROL TASK took 142 us, mem free pre gc: 46224 B post gc: 46352 B, freed 128 B
Task ____ BOOT CTRL TASK took 120 us, mem free pre gc: 46224 B post gc: 46352 B, freed 128 B
Task GPS SENDING TASK took 120 us, mem free pre gc: 46224 B post gc: 46352 B, freed 128 B
Task GPS info packet generator took 129 us, mem free pre gc: 46224 B post gc: 46352 B, freed 128 B
Task NMEA composite dictionary generator took 132 us, mem free pre gc: 46224 B post gc: 46352 B, freed 128 B
Task NMEA strings parser took 1509 us, mem free pre gc: 44688 B post gc: 45200 B, freed 512 B
Task File Downloader took 292 us, mem free pre gc: 45008 B post gc: 45200 B, freed 192 B
Task ____ REGISTRATION TASK took 210 us, mem free pre gc: 45072 B post gc: 45200 B, freed 128 B
Task ____ MONITOR TASK took 157 us, mem free pre gc: 45072 B post gc: 45200 B, freed 128 B
Task ___ TASK took 157 us, mem free pre gc: 45072 B post gc: 45200 B, freed 128 B
Task BATTERY TASK took 168 us, mem free pre gc: 45072 B post gc: 45200 B, freed 128 B
PACKET TERMINATOR CLIPPED BY BUFFER, DROPPING
UNHANDLED EXCEPTION: Network Router Task
WARNING: Descheduling task
Traceback (most recent call last):
File "core/scheduler.py", line 83, in _run_next
File "softmodules/router.py", line 253, in run
File "softmodules/router.py", line 155, in rx_network
File "softmodules/router.py", line 49, in frame_packet
MemoryError: memory allocation failed, allocating 581 bytes
Traceback (most recent call last):
File "main.py", line 50, in <module>
File "core/scheduler.py", line 55, in run
File "core/scheduler.py", line 93, in _run_next
File "core/scheduler.py", line 91, in _run_next
File "core/scheduler.py", line 83, in _run_next
File "softmodules/router.py", line 253, in run
File "softmodules/router.py", line 155, in rx_network
File "softmodules/router.py", line 49, in frame_packet
**MemoryError: memory allocation failed, allocating 581 bytes**
SYSTEM RESETTING IN LESS THAN 30 SECONDS
We're failing to allocate 581 bytes of memory when we have over 45k of free memory, what's going on? I can only assume this is some kind of severe fragmentation?
I've done a bit of investigation and uncovered the following discussions. My understanding is that memory fragmentation is a known issue and isn't currently being worked on. Progress towards a solution occurs slowly and in small increments.
#1161
#1168
#1604
http://forum.micropython.org/viewtopic.php?f=3&t=455
https://github.com/micropython/micropython/wiki/Memory-Manager
Do you guys have any suggestions for improving robustness or techniques we can use to avoid this? Is there a way to measure memory fragmentation and restart the device safely when fragmentation approaches a threshold? Are there certain objects that significantly contribute to fragmentation that I should avoid using when writing code?
I like @dpgeorge 's ideas on the forum. Either having a memory-manager that defrags the memory or is able to allocate in fragmented memory using filesystem-like blocks.