Description
This issue was brought up by @CrizEo in the forum. Creating (and using) a btree database fails, if once it gets a reasonable size. The following code fails on an ESP8266:
import btree
import os, urandom # for random bytestring + db size
import gc # for showing free heap
from ubinascii import hexlify
DATABASE_FILENAME = "btree.db"
def randbytes():
length = 1 + urandom.getrandbits(3) # 1 to 16
return os.urandom(length)
try:
dbfile = open(DATABASE_FILENAME, 'r+b')
except OSError:
dbfile = open(DATABASE_FILENAME, 'w+b')
db = btree.open(dbfile, cachesize=2560)
counter_entries = 0 # number of entries (pairs of k+v) in db
counter_bytes = 0 # number of bytes (keys + values) in database
for i in range(500):
key, value = "%d" % counter_entries, hexlify(randbytes())
db[key] = value # here is where the error occurs
counter_entries += 1
counter_bytes += len(key) + len(value)
if counter_entries % 10 == 0:
print("total: %d entries (%d B); db size: %d B; free heap: %d B" %
(counter_entries, counter_bytes, os.stat(DATABASE_FILENAME)[6], gc.mem_free()))
gc.collect()
db.close()
dbfile.close()
The log output is:
>>> import test_btree
total: 10 entries (104 B); db size: 0 B; free heap: 12576 B
total: 20 entries (206 B); db size: 0 B; free heap: 12528 B
total: 30 entries (344 B); db size: 0 B; free heap: 12544 B
total: 40 entries (476 B); db size: 0 B; free heap: 12544 B
total: 50 entries (592 B); db size: 0 B; free heap: 12544 B
total: 60 entries (698 B); db size: 0 B; free heap: 12544 B
total: 70 entries (808 B); db size: 0 B; free heap: 12544 B
total: 80 entries (914 B); db size: 0 B; free heap: 12544 B
total: 90 entries (982 B); db size: 0 B; free heap: 12544 B
total: 100 entries (1124 B); db size: 0 B; free heap: 12528 B
total: 110 entries (1258 B); db size: 0 B; free heap: 12544 B
total: 120 entries (1392 B); db size: 0 B; free heap: 12544 B
total: 130 entries (1548 B); db size: 0 B; free heap: 12560 B
total: 140 entries (1644 B); db size: 0 B; free heap: 12544 B
total: 150 entries (1762 B); db size: 0 B; free heap: 12544 B
total: 160 entries (1886 B); db size: 0 B; free heap: 12544 B
total: 170 entries (2018 B); db size: 0 B; free heap: 4272 B
total: 180 entries (2130 B); db size: 0 B; free heap: 4288 B
total: 190 entries (2264 B); db size: 0 B; free heap: 4272 B
total: 200 entries (2400 B); db size: 0 B; free heap: 4272 B
total: 210 entries (2514 B); db size: 0 B; free heap: 4288 B
total: 220 entries (2646 B); db size: 0 B; free heap: 4288 B
total: 230 entries (2794 B); db size: 0 B; free heap: 4272 B
total: 240 entries (2936 B); db size: 0 B; free heap: 4288 B
total: 250 entries (3050 B); db size: 0 B; free heap: 4288 B
total: 260 entries (3164 B); db size: 0 B; free heap: 4288 B
total: 270 entries (3290 B); db size: 0 B; free heap: 4272 B
total: 280 entries (3420 B); db size: 0 B; free heap: 4288 B
total: 290 entries (3542 B); db size: 0 B; free heap: 4288 B
total: 300 entries (3674 B); db size: 0 B; free heap: 4272 B
total: 310 entries (3790 B); db size: 0 B; free heap: 4272 B
total: 320 entries (3916 B); db size: 0 B; free heap: 4288 B
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "test_btree.py", line 24, in <module>
OSError: 0
The same code runs fine on Linux Micropython with a constant value for free heap. If the database is created on a different machine and copied to the ESP8266, it fails too.
(sysname='esp8266', nodename='esp8266', release='2.0.0(5a875ba)', version='v1.9.1-208-g4dc7c56-dirty on 2017-08-07', machine='ESP module with ESP8266')