forked from micropython/micropython
-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Add time capsule to store strings between supervisor reloads #4597
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| /* | ||
| * This file is part of the Micro Python project, http://micropython.org/ | ||
| * | ||
| * The MIT License (MIT) | ||
| * | ||
| * Copyright (c) 2021 Jonah Yolles-Murphy (TG-Techie) | ||
| * | ||
| * Permission is hereby granted, free of charge, to any person obtaining a copy | ||
| * of this software and associated documentation files (the "Software"), to deal | ||
| * in the Software without restriction, including without limitation the rights | ||
| * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
| * copies of the Software, and to permit persons to whom the Software is | ||
| * furnished to do so, subject to the following conditions: | ||
| * | ||
| * The above copyright notice and this permission notice shall be included in | ||
| * all copies or substantial portions of the Software. | ||
| * | ||
| * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
| * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
| * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
| * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN | ||
| * THE SOFTWARE. | ||
| */ | ||
|
|
||
| // TODO: add description of module | ||
|
|
||
| #include "py/objstr.h" | ||
| #include "py/runtime.h" | ||
| #include "shared-bindings/capsuleio/__init__.h" | ||
|
|
||
| // for now only string and none can be saved. in the future it may be tuples, floats, ints (maybe) | ||
| STATIC mp_obj_t capsule_bury(mp_obj_t obj) { | ||
| capsule_result_t result = capsuleio_bury_obj(obj); | ||
| switch (result) { | ||
| case CAPSULEIO_STRING_TO_LONG: | ||
| mp_raise_ValueError(translate("too long to store in time capsule")); | ||
| break; // is this needed? the above is noreturn | ||
| case CAPSULEIO_TYPE_CANNOT_BE_BURIED: | ||
| mp_raise_TypeError(translate("can only save a string or None in the time capsule")); | ||
| break; // is this needed? the above is noreturn | ||
| case CAPSULEIO_OK: | ||
| default: | ||
| return mp_const_none; | ||
| } | ||
| } | ||
|
|
||
| MP_DEFINE_CONST_FUN_OBJ_1(capsuleio_bury_fnobj, capsule_bury); | ||
| // this function requires no runtime wrapper | ||
| MP_DEFINE_CONST_FUN_OBJ_0(capsuleio_unearth_fnobj, capsuleio_unearth_new_obj); | ||
|
|
||
| STATIC const mp_rom_map_elem_t mp_module_capsuleio_globals_table[] = { | ||
| { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_capsuleio) }, | ||
| { MP_ROM_QSTR(MP_QSTR_bury), MP_ROM_PTR(&capsuleio_bury_fnobj) }, | ||
| { MP_ROM_QSTR(MP_QSTR_unearth), MP_ROM_PTR(&capsuleio_unearth_fnobj) }, | ||
| }; | ||
|
|
||
| STATIC MP_DEFINE_CONST_DICT(mp_module_capsuleio_globals, mp_module_capsuleio_globals_table); | ||
| < 10000 /td> |
|
|
|
|
||
| const mp_obj_module_t capsuleio_module = { | ||
| .base = { &mp_type_module }, | ||
| .globals = (mp_obj_dict_t *)&mp_module_capsuleio_globals, | ||
| }; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| /* | ||
| * This file is part of the Micro Python project, http://micropython.org/ | ||
| * | ||
| * The MIT License (MIT) | ||
| * | ||
| * Copyright (c) 2021 Jonah Yolles-Murphy (TG-Techie) | ||
| * | ||
| * Permission is hereby granted, free of charge, to any person obtaining a copy | ||
| * of this software and associated documentation files (the "Software"), to deal | ||
| * in the Software without restriction, including without limitation the rights | ||
| * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
| * copies of the Software, and to permit persons to whom the Software is | ||
| * furnished to do so, subject to the following conditions: | ||
| * | ||
| * The above copyright notice and this permission notice shall be included in | ||
| * all copies or substantial portions of the Software. | ||
| * | ||
| * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
| * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
| * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
| * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN | ||
| * THE SOFTWARE. | ||
| */ | ||
|
|
||
| #ifndef MICROPY_INCLUDED_SHARED_BINDINGS_CAPSULEIO_H | ||
| #define MICROPY_INCLUDED_SHARED_BINDINGS_CAPSULEIO_H | ||
|
|
||
| #include "shared-module/capsuleio/__init__.h" | ||
|
|
||
|
|
||
| #endif // MICROPY_INCLUDED_SHARED_BINDINGS_CAPSULEIO_H |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| /* | ||
| * This file is part of the MicroPython project, http://micropython.org/ | ||
| * | ||
| * The MIT License (MIT) | ||
| * | ||
| * Copyright (c) 2021 Jonah Yolles-Murphy (TG-Techie) | ||
| * | ||
| * Permission is hereby granted, free of charge, to any person obtaining a copy | ||
| * of this software and associated documentation files (the "Software"), to deal | ||
| * in the Software without restriction, including without limitation the rights | ||
| * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
| * copies of the Software, and to permit persons to whom the Software is | ||
| * furnished to do so, subject to the following conditions: | ||
| * | ||
| * The above copyright notice and this permission notice shall be included in | ||
| * all copies or substantial portions of the Software. | ||
| * | ||
| * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
| * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
| * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
| * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN | ||
| * THE SOFTWARE. | ||
| */ | ||
|
|
||
| #include "shared-bindings/capsuleio/__init__.h" | ||
| #include "py/objstr.h" | ||
| #include <string.h> | ||
|
|
||
|
|
||
| // the time capsule memory reservation | ||
| capsuleio_capsule_t capsuleio_capsule; | ||
|
|
||
| STATIC bool capsuleio_bury_string(const byte *data, size_t length) { | ||
| // the only fail mode is the string is too long | ||
| // make sure the data will fit | ||
| if (length > CIRCUITPY_CAPSULEIO_AMOUNT_BYTES) { | ||
| return false; | ||
| } else { | ||
| // set the type of the data stored | ||
| capsuleio_capsule.kind = CAPSULEIO_STRING; | ||
| // copy the string data in | ||
| memcpy(&capsuleio_capsule.data, data, length); | ||
| // write the null byte | ||
| capsuleio_capsule.data[length] = 0; | ||
| return true; | ||
| } | ||
| } | ||
|
|
||
| STATIC void capsuleio_bury_none(void) { | ||
| capsuleio_capsule.kind = CAPSULEIO_NONE; | ||
| memset(&capsuleio_capsule.data, 0, CIRCUITPY_CAPSULEIO_AMOUNT_BYTES); | ||
| return; | ||
| } | ||
|
|
||
| mp_obj_t capsuleio_unearth_new_obj(void) { | ||
| switch (capsuleio_capsule.kind) { | ||
| case CAPSULEIO_NONE: | ||
| return mp_const_none; | ||
| case CAPSULEIO_STRING: | ||
| return mp_obj_new_str_copy( | ||
| &mp_type_str, | ||
| (const byte *)&capsuleio_capsule.data, | ||
| strlen((const char *)&capsuleio_capsule.data)); | ||
| default: | ||
| // this should never be reached, but just in case | ||
| return mp_const_none; | ||
| } | ||
| } | ||
|
|
||
| capsule_result_t capsuleio_bury_obj(mp_obj_t obj) { | ||
| if (obj == mp_const_none) { | ||
| capsuleio_bury_none(); | ||
| return CAPSULEIO_OK; | ||
| } else if (MP_OBJ_IS_STR(obj)) { | ||
| GET_STR_DATA_LEN(obj, data, length); // defines locals data and length | ||
| bool bury_worked = capsuleio_bury_string(data, length); | ||
| if (!bury_worked) { | ||
| return CAPSULEIO_STRING_TO_LONG; | ||
| } else { | ||
| return CAPSULEIO_OK; | ||
| } | ||
| } else { | ||
| return CAPSULEIO_TYPE_CANNOT_BE_BURIED; | ||
| } | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do not think think this is a reasonable thing to do. Have you considered such boards as the QT PY? This will take 1/32 of its total memory. Probably many folks are running those right on the edge and 1kb would push them over the edge and unable to update.
What does the data sheet say on typical circuitpython boards w.r.t. program/erase cycles on the embedded flash, and what’s the typical cycle count on express board flash? I imagine if you take into account whatever load leveling is there you’ll find that it’s expected to live decades or more with a 1kb text file written occasionally. (I.e., is this solving a real problem?)
Maybe instead you want to learn more about the garbage collector & memory allocator and mark a capsuled tree as uncollectible? In this way the memory cost would be dynamic and opt-in rather than a high-water mark for the largest allocation somebody changes the capsuleio_amount_bytes const to.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
alarm.sleep_memoryandmicrocontroller.nvmboth provide persistent storage with a simple bytearray-style of storage. Rather than pass a string, arbitrary bytes is more flexible, though a bit less convenient. Anything can be converted to bytes. The API of those modules would be what to follow. This would be calledsupervisor.persistent_memoryor something like that.@WarriorOfWire I have seen use cases of needing persistent storage and restarting something every few seconds. For some chips I calculated this would reach the specified flash lifetime after a few weeks. (Useful rule of thumb: a million seconds is about 11 and a half days.)
Rather than reserving a chunk of RAM permanently, the data in question can be copied out of the heap to a safe place when the VM exits, and then put back after the restart. The "safe place" might be on the stack, or just someplace that's untouched while the VM is not running. It might even work to leave it in its old heap location. As soon as the heap is re-created, a new heap object of the right size can be created and the old data copied there.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
API
@dhalbert That sounds like a much better interface, especially since that mirrors the literal, physical underlying hardware. I really like your suggestion of putting it with the supervisor namespace as the "time capsule"* should share the same lifetime.
@WarriorOfWire: Agreed 1k is not a reasonable amount of memory for non-full builds of cp. The ..._CAPSULE_AMOUNT_BYTES should definitely change for the size of the build (maybe m0 = 128, m4/nrf=512?, imx=even more?).
It's also not uncommon for 'niceties' like gamepad, json, or ulab to be excluded from small builds, maybe only full builds of cp should include this?
Implementation
Opt-in does sound a lot better than consistent memory hit and at the same time I'd be wary to mark an arbitrary tree of python objects as 'do not collect'? Though the
supervisor.time_capsule*interface would not have that option.Could you elaborate, please? What is the advantage of copying in and out of the safe place vs storing directly to the safe place?
separate thought
A side note on naming, 'persistent' can be another name for 'non-volatile' in the memory hierarchy. IIRC previous conversations have discussed calling it "scratchpad" but that may not be clear.
An additional difference is on disk and
.nvmwill persist between resets, not just reloads.The guarantee that the "time capsule"* will always init to all zeros on reset. I think there is some safety there, though that was not a motivating reason to implement this.
* whatever the final name becomes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It doesn't have to be pre-allocated and use up precious space in RAM if not used.
I have a somewhat better API idea:
Don't provide a fixed-size area but to instead just do something like
supervisor.persisted_bytes = some bytearray or bytes. So it could be as large or small as you want. When the VM exits, the bytes are either left where they are (as described above) or copied to someplace temporary (like the stack). When the VM restarts, a new object with the bytes is created, you can fetch it viasupervisor.persisted_bytes. If you want to throw away what you persisted, you can dosupervisor.persisted_bytes = None.Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As @Neradoc pointed
alarm.sleep_memoryhas the same functionality.The main differences appear to be:
Possible Solution
merge alarm.sleep_memory and capsule as a
supervisor.saved_memory*singleton with a max length that is included in only full builds of python.For boards that do not have specialized features, a common implementation could do the on-stack (or safe place or pystack) swap. Here having a max length means you can't request to save something so large it would overflow the stack.
For ports that have specialized peripherals/memories, it is implemented on top of those.
The max length would vary per port.
*still working title
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
supervisor.persisted_byteswas proposed as a name to use. I +1 that name.This is not what I attempted to communicate - I intended to represent that statically allocating a buffer that will be rarely used and is not even in a hot path is hostile to users’ memory. It is hostile whether it is a full build or not. 128 bits is 10 vectorio circles with xy locations. It’s not insignificant on any build and this memory cost ought only affect users of the api.
Choosing different buffer sizes per board also makes it more painful to use - and to Dan’s point which I share, it’s needless right?
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for re-framing your point and elaborating on Dan's. I now see how I miss interpreted both of your statements and guidance.
I hear you regarding statically allocating and how having the persisted data be any size tailors any cost involved not to the board or build but rather to the specific use case/call (?). That went right over my head for a bit: thank you for explaining.
I have a few questions to make sure I head in the right direction:
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"the right direction" being towards
supervisor. persisted_bytesas Dan has suggested above.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We would store
.persistent_bytesin thesupervisormodule dictionary, which we would make mutable. We would treat it similarly to the.wake_alarmattribute inalarm, whose module dictionary is also mutable.gc_collect_ptr()is called specially on the.wake_alarmobject, for instance.circuitpython/shared-bindings/alarm/__init__.c
Line 204 in 9a3f04a
I do not remember if anything is done with the heap when the VM is stopped. That would have to be researched. If the heap is left alone, we don't need to copy the bytes until we start up the VM again, before we reinitialize the heap. The bytes could be copied onto the stack, the heap would be initialized, and then a
bytesobject would be allocated for the saved bytes and they would be copied in. Since the original object might be anywhere in heap storage, I don't think we can rely on it not being smashed up in some way when the heap is initialized (but I don't know the details of heap initialization either).This is somewhat complicated and will require tracing through some internals stuff to make sure. I cannot say for sure "put some new code here and there" without re-understanding the detail VM and heap shutdown and startup code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think others are better informed about the lifecycle of the Circuitpython init/teardown than me, though I do need an excuse to use this Segger. I’d love to see what Dan has to say there ❤️
Also to be clear, I don’t particularly care whether it’s a copy or marked objects. My care is that users should only “pay for what they use” as far as ram goes and just about any dynamic allocation strategy will accomplish that.