8000 Tool to generate native .mpy files from a .elf file (dynamically loadable native code) by dpgeorge · Pull Request #5083 · micropython/micropython · GitHub
[go: up one dir, main page]
Skip to content

Tool to generate native .mpy files from a .elf file (dynamically loadable native code) #5083

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 20 commits into from
Dec 12, 2019
Merged

Tool to generate native .mpy files from a .elf file (dynamically loadable native code) #5083

merged 20 commits into from
Dec 12, 2019

Conversation

dpgeorge
Copy link
Member
@dpgeorge dpgeorge commented Sep 9, 2019 •
edited
Loading

This PR adds a tool elf2mpy.py and associated build scripts/headers to build .mpy files from C source code. Such .mpy files can then be dynamically imported as though they were a normal Python module, even though they are implemented in C.

Included in this PR are some examples: ex0, ex1 and ex2 are basic examples, and the qr example shows how to take an existing C library "off the shelf" and wrap it up into a .mpy file (to generate QR codes, work done in collaboration with @mattytrentini ).

Currently the tool supports targets of unix x86-64, ARM Thumb2 (Cortex M3 and above, eg stm32 boards), and partial support for esp32. esp32 support is non trivial and the qr example does not work (at this stage I'm unsure why). esp32 support relies on #5082

To try this PR out do the following:

  • go to the tools/elf2mpy/ex0 directory
  • edit Make 8000 file to select the architecture (unix, stm32, esp32)
  • run make
  • copy the resulting ex0.mpy file to the board
  • run import ex0
  • run ex0.foo(1, 2) (should return 4)
  • run ex0.factorial(5) (should return 120)

Then try out ex1, ex2 and qr. Also use one of these examples as a template to write your own C-based module.

There's still a bit of work to do to clean this up (eg tool/file naming, directory structure, docs). Also freezing the resulting mpy files won't work at the moment because the code relocates itself (writes to itself) when it is imported, and frozen code is read-only. Eventually this can be fixed.

Edit: for past discussion and PR on this idea see #1627

@dpgeorge
Copy link
Member Author
dpgeorge commented Oct 4, 2019

I redid this, changing it so that it builds the .mpy directly from .o files rather than .elf. The new tool is called ldmpy.py and it's essentially a linker that outputs .mpy files. This allows the resulting .mpy to be more efficient because it can skip PLT indirection. It also fixes all the problems with esp32 (by rebuilding the GOT in a convenient form).

So now all included examples work on all supported targets (x86-64, ARM Thumb2, xtensa-esp32).

@dpgeorge dpgeorge force-pushed the tools-elf2mpy branch 7 times, most recently from 86d3c70 to 30bfdf7 Compare October 10, 2019 01:46
@dpgeorge
Copy link
Member Author

I've updated this further. New features include:

  • global function table pointer mp_fun_table (fixed up in GOT) means that it's now possible to call MicroPython API functions from arbitrary C functions, ie no need to pass around a pointer to the mp_fun_table
  • support for bss data in the C code
  • support for calling mp_printf directly
  • support for combining Python code with C code in the same .mpy file
  • preliminary support for freezing these .mpy files

@dpgeorge dpgeorge added the py-core Relates to py/ directory in source label Oct 10, 2019
This was referenced Oct 16, 2019
Closed
Closed
Closed
@jimmo jimmo mentioned this pull request Oct 27, 2019
Closed
@dpgeorge dpgeorge mentioned this pull request Oct 29, 2019
Closed
@dpgeorge dpgeorge force-pushed the tools-elf2mpy branch 4 times, most recently from 43bafff to 33dc35a Compare November 5, 2019 02:05
@dpgeorge dpgeorge mentioned this pull request Nov 11, 2019
Closed
@dpgeorge dpgeorge force-pushed the tools-elf2mpy branch 2 times, most recently from 81cd7a5 to e7d9532 Compare November 28, 2019 02:59
@dpgeorge
Copy link
Member Author

@stinos the AppVeyor build is failing here because the compiler doesn't like NORETURN used in a typedef, declaring a function that doesn't return; see line 88 of py/nativeglue.h and the 2 most recent commits here.

Do you know how, in MSVC, to declare a function type (using typedef) that does not return? It works under gcc...

@stinos
Copy link
Contributor
stinos commented Nov 28, 2019

Do you know how, in MSVC, to declare a function type (using typedef) that does not return? It works under gcc...

Unless I'm missing something this cannot be done, this is a gcc extension. For msvc (and C's _Noreturn as well), noreturn behaves like inline i.e. it is not part of the type. As such I also cannot think of a usable workaround..

@dpgeorge
Copy link
Member Author

Unless I'm missing something this cannot be done,

OK.... I'll have to think of a way around it. Possibly just put for(;;); at the end of the no-return function when building with a non-gcc compiler.

@amirgon amirgon mentioned this pull request Nov 28, 2019
Closed
@dpgeorge
extmod: Add dynamic-runtime guards to btree/framebuf/uheapq/ure/uzlib.
bbeaafd 
So they can be built as dynamic native modules, as well as existing static
native modules.
@dpgeorge
examples/natmod: Add btree example.
37817ab
@dpgeorge
examples/natmod: Add uheapq example.
83f9fb1
@dpgeorge
examples/natmod: Add uzlib example.
16e591e
@dpgeorge
examples/natmod: Add framebuf example.
2a485e1
@dpgeorge
examples/natmod: Add ure example.
42c1aed
@dpgeorge
py/persistentcode: Make ARM Thumb archs support multiple sub-archs.
9ac949c
@dpgeorge
py/nativeglue: Add float new/get functions with both single and double.
ff58961
@dpgeorge
py/dynruntime: Add support for float API to make/get floats.
abc6429 
We don't want to add a feature flag to .mpy files that indicate float
support because it will get complex and difficult to use.  Instead the .mpy
is built using whatever precision it chooses (float or double) and the
native glue API will convert between this choice and what the host runtime
actually uses.
@dpgeorge
tests/import: Add test for importing viper code with additional flags.
111d1ff
@dpgeorge
tools/mpy-tool.py: Raise exception if trying to freeze relocatable mpy.
fc97d6d
@dpgeorge
tests: Add script to run dynamic-native-module tests.
4eef940
@dpgeorge
examples/natmod: Add features1 and features2 examples.
60c3c22
@dpgeorge
travis: Add tests for building and importing dynamic native modules.
ac76967
@dpgeorge dpgeorge merged commit ac76967 into micropython:master Dec 12, 2019
@dpgeorge
Copy link
Member Author

FINALLY THIS IS DONE AND MERGED!

Still to come: docs.

@dpgeorge dpgeorge deleted the tools-elf2mpy branch December 12, 2019 11:07
@amirgon
Copy link
Contributor
amirgon commented Dec 13, 2019

Question (ESP32):
After I copy the mpy file containing my C library to Flash, will it run from Flash or first copied to RAM? Will the data section be copied to RAM or remain on Flash?
If it's all copied to RAM this may limit the usage of this feature since RAM is usually a more limited resource.

@dpgeorge
Copy link
Member Author

After I copy the mpy file containing my C library to Flash, will it run from Flash or first copied to RAM? Will the data section be copied to RAM or remain on Flash?

The code (text) will be copied to iRAM and executed from there. BSS and rodata are copied to normal RAM.

If it's all copied to RAM this may limit the usage of this feature since RAM is usually a more limited resource.

Yes, but note that not all flash can be executed from, only a limited part of it. And similarly with RAM, only iRAM can be executed from. Also note that the code must be relocated and linked in to the firmware, ie modified, so it must go in to RAM.

There is a plan to eventually support putting .mpy files in flash, to be executed from there, saving (i)RAM. The idea is to set aside a region of flash for user modules to be "dynamically frozen" to, and to freeze them at runtime via a call like micropython.freeze(['lib1.mpy', 'lib2.mpy']).

@amirgon
Copy link
Contributor
amirgon commented Dec 13, 2019

Thank you for the explanation.

There is a plan to eventually support putting .mpy files in flash, to be executed from there, saving (i)RAM. The idea is to set aside a region of flash for user modules to be "dynamically frozen" to, and to freeze them at runtime via a call like micropython.freeze(['lib1.mpy', 'lib2.mpy']).

This could be very useful, especially for dynamically loading large libraries.

@amirgon amirgon mentioned this pull request Dec 31, 2019
Closed
@traverseda traverseda mentioned this pull request Jan 23, 2020
Open
@amirgon amirgon mentioned this pull request Feb 21, 2020
Closed
@dlech dlech mentioned this pull request Aug 12, 2020
Closed
@rlourette
Copy link

What version of micropython is this tool (elf2mpy.py) a part of? Thanks for the work @dpgeorge

@dpgeorge
Copy link
Member Author
dpgeorge commented Feb 1, 2021

What version of micropython is this tool (elf2mpy.py) a part of?

It's now called tools/mpy_ld.py and was added in aad79ad, so first appeard in v1.12.

@rlourette
Copy link
rlourette commented Feb 1, 2021 via email

tannewt pushed a commit to tannewt/circuitpython that referenced this pull request Aug 4, 2021
@microdev1
Merge pull request micropython#5083 from hierophect/stm32-sleepmem
98cd989 
STM32: add SleepMemory
@jonnor jonnor mentioned this pull request Sep 29, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
py-core Relates to py/ directory in source
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@dpgeorge @stinos @amirgon @rlourette
0