8000 mp_port_fun_table: port-independent code to support port-specific fun… · micropython/micropython@4a15c62 · GitHub
[go: up one dir, main page]
Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 4a15c62

Browse files
tvetve
committed
mp_port_fun_table: port-independent code to support port-specific functions in native modules
1 parent 2883369 commit 4a15c62

File tree

5 files changed

+138
-22
lines changed

5 files changed

+138
-22
lines changed

‎docs/develop/natmod.rst

Lines changed: 30 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -68,14 +68,36 @@ So, if your C code has writable data, make sure the data is defined globally,
6868
without an initialiser, and only written to within functions.
6969

7070
Linker limitation: the native module is not linked against the symbol table of the
71-
full MicroPython firmware. Rather, it is linked against an explicit table of exported
72-
symbols found in ``mp_fun_table`` (in ``py/nativeglue.h``), that is fixed at firmware
73-
build time. It is thus not possible to simply call some arbitrary HAL/OS/RTOS/system
74-
function, for example.
75-
76-
New symbols can be added to the end of the table and the firmware rebuilt.
77-
The symbols also need to be added to ``tools/mpy_ld.py``'s ``fun_table`` dict in the
78-
same location. This allows ``mpy_ld.py`` to be able to pick the new symbols up and
71+
full MicroPython firmware.
72+
It is thus not possible to simply call some arbitrary HAL/OS/RTOS/system
73+
function. However, there are two limited linking mechanisms available:
74+
via the ``mp_fun_table`` and on some ports (initially esp32 only) the
75+
``mp_port_fun_table``.
76+
77+
The ``mp_fun_table`` is the same on all ports and is defined in ``py/nativeglue.h``.
78+
It contains references to close to a hundred functions in MicroPython's core
79+
that can be used by native code. The functions are defined as a set of macros
80+
that turn what looks like a direct call (e.g. ``mp_printf(...)``) into
81+
indirect calls via ``mp_fun_table`` (e.g. ``(*mp_fun_table[72])(...)``).
82+
83+
The ``mp_port_fun_table`` is a table available on some ports that
84+
exposes additional port-specific functions, typically ones coming from large
85+
RTOS or networking libraries, such as ESP-IDF in the esp32 case.
86+
The ``mp_port_fun_table`` may also expose some commonly used functions from
87+
the standard C library (libc).
88+
The ``mp_port_fun_table`` contains the addresses of the exposed functions
89+
and references in the native modules are "fixed-up" as part of the dynamic
90+
linking process.
91+
If a native module calls a function that does not exist (for example
92+
due to varying sets of libraries built into the firmware) the call
93+
will raise a ``RunTimeError``.
94+
A sample native module using the ``mp_port_fun_table`` can be found in
95+
``examples/natmod/esp32-heap``.
96+
97+
New symbols can be added to the end of either table and the firmware rebuilt.
98+
In the case of the ``mp_fun_table`` the symbols also need to be added to
99+
``tools/mpy_ld.py``'s ``fun_table`` dict in the same location.
100+
This allows ``mpy_ld.py`` to be able to pick the new symbols up and
79101
provide relocations for them when the mpy is imported. Finally, if the symbol is a
80102
function, a macro or stub should be added to ``py/dynruntime.h`` to make it easy to
81103
call the function.

‎py/dynruntime.mk

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,11 @@ CFLAGS += -fpic -fno-common
3333
CFLAGS += -U _FORTIFY_SOURCE # prevent use of __*_chk libc functions
3434
#CFLAGS += -fdata-sections -ffunction-sections
3535

36+
ifneq ($(PORT),)
37+
CFLAGS += -I$(MPY_DIR)/ports/$(PORT)
38+
CFLAGS += -DMICROPY_PORT_FUN_TABLE
39+
endif
40+
3641
MPY_CROSS_FLAGS += -march=$(ARCH)
3742

3843
SRC_O += $(addprefix $(BUILD)/, $(patsubst %.c,%.o,$(filter %.c,$(SRC))))
@@ -137,7 +142,7 @@ $(BUILD)/%.mpy: %.py
137142
# Build native .mpy from object files
138143
$(BUILD)/$(MOD).native.mpy: $(SRC_O)
139144
$(ECHO) "LINK $<"
140-
$(Q)$(MPY_LD) --arch $(ARCH) --qstrs $(CONFIG_H) -o $@ $^
145+
$(Q)$(MPY_LD) --arch $(ARCH) --qstrs $(CONFIG_H) $(PORT_FUN) -o $@ $^
141146

142147
# Build final .mpy from all intermediate .mpy files
143148
$(MOD).mpy: $(BUILD)/$(MOD).native.mpy $(SRC_MPY)

‎py/mkenv.mk

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,8 @@ endif
66
# invoked) to the top of the tree. $(lastword $(MAKEFILE_LIST)) returns
77
# the name of this makefile relative to where make was invoked.
88
#
9-
# We assume that this file is in the py directory so we use $(dir ) twice
10-
# to get to the top of the tree.
9+
# Note that the construction of TOP fails if there is a space in the path to this makefile,
10+
# in that case define TOP explicitly in the parent makefile or environment.
1111

1212
THIS_MAKEFILE := $(lastword $(MAKEFILE_LIST))
1313
TOP := $(patsubst %/py/mkenv.mk,%,$(THIS_MAKEFILE))

‎py/persistentcode.c

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,10 @@
3636
#include "py/objstr.h"
3737
#include "py/mpthread.h"
3838

39+
#if MICROPY_PORT_FUN_TABLE
40+
#include "portnativeglue.h"
41+
#endif
42+
3943
#if MICROPY_PERSISTENT_CODE_LOAD || MICROPY_PERSISTENT_CODE_SAVE
4044

4145
#include "py/smallint.h"
@@ -189,6 +193,13 @@ STATIC void arch_link_qstr(uint8_t *pc, bool is_obj, qstr qst) {
189193
#endif
190194
}
191195

196+
#if MICROPY_PORT_FUN_TABLE
197+
// mp_port_fun_does_not_exist is used for mp_port_fun_table slots whose function doesn't exist
198+
NORETURN void mp_port_fun_does_not_exist(void) {
199+
mp_raise_msg(&mp_type_RuntimeError, MP_ERROR_TEXT("port fun unavailable"));
200+
}
201+
#endif
202+
192203
void mp_native_relocate(void *ri_in, uint8_t *text, uintptr_t reloc_text) {
193204
// Relocate native code
194205
reloc_info_t *ri = ri_in;
@@ -225,6 +236,17 @@ void mp_native_relocate(void *ri_in, uint8_t *text, uintptr_t reloc_text) {
225236
} else if (op == 6) {
226237
// Destination is mp_fun_table itself
227238
dest = (uintptr_t)&mp_fun_table;
239+
} else if (op == 126) {
240+
#if MICROPY_PORT_FUN_TABLE
241+
size_t index = read_uint(ri->reader, NULL);
242+
if (index < mp_port_fun_table_sz) {
243+
dest = ((uintptr_t *)&mp_port_fun_table)[index];
244+
} else {
245+
dest = (uintptr_t)&mp_port_fun_does_not_exist;
246+
}
247+
#else
248+
mp_raise_ValueError(MP_ERROR_TEXT("no mp_port_fun_table"));
249+
#endif
228250
} else {
229251
// Destination is an entry in mp_fun_table
230252
dest = ((uintptr_t *)&mp_fun_table)[op - 7];

‎tools/mpy_ld.py

Lines changed: 78 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -428,7 +428,7 @@ def populate_got(env):
428428
if got_entry.name == "mp_fun_table":
429429
dest = "mp_fun_table"
430430
elif got_entry.name.startswith("mp_fun_table+0x"):
431-
dest = int(got_entry.name.split("+")[1], 16) // env.arch.word_size
431+
dest = ("fun_table", int(got_entry.name.split("+")[1], 16)) // env.arch.word_size
432432
elif got_entry.sec_name.startswith(".text"):
433433
dest = ".text"
434434
elif got_entry.sec_name.startswith(".rodata"):
@@ -437,6 +437,8 @@ def populate_got(env):
437437
dest = ".data.rel.ro"
438438
elif got_entry.sec_name.startswith(".bss"):
439439
dest = ".bss"
440+
elif got_entry.sec_name == ".external.mp_port_fun_table":
441+
dest = ("port_fun_table", got_entry.sym.port_fun_offset)
440442
else:
441443
assert 0, (got_entry.name, got_entry.sec_name)
442444
env.mpy_relocs.append((".text", env.got_section.addr + got_entry.offset, dest))
@@ -651,7 +653,7 @@ def do_relocation_data(env, text_addr, r):
651653
kind = sec.name
652654
elif sec.name == ".external.mp_fun_table":
653655
assert addr == 0
654-
kind = s.mp_fun_table_offset
656+
kind = ("fun_table", s.mp_fun_table_offset)
655657
else:
656658
assert 0, sec.name
657659
if env.arch.separate_rodata:
@@ -777,6 +779,7 @@ def link_objects(env, native_qstr_vals_len, native_qstr_objs_len):
777779
]
778780
)
779781
}
782+
port_fun_sec = Section(".external.mp_port_fun_table", b"", 0)
780783
for sym in env.unresolved_syms:
781784
assert sym["st_value"] == 0
782785
if sym.name == "_GLOBAL_OFFSET_TABLE_":
@@ -789,12 +792,20 @@ def link_objects(env, native_qstr_vals_len, native_qstr_objs_len):
789792
sym.section = env.qstr_obj_section
790793
elif sym.name in env.known_syms:
791794
sym.resolved = env.known_syms[sym.name]
795+
elif sym.name in fun_table:
796+
sym.section = mp_fun_table_sec
797+
sym.mp_fun_table_offset = fun_table[sym.name]
798+
elif sym.name in port_fun_entries:
799+
sym.section = port_fun_sec
800+
sym.port_fun_offset = port_fun_entries[sym.name]
801+
log(
802+
LOG_LEVEL_3,
803+
"Port_fun_table reference: {} -> port_fun_sec+{}".format(
804+
sym.name, sym.port_fun_offset
805+
),
806+
)
792807
else:
793-
if sym.name in fun_table:
794-
sym.section = mp_fun_table_sec
795-
sym.mp_fun_table_offset = fun_table[sym.name]
796-
else:
797-
raise LinkError("{}: undefined symbol: {}".format(sym.filename, sym.name))
808+
raise LinkError("{}: undefined symbol: {}".format(sym.filename, sym.name))
798809

799810
# Align sections, assign their addresses, and create full_text
800811
env.full_text = bytearray(env.arch.asm_jump(8)) # dummy, to be filled in later
@@ -868,13 +879,35 @@ def write_qstr(self, s):
868879
self.write_bytes(s)
869880

870881
def write_reloc(self, base, offset, dest, n):
882+
"""
883+
Write a relocation entry into the mpy that the dynamic linker will have to resolve
884+
when the mpy is loaded. A relocation entry consists of a kind/op byte, followed by an
885+
optional offset word, followed by an optional count word.
886+
- base+offset is the location where the fixup is to be done, base is '.text' or 'rodata'.
887+
- dest is the target whose address needs to be placed into the fixup: 0=string table,
888+
1=rodata_const_table, 2=bss_const_table, 3..5: unused, 6=mp_fun_table,
889+
7..126:entry in mp_fun_table, 1000..66535:entry in mp_port_fun_table.
890+
- n is number of consecutive words to fix up.
891+
"""
871892
need_offset = not (base == self.prev_base and offset == self.prev_offset + 1)
872893
self.prev_offset = offset + n - 1
873-
if dest <= 2:
894+
index = None
895+
if isinstance(dest, tuple):
896+
if dest[0] == "port_fun_table":
897+
# offset into mp_port_fun_table
898+
assert dest[1] < 65536
899+
index = dest[1]
900+
dest = 126 # magic number for the loader to refer to mp_port_fun_table
901+
elif dest[0] == "fun_table":
902+
assert 6 <= dest[1] < 126
903+
assert n == 1
904+
dest = dest[1]
905+
else:
906+
assert 0, dest
907+
elif dest <= 2:
874908
dest = (dest << 1) | (n > 1)
875909
else:
876-
assert 6 <= dest <= 127
877-
assert n == 1
910+
assert dest == 6, dest # only case left: mp_fun_table itself
878911
dest = dest << 1 | need_offset
879912
assert 0 <= dest <= 0xFE, dest
880913
self.write_bytes(bytes([dest]))
@@ -884,6 +917,8 @@ def write_reloc(self, base, offset, dest, n):
884917
elif base == ".rodata":
885918
base = 1
886919
self.write_uint(offset << 1 | base)
920+
if index is not None:
921+
self.write_uint(index)
887922
if n > 1:
888923
self.write_uint(n)
889924

@@ -966,8 +1001,12 @@ def build_mpy(env, entry_offset, fmpy, native_qstr_vals, native_qstr_objs):
9661001
kind = bss_const_table_idx
9671002
elif kind == "mp_fun_table":
9681003
kind = 6
1004+
elif isinstance(kind, tuple) and kind[0] == "fun_table":
1005+
kind = (kind[0], 7 + kind[1])
1006+
elif isinstance(kind, tuple) and kind[0] == "port_fun_table":
1007+
pass
9691008
else:
970-
kind = 7 + kind
1009+
assert 0, kind
9711010
assert addr % env.arch.word_size == 0, addr
9721011
offset = addr // env.arch.word_size
9731012
if kind == prev_kind and base == prev_base and offset == prev_offset + 1:
@@ -989,6 +1028,28 @@ def build_mpy(env, entry_offset, fmpy, native_qstr_vals, native_qstr_objs):
9891028
out.close()
9901029

9911030

1031+
################################################################################
1032+
# Port-specific relocation table, which contains the addresses of a number of port-specific
1033+
# functions compiled into the firmware. These entries are used by the dynamic linker to fix-up
1034+
# relocation entries in dynamically loaded mpy modules.
1035+
port_fun_entries = {}
1036+
1037+
1038+
def load_port_fun(portfun_file):
1039+
"""
1040+
Load list of port-specific functions from file used for C-#include: has a function name
1041+
followed by a comma on each line
1042+
"""
1043+
ix = len(port_fun_entries)
1044+
with open(portfun_file) as f:
1045+
for l in f:
1046+
m = re.match(r"^([A-Za-z0-9_][A-Za-z0-9_]*),$", l)
1047+
if m:
1048+
port_fun_entries[m.group(1)] = ix
1049+
ix += 1
1050+
# print("port_fun_entries=", port_fun_entries)
1051+
1052+
9921053
################################################################################
9931054
# main
9941055

@@ -1062,6 +1123,9 @@ def main():
10621123
cmd_parser.add_argument("--arch", default="x64", help="architecture")
10631124
cmd_parser.add_argument("--preprocess", action="store_true", help="preprocess source files")
10641125
cmd_parser.add_argument("--qstrs", default=None, help="file defining additional qstrs")
1126+
cmd_parser.add_argument(
1127+
"--portfun", default=None, help="file defining port-specific functions"
1128+
)
10651129
cmd_parser.add_argument(
10661130
"--output", "-o", default=None, help="output .mpy file (default to input with .o->.mpy)"
10671131
)
@@ -1071,6 +1135,9 @@ def main():
10711135
global log_level
10721136
log_level = args.verbose
10731137

1138+
if args.portfun:
1139+
load_port_fun(args.portfun)
1140+
10741141
if args.preprocess:
10751142
do_preprocess(args)
10761143
else:

0 commit comments

Comments
 (0)
0