8000 py/persistentcode: Add architecture flags compatibility checks. · micropython/micropython@a6bc1cc · GitHub
[go: up one dir, main page]
Skip to content

Commit a6bc1cc

Browse files
agattiagatti
committed
py/persistentcode: Add architecture flags compatibility checks.
This commit extends the MPY file format in a backwards-compatible way to store an encoded form of architecture-specific flags that have been specified in the "mpy-cross" command line, or that have been explicitly set as part of a native emitter configuration. The file format changes are as follows: * The features byte, previously containing the target native architecture and the minor file format version, now claims bit 6 as a flag indicating the presence of an encoded architecture flags integer * If architecture flags need to be stored, they are placed right after the MPY file header. This means that properly-written MPY parsers, if encountering a MPY file containing encoded architecture flags, should raise an error since no architecture identifiers have been defined that make use of bits 6 and 7 in the referenced header byte. This should give enough guarantees of backwards compatibility when this feature is used (improper parsers were subjected to breakage anyway). The encoded architecture flags could have been placed at the end, but: * Having them right after the header makes the architecture compatibility checks occur before having read the whole file in memory (which still happens on certain platforms as the reader may be backed by a memory buffer), and prevents eventual memory allocations that do not take place if the module is rejected early * Properly-written MPY file parsers should have checked the upper two bits of the flags byte to be actually zero according to the format specification available right before this change, so no assumptions should have been made on the exact order of the chunks for an unexpected format. The meaning of the architecture flags value is backend-specific, with the only common characteristic of being a variable-encoded unsigned integer for the time being. The changes made to the file format effectively limit the number of possible target architectures to 16, of which 13 are already claimed. There aren't that many new architectures planned to be supported for the lifetime of the current MPY file format, so this change still leaves space for architecture updates if needed. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
1 parent 7373338 commit a6bc1cc

File tree

6 files changed

+80
-7
lines changed

6 files changed

+80
-7
lines changed

‎docs/reference/mpyfiles.rst‎

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,10 @@ If importing an .mpy file fails then try the following:
8080
above, or by inspecting the ``MPY_CROSS_FLAGS`` Makefile variable for the
8181
port that you are using.
8282

83+
* If the third byte of the .mpy file has bit #6 set, then check whether the
84+
encoded architecture-specific flag bits vuint is compatible with the
85+
target you're importing the file on.
86+
8387
The following table shows the correspondence between MicroPython release
8488
and .mpy version.
8589

@@ -153,10 +157,31 @@ size field
153157
====== ================================
154158
byte value 0x4d (ASCII 'M')
155159
byte .mpy major version number
156-
byte native arch and minor version number (was feature flags in older versions)
160+
byte feature flags, native arch, minor version number (was feature flags in older versions)
157161
byte number of bits in a small int
158162
====== ================================
159163

164+
The third byte is split as follows (MSB first):
165+
166+
====== ================================
167+
bit meaning
168+
====== ================================
169+
7 reserved, must be 0
170+
6 an architecture-specific flags vuint follows the header
171+
5..2 native arch number
172+
1..0 minor version number
173+
====== ================================
174+
175+
Architecture-specific flags
176+
~~~~~~~~~~~~~~~~~~~~~~~~~~~
177+
178+
If bit #6 of the header's feature flags byte is set, then a vuint containing
179+
optional architecture-specific information will follow the header. The contents
180+
of this integer depends on which native architecture the file is meant for.
181+
182+
See also ``mpy-tool.py``'s ``-march-flags`` command-line option to set this
183+
value when creating MPY files.
184+
160185
The global qstr and constant tables
161186
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
162187

‎mpy-cross/main.c‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,7 @@ static int compile_and_save(const char *file, const char *output_file, const cha
9393
mp_parse_tree_t parse_tree = mp_parse(lex, MP_PARSE_FILE_INPUT);
9494
mp_compiled_module_t cm;
9595
cm.context = m_new_obj(mp_module_context_t);
96+
cm.arch_flags = 0;
9697
mp_compile_to_raw_code(&parse_tree, source_name, false, &cm);
9798

9899
if ((output_file != NULL && strcmp(output_file, "-") == 0) ||

‎py/bc.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -220,6 +220,7 @@ typedef struct _mp_compiled_module_t {
220220
bool has_native;
221221
size_t n_qstr;
222222
size_t n_obj;
223+
size_t arch_flags;
223224
#endif
224225
} mp_compiled_module_t;
225226

‎py/persistentcode.c‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -471,7 +471,7 @@ void mp_raw_code_load(mp_reader_t *reader, mp_compiled_module_t *cm) {
471471
|| header[3] > MP_SMALL_INT_BITS) {
472472
mp_raise_ValueError(MP_ERROR_TEXT("incompatible .mpy file"));
473473
}
474-
if (MPY_FEATURE_DECODE_ARCH(header[2]) != MP_NATIVE_ARCH_NONE) {
474+
if (arch != MP_NATIVE_ARCH_NONE) {
475475
if (!MPY_FEATURE_ARCH_TEST(arch)) {
476476
if (MPY_FEATURE_ARCH_TEST(MP_NATIVE_ARCH_NONE)) {
477477
// On supported ports this can be resolved by enabling feature, eg
@@ -483,6 +483,12 @@ void mp_raw_code_load(mp_reader_t *reader, mp_compiled_module_t *cm) {
483483
}
484484
}
485485

486+
size_t arch_flags = 0;
487+
if (MPY_FEATURE_ARCH_FLAGS_TEST(header[2])) {
488+
(void)arch_flags;
489+
mp_raise_ValueError(MP_ERROR_TEXT("incompatible .mpy file"));
490+
}
491+
486492
size_t n_qstr = read_uint(reader);
487493
size_t n_obj = read_uint(reader);
488494
mp_module_context_alloc_tables(cm->context, n_qstr, n_obj);
@@ -504,6 +510,7 @@ void mp_raw_code_load(mp_reader_t *reader, mp_compiled_module_t *cm) {
504510
cm->has_native = MPY_FEATURE_DECODE_ARCH(header[2]) != MP_NATIVE_ARCH_NONE;
505511
cm->n_qstr = n_qstr;
506512
cm->n_obj = n_obj;
513+
cm->arch_flags = arch_flags;
507514
#endif
508515

509516
// Deregister exception handler and close the reader.
@@ -672,7 +679,7 @@ void mp_raw_code_save(mp_compiled_module_t *cm, mp_print_t *print) {
672679
byte header[4] = {
673680
'M',
674681
MPY_VERSION,
675-
cm->has_native ? MPY_FEATURE_ENCODE_SUB_VERSION(MPY_SUB_VERSION) | MPY_FEATURE_ENCODE_ARCH(MPY_FEATURE_ARCH_DYNAMIC) : 0,
682+
(cm->arch_flags != 0 ? MPY_FEATURE_ARCH_FLAGS : 0) | (cm->has_native ? MPY_FEATURE_ENCODE_SUB_VERSION(MPY_SUB_VERSION) | MPY_FEATURE_ENCODE_ARCH(MPY_FEATURE_ARCH_DYNAMIC) : 0),
676683
#if MICROPY_DYNAMIC_COMPILER
677684
mp_dynamic_compiler.small_int_bits,
678685
#else
@@ -681,6 +688,10 @@ void mp_raw_code_save(mp_compiled_module_t *cm, mp_print_t *print) {
681688
};
682689
mp_print_bytes(print, header, sizeof(header));
683690

691+
if (cm->arch_flags) {
692+
mp_print_uint(print, cm->arch_flags);
693+
}
694+
684695
// Number of entries in constant table.
685696
mp_print_uint(print, cm->n_qstr);
686697
mp_print_uint(print, cm->n_obj);

‎py/persistentcode.h‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@
4545

4646
// Macros to encode/decode native architecture to/from the feature byte
4747
#define MPY_FEATURE_ENCODE_ARCH(arch) ((arch) << 2)
48-
#define MPY_FEATURE_DECODE_ARCH(feat) ((feat) >> 2)
48+
#define MPY_FEATURE_DECODE_ARCH(feat) (((feat) >> 2) & 0x2F)
4949

5050
// Define the host architecture
5151
#if MICROPY_EMIT_X86
@@ -85,6 +85,10 @@
8585
#define MPY_FILE_HEADER_INT (MPY_VERSION \
8686
| (MPY_FEATURE_ENCODE_SUB_VERSION(MPY_SUB_VERSION) | MPY_FEATURE_ENCODE_ARCH(MPY_FEATURE_ARCH)) << 8)
8787

88+
// Architecture-specific flags are present in the .mpy file
89+
#define MPY_FEATURE_ARCH_FLAGS (0x40)
90+
#define MPY_FEATURE_ARCH_FLAGS_TEST(x) (((x) & MPY_FEATURE_ARCH_FLAGS) == MPY_FEATURE_ARCH_FLAGS)
91+
8892
enum {
8993
MP_NATIVE_ARCH_NONE = 0,
9094
MP_NATIVE_ARCH_X86,

‎tools/mpy-tool.py‎

Lines changed: 34 additions & 3 deletions
1692
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,8 @@ class Config:
120120
MP_BC_FORMAT_VAR_UINT = 2
121121
MP_BC_FORMAT_OFFSET = 3
122122

123+
MP_NATIVE_ARCH_FLAGS_PRESENT = 0x40
124+
123125
mp_unary_op_method_name = (
124126
"__pos__",
125127
"__neg__",
@@ -542,6 +544,7 @@ def __init__(
542544
mpy_source_file,
543545
mpy_segments,
544546
header,
547+
arch_flags,
545548
qstr_table,
546549
obj_table,
547550
raw_code,
@@ -554,6 +557,7 @@ def __init__(
554557
self.mpy_segments = mpy_segments
555558
self.source_file = qstr_table[0]
556559
self.header = header
560+
self.arch_flags = arch_flags
557561
self.qstr_table = qstr_table
558562
self.obj_table = obj_table
559563
self.raw_code = raw_code
@@ -1339,7 +1343,7 @@ def read_mpy(filename):
13391343
if header[1] != config.MPY_VERSION:
13401344
raise MPYReadError(filename, "incompatible .mpy version")
13411345
feature_byte = header[2]
1342-
mpy_native_arch = feature_byte >> 2
1346+
mpy_native_arch = (feature_byte >> 2) & 0x2F
13431347
if mpy_native_arch != MP_NATIVE_ARCH_NONE:
13441348
mpy_sub_version = feature_byte & 3
13451349
if mpy_sub_version != config.MPY_SUB_VERSION:
@@ -1350,6 +1354,11 @@ def read_mpy(filename):
13501354
raise MPYReadError(filename, "native architecture mismatch")
13511355
config.mp_small_int_bits = header[3]
13521356

1357+
arch_flags = 0
1358+
# Read the architecture-specific flag bits if present.
1359+
if (feature_byte & MP_NATIVE_ARCH_FLAGS_PRESENT) != 0:
1360+
arch_flags = reader.read_uint()
1361+
13531362
# Read number of qstrs, and number of objects.
13541363
n_qstr = reader.read_uint()
13551364
n_obj = reader.read_uint()
@@ -1378,6 +1387,7 @@ def read_mpy(filename):
13781387
filename,
13791388
segments,
13801389
header,
1390+
arch_flags,
13811391
qstr_table,
13821392
obj_table,
13831393
raw_code,
@@ -1673,25 +1683,39 @@ def merge_mpy(compiled_modules, output_file):
16731683
merged_mpy.extend(f.read())
16741684
else:
16751685
main_cm_idx = None
1686+
arch_flags = 0
16761687
for idx, cm in enumerate(compiled_modules):
16771688
feature_byte = cm.header[2]
1678-
mpy_native_arch = feature_byte >> 2
1689+
mpy_native_arch = (feature_byte >> 2) & 0x2F
16791690
if mpy_native_arch:
16801691
# Must use qstr_table and obj_table from this raw_code
1681
if main_cm_idx is not None:
16821693
raise Exception("can't merge files when more than one contains native code")
16831694
main_cm_idx = idx
1695+
arch_flags = cm.arch_flags
16841696
if main_cm_idx is not None:
16851697
# Shift main_cm to front of list.
16861698
compiled_modules.insert(0, compiled_modules.pop(main_cm_idx))
16871699

1700+
if config.arch_flags is not None:
1701+
arch_flags = config.arch_flags
1702+
16881703
header = bytearray(4)
16891704
header[0] = ord("M")
16901705
header[1] = config.MPY_VERSION
1691-
header[2] = config.native_arch << 2 | config.MPY_SUB_VERSION if config.native_arch else 0
1706+
header[2] = (
1707+
(MP_NATIVE_ARCH_FLAGS_PRESENT if arch_flags != 0 else 0)
1708+
| config.native_arch << 2
1709+
| config.MPY_SUB_VERSION
1710+
if config.native_arch
1711+
else 0
1712+
)
16921713
header[3] = config.mp_small_int_bits
16931714
merged_mpy.extend(header)
16941715

1716+
if arch_flags != 0:
1717+
merged_mpy.extend(mp_encode_uint(arch_flags))
1718+
16951719
n_qstr = 0
16961720
n_obj = 0
16971721
for cm in compiled_modules:
@@ -1823,6 +1847,12 @@ def main(args=None):
18231847
default=16,
18241848
help="mpz digit size used by target (default 16)",
18251849
)
1850+
cmd_parser.add_argument(
1851+
"-march-flags",
1852+
metavar="F",
1853+
type=int,
1854+
help="architecture flags value to set in the output file (strips existing flags if not present)",
1855+
)
18261856
cmd_parser.add_argument("-o", "--output", default=None, help="output file")
18271857
cmd_parser.add_argument("files", nargs="+", help="input .mpy files")
18281858
args = cmd_parser.parse_args(args)
@@ -1835,6 +1865,7 @@ def main(args=None):
18351865
}[args.mlongint_impl]
18361866
config.MPZ_DIG_SIZE = args.mmpz_dig_size
18371867
config.native_arch = MP_NATIVE_ARCH_NONE
1868+
config.arch_flags = args.march_flags
18381869

18391870
# set config values for qstrs, and get the existing base set of qstrs
18401871
# already in the firmware

0 commit comments

Comments
 (0)
0