8000 extmod/nimble: Run the eventq in micropython.schedule. by andrewleech · Pull Request #1 · jimmo/micropython · GitHub
[go: up one dir, main page]
Skip to content

extmod/nimble: Run the eventq in micropython.schedule. #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 141 commits into
base: modbluetooth-1.13-fixes
Choose a base branch
from
Open

extmod/nimble: Run the eventq in micropython.schedule. #1

wants to merge 141 commits into from

Conversation

andrewleech
Copy link

Hi Jim,
I'm testing out these changes to run the nimble eventq handler via schedule() and remove the ringbuffer system. I've only tested the peripheral mode running my application on reader so far, but it appears to be running very reliably - I've had no transmission sync/order/race issues for over an hour where previously I'd have a glitch at least every 10-20 minutes.

I'm looking at how to achieve the same in btstack and it appears far less straightforward there. I can't find anything that looks like a similar functional boundary to split across to schedule().
Event callbacks in btstack_packet_handler() are typically run during the processing of mp_bluetooth_hci_poll_h4(), not going via any sort of event queue.

Perhaps I just need to run the entire mp_bluetooth_hci_poll() function via schedule?

Have you looked at this kind of change in enough detail in the past to have any suggestions here?

MaureenHelm and others added 30 commits August 21, 2020 11:23
@MaureenHelm @dpgeorge
zephyr: Include storage/flash_map.h unconditionally.
ac94e06 
Include storage/flash_map.h unconditionally so we always have access to the
FLASH_AREA_LABEL_EXISTS macro, even if CONFIG_FLASH_MAP is not defined.

This fixes a build error for the qemu_x86 board:

main.c:108:63: error: missing binary operator before token "("
  108 |     #elif defined(CONFIG_FLASH_MAP) && FLASH_AREA_LABEL_EXISTS(storage)
      |                                                               ^
../../py/mkrules.mk:88: recipe for target 'build/genhdr/qstr.i.last' failed

Signed-off-by: Maureen Helm <maureen.helm@nxp.com>
@MaureenHelm @dpgeorge
travis: Add zephyr build to CI.
17689a7 
Adds a job to build the zephyr port in CI using the same docker container
that the zephyr project uses for its own CI.

Always make clean zephyr builds to ensure we don't just rebuild C code, but
we also rebuild Kconfig and dts.  This is required when switching between
boards, which have different Kconfigs and device trees.

Uses the tagged zephyr 2.3.0 release.

Signed-off-by: Maureen Helm <maureen.helm@nxp.com>
@agniutkarsh @dpgeorge
docs: Change \* to * in argument lists.
e76c746 
Latest versions of Sphinx (at least 3.1.0) do not need the `*` escaped and
will render the `\` in the output if it is there, so remove it.

Fixes issue micropython#6209.
@davehylands @dpgeorge
stm32/pin_defs_stm32: Fix pin printing to show IN mode correctly.
8727c4e 
Prior to this commit, if you configure a pin as an output type (I2C in this
example) and then later configure it back as an input, then it will report
the type incorrectly.  Example:

    >>> import machine
    >>> b6 = machine.Pin('B6')
    >>> b6
    Pin(Pin.cpu.B6, mode=Pin.IN)
    >>> machine.I2C(1)
    I2C(1, scl=B6, sda=B7, freq=420000)
    >>> b6
    Pin(Pin.cpu.B6, mode=Pin.ALT_OPEN_DRAIN, pull=Pin.PULL_UP, af=Pin.AF4_I2C1)
    >>> b6.init(machine.Pin.IN)
    >>> b6
    Pin(Pin.cpu.B6, mode=Pin.ALT_OPEN_DRAIN, af=Pin.AF4_I2C1)

With this commit the last print now works:

    >>> b6
    Pin(Pin.cpu.B6, mode=Pin.IN)
@mbuesch @dpgeorge
tools/pyboard.py: Replace eval() of received data with alternative.
60cf2c0 
Prior to this commit, pyboard.py used eval() to "parse" file data received
from the board.  Using eval() on received data from a device is dangerous,
because a malicious device may inject arbitrary code execution on the PC
that is doing the operation.

Consider the following scenario:

Eve may write a malicious script to Bob's board in his absence.  On return
Bob notices that something is wrong with the board, because it doesn't work
as expected anymore.  He wants to read out boot.py (or any other file) to
see what is wrong.  What he gets is a remote code execution on his PC.

Proof of concept:

Eve:

  $ cat boot.py
  _print = print
  print = lambda *x, **y: _print("os.system('ls /; echo Pwned!')", end="\r\n\x04")
  $ ./pyboard.py -f cp boot.py :
  cp boot.py :boot.py

Bob:

  $ ./pyboard.py -f cp :boot.py /tmp/foo
  cp :boot.py /tmp/foo
  bin   chroot  dev  home  lib32  media  opt   root  sbin  sys  usr
  boot  config  etc  lib   lib64  mnt    proc  run   srv   tmp  var
  Pwned!

There's also the possibility that the device is malfunctioning and sends
random and possibly dangerous data back to the PC, to be eval'd.

Fix this problem by using ast.literal_eval() to parse the received bytes,
instead of eval().

Signed-off-by: Michael Buesch <m@bues.ch>
@mmilata @dpgeorge
tools/mpy-tool.py: Fix offset of line number info.
492cf34 
Signed-off-by: Martin Milata <martin@martinmilata.cz>
@dpgeorge
tools/makemanifest.py: Use os.makedirs to make path for generated files.
448319a 
The existing implementation of mkdir() in this file is not sophisticated
enough to work correctly on all operating systems (eg Mac can raise
EISDIR).  Using the standard os.makedirs() function handles all cases
correctly.

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
py/runtime: Fix builtin compile() in "single" mode so it prints exprs.
5f9b105 
As per CPython behaviour, compile(stmt, "file", "single") should create
code which prints to stdout (via __repl_print__) the results of any
expressions in stmt.

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
tests/extmod: Add test for uasyncio.sleep of a negative time.
20948a3 
It should take 0 time to await on a negative sleep.

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
extmod/uasyncio: Truncate negative sleeps to 0.
55c76ea 
Otherwise a task that continuously awaits on a large negative sleep can
monopolise the scheduler (because its wake time is always less than
everything else in the pairing heap).

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
unix/fatfs_port: Implement get_fattime.
9289935 
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
lib/timeutils: Add helper functions to deal with nanosecs since 1970.
badd351 
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
py/mphal.h: Introduce mp_hal_time_ns and implement on various ports.
ee50a6e 
This should return a 64-bit value being the number of nanoseconds since
1970/1/1.

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
extmod/vfs_lfs: Add mtime support to littlefs files.
2acc087 
This commit adds support for modification time of files on littlefs v2
filesystems, using file attributes.  For some background see issue micropython#6114.

Features/properties of this implementation:
- Only supported on littlefs2 (not littlefs1).
- Uses littlefs2's general file attributes to store the timestamp.
- The timestamp is 64-bits and stores nanoseconds since 1970/1/1 (if the
  range to the year 2554 is not enough then additional bits can be added to
  this timestamp by adding another file attribute).
- mtime is enabled by default but can be disabled in the constructor, eg:
  uos.mount(uos.VfsLfs2(bdev, mtime=False), '/flash')
- It's fully backwards compatible, existing littlefs2 filesystems will work
  without reformatting and timestamps will be added transparently to
  existing files (once they are opened for writing).
- Files without timestamps will open correctly, and stat will just return 0
  for their timestamp.
- mtime can be disabled or enabled each mount time and timestamps will only
  be updated if mtime is enabled (otherwise they will be untouched).

Signed-off-by: Damien George <damien@micropython.org>
@jimmo @dpgeorge
extmod/btstack: Implement GAP scan duration_ms parameter.
3d9a7ed 
This commit makes scanning work when duration_ms is set to zero.  Prior to
this it would not work with duration_ms set to zero.
@jimmo @dpgeorge
extmod/modbluetooth: Fix race between READ_REQUEST and other IRQs.
0bc2c1c 
The READ_REQUEST callback is handled as a hard interrupt (because the BLE
stack needs an immediate response from it so it can continue) and so calls
to Python require extra protection:

- the caller-owned tuple passed into the callback must be separate from the
  tuple used by other callback events (which are soft interrupts);

- the GC and scheduler must be locked during callback execution.
@pi-anl @dpgeorge
extmod/bluetooth: Support active scanning in BLE.gap_scan().
a80a146 
This adds an additional optional parameter to gap_scan() to select active
scanning, where scan responses are returned as well as normal scan results.
This parameter is False by default which retains the existing behaviour.
@dpgeorge
tests/extmod: Make uasyncio_fair test more reliable by adjusting sleeps.
5fb276d 
With sleep(0.2) a multiple of sleep(0.1), the order of task 2 and 3
execution is not well defined, and depends on the precision of the system
clock and how fast the rest of the code runs.  So change 0.2 to 0.18 to
make the test more reliable.

Also fix a typo of t3/t4, and cancel t4 at the end.

Signed-off-by: Damien George <damien@micropython.org>
@rcolistete @glennrub
nrf/Makefile: Improve user C modules support.
91c5d16 
Add CFLAGS_EXTRA to CFLAGS. Include LDFLAGS_MOD to the compilation.
And, add SRC_MOD to SRC_QSTR.
@stinos @dpgeorge
tests/run-tests: Use absolute paths where possible.
405893a 
Replace some usages of paths relative to the current working directory
with absolute paths relative to the tests directory.

Fixes and resulting changes:
- default values of MICROPYTHON and MPYCROSS are absolute paths and
  always correct
- likewise, the correct full paths for tools and extmod directories
  are appended to sys.path
- printing/cleaning failures works properly since it expects the .exp
  and .out files in the tests directory which is also where they
  are written to now, plus no more need for changing directories

This fixes micropython#5872 and allows running custom tests which use run-tests
without having to cd to the tests directory first, and the test output
still is in the tests/ directory instead of the current working directory.

Discovery of tests and all skip test logic based on paths relative to
the current working directory remains unchanged which essentially means
that for running most of MicroPython's own tests, run-tests must still
be ran from within it's directory, so document that.
@stinos @dpgeorge
tests/run-tests: Make test output directory configurable.
0c3f9d5 
A configurable result directory is advantageous because it enables
using a dedicated location, eventually outside of the source tree,
instead of forcing the output files into a fixed directory which might
also contain other files already. For that reason the default output
directory also has been changed to tests/results/.
@mbuesch @dpgeorge
extmod/machine_i2c: Fix buffer overrun if 'addrsize' is bigger than 32.
cef678b 
The memory operation functions read_mem() and write_mem() create a
temporary buffer on the local C stack for the address bytes with the size
of 4 bytes.  This buffer is filled in a loop from the user supplied address
and address length.  If the user supplied 'addrsize' is bigger than 32, the
local buffer is overrun.

Fix this by raising an exception for invalid 'addrsize' values.

Signed-off-by: Michael Buesch <m@bues.ch>
@dpgeorge
zephyr/README: Update required Zephyr version and mention new features.
a93a378 
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
LICENSE,docs: Update copyright year range to include 2020.
338b12d 
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
stm32/powerctrl.h: Include stdbool.h to get definition of bool.
40d174a 
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
tests: Split out complex reverse-op tests to separate test file.
0c7354a 
So they can be skipped if __rOP__'s are not supported on the target.  Also
fix the typo in the complex_special_methods.py filename.

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
all: Update Python code to conform to latest black formatting.
0665907 
Updating to Black v20.8b1 there are two changes that affect the code in
this repository:

- If there is a trailing comma in a list (eg [], () or function call) then
  that list is now written out with one line per element.  So remove such
  trailing commas where the list should stay on one line.

- Spaces at the start of """ doc strings are removed.

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
extmod/modlwip: Fix error return for TCP recv when not connected.
d1995e5 
This commit fixes the cases when a TCP socket is in STATE_NEW,
STATE_LISTENING or STATE_CONNECTING and recv() is called on it.  It now
raises ENOTCONN instead of a random error code due to it previously
indexing beyond the start of error_lookup_table[].

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
unix/variants: Fix fast and freedos variants so they build again.
836bca9 
This regressed in bd2fff6

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
esp32/mphalport: Fix mp_hal_time_ns offset.
40153b8 
gettimeofday returns seconds since 2000/1/1 so needs to be adjusted to
seconds since 1970/1/1 to give the correct return value of mp_hal_time_ns.

Signed-off-by: Damien George <damien@micropython.org>
MaureenHelm and others added 25 commits September 30, 2020 23:21
@Maure 10000 enHelm @dpgeorge
zephyr: Replace zephyr integer types with C99 types.
ce49be4 
Zephyr v2.4.0 stopped using custom integer types in favor of C99 types
instead.

Signed-off-by: Maureen Helm <maureen.helm@nxp.com>
@MaureenHelm @dpgeorge
zephyr: Const-ify struct device instance pointers.
f842e32 
Zephyr v2.4.0 added a const qualifier to usages of struct device to
allow storing device driver instances exclusively in flash and thereby
reduce ram footprint.

Signed-off-by: Maureen Helm <maureen.helm@nxp.com>
@MaureenHelm @dpgeorge
travis: Update zephyr build to v2.4.0.
c2a7aac 
Updates CI to use the latest zephyr release tag.

Signed-off-by: Maureen Helm <maureen.helm@nxp.com>
@MaureenHelm @dpgeorge
zephyr: Update build instructions to v2.4.0.
997ec9e 
Updates the zephyr port build instructions to use the latest zephyr
release tag.

Signed-off-by: Maureen Helm <maureen.helm@nxp.com>
@dlech @dpgeorge
docs/reference/packages.rst: Fix typo, remove duplicate "port".
ee7568c 
Fixes micropython#6485.
@mikewadsten @dpgeorge
py/makeversionhdr.py: Match only git tags which look like versions.
c711c00 
Some downstream projects may use tags in their repositories for more than
just designating MicroPython releases.  In those cases, the
makeversionhdr.py script would end up using a different tag than intended.
So tell `git describe` to only match tags that look like a MicroPython
version tag, such as `v1.12` or `v2.0`.
@dpgeorge
extmod/machine_i2c: Rename type to SoftI2C and add custom print method.
c35deb2 
Also rename machine_i2c_type to mp_machine_soft_i2c_type.  These changes
make it clear that it's a soft-I2C implementation, and match SoftSPI.

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
extmod/machine_i2c: Remove "id" arg in SoftI2C constructor.
aaed338 
The SoftI2C constructor is now used soley to create SoftI2C instances, it
can no longer delegate to create a hardware-based I2C instance.

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
extmod/machine_spi: Remove "id" arg in SoftSPI constructor.
9e0533b 
The SoftSPI constructor is now used soley to create SoftSPI instances, it
can no longer delegate to create a hardware-based SPI instance.

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
ports: Add SoftI2C and SoftSPI to machine module where appropriate.
39d50d1 
Previous commits removed the ability for one I2C/SPI constructor to
construct both software- or hardware-based peripheral instances.  Such
construction is now split to explicit soft and non-soft types.

This commit makes both types available in all ports that previously could
create both software and hardware peripherals: machine.I2C and machine.SPI
construct hardware instances, while machine.SoftI2C and machine.SoftSPI
create software instances.

This is a breaking change for use of software-based I2C and SPI.  Code that
constructed I2C/SPI peripherals in the following way will need to be
changed:

    machine.I2C(-1, ...)            ->  machine.SoftI2C(...)
    machine.I2C(scl=scl, sda=sda)   ->  machine.SoftI2C(scl=scl, sda=sda)

    machine.SPI(-1, ...)            ->  machine.SoftSPI(...)
    machine.SPI(sck=sck, mosi=mosi, miso=miso)
                        ->  machine.SoftSPI(sck=sck, mosi=mosi, miso=miso)

Code which uses machine.I2C and machine.SPI classes to access hardware
peripherals does not need to change.

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
ports: Support legacy soft I2C/SPI construction via id=-1 arg.
71f3ade 
With a warning that this way of constructing software I2C/SPI is
deprecated.  The check and warning will be removed in a future release.

This should help existing code to migrate to the new SoftI2C/SoftSPI types.

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
docs: Update I2C and SPI docs to add reference to SoftI2C and SoftSPI.
98182a9 
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
unix,windows: Implement mp_hal_time_ns using gettimeofday.
905a18a 
This provides microsecond accuracy.

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
extmod/utime_mphal: Add generic utime.time_ns() function.
d4b61b0 
It requires mp_hal_time_ns() to be provided by a port.  Th
67F4
is function
allows very accurate absolute timestamps.

Enabled on unix, windows, stm32, esp8266 and esp32.

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
py/parse: Expose rule-name printing as MICROPY_DEBUG_PARSE_RULE_NAME.
843dcd4 
So it can be enabled without modifying the source.

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
unix/variants: Enable MICROPY_DEBUG_PARSE_RULE_NAME on coverage build.
817b80a 
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
stm32/Makefile: Allow boards to extend SRC_C, SRC_O and OBJ variables.
0fff2e0 
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
esp32: Use path relative to root for netutils/timeutils headers.
1dc6435 
Signed-off-by: Damien George <damien@micropython.org>
@iabdalkader @dpgeorge
stm32/sdio: Don't change any DMA2 settings on H7 MCUs.
7497d89 
DMA2 clock and registers should be left in their current state in the H7
build.
@iabdalkader @dpgeorge
stm32/sdcard: Fix H7 build when using SDMMC2.
9855b9c 
Changes are:
- Fix missing IRQ handler when SDMMC2 is used instead of 
A3D4
SDMMC1 with H7
  MCUs.
- Removed outdated H7 series compatibility macros.
- Defined common IRQ handler macro for F4 series.
@jimmo @dpgeorge
stm32/rfcore: Add Python API for basic rfcore operations.
7c76a2d 
The new functions provide FUS/WS status, version and SYS HCI commands:
- stm.rfcore_status()
- stm.rfcore_fw_version(fw_id)
- stm.rfcore_sys_hci(ogf, ocf, cmd)
@jimmo @dpgeorge
stm32/boards/NUCLEO_WB55: Add standalone WB55 FUS/WS firmware updater.
222ec1a 
This commit adds a script that can be run on-device to install FUS and WS
binaries from the filesystem.  Instructions for use are provided in
the rfcore_firmware.py file.

The commit also removes unneeded functionality from the existing rfcore.py
debug script (and renames it rfcore_debug.py).
@jimmo @dpgeorge
stm32/rfcore: Update to support WS=1.9.0.0.4.
fa12bfc 
This WS update to 1.9.0.0.4 broke the workaround used in rfcore for
OCF_CB_SET_EVENT_MASK2, so fix it to support WS 1.8 and 1.9.
@jimmo @dpgeorge
py/objdict: Add mp_const_empty_dict_obj, use it for mp_const_empty_map.
880875b
@jimmo @dpgeorge
py/objtype: Handle __dict__ attribute when type has no locals.
b137d06
jimmo pushed a commit that referenced this pull request Jun 22, 2021
@jepler @dpgeorge
py/repl: Don't read past the end of import_str.
d67f411 
asan considers that memcmp(p, q, N) is permitted to access N bytes at each
of p and q, even for values of p and q that have a difference earlier.
Accessing additional values is frequently done in practice, reading 4 or
more bytes from each input at a time for efficiency, so when completing
"non_exist<TAB>" in the repl, this causes a diagnostic:

    ==16938==ERROR: AddressSanitizer: global-buffer-overflow on
    address 0x555555cd8dc8 at pc 0x7ffff726457b bp 0x7fffffffda20 sp 0x7fff
    READ of size 9 at 0x555555cd8dc8 thread T0
        #0 0x7ffff726457a  (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xb857a)
        #1 0x555555b0e82a in mp_repl_autocomplete ../../py/repl.c:301
        #2 0x555555c89585 in readline_process_char ../../lib/mp-readline/re
        #3 0x555555c8ac6e in readline ../../lib/mp-readline/readline.c:513
        #4 0x555555b8dcbd in do_repl /home/jepler/src/micropython/ports/uni
        #5 0x555555b90859 in main_ /home/jepler/src/micropython/ports/unix/
        #6 0x555555b90a3a in main /home/jepler/src/micropython/ports/unix/m
        #7 0x7ffff619a09a in __libc_start_main ../csu/libc-start.c:308
        micropython#8 0x55555595fd69 in _start (/home/jepler/src/micropython/ports/uni

    0x555555cd8dc8 is located 0 bytes to the right of global variable
    'import_str' defined in '../../py/repl.c:285:23' (0x555555cd8dc0) of
    size 8
      'import_str' is ascii string 'import '

Signed-off-by: Jeff Epler <jepler@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
0