Description
TLTR: taking advantage of esp-idf facilities might improve MP performance and open the way to higher demanding projects.
About 2 years ago I placed my esp32 activities in standby waiting for better times. Recently I noticed major improvements to both esp-idf and micropython so I wanted to give it a second shot.
For prototyping and simple applications the esp32+micropython setup has always been pretty straight forward, but for complex / high speed / exact timing apps it isn't good enough (yet). Partly can't be resolved because of python inner nature, but given the huge amount of code produced in the past years, it's a pity to not being able to pack it into something efficient and feature complete.
The first problem I see is exploiting the cores. It is already hard enough using esp-idf only; by adding micropython on top becomes a nightmare. As an example the forum is full of messages telling the new users about Micropython running on core1, and it was, but then MP_TASK_COREID has been switched to core0 to fix some problems in esp-idf v3 making core1 99% unused (1% is the IPC task; it should be running on core1 anyway, and esp-idf might be still doing some use of it to work around some hw issues in some chip revisions).
The second problem I see is peripheral support (as usual). With RMT mainlined, I2S close to be, MCPWM drafted, and a few more snippets already existing (counters, sigma-delta, ecc), micropython might support 100% of the chip soon. All these drivers are far from being full featured (ex: I2S doesn't include an LCD/CAM parallel mode), but Rome wasn't built in a day... it will...
The third problem is kind of normal for open source projects and can be read in RFC. Basically there are a gazillion code snipplets available all around to do anything, but nobody knows how to pack all of those properly. And this introduces a lot of code management overhead ... a lot of talking ... a lot of people producing magnificent code but ... hard to merge in mainline. It's kind of wasteful.
I wanted to open this issue because of an idea bugging my head and probably can fix all of the three problems above: what about a mailbox similar to RPi/BCM one? On RPi the mailbox is the only way to make ARM core and the VPU+QPUs cores work together because the VPU is missing some of the ARM features to sync the cores, shared memory access locking and so on. Here we have a similar situation, software only: ARM->FreeRTOS, VPU->Micropython, Mailbox->Queues.
At the time of writing I don't have a detailed idea in mind. But FreeRTOS offers all the needed tools and esp-idf does the rest (ex: 2 IPC high prio tasks, 1 per core). Probably using those tools to produce a middle-ware would harmonize resources utilization and solve the problems stated above.
As an example the I2S driver needs a custom FreeRTOS task and looks like the task can run on core0 while MP is running on core1 and probably other drivers would benefit from such setup too. Moreover, esp-idf creates IPC tasks on boot; they could/should be used to do this job... Inter Process Communication. Currently MP isn't taking advantage of it. And the IPC mechanism could be extended, creating more tasks with different prios/resolutions (IPC prio22, 1s->prio21, 1ms->prio20, 1us->prio19...); then the drivers registering their jobs according to their timing needs. This would give a common interface to MP drivers developers, reducing the administrative overhead.
Does it sound dumb?