ESP8266 port needs wifi #992
Comments
|
After some playing with the ESP wifi API, I have concluded that it won't be possible to implement standard sockets at the Python level. The ESP runs a very limited OS, which schedules tasks in a cooperative, event-triggered/callback fashion. This means that any code you want to run can only run for a short time and must then return complete control (ie doesn't save the stack) to the OS, so that other tasks can be called. For example, to list/scan the available access points, one would want to do: aps = wifi.scanap()But this won't work, because the scanap() call must return control to the OS for the scan to finish, then the OS does a callback to the application giving the result of the scan. Similarly: # s is a socket
s.send(b'some data')
result = s.recv(100) # wait for responseThis can't be implemented on the ESP because you need to return complete control to the OS between the send() and recv() calls. One way to do it is to implement some kind of task switching by allocating some heap RAM to save the entire stack of the Python "task", and returning to the OS when needed. Th 8000 en when the OS calls us again, we restore the stack and continue the Python code. This would require at least 8-16k RAM for the stack, which uses nearly all the available heap (which is about 20k). The other way to do it is go Javascript style and make everything a callback. Eg, to scan access points: wifi.scanap(lambda aplist: print(aplist))This is how the eLua-on-ESP8266 does it. |
Yep, that's what I already heard at pfalcon/esp-open-sdk#1 (comment)
Or do it Python-style, by implementing suitable asyncio event loop (in C). But latest news is that they released SDK integrated with FreeRTOS, which got to be pre-emptive thus: https://github.com/espressif/esp_iot_rtos_sdk . But it goes under explicit GPLv3: https://github.com/espressif/esp_iot_rtos_sdk/blob/master/LICENSE . That immediately puts themselves into non-compliance (as they keep shipping binary blobs of course). It also puts everyone else who redistributes binaries built with it into non-compliance. And it's actually old news, because even 0.9.3 went with redacted GPLv3 (the only difference is that nobody reviewed which exact clauses were removed there). I submitted compliance request: espressif/ESP8266_RTOS_SDK#1 . |
Isn't this the same as what I suggested above: saving the entire C stack at points where we need to return control to the OS, and then restoring the stack when we get called again? Whatever this implementation is called, it needs a fair bit of RAM for the stack.
I should have read that earlier...
This is just going to eat up even more of the precious RAM. The uPy task will need a large stack. It seems that to do proper sockets we need to have some kind of task switching, with a large state/stack that needs to be saved. Might not be worth it if all we are left with is 10k heap RAM for Python code and objects. Alternative is to use original 0.9.2/0.9.3 esp-sdk, and have the Python wifi API be event driven and use callbacks everwhere. |
I cannot assess details now, but generally, Python coroutines are analogs of callbacks, where control flow destination is just implicit instead of explicit. CPython's asyncio implementation is layered exactly that way: on event loop level, there're only callbacks, coroutine support is encapsulated in Task object, which would work with any compliant implementation of event loop. |
Ignoring all details, the main thing to compare is how state is saved between callbacks. asyncio uses yield in abundance, and any function that has a yield in it is a closure and must allocate its state on the heap. Implementing similar coroutines in C would also require saving state/stack on the heap. In this case the state is large since it is the entire uPy stack. |
|
Btw (don't feel like opening another ticket - there can be easily dozen of those) what's the plan regarding filesystem on esp8266? |
You can apparently read/write from/to the flash. So I would use part of the flash (say 40k, or whatever is left after the binary is downloaded) for a fatfs filesystem. Then you can use a protocol like zmodem to upload files :) |
Yes, that's what I asked about. Don't you think that fatfs is too bloated - both code-wise and space overhead wise? Do you have other ideas for compact read/write filesystem? |
It was originally designed to fit on small media. The reason it's used in stmhal port is so that the pyboard can provide raw block access to the PC, and the filesystem is visible to the PC. In the ESP case, we don't need that feature (PC visibility), so we are free to create our own FS. fatfs is a simple choice and it's plug-and-play. Is there an alternative out there that implements a simple FS? An easy thing to do would be provide a single-level FS (no directories, just root), a fixed file-allocation-table (fixed max number of files), and use simple block chaining to allocate space for the files. Or if you want more flexibility, could implement a bare-bones version of ext2fs. |
That's what I'm trying to figure (wondered, maybe you have something in store ;-) ). Google doesn't turn up anything conclusive, something to look on github for sure. (e.g. https://github.com/pellepl/spiffs).
Yeah, that sounds simple enough (though single-level sounds too limiting). But even with that, there should be bunch of details to scope-creep it. One good thing from it can be implementing support for proverbial frozen modules. Well, I guess you're right - using fatfs seems like solution of least resistance, which can be optimized later on need. |
|
Hi guys, it is not my intention to high jack this thread, but why not instead help with the CC3200 port? I know the CC3200 is not as cheap as the ESP8266, but it is still quite cheap, it is very well documented and doesn't have all the limitations of the ESP8266. I have the code here: https://github.com/danicampora/micropython you can take a look at the wifi branch, where you can already play with the Micropython REPL using telnet. With all the functionality implemented (UART, utime, RTC, WiFi, GPIO), there is around 92K of heap available for the Python code, and apart from that there is already 8K of stack reserved for the Python interpreter. Everything is running on top of FreeRTOS. The things implemented are not yet fully tested, and many things can be improved, but I believe that we are on the right path. What do you think guys? Regards, Daniel |
|
ESP development has stalled, pending a decision whether or not to move to the new esp rtos sdk. Good to hear that CC3200 dev is coming along. Will be nice to support both that and the ESP in the end, so just depends where the effort is spent first. I'd guess that many more people have an ESP than a CC3200 dev board, so that's why I'm inclined to go for the ESP. But as you say, CC3200 has much better ecosystem. Would be good if we can share components among these 2 ports, since they are so similar. |
|
@danicampora seems you don't have issues enabled on your repo... I just wanted to suggest that you try to use common code from stmhal for the cc3200 port: readline, bufhelper and pyexec. These are used directly by the esp8266 port, and eventually will go in a common directory (not stmhal). If they don't compile against the cc3200 port then we should fix them (or the port) until they do. To coordinate activity, perhaps it's best to pull in your cc3200 port as a branch of this main repo? |
|
@danicampora : You're right saying that there's risk of spreading too thin, but well, ESP8266 port is just a diversion to try something new on a cute hyped module, which thus has many active users. The port was started up based on request from community, and with the hope that community will pick up its development (which in turn depends on good publicity, which again requires community spreading the word). It's good to hear that CC3200 is coming along! From my side as a regular dev, think that for core devs so fra it's better to concentrate on core functionality, docs, and reference port, before spending too much effort on other ports (I have ~20 boards lying around to which it would be nice to port uPy). |
|
WICED platform port (GNU SDK) would be even more handy because of interesting HW, e.g.: |
|
And tomorrow there will be something even more interesting, and the day after tomorrow what was interesting yesterday will be in ashes. In that respect, concentrating on CC3200 is not a bad choice, as there's at least reputable vendor behind it. And for MicroPython, to be a reputable project, it makes sense to concentrate on its hardware which the project already offered to users and related aspects (coverage, stability, docs). Weekend hacks are free for everyone of course. |
|
@pfalcon is right: everyone is now bringing out thier IoT killer device, that's lower power, lower cost, and easier to hack. uPy should support 1 or 2 of these, and the 1 or 2 should be chosen based on openness and robustness. Edit: Broadcom WICED seems very robust and has good open SDK. I have their dev board and, if/when I have time, will look into it in more detail. |
|
@dpgeorge: Yes I agree with using the common code. I had to change some of the readline because it was now working smoothly with telnet, but I am sure I can come up with a common solution merging my changes into the stmhal files. The changes in the other modules are probably minor and maybe some are even superfluous. Regarding the issues I will enable them now, but I also think that is best to pull my branch into the main repo. @pfalcon I agree 100% to what you say about core devs. |
|
@dpgeroge Ok, so I guess that means not bringing the CC3200 into the main repo yet... |
|
I will keep working on the CC3200 port, and I'll keep it synchronized with the main repo. I'll work on using as much as possible common code. The CC3000 has been a very popular device since it was launched, and the CC3200/CC3100 are just its evolution, so I though that it would be a good choice. |
|
@danicampora Ok. Look at esp8266 port for hints how to use readline and pyexec. We will merge your code once it's got good functionality. |
|
@dpgeorge Ok, sounds good. |
|
Broadcom is reputable vendor at least for me ;-) they are in Wi-Fi business pretty long time. |
|
I haven't used it myself, but the WiFi API looks pretty straightforward -- e.g. see the now open-source Lua firmware. The socket code looks a bit messy though. Would a rudimentary API be doable, maybe in a non-standard module (for now)? |
|
Yep, nodemcu went open-source, possibly after alternative implementation surfaced: https://github.com/haroldmars/nodelua . WiFi API was always pretty straightforward - @dpgeorge did all the hard work, so individual functions can be just added one by one as needed. |
NodeMcu uses spiffs (https://github.com/pellepl/spiffs) Spiffs is a file system intended for SPI NOR flash devices on embedded targets. Spiffs is designed with following characteristics in mind:
|
Ok, telling that, listening to it, and knowing it theoretically is apparently not enough to grasp the depth of it. Because everything, everything is even-driven. Starting from user_init() - it's not a funky name for main(), it's a callback user may use to initialize his stuff - even before system is fully initialized, like wifi subsystem, for example call to wifi_station_scan() fails. Ok, there's workaround to get control after wifi is initialized - system_init_done_cb(). From its callback, wifi_station_scan() can be called, but again, it's just a callback and should return quickly. Staying in system_init_done_cb()'s callback causes havoc, for example, wifi_station_scan()'s callback locks up. So, next step would be to make repl a coroutine - instead of calling stdin_rx_chr() in a loop, it will be fed character by character from external event loop. |
|
A couple knowledgeable folks have told me the 8266 lacks some features Any info about this? On Tue, Jan 6, 2015 at 7:12 PM, Paul Sokolovsky notifications@github.com
|
Yep, the ESP8266 is pretty deeply set in its ways. Yes, I tried using system_init_done_cb but, as you point out, it doesn't work either.
Yes, that's the thought I had and where I stopped to move on to other things :) Seriously, is it worth doing this? Alternatives: 1) use RTOS version of SDK; 2) abandon it in favour of a better wifi solution. |
|
I vote for option number 2 ;-). The CC3200 port is almost ready to go live, the FTP server is already working super solid. Sent from my iPhone
|
I don't know much about TI's retail activities. The CC3200 is getting support from @danicampora which will hopefully make it into this repo soon. Following that, CC3100 should be relatively easy to support. Do you know of good CC3100 modules out there that are cheap and small (like the Adafruit CC3000 breakout)? |
Refactoring one function? Why not? ;-) Though I dunno who/when would do it. But it's interesting exercise in program control flow transformation ;-).
As you point yourself, that may solve some problems, but bring other. Also, it's unclear how tested and stable it is, how well supported, etc.
Well, I didn't start esp8266 port ;-). I treat it as a testing ground for uPy portability and scalability (just as many other ports). As for better wifi, ah, they all the same... |
|
The CC3100 modules are not yet available, but they should be soon. Remember that both chips (CC3200 and CC3100) are quite new and productions quality samples were made available only 3 months ago. |
|
I saw TI CC3100 dev boards yesterday for about $45 listed from TI. I didn't No stock Modules not available Oh well. On Wed, Jan 7, 2015 at 4:46 AM, Daniel Campora notifications@github.com
|
Strange, I have two of the CC3100BOOST modules which I purchased from Digikey about two months ago. I even managed to test them with the pyboard thanks to some code provided by user nelfata from the micropython forum: http://forum.micropython.org/viewtopic.php?f=2&t=401 He ask me to post it to github for him: https://github.com/blmorris/micropython-cc3100 Unfortunately it is a bit beyond me to extend it, so I have moved on for now. The boards belong to my employer but I might be able to pass them along to someone better able than me to develop the micropython drivers for them. |
|
Yes the CC3100 BOOST is available since a while ago, but I think that @dpgeorge was talking about something like the adafruit module, which could be used for production. The CC3100 BOOST is more like a development board. But yes, that CC3100 board is one of the options available now.
|
Cool, that can be combined with what I have done for the CC3200. |
Great, I hope it helps. Nadim will be happy to know his code is getting used. And as I suggested to him, I can compile and test code for the CC3100, and even tweak things a bit for debugging, though my C coding skills are not very strong. At risk of straying even further off the original topic of the thread, I wonder if there is a reason that CC3100 dev boards have been slow in coming. The Adafruit board uses the CC3000 module from TI rather than the chip itself, it seems like it would be a minor design change to the CC3000 breakout board to adapt it to the CC3100 module. Maybe the hobby hardware developers have just been waiting for the modules to reach the market to develop their breakout boards. If I had known two months ago that this would be the situation I might have adapted the Adafruit design for the CC3100MOD and built up a few myself. I suppose I could just ask them directly… |
|
Regarding task switching with the ESP SDK: I have implemented something similar in my Arduino port for esp8266 to support blocking reads and writes. I'm not familiar with python internals (I.e. REPL) but I have a feeling that this approach should work here as well. Basically I save registers and move the stack pointer on entry/exit to the user task. This way it is possible to yield(); to the OS at any point in the code. |
|
@dpgeorge: Another issue with ESP port is that vendor SDK tries to define its own versions of C99 types. I see that in uart.c, you avoid that by not mixing uPy and esp8266 sdk includes. But following this route further may be cumbersome. Would you prefer me to patch vendor includes in esp-open-sdk (one drawback of that is it will complicate vendor sdk version upgrades a bit - btw, I've set 0.9.4 as current in esp-open-sdk, now that I see basic wifi functionality working when built against it). |
|
@pfalcon Something along these lines should do the trick: https://github.com/igrr/atproto/blob/master/target/esp8266/c_types.h.diff |
|
@igrr: Sure. The question whether/how to do it. |
Yes, this is certainly do-able. You are implementing your own task-switching code.
Yeah, it's a right pain that!
There is no elegant solution to this, so, yes, if you are willing to maintain this patch then please go ahead and do this. It would certainly help a lot interfacing to their library code. |
|
Ok, so esp8266 port now has basic facilities to properly integrate into module OS cooperative multitasking. Further steps required:
|
Ok, this is important topic which I didn't want to discuss without ability to test actual usecases. Well, now there's something to play it. Unfortunately, my results are less inspiring than I wanted. As this ticket is already overloaded, and because any solution to the issue appears to require extensions to Python, I posted it here: http://forum.micropython.org/viewtopic.php?f=3&t=501 |
Patch to fix that pushed to esp-open-sdk. |
|
Btw, I've got an idea how to support normal POSIX sockets w/o RTOS and bloat like that: just implement no-blocking mode. That's part of the API, so who would dare to say that's not real sockets? ;-) |
|
Fair enough! |
|
That sounds good, but it would also be nice to have callbacks. |
|
Heads up that Arduino now has ESP8266 support, including WiFi. Still hoping someone can add a WiFi driver to MicroPython soon :) |
|
ESP8266 SDK 1.1.0 was (re)released under MIT license today: http://bbs.espressif.com/viewtopic.php?f=5&t=481 . |
|
@dpgeorge, @atalax: I'm again hit by an issue when esp8266 serial programming speed is awful. I'd appreciate if you could post your figures at espressif/esptool#44 |
|
ESP8266 port now has proper wifi. |
nrf: Fix I2C transfers with data larger than HW allows
The ESP8266 port gives a Python REPL over UART, but there is currently no interface to the wifi capabilities of the chip. This issue is intended to point potential contibutors in the right direction for implementing wifi.
Wifi bindings will be most useful if they fit in the standard POSIX mould of socket, connect, accept, bind listen, send, recv, etc. These functions can then be made available in the standard Python socket library.
Configuration functions, like list-access-points, and set-wpa-password, need to go in a dedicated ESP wifi driver, which is available to Python scripts from the network module.
Before any Python bindings are made, there are 2 main categories of C functions that need to be written:
Once these C functions exist, bindings to Python scripts is straight forward (and can be copied from stmhal port).
Usage of the wifi within a script will then look something like:
See issue #876 for background on this scheme.
The text was updated successfully, but these errors were encountered: