extmod/modlwip: lwIP module for micropython #1517

galenhz · 2015-10-18T03:21:20Z

General lwIP module for micropython. Not perfect; probably needs adjustments for other ports, memory utilization, and other minor issues, but it does its job for straightforward cases, and future improvements should be simple.

To use, you'll need to configure a SLIP connection on your friendly linux box. I do it as follows, your mileage may vary.

slattach -L -p slip -s 115200 /dev/ttyUSB0 &
ifconfig sl0 192.168.5.1 pointopoint 192.168.5.2 mtu 1500

Run the following code on the pyboard to enable it:

import lwip
import pyb
lwip.reset()
tim = pyb.Timer(7)
tim.init(freq=20)
tim.callback(lambda t:lwip.callback())

u = pyb.UART(2, 115200, read_buf_len=3100)
sl = lwip.slip(2, "192.168.5.2", "192.168.5.1")

This will get a SLIP connection going on serial port 2, with the board configured as 192.168.5.2 and the linux host as 192.168.5.1.

The lwip.socket module works more or less like you'd expect.

sock = lwip.socket(lwip.AF_INET, lwip.SOCK_STREAM)
sock.connect(("192.168.5.1", 22))
print(sock.recv(1024))
sock.close()

That code will print the protocol version string from your local ssh server.

pfalcon · 2015-10-18T07:17:17Z

Thanks for submitting this!

This will get a SLIP connection going on serial port 2,

Just to clarify - this requires connecting another USB-UART adapter to UART2 of pyboard, while normal USB connection is used for REPL, right?

If you have instructions how to bring up SLIP on standard USB serial, feel free to post them - they might help folks with testing by cutting on hardware setup required.

pfalcon · 2015-10-18T07:17:53Z

Build instructions (after checking out lwip-prototype2 branch):

git submodule update --init
make MICROPY_PY_LWIP=1 MICROPY_PY_LWIP_SLIP=1

I can confirm this builds for me.

pfalcon · 2015-10-18T12:58:04Z

@galenhz : Please also rebase onto latest master (there's conflict in .gitmodules, as I added another submodule in the meantime, but it's easy to fix).

pfalcon · 2015-10-18T21:24:14Z

Ok, after bunch of refactoring, figuring out how to run slattach on a non-serial line, and banging head on stupid mistakes, I finally got ping response from it, running in unix version:

# ping 192.168.5.2
PING 192.168.5.2 (192.168.5.2) 56(84) bytes of data.
64 bytes from 192.168.5.2: icmp_seq=1 ttl=255 time=4.49 ms
64 bytes from 192.168.5.2: icmp_seq=2 ttl=255 time=9.09 ms
64 bytes from 192.168.5.2: icmp_seq=3 ttl=255 time=12.5 ms
64 bytes from 192.168.5.2: icmp_seq=4 ttl=255 time=19.2 ms
64 bytes from 192.168.5.2: icmp_seq=5 ttl=255 time=20.6 ms
64 bytes from 192.168.5.2: icmp_seq=6 ttl=255 time=21.3 ms
64 bytes from 192.168.5.2: icmp_seq=7 ttl=255 time=23.9 ms
64 bytes from 192.168.5.2: icmp_seq=8 ttl=255 time=27.5 ms
64 bytes from 192.168.5.2: icmp_seq=9 ttl=255 time=27.5 ms

The refactoring I made reuses Python stream protocol and thus completely port and device independent. With build files refactoring, there would be zero changes to stmhal port, and any port would leverage the module with just defining MICROPY_PY_LWIP to 1.

pfalcon · 2015-10-18T21:26:03Z

For reference, I had to use slattach with pseudoterminal support available at https://github.com/Granjow/net-tools . slattach as shipped with Ubuntu 14.04 lacks such support.

pfalcon · 2015-10-18T21:33:25Z

949c5c9 introduces HAL_Delay()/HAL_GetTick() for unix port.

dpgeorge · 2015-10-18T21:39:15Z

With build files refactoring, there would be zero changes to stmhal port

With your changes, how do you connect a UART to lwip, as in the example in the first post here? Ie, what does this become:

u = pyb.UART(2, 115200, read_buf_len=3100)
sl = lwip.slip(2, "192.168.5.2", "192.168.5.1")

?
Can you now pass u to lwip.slip directly instead of specifying 2?

pfalcon · 2015-10-18T22:16:23Z

Can you now pass u to lwip.slip directly instead of specifying 2?

Yes, exactly, it's:

sl = lwip.slip(u, "192.168.5.2", "192.168.5.1")

lwIP's SLIP implementation internally identifies serial interfaces by uint8_t, so it's not directly possible to pass around arbitrary object, so it's static singleton (and @galenhz supports only singleton SLIP interface anyway, and of course that's hardly a problem).

I'll post my implementation a bit later after cleaning up, and hope that @galenhz rebases this patch in the meantime.

dpgeorge · 2015-10-18T22:34:48Z

it's not directly possible to pass around arbitrary object, so it's static singleton

Be careful with root pointers then.

pfalcon · 2015-10-18T22:44:49Z

Be careful with root pointers then.

Hmm, right, need to put it there, even though it's not too beautiful. Btw, I wanted to ask why stmhal reserved root pointer space for each of possible device object, but it's the same reason, right - (almost) any of them can get callback attached and then can get out of scope, right?

pfalcon · 2015-10-18T22:53:52Z

Hmm, right, need to put it there, even though it's not too beautiful.

So, that's probably will be the part which breaks "and any port would leverage the module with just defining MICROPY_PY_LWIP to 1". As of now, there're deficiencies in build system which preclude that, by they can be fixed. But root pointers really should be in port's mpconfigport.h (or core should know about extmod's, and we wanted to avoid that). Perhaps we can streamline that by indirection, when mpconfigport.h includes some header from under extmod/ .

dpgeorge · 2015-10-18T23:02:20Z

Hmm, right, need to put it there, even though it's not too beautiful.

Yeah, root pointers get ugly quickly, especially since sometimes you need to expose the type of the data, which can lead to more headers being included.

why stmhal reserved root pointer space for each of possible device object

Main reason is because global variables are needed to implement IRQs. Since the global variables are uPy objects, they must go in the root pointer space.

But it also serves a second purpose: to allow persistent peripheral objects, so "constructing" a UART the second time just reuses the existing instance (which makes sense because there is only 1 underlying physical peripheral for a given UART id).

galenhz · 2015-10-19T05:55:18Z

I've rebased the patch. Still builds. Sorry I took so long, was without internet access for a while.

dpgeorge · 2015-10-19T12:00:38Z

Do we want to merge this as-is, or wait for @pfalcon's code so that it can be part of core instead of stmhal specific?

pfalcon · 2015-10-19T13:04:45Z

@dpgeorge : Can you please reply to #1518 ? I'd personally like this to be merged ASAP, but I'd target it to after the release. And in either case, please wait till I post my changes later today, so you and @galenhz decided what's the best approach - merge his commits, then mine, or squash them, to avoid stuff being added to stmhal just to remove in next commit.

Generally, ports should inherit INC from py.mk, append to it, not overwrite it. TODO: Likely should do the same for other vars too.

galenhz · 2015-10-19T18:48:31Z

Can you take only some commits from the pull request? I separated out the commits for the module itself, as opposed to the stmhal integration.

pfalcon · 2015-10-19T19:13:54Z

Yes, that's possible.

Here're my changes: pfalcon/pycopy@f657667 (still has debug output, but at least independent build system changes are merged).

The test script I used with unix port is:

import sys
import os
import lwip
import utime

fd = os.open("/dev/pts/115", os.O_RDWR | os.O_NONBLOCK)
f = open(fd, "rb")

lwip.reset()
sl = lwip.slip(f, "192.168.5.2", "192.168.5.1")
while 1:
    lwip.callback()
#    print(".", end="")
    utime.sleep(1 / 20)

Where /dev/pts/115 is slave pseudoterminal as returned by patched slattach mentioned above.

pfalcon · 2015-10-20T21:59:52Z

@galenhz : So, I hope you can test my patch above with your pyboard setup when you have time.

galenhz · 2015-10-21T01:11:50Z

Doesn't build. It looks like most of the lwip files are entered twice in the final link, so I get a ton of duplicate symbol errors. Looking at it now.

galenhz · 2015-10-21T01:13:52Z

Part of the problem was that your branch still had my build stuff in stmhal/Makefile. I removed that. Now the only duplicate symbols are in netutils.

galenhz · 2015-10-21T01:22:54Z

Purged the netutils references from stmhal/Makefile just to get it to build. (Not a real solution in the long run.) Now I can set up the slip interface correctly on the F4 discovery board, and pings work, but once I do that the REPL stops responding.

That breaks the way I've been testing, unfortunately (running commands on REPL while a FTDI Friend connected to serial port 2 handled SLIP traffic).

pfalcon · 2015-10-21T12:57:12Z

Not a real solution in the long run.

Ack. And I don't see a scalable solution now except to release turn netutils (and other stuff under lib/) into lib*.a.

That breaks the way I've been testing, unfortunately (running commands on REPL while a FTDI Friend connected to serial port 2 handled SLIP traffic).

Do you use my test script or yours? I had to do it that way because unix port lacks Timer (again, need to implement it). Please keep using your script, just instead of uart port number, pass uart object to lwip.slip().

pfalcon · 2015-10-21T19:53:41Z

So, with 1.5 tagged, we can start merging this. I'd merge generic part (without stmhal changes), though I'd prefer to squash dd579da to keep history cleaner. @galenhz : Let me know if you have other suggestions.

pfalcon · 2015-10-22T21:36:58Z

Ok, so I pushed 22cb7cd (the submodule) to mainline for starters.

dpgeorge · 2015-10-22T21:45:56Z

Ok, so I pushed 22cb7cd (the submodule) to mainline for starters.

Good.

I'd merge generic part (without stmhal changes)

Yes, I'd prefer not to touch stmhal if it's just temporary.

galenhz · 2015-10-23T15:31:40Z

I can't squash the second modlwip commit without blowing away the middle commits (to stmhal) entirely. I'll do it if you want.

The fact that SLIP breaks REPL is a pretty severe problem from my perspective. Note that my version of the test runs the callback function in a timer interrupt. Is it possible that the sio functions you wrote don't behave well in interrupt context?

pfalcon · 2015-10-23T15:45:28Z

I can't squash the second modlwip commit without blowing away the middle commits (to stmhal) entirely. I'll do it if you want.

I'll do it myself, just wanted to be sure you don't have strong opinion against it. It's easy to that vai interactive rebase btw (rebase -i).

The fact that SLIP breaks REPL is a pretty severe problem from my perspective. Note that my version of the test runs the callback function in a timer interrupt. Is it possible that the sio functions you wrote don't behave well in interrupt context?

Can you describe the symptoms in more detail? I plan to test it on pyboard, but I won't be able to reproduce your setup with separate ISB and UART connection - my board doesn't have any headers, and I don't have right kind of headers to solder now (need female ones to be able to test other boards).

galenhz · 2015-10-23T15:56:07Z

You can go ahead and do the squash. No objections, just wanted to make sure stmhal stuff didn't go completely into the memory hole, just in case I need to revive it at some point.

The symptoms are pretty simple. I run my usual start sequence to enable lwip and slip. the lwip initialization step completes, comes back with prompt. At that point, the REPL prompt becomes completely unresponsive. I type, nothing happens. I disconnect and reconnect miniterm, no change. I unplug the USB connection (it's separate from power) and reconnect it, no change. Meanwhile, I can ping the board through the slip interface, so it's not actually locked up as far as I can tell.

dhylands · 2015-10-23T16:00:15Z

Usually, that means that there is an interrupt which is firing continuously.

pfalcon · 2015-10-23T16:32:19Z

Ok, pushed https://github.com/galenhz/micropython/commit/24f686958ec80703f0f1ef5d172d43a6337237ef and https://github.com/galenhz/micropython/commit/dd579da55701ed372f58cd7577f9394c7ae9aedb , squashed together and with some whitespace fixes.

pfalcon · 2015-10-23T16:34:09Z

No objections, just wanted to make sure stmhal stuff didn't go completely into the memory hole, just in case I need to revive it at some point.

Nope, no worries, the idea is to make this module available and suitable for any port, even unix, to ease testing. Baremetal ports and first of all stmhal are the main clients of it still, so I'll be testing and figuring it out.

Not closing this issue thus.

pfalcon · 2015-10-23T17:06:39Z

@galenhz , thinking about it, my refactor relies on stream being in non-blocking mode (while you explicitly tested with uart.any()). Please retry testing with:

u = pyb.UART(2, 115200, read_buf_len=3100, timeout=0)

@dpgeorge : I suggest adding note to the docs that timeout=0 is a way to achieve fully non-blocking mode on UART.

galenhz · 2015-10-24T00:48:03Z

It stops the lockup, but now pings do not get responses. Wireshark shows I can't transmit UDP packets, nor recieve them. Is the pyb UART object compatible with streaming?

pfalcon · 2015-10-24T13:46:00Z

Is the pyb UART object compatible with streaming?

It should be, that was the idea. But it unlikely was tested much in all modes. So, well, need to do this now.

Based on the original patch by Galen Hazelwood micropython/micropython#1517

pfalcon · 2015-10-25T11:25:15Z

Ok, so I can't reproduce any of the REPL issues with the patch at #1528

Based on the original patch by Galen Hazelwood: #1517 .

pfalcon · 2015-10-26T22:44:55Z

@galenhz : Before going for semantic changes, I wanted to resolve some smallish codestyle issues (to avoid inconsistent styling within one file still): 7621706 , fa87e90 , 858ed6d

galenhz · 2015-10-26T23:46:38Z

These look perfectly reasonable to me. Were there any other changes you wanted to make?

My next step was going to be implementing one of the other ideas you suggested; replacing void pointers with unions, as you suggested. Should I hold off until you're done?

pfalcon · 2015-10-27T00:14:38Z

On codestyle front, that's all, even if something left, it's minor. Next change I had in mind is that incoming, maybe pcb pointers in socket structure should be volatile, because otherwise compiler may optimize

            while (socket->incoming == NULL) {
                LWIP_DELAY(100);
            }

to infinite loop. After such change, I'd proceed with actual testing of the module (not just pings as handled by lwip) - without it, I don't even dare ;-). (And to do that, I'd need to implement Timer for unxi port, which is an excursion on its own.)

replacing void pointers with unions, as you suggested. Should I hold off until you're done?

That's good, please proceed, you know code layout better to do it.

pfalcon · 2015-10-31T16:45:13Z

extmod/modlwip.c

+        // backlog setting kind of pointless. FIXME
+        return ERR_BUF;
+    } else {
+        socket->incoming = (void *)newpcb;


@galenhz : So, here very suspicious typesafety problem, masked by void*: socket->incoming is generally pbuf, but here you store pcb to it, and that's very suspicious, because there's already socket->pcb. What's that? Optimization? But then it should be documented by both words, and of proper unions.

The reason why is that "incoming" can mean one of two different things, depending on what kind of socket this is. For normal sockets, it's a pbuf, but sockets in TCP listen mode get new pcb sockets instead, which are the new connections coming in. I'll fix this along with the pcb type this weekend.

pfalcon · 2015-10-31T16:52:31Z

@galenhz : here's the change I meant few days ago: 4deb493

pfalcon · 2015-10-31T16:53:49Z

@galenhz: Another issue is all the bare numbers used throughout the code, like -16 error, etc. - would be nice to have symbolic constants for them.

galenhz · 2015-11-02T16:42:30Z

Opened a new pull request for the pointer fix:

#1575

pfalcon · 2016-03-24T10:04:33Z

I kept this PR open to discuss further points/issues, but we've been doing quite a lot of changes to modlwip while working on esp8266 port, so closing this now. Thanks @galenhz !

bitvijays · 2020-06-15T13:27:54Z

@galenhz @pfalcon @dpgeorge Hope you are doing well. Thank you for working on this. Just curious, is this still working? Is it possible to connect to Pycom devices using LWIP? Is there any documentation to get this working apart from the above mentioned in the PR? Thank you :)

dpgeorge · 2020-06-18T14:17:48Z

is this still working?

Yes the lwip bindings have been used extensively on a few different ports now.

Is it possible to connect to Pycom devices using LWIP?

You'll need to ask Pycom about that.

galenhz added 5 commits October 18, 2015 22:47

lib/lwip: Add LwIP stack as a submodule in the library directory

e559531

extmod/modlwip: Initial commit of the lwip network stack module

24f6869

stmhal: Commit build infrastructure and serial port hooks for modlwip

9cb645b

stmhal/mpconfigport.h: Use less awkward pattern for adding lwip module

0741fa6

extmod/modlwip: Optimize structure, remove some repeated messy casts

dd579da

pfalcon referenced this pull request Oct 19, 2015

all: Make netutils.h available to all ports by default.

e0f5df5

Generally, ports should inherit INC from py.mk, append to it, not overwrite it. TODO: Likely should do the same for other vars too.

pfalcon added a commit to pfalcon/pycopy that referenced this pull request Oct 24, 2015

extmod/modlwip: slip: Use stream protocol and be port-independent.

b901508

Based on the original patch by Galen Hazelwood micropython/micropython#1517

pfalcon added a commit that referenced this pull request Oct 26, 2015

extmod/modlwip: slip: Use stream protocol and be port-independent.

e0d7740

Based on the original patch by Galen Hazelwood: #1517 .

pfalcon reviewed Oct 31, 2015
View reviewed changes

pfalcon closed this Mar 24, 2016

bitvijays mentioned this pull request Jun 19, 2020

Lwip support for SLIP | Pycom devices | FiPy/ LoPy pycom/pycom-micropython-sigfox#458

Open

Uh oh!

extmod/modlwip: lwIP module for micropython #1517

extmod/modlwip: lwIP module for micropython #1517

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh 8000 !

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!