Using LWIP on Windows #6463

gvanem · 2020-09-21T08:54:34Z

First off, building MicroPython with MSVC and the ports/windows/micropython.vcxproj was exceptionally easy. Good job!

But when trying to add support for LwIP with:

-DMICROPY_PY_LWIP=1
-DMICROPY_ENABLE_SCHEDULER=1
Add extmod/modlwip.c and lib/netutils/netutils.c.

And trying to figure out where to add that in the .vcxproj files was not so easy. So I hacked up my own GNU-makefile.
And then getting to the link stage, I got this error:

lwip32.lib(sys_arch.obj) : error LNK2005: _sys_arch_protect already defined in MicroPython.lib(modlwip.obj)
lwip32.lib(sys_arch.obj) : error LNK2005: _sys_arch_unprotect already defined in MicroPython.lib(modlwip.obj)

Is there another define that should be set to avoid this? Adding -DNO_SYS=1 did not help.

Or are we supposed to link with a dynamic LwIP import-library? Since that works.

The text was updated successfully, but these errors were encountered:

stinos · 2020-09-21T08:58:54Z

So I hacked up my own GNU-makefile.

The .vcxproj file is for msbuild and the cl compiler, now you're talking about make, so I assume you're also using a different compiler i.e. gcc? In any case, will be hard to provide any help without seeing that makefile.

gvanem · 2020-09-21T09:11:23Z

so I assume you're also using a different compiler i.e. gcc?

No, using cl with GNU-make works just fine. Here is a preliminary version.

Edit2: here is an updated version.

stinos · 2020-09-22T07:59:37Z

Sorry don't have time to figure out what's wrong in a 1k LOC Makefile :) Plus if I run make on it I get this:

$ make -v
GNU Make 4.3
Built for x86_64-pc-msys
...
$ make -f ./Makefile.windows CC=cl
Makefile.windows:426: *** missing separator.  Stop.

Anyway, some pointers:

you seem to be using lwip from mingw, as a library. Apart from mingw not being supported (though I have no idea in what way that affects using lwip from it), MicroPython has lwip as a submodule and the ports which do use lwip use the submodule's source code. So there might be version differences with what you have.
in any case, since you're linking against lwip32.lib which already has sys_arch_protect there's not much choice here: either get rid of the definition in modlwip.c (Windows port doesn't have threading on so wouldn't really matter) and hope the one in lwip32.lib does something sensible, or don't use lwip32.lib and instead build from lwip source
adding flags like MICROPY_HAVE_SYS_ARCH_PROTECT has no use, MicroPython doesn't use those definitions
the Windows port already comes with a Makefile so an alternative way to build it is using some gcc-like frontend for cl - pretty sure that exists, seem to recall ffmpeg uses that, don't know if it's worth it though
alternatively you could try adding lwip source to msvc/sources.props, add preprocessor definitions in mpconfigoprt.h and additional include directories in msvc/common.props

gvanem · 2020-09-22T10:52:45Z

Makefile.windows:426

In rule mk_dirs in this one?

you seem to be using lwip from mingw,

I was a big fan of MinGW ~ years 10 ago. Now, I hate it.

the Windows port already comes with a Makefile

Tried it. Doesn't work here. E.g a make V=1 in ports/windows:

python3 ../../py/makeversionhdr.py build/genhdr/mpversion.h
GEN build/genhdr/qstrdefs.generated.h
cat ../../py/qstrdefs.h ../unix/qstrdefsport.h build/genhdr/qstrdefs.collected.h | sed 's/^Q(.*)/"&"/' | gcc -E -I. -I..
/.. -Ibuild -Wall -Wpointer-arith -Wdouble-promotion -Werror -std=gnu99 -DUNIX -D__USE_MINGW_ANSI_STDIO=1 
-DFFCONF_H=\"lib/oofatfs/ffconf.h\" -DMICROPY_USE_READLINE=1 -Os   - | 
sed 's/^\"\(Q(.*)\)\"/\1/' > build/genhdr/qstrdefs.preprocessed.h
sed: -e expression #1, char 16: unterminated `s' command
<stdin>:30: fatal error: when writing output to : Invalid argument

Gave up on it. Hence my reason for creating my own Makefile I can understand.

stinos · 2020-09-22T12:30:46Z

In rule mk_dirs in this one?

yes

sed: -e expression #1, char 16: unterminated `s' command

Looks like an escape character problem, is it possible you're not invoking this from a posix-like shell (msys2/cygwin/bash/...) but from cmd or so?

gvanem · 2020-09-22T12:50:34Z

In rule mk_dirs in this one?
yes

The TABs got lost between github and your editor. They're all in there. Click here and save to...

from cmd or so?

I use 4NT from JPsoft.com. So what? CMD, Bash? Puke.

stinos · 2020-09-22T15:19:20Z

The TABs got lost

Aha. Now:

make: *** No rule to make target 'ports/windows/fmode.c', needed by 'objects/genhdr/moduledefs.h'.  Stop.

4NT

Looks like cmd.

Anyway this is probably more suited for the forum since it's not one specific clearly aligned issue.

gvanem · 2020-09-22T17:37:30Z

No rule to make target 'ports/windows/fmode.c',

Huh. That file is in the repo. What make version is this? What does make -f Makefile.Windows CC=cl -n say?

stinos · 2020-09-22T18:33:11Z

$ make -v
GNU Make 4.3
Built for x86_64-pc-msys
Copyright (C) 1988-2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ make -f Makefile.Windows CC=cl -n
mkdir --parents objects/genhdr/qstr/ports/windows/msvc
mkdir objects/genhdr/qstr/extmod          \
             objects/genhdr/qstr/lib             \
             objects/genhdr/qstr/lib/embed       \
             objects/genhdr/qstr/lib/mp-readline \
             objects/genhdr/qstr/lib/netutils    \
             objects/genhdr/qstr/lib/oofatfs     \
             objects/genhdr/qstr/lib/utils       \
             objects/genhdr/qstr/ports/unix      \
             objects/genhdr/qstr/py
make: *** No rule to make target 'ports/windows/fmode.c', needed by 'objects/genhdr/moduledefs.h'.  Stop.

So perhaps it wants the makefile in the MicroPython root directory?

$ mv ./Makefile.windows ../../
$ cd ../../
$ make -f Makefile.Windows CC=clang-cl -n
mkdir --parents objects/genhdr/qstr/ports/windows/msvc
mkdir objects/genhdr/qstr/extmod          \
             objects/genhdr/qstr/lib             \
             objects/genhdr/qstr/lib/embed       \
             objects/genhdr/qstr/lib/mp-readline \
             objects/genhdr/qstr/lib/netutils    \
             objects/genhdr/qstr/lib/oofatfs     \
             objects/genhdr/qstr/lib/utils       \
             objects/genhdr/qstr/ports/unix      \
             objects/genhdr/qstr/py
Makefile.Windows:490: *** open: objects/genhdr/moduledefs.h: No such file or directory.  Stop.

gvanem · 2020-09-22T20:08:33Z

So perhaps it wants the makefile in the MicroPython root directory?

Absolutely. The failed ** open is due to $(file ..). A -n does nothing for mk_dirs. Try w/o a -n.

stinos · 2020-09-23T08:04:44Z

Ok after removing some things which don't compile with plain msvc (mainly socket stuff) this builds an executbale now. Cool to see this can be done.

gvanem · 2020-09-23T08:58:46Z

... don't compile with plain msvc (mainly socket stuff)

Exactly. My motivation behind this issue was to be able to do micropython examples\network\http_client.py. (Linking to LwIP).

I've modified ports/unix/modusocket.c etc. to suite, but all my efforts so far has only led to
ImportError: no module named 'socket'. Help!

Edit: here are the diffs mentioned above.

stinos · 2020-09-23T09:08:53Z

Sorry but it's hard to provide help without knowing exactly what you are doing. And to add to the confiusion different unrelated issues keep on piling up (started with lwip for msvc, then there's the clang problem, now it's about mouscoket but we don't know with which compiler nor what your source code looks like). So I'd suggest you head over to the forum and make a dedicated post per issue in which you provide all possible information to be able to reproduce it (complete build environment information, source code modifications).

gvanem · 2020-09-23T10:17:01Z

And to add to the confiusion different unrelated issues keep on piling up

IMHO, the issues are related to why LwIP won't run. But I've managed with micropython examples\network\http_client.py
to actually call into LwIP, but it aborts on an un-initialised mutex:

Assertion "mutex->mut != NULL" failed at line 375 in contrib/ports/win32/sys_arch.c
Got SIGABRT. Backtrace:
  63AA363D (exc-abort): (filename not available): (function-name not available)
  63AA1406 (exc-abort): (filename not available): (function-name not available)
  7594D218 (ucrtbase): (filename not available): raise
  7594E6D2 (ucrtbase): (filename not available): abort
  mem.c (870): mem_malloc
  memp.c (254): do_memp_malloc_pool_fn
  memp.c (352): memp_malloc_fn
  api_msg.c (714): netconn_alloc
  api_lib.c (155): netconn_new_with_proto_and_callback
  sockets.c (1753): lwip_socket
  modusocket.c (466): socket_make_new
  runtime.c (666): mp_call_method_n_kw
  vm.c (1085): mp_execute_bytecode
  objfun.c (288): fun_bc_call
  vm.c (1000): mp_execute_bytecode
  objfun.c (288): fun_bc_call
  runtime.c (624): mp_call_function_0
  main.c (148): execute_from_lexer
  main.c (687): main_

Seems I have to create something better than ports/unix/main.c for this. Use ports/stm32/main.c as a boiler-plate?

stinos · 2020-09-23T11:08:08Z

IMHO, the issues are related to why LwIP won't run.

It's rather unlikely the ImportError for the socket module is related - unless that's not the exact error.

Seems I have to create something better than ports/unix/main.c for this. Use ports/stm32/main.c as a boiler-plate?

Well, depends on what you mean with boiler plate. There's only a couple of mutexs in lwip, they're not initialized, so solution is to initialize them. Brief glance over the lwip source code lets me think that happens in lwip_init, depedning on flags. So: call that function and make sure the preprocessor flags you use in the MicroPython build are the same preprocessor flags used for the lwip build.

gvanem · 2020-09-23T11:36:12Z

It's rather unlikely the ImportError for the socket module is related

I've progressed a bit further than that. Now a micropython examples\network\http_client.py gives:

Traceback (most recent call last):
  File "examples\network\http_client.py", line 30, in <module>
  File "examples\network\http_client.py", line 10, in main
OSError: [addrinfo error 202]

202 should be ERROR_INFLOOP_IN_RELOC_CHAIN which is highly unlikely. And a micropython -v -vis of absolutely no help.

Edit. 202 == EAI_FAIL. It confused me that MicroPython said OSError.

Brief glance over the lwip source code lets me think that happens in lwip_init

No, lwip_init() is for NO_SYS=1 only. The BSD-API in LwIP needs a NO_SYS=0. This (a lwip_init_stuff() in main.c):

  err_t err = sys_sem_new (&init_sem, 0);
  tcpip_init (test_init, &init_sem);
  sys_sem_wait (&init_sem);
  sys_sem_free (&init_sem);

works kinda, but a lot of LwIP stuff is still missing. It's a real pain of a tcp/ip-stack.

stinos · 2020-09-23T11:44:45Z

No, lwip_init() is for NO_SYS=1 only

Not exactly, tcpip_init calls lwip_init, it's only sys_init which gets skipped. So if you're calling that and the mutex is still not initialized it could still be the problem I mentioned with PP flags.

gvanem · 2020-09-23T12:44:43Z

Finally got it working; had a crash with LWIP_ACD (Address Conflict Detection). No idea why.
Disabled that, rebuilt everything. Now the http_client.py works!!

So I feel like putting together a PR of my patches. But I'm a n00b when it comes to Github and pull requests.

Edit: And with MbedTLS, now even the http_client_ssl.py works!

stinos · 2020-09-23T19:08:43Z

Now the http_client.py works!

That's nice!

So I feel like putting together a PR of my patches

Surely there are a ton of tutorials out there, but practically it comes down to (of which you may have done parts already):

fork this repository (by clicking Fork in the top right I guess)
clone your fork: git clone https://github.com/gvanem/micropython.git, or with ssh that's git clone git@github.com:gvanem/micropython.git
add a the original repository as remote so you can keep track of changes: git remote add upstream https://github.com/micropython/micropython.git (or the ssh version again)
create a branch: git checkout -b lwip
read CODECONVENTIONS and CONTIBUTING in this repository's root, eventually look at other PRs to see what they look like
apply whatever code changes you want to see merged; this should be incremental i.e. each logical standalone change should be one commit with an explanation of why it's needed
push the branch and github will automatically show a button to create a PR now, do that and people will be able to review your changes

Catch main up to 7.3.x

tannewt added a commit to tannewt/circuitpython that referenced this issue Jun 6, 2022

Merge pull request micropython#6463 from adafruit/7.3.x

2baf476

Catch main up to 7.3.x

jonnor added the port-windows label Sep 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Using LWIP on Windows #6463

Using LWIP on Windows #6463

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Using LWIP on Windows #6463

Using LWIP on Windows #6463

Comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!