Strange ETIMEDOUT with urequests lib #14105

vdomos · 2024-03-16T09:42:10Z

vdomos
Mar 16, 2024

Hello,

I have a strange problem with the urequests module.

The code runs on a Raspberry Pico-W and a Micropython:
recently updated:

MicroPython v1.22.2 on 2024-02-22; Raspberry Pi Pico W with RP2040

The urequests library is in version 0.8.0

I'm trying to access a URL with json data in return.

This request works with various URLs tested but not with the ESPHOME REST API.

Here's the code used:

import time
from machine import UART, Pin
import network
import rp2
import urequests

country = 'FR'
ssid = 'xxxxxxxxxxxxxx'
password = 'yyyyyyyyyyyyy'

rp2.country(country)  
wlan = network.WLAN(network.STA_IF)
wlan.active(True)

print('Connecting to %s WIFI AP' % (ssid))
wlan.connect(ssid, password)
while not wlan.isconnected() and wlan.status() >= 0:
  print("Waiting to connect...")
  time.sleep(1)
print(f"Connected to WIFI: {wlan.ifconfig()}")

URL = "http://192.168.0.x:yyyy/metrics"

req = urequests.get(URL, timeout=10)
jdata = req.json()
print(f"Data obtained: {jdata}")

this works well with an internal URL, for example:

Connecting to xxxxxxxx WIFI AP
Connected to WIFI: ('192.168.0.x', '255.255.255.0', '192.168.0.y', '192.168.0.z')
Data obtained: {'temperature': 11.4, 'ts': 1.710534e+09, 'dailyrain': 0, 'pressure': 1012.8, 'humidity': 88.3, 'daytempmax': 18.9, 'luminance': 0.0, 'counter': 257, 'hourlyrain': 0, 'daytempmin': 11.2}

But with a ESPHOME API REST URL,

URL = "http://192.168.0.x/sensor/huzzah_weather_pressure"

I have a ETIMEDOUT error

Connecting to xxxxxxx WIFI AP
Connected to WIFI: ('192.168.0.x', '255.255.255.0', '192.168.0.y', '192.168.0.z')
Traceback (most recent call last):
  File "<stdin>", line 33, in <module>
  File "requests/__init__.py", line 33, in json
  File "requests/__init__.py", line 20, in content
OSError: [Errno 110] ETIMEDOUT

Yet this request works all the time with a curl under Linux for example.
The only difference I see with curl in verbose mode is that the connection seems to close before the data is available.

With a URL that works with curl and urequests:

$ curl -vvv "http://192.168.0.x:9108/metrics"; echo
*   Trying 192.168.0.x:9108...
* Connected to 192.168.0.x (192.168.0.x) port 9108
> GET /metrics HTTP/1.1
> Host: 192.168.0.x:9108
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: Werkzeug/3.0.1 Python/3.11.7
< Date: Fri, 15 Mar 2024 20:30:27 GMT
< Content-Type: application/json
< Content-Length: 168
< Connection: close
<
{"counter":257,"dailyrain":0,"daytempmax":18.9,"daytempmin":11.2,"hourlyrain":0,"humidity":88.3,"luminance":0.0,"pressure":1012.8,"temperature":11.4,"ts":1710534005.0}
* Closing connection

With ESPHOME API REST URL:

$ curl -vvv "http://192.168.0.y/sensor/huzzah_weather_pressure";echo
*   Trying 192.168.0.y:80...
* Connected to 192.168.0.y (192.168.0.y) port 80
> GET /sensor/huzzah_weather_pressure HTTP/1.1
> Host: 192.168.0.y
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Length: 76
< Content-Type: application/json
< Access-Control-Allow-Origin: *
< Connection: close
< Accept-Ranges: none
<
* Closing connection                                                                                                 
{"id":"sensor-huzzah_weather_pressure","value":1013.73,"state":"1013.7 hPa"}

glenn20 · 2024-03-17T07:48:12Z

glenn20
Mar 17, 2024

This is a strange problem. When you contrast with an "internal" ip address, do you mean that your devices are on different subnet ranges, in which case you might want to dounle check your firewall routing between subnets. Also worth trying is to update requests to version 0.90 to see if that makes any difference (see https://github.com/micropython/micropython-lib/blob/master/python-ecosys/requests/manifest.py).

1 reply

vdomos Mar 17, 2024
Author

Thank for your reply,

Sorry, when I say internal, this is just on my own lan network.
I have no subnet.

I've also tested it on an internet URL like http://api.open-meteo.com/v1/forecast?..., and it works fine.
This only seems to happen with a URL from the ESPHOME REST API, and I've also opened a issue for it.

vdomos · 2024-03-17T13:14:30Z

vdomos
Mar 17, 2024
Author

Can you guide me to update urequests to 0.9.0.

I don't have pip installed on this version or I'm doing it wrong.

6 replies

vdomos Mar 22, 2024
Author

thank you, I was able to reinstall a new version of urequest with mip.
Unfortunately I still have th error accessing the esphome api.

glenn20 Mar 25, 2024

Sorry to hear that. My first suspicion is that the socket is not being closed correctly at the other end. I suggest trying the following to try to diagnose the problem:

After req = urequests.get(URL, timeout=10), insert the following code:

s = req.raw
while (c := s.read(1)):
    print("Got ", c)

This will print each character of the json string as it arrives from the socket.You can then see if you get any data at all, or if the server is just not responding with any data after accepting the connection.

(whoops - added an important correction to the sample code).

vdomos Mar 30, 2024
Author

Hello,

Here is the result of your test solution:

Connecting to WIFI AP
Connected to WIFI: ('192.168.0.x', '255.255.255.0', '192.168.0.y', '192.168.0.z')
Got  b'{'
Got  b'"'
Got  b'i'
Got  b'd'
Got  b'"'
Got  b':'
Got  b'"'
Got  b's'
Got  b'e'
Got  b'n'
Got  b's'
Got  b'o'
Got  b'r'
Got  b'-'
Got  b'h'
Got  b'u'
Got  b'z'
Got  b'z'
Got  b'a'
Got  b'h'
Got  b'_'
Got  b'w'
Got  b'e'
Got  b'a'
Got  b't'
Got  b'h'
Got  b'e'
Got  b'r'
Got  b'_'
Got  b'p'
Got  b'r'
Got  b'e'
Got  b's'
Got  b's'
Got  b'u'
Got  b'r'
Got  b'e'
Got  b'"'
Got  b','
Got  b'"'
Got  b'v'
Got  b'a'
Got  b'l'
Got  b'u'
Got  b'e'
Got  b'"'
Got  b':'
Got  b'9'
Got  b'9'
Got  b'6'
Got  b'.'
Got  b'0'
Got  b'4'
Got  b','
Got  b'"'
Got  b's'
Got  b't'
Got  b'a'
Got  b't'
Got  b'e'
Got  b'"'
Got  b':'
Got  b'"'
Got  b'9'
Got  b'9'
Got  b'6'
Got  b'.'
Got  b'0'
Got  b' '
Got  b'h'
Got  b'P'
Got  b'a'
Got  b'"'
Got  b'}'
Traceback (most recent call last):
  File "<stdin>", line 31, in <module>
OSError: [Errno 110] ETIMEDOUT

The json is well received, here is the expected answer:

{"id":"sensor-huzzah_weather_pressure","value":996.14,"state":"996.1 hPa"}

But the connection ends in timeout.

glenn20 Apr 15, 2024

Mmmm - this seems to confirm my suspicion that the remote is not closing the socket after transmitting the data or that the local does not recognise that the socket has been closed. There are various responses to this: investigate the underlying cause or hack the code to work when the socket is not closed (or detected as closed).

glenn20 Apr 15, 2024

eg. a hack that should work for this very specific case could be (warning: untested code fragment):

s = req.raw
resp = b""
while (c := s.read(1)):
    resp += c
    if c == "}":
        break
req.close()
jdata = json.loads(resp)

You would need to adapt if the response json can include nested brace ("{}") pairs.

shariltumin · 2024-04-01T20:47:14Z

shariltumin
Apr 1, 2024

I was experiencing similar problems (ECONNRESET) in getting response from a simple asyncio-based http server on an esp32 and rp2040 clients using requests. I ended up modifying the content() method of the library as follows:

    @property
    def content(self):
        if self._cached is None:
            blk=200 # 2 is min blk size
            self._cached = b''
            try:
                while True:
                   data = self.raw.recv(blk)
                   self._cached += data
                   if len(data)<blk: break
            finally:
                self.raw.close()
                self.raw = None
        return self._cached

As you can see, from the original socket.read() I am using socket.recv(blk) instead.

Try it, it might work for you as well.

0 replies

vdomos · 2024-04-14T16:45:14Z

vdomos
Apr 14, 2024
Author

Hi,
@shariltumin , thank you for your reply
Sorry for this question, but how to modify the urequests.py file.
The biliotheque is integrated in Micropython, so it's impossible to access it.

I tried to install another version of urequest in /lib, but it is not loaded

2 replies

shariltumin Apr 14, 2024

You can find the slightly modified version of the library here. Please read the disclaimer. Hopefully it will help.

saliWd Apr 15, 2025

had the same issue, modified version did help as well. Thanks a lot

vdomos · 2024-04-18T18:50:35Z

vdomos
Apr 18, 2024
Author

Hi,

thanks @shariltumin, @glenn20 for the time you took to help me and the analyses and solutions you provided

The modified version of the library works fine.

0 replies

roberterdin · 2025-03-24T02:48:12Z

roberterdin
Mar 24, 2025

FWIW, I ran into the same issue in an even more nondeterministic way.
I'm making 2 subsequent calls to the same API with a different body. The 2nd call would always fail with ETIMEDOUT. Strangely, if I make the same 2 calls in a standalone application (i.e. I just have 2 requests.post statements with the exact same payload hard-coded as in my application), both calls can be made successfully.
Either way, the fix provided by @shariltumin works for me too. But I wonder if it's worth patching the standard library?

0 replies

Strange ETIMEDOUT with urequests lib #14105

Uh oh!

Uh oh!

Replies: 6 comments · 9 replies

Uh oh!

Uh oh!

vdomos Mar 17, 2024 Author

Uh oh!

vdomos Mar 17, 2024 Author

Uh oh!

vdomos Mar 22, 2024 Author

Uh oh!

Uh oh!

Uh oh!

vdomos Mar 30, 2024 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vdomos Apr 14, 2024 Author

Uh oh!

Uh oh!

Uh oh!

vdomos Apr 18, 2024 Author

Uh oh!

Replies: 6 comments 9 replies

vdomos Mar 17, 2024
Author

vdomos
Mar 17, 2024
Author

vdomos Mar 22, 2024
Author

vdomos Mar 30, 2024
Author

vdomos
Apr 14, 2024
Author

vdomos
Apr 18, 2024
Author