[ESP8266] No non-blocking read on uart0 if dupterm is active #4849

Winkelkatze · 2019-06-15T22:16:43Z

As far as I can see, there is no non-blocking way to read the input from the uart0, if this uart is also used for REPL through dupterm.
In version 1.9 and before it is possible to use the UART.read() function for this, however since commit afd0701 the uart buffer is bypassed when UART 0 is used for REPL, so that UART.read() will never receive any data.
The input data is now directly put into the stdin buffer, which has (as far as I know) no function to check if any data is available.

The only workaround for this would be to disable dupterm for the uart0 while intending to read data. Yet, this is somewhat inconvenient, because then print() also won't output any data on the uart.

I think, the best option is to add a non-blocking read for stdin (or at least a way to check, if data is available for read).
This is not the same as a non-blocking read on the uart, but for most applications, this should be sufficient.

The text was updated successfully, but these errors were encountered:

dpgeorge · 2019-06-18T13:35:54Z

Thanks for the report of the issue. Indeed the esp8266 handles this case of UART attached to REPL differently to other ports, in order to be efficient with RAM usage. The stdin ring buffer is "large" (eg so pasting code to the REPL works) and used for both UART REPL and WebREPL, so that they don't both need separate large input buffers.

The only workaround for this would be to disable dupterm for the uart0 while intending to read data. Yet, this is somewhat inconvenient, because then print() also won't output any data on the uart.

Yes this is a good workaround and I don't think that inconvenient, eg use this helper function:

def uart_read(n):
    uart = uos.dupterm(None, 1)
    data = uart.read(n)
    uos.dupterm(uart, 1)
    return data

I think, the best option is to add a non-blocking read for stdin (or at least a way to check, if data is available for read).

Adding a check (poll) for data using the select module would be possible.

Winkelkatze · 2019-06-18T17:26:42Z

I don't think your workaround mimics the behavior of the UART.read() function in the older versions. With your method, any data that is received, while not inside this function would be lost because it is still forwarded to stdin.
I think in the old versions, the UART buffer was only read to stdin while REPL is active. Therefore, a UART.read() call will return the data from the ringbuffer, which is filled in the background, even if the code is not currently in the read() function. This way, read() can be called infrequently and it will still not lose any data.

But yes I also think, adding support to use select on stdin would be the best solution to fix this problem.

dpgeorge · 2019-06-19T03:52:49Z

With your method, any data that is received, while not inside this function would be lost because it is still forwarded to stdin.

Yes, that is true.

I guess the point is you want to use the UART for the REPL (output printing) and for reading in at the same time. You can do this by making a custom TX-only UART, eg:

import io
class UARTHalf(io.IOBase): 
    def __init__(self, uart):
        self.uart = uart
    def write(self, data):
        return self.uart.write(data)

def test():
    uart = os.dupterm(None, 1)
    uart_half = UARTHalf(uart)
    os.dupterm(uart_half, 1)
    print('testing')
    while True:
        data = uart.read(10)
        if data:
            print('got', data)
        if data == b'q':
            break
    os.dupterm(uart, 1)
test()

Type text at the prompt and press "q" to finish.

I think in the old versions, the UART buffer was only read to stdin while REPL is active

The REPL input is always active. Eg you want to be able to stop a program using ctrl-C while it's running, and that requires processing stdin during program execution. That is what makes it tricky.

dpgeorge · 2019-06-19T04:12:51Z

adding support to use select on stdin would be the best solution to fix this problem.

See #4859 for a proposal to do that (and example code to test it on esp8266). Note that this is not really the best solution because you can only ever read 1 char if you want to guarantee non-blocking behaviour (because select/poll can only tell if 1 or more chars is available, not the number of chars/bytes). As such, in that PR I also made sys.stdin.buffer.read(n) non-blocking, but kept sys.stdin.read(n) blocking. This would be a breaking change in terms of behaviour of buffer read.

(On CPython, sys.stdin.read(n) waits for an entire line before returning.)

robert-hh · 2019-06-19T07:12:04Z

On CPython, sys.stdin.read(n) waits for an entire line before returning.

That depends on the tty mode setting, e.g. in linux et. al.. If tty is set to raw mode, it does not wait for an entire line.

dpgeorge · 2019-06-19T07:19:57Z

If tty is set to raw mode, it does not wait for an entire line.

I guess you'd have to reconfigure it as part of the script, and then restore configuration before the script finishes to make sure the REPL still works.

In the end the question is whether to make sys.stdin.buffer non-blocking, or somehow make it configurable by the user, or leave it blocking and use something like UARTHalf as described above.

robert-hh · 2019-06-19T07:27:08Z

I guess you'd have to reconfigure it as part of the script, and then restore configuration before the script finishes to make sure the REPL still works.

That's what I do in the Linux variant of the pye editor.
Besides that, I have no preference, and in matters of hardware platform specific behavior I also do not stick too much to CPython compatibility, given that it is documented(!)

dpgeorge · 2019-06-19T07:36:22Z

That's what I do in the Linux variant of the pye editor.

I see, thanks for the pointer. It looks like you use os.read and os.write on Linux to do the actual IO (rather than using sys.stdin). And on a bare-metal target you use sys.stdin.read(1) to get data, and in fact rely on the blocking behaviour of that call to wait for the user to press a key.

But note that sys.stdin.read(1) to read exactly one byte is already not how CPython works, at least not without reconfiguring the STDIN file. And MicroPython (on bare-metal) doesn't have a way to reconfigure it, so that means the code already needs to be different for the different platforms.

Of course it would be good to keep things as similar to CPython as possible, but in the case of file IO and tty's it's hard. I don't think MicroPython will ever go the way of providing actual file descriptors/integers so things will always be different when it comes to these things.

Winkelkatze · 2019-06-20T16:32:08Z

Thanks for the PR @dpgeorge, this implements exactly the functionality I was missing. Also your proposed workaround seems to be a good solution until the non-blocking buffer read gets upstream.

dpgeorge · 2019-06-24T07:02:48Z

until the non-blocking buffer read gets upstream.

It's not certain that non-blocking capability will be merged, at least not in the form given in that PR. But polling should be there and that already solves the problem of this issue.

dpgeorge · 2019-07-03T04:39:22Z

Polling of sys.stdin was added in c80614d

dpgeorge added the port-esp8266 label Jun 18, 2019

dpgeorge mentioned this issue Jun 19, 2019

sys.stdio: add support for ioctl and make sys.stdin.buffer.read non-blocking #4859

Closed

dpgeorge closed this as completed Jul 3, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[ESP8266] No non-blocking read on uart0 if dupterm is active #4849

[ESP8266] No non-blocking read on uart0 if dupterm is active #4849

Uh 8000 oh!

Uh oh!

Uh oh!

Uh oh 8000 !

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[ESP8266] No non-blocking read on uart0 if dupterm is active #4849

[ESP8266] No non-blocking read on uart0 if dupterm is active #4849

Comments

Uh 8000 oh!

Uh oh!

Uh oh!

Uh oh 8000 !

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!