8000 esp8266: Characters missing when reading from UART · Issue #4153 · micropython/micropython · GitHub
[go: up one dir, main page]

Skip to content

esp8266: Characters missing when reading from UART #4153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
freol35241 opened this issue Sep 16, 2018 · 17 comments
Closed

esp8266: Characters missing when reading from UART #4153

freol35241 opened this issue Sep 16, 2018 · 17 comments

Comments

@freol35241
Copy link

Hi,

First, thanks for micropython, its very handy!

But today I got stuck on something weird, it seem that when reading from a serial connection with the UART module it skips some characters. Actually it seems to skip every 16th character so that when calling either uart.readline() or uart.read(value>15) characters are missing.

Could potentially be a problem with an internal buffer that is not emptied/read properly?

For info,
Hardware is Weemos D1 mini pro (ESP8266ex)
Micropython version (latest): esp8266-20180907-v1.9.4-494-ge814db592

@adritium
Copy link

This is not the forum but a bug, feature tracker.
Unless you have a strong suspicion it’s a bug, move this to the forum.

@freol35241
Copy link
Author

Well, I have not found a way to read the 16th character. This is not expected behavior based on the current documentation 10000 and therefore I would define it as a "bug" in either code or docs.

@robert-hh
Copy link
Contributor

What kind of data are you reading at which speed?

@freol35241
Copy link
Author
freol35241 commented Sep 16, 2018

baudrate 9600

Using readline()-method

received data: b' 21650;   230; * 140;   507;   *14;   435;   23*;   219;  -512;*;0;0;0;0;0;0;1;*;0;0;   185;   *30;   245;   26*;   495;   520;*  545;   220;  *245;   270;  
 3*0;   240;     0*     0;     0; *   0;     0\r\n'

Missing characters are shown by *

First * is a space
Second * is digit
Third * is a digit
etc.

@robert-hh
Copy link
Contributor

9600 Baud is pretty slow. if you are reading fast enough, then there should not be a problem. The ringbuffer for UART has a size of 16 bytes. That seems to indicate, that there is a problem. But looking at the code at py/ringbuf.h, it is not obvious to me. I have coded almost identical buffers in the past with no data loss at all, as long as I read out fast enough,

@peterhinch
Copy link
Contributor

To back up the comment from @adritium there is a limited audience here consisting of MicroPython maintainers and contributors.

Your query will be seen by many more people, some of whom will have used the UART on ESP8266, in the forum. In my opinion it's unlikely to be a bug as the single bidirectional UART is widely used for the REPL and for uploading substantial programs. In that context it works fine.

@freol35241
Copy link
Author

Hi all, thank you for you interest in this topic. I decided to do an extra check before following the suggestion to post in the forum instead. Below is a copy-paste from the webrepl where i first detach the default UART and then attach a new UART with the right baudrate for my incoming traffic. The incoming traffic is now posted directly to stdin and all characters are showing (nothing is missing):

WebREPL connected                                                                                                                                     
>>> import machine, uos                                                                                                                               
>>> uos.dupterm(None, 1)                                                                                                                              
UART(0, baudrate=115200, bits=8, parity=None, stop=1, timeout=0, timeout_char=1)                                                                      
>>> uos.dupterm(machine.UART(0,9600), 1)                                                                                                              
>>>  22885;   235;   152;   500;   524;   524;   233;   238;  -512;0;0;0;0;0;0;1;0;0;0;0;   176;   221;   236;   251;   495;   520;   545;   211;   23
6;   261;   330;   240;     0;     0;     0;     0;     0 

Doing the same but not directing to stdin:

WebREPL connected                                                                                                                                     
>>> import machine, uos                                                                                                                               
>>> uos.dupterm(None, 1)                                                                                                                              
UART(0, baudrate=115200, bits=8, parity=None, stop=1, timeout=0, timeout_char=1)                                                                      
>>> ser = machine.UART(0, 9600)                                                                                                                       
>>> while True:                                                                                                                                       
...     ser.readline()                                                                                                                                
...                                                                                                                                                   
...                                                                                                                                                   
...                                                                                                                                                   
b' 22898;   235; '                                                                                                                                    
b' 149;   500;   '                                                                                                                                    
b'21;   523;   23'                                                                                                                                    
b';   236;  -512;'                                                                                                                                    
b';0;0;0;0;0;1;0;'                                                                                                                                    
b';0;0;   178;   '                                                                                                                                    
b'23;   238;   25'                                                                                                                                    
b';   495;   520;'                                                                                                                                    
b'  545;   213;  '                                                                                                                                    
b'238;   263;   3'                                                                                                                                    
b'0;   240;     0'                                                                                                                                    
b'     0;     0; '                                                                                                                                    
b'   0;     0\r\n'    

In the bottom example, every 16th character is missing. This leads me to believe that there has to be a difference in how the connections going to stdin and the ones going into another buffer is handled?

I will be happy to post this question in the forum instead if you still think this belongs there.

@robert-hh
Copy link
Contributor
robert-hh commented Sep 17, 2018

For me it looks more like an issue in combination with ser.readline() and the internal buffer mechanism. Could you try a combination of ser.any(()) and ser.read() instead?
It might have been addressed already in the past.
Edit: The difference between stdin and uart is the size of the ring buffer, which is 256 for stdin and 16 for uart.

@peterhinch
Copy link
Contributor

I would suggest that the rxbuf argument to the UART constructor should specify a buffer big enough to hold the longest line which might be transmitted. It's not obvious (to me) what readline() is supposed to do when the buffer fills up before a newline character is received.

@robert-hh
Copy link
Contributor

I do not see in the code that an rxuf argument would be supported. I have no ESP8266 here at this very moment, so I cannot try. The only buffer I could find is this small ringbuffer for incoming characters:
https://github.com/micropython/micropython/blob/master/ports/esp8266/uart.c#L39
And a piece of code in readline, which drops a character in certain error conditions around this place: https://github.com/micropython/micropython/blob/master/py/stream.c#L382
But I could not get further (yet).

@peterhinch
Copy link
Contributor

Ah. My fault for reading the docs rather than studying the code :) The docs suggest that init() can be called with an rxbuf arg, but a quick try (and a look at the code) suggests it's not actually supported.

It would therefore seem that readline() is only suitable for lines of <= 16 chars (including any end of line characters). So I'd agree with your suggestion of reading longer lines on a character by character basis. The approach I'd normally advocate would be to use uasyncio's StreamReader and StreamWriter objects, but I haven't tried this on ESP8266. I have serious doubt that a 16 character buffer is enough to cope with the latency implicit in using uasyncio (with other tasks competing for the CPU).

So the issue to raise is the fact that the docs don't agree with reality. Either the docs or the code should be changed. Preferably the latter; a fixed size 16 byte buffer is of limited use, especially on a device with the amount of latency as the ESP8266. How does it reliably handle file uploads via rshell at 115200 baud?

@robert-hh
Copy link
Contributor

Hi Peter- It is still and issue, because stream readline() manages its own buffer of abritrary length. So it should work also for longer lines. its is only the ISR's circular buffer with a length of 16, being able to buffer 15 characters.
b.t.w.: readline can be call with a numeric parameter, which is the buffer size for readline. But that should not change the picture. For me it looks more like the interaction between readline and the uart receive functions.
And yes: there is also the issue of documentation not matching the implementation.

@freol35241
Copy link
Author
freol35241 commented Sep 18, 2018

Some further updates

ser.any() in combination with ser.read() gives the same result as with ser.readline()

WebREPL connected                                                                                                                                     
>>> import machine, uos                                                                                                                               
>>> uos.dupterm(None, 1)                                                                                                                              
UART(0, baudrate=115200, bits=8, parity=None, stop=1, timeout=0, timeout_char=1) 
>>> ser = machine.UART(0, 9600)                                                                                                                       
>>> while True:                                                                                                                                       
...     if ser.any():                                                                                                                                 
...         ser.read()                                                                                                                                
...                                                                                                                                                   
...                                                                                                                                                   
...                                                                                                                                                 
b' 24432;   239; '                                                                                                                                    
b' 195;   504;   '                                                                                                                                    
b'26;   527;   40'                                                                                                                                    
b';   249;  -512;'                                                                                                                                    
b';0;0;0;0;0;1;0;'                                                                                                                                    
b';0;0;   143;   '                                                                                                                                    
b'88;   203;   21'                                                                                                                                    
b';   495;   520;'                                                                                                                                    
b'  545;   178;  '                                                                                                                                    
b'203;   228;   3'                                                                                                                                    
b'0;   240;     0'                                                                                                                                    
b'     0;     0; '                                                                                                                                    
b'   0;     0\r\n' 

FYI, I managed to solve my immediate problem by using sys.stdin as a workaround:

WebREPL connected                                                                                                                                     
>>> import machine, uos, sys                                                                                                                               
>>> uos.dupterm(None, 1)                                                                                                                              
UART(0, baudrate=115200, bits=8, parity=None, stop=1, timeout=0, timeout_char=1)        
>>> uos.dupterm(machine.UART(0,9600), 1)                                                                                                                                                                                                                                                
>>> sys.stdin.readline()                                                                                                                              
' 24439;   239;   194;   504;   524;   526;   400;   249;  -512;0;0;0;0;0;0;1;0;0;0;0;   144;   189;   204;   219;   495;   520;   545;   179;   204; 
  229;   330;   240;     0;     0;     0;     0;     0\n'

@robert-hh
Copy link
Contributor

Sadly, that was my suspicion. sys.stdin uses the larger buffer of 256 bytes.

@freol35241
Copy link
Author

For the record, the lines I am reading are smaller than 256 bytes (~200 characters) and the same issue should therefore not be visible anyway when using sys.stdin

@robert-hh
Copy link
Contributor

It might be interesting if the issue reappears after 255 characters for longer lines.

@dpgeorge
Copy link
Member
dpgeorge commented Dec 5, 2018

Indeed the issue here is with the UART RX buffer. When attached to dupterm as stdin/stdout the RX buffer that is used for incoming UART characters in the stdin buffer, which is 256 bytes, so 255 usable entries. When the UART is detached from the REPL the buffer is only 16 bytes, or 15 usable entries. So that explains all the behaviour seen/described above.

In commit 52bec93 I implemented the rxbuf argument to the UART constructor and init() method, so now you can set a large enough buffer for any given application (the buffer will come from the heap). Eg:

uart.init(rxbuf=200)

I think that should fully solve this issue.

@dpgeorge dpgeorge closed this as completed Dec 5, 2018
@dpgeorge dpgeorge changed the title Characters missing when reading from UART esp8266: Characters missing when reading from UART Dec 5, 2018
tannewt pushed a commit to tannewt/circuitpython that referenced this issue Jun 23, 2021
tannewt added a commit to tannewt/circuitpython that referenced this issue Jun 23, 2021
resolves micropython#4153: Fixed build issue when CIRCUITPY_USB is off
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants
0