ValueError: can't have unbuffered text I/O for io.open(1, 'wt', 0) #61606

rbtcollins · 2013-03-12T14:42:41Z

BPO	17404
Nosy	@amauryfa, @pitrou, @rbtcollins, @benjaminp, @hynek, @vadmium, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-03-12.14:42:40.965>
labels = ['3.10', 'type-bug', '3.8', '3.9', 'expert-IO']
title = "ValueError: can't have unbuffered text I/O for io.open(1,\t'wt', 0)"
updated_at = <Date 2020-11-06.22:51:34.480>
user = 'https://github.com/rbtcollins'

bugs.python.org fields:

activity = <Date 2020-11-06.22:51:34.480>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['IO']
creation = <Date 2013-03-12.14:42:40.965>
creator = 'rbcollins'
dependencies = []
files = []
hgrepos = []
issue_num = 17404
keywords = []
message_count = 7.0
messages = ['184025', '184040', '184041', '184045', '184072', '184073', '332739']
nosy_count = 8.0
nosy_names = ['amaury.forgeotdarc', 'pitrou', 'rbcollins', 'benjamin.peterson', 'stutzbach', 'hynek', 'martin.panter', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue17404'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

rbtcollins · 2013-03-12T14:42:40Z

The io library rejects unbuffered text I/O, but this is not documented - and in fact can be manually worked around:
binstdout = io.open(sys.stdout.fileno(), 'wt', 0)
sys.stdout = io.TextIOWrapper(binstdout, encoding=sys.stdout.encoding)
will get a sys.stdout that is unbuffered.

Note that writing to a pipe doesn't really need to care about buffering anyway, if the user writes 300 characters, the codec will output a single block and the IO made will be one write:

This test script:
import sys
import io
stream = io.TextIOWrapper(io.open(sys.stdout.fileno(), 'wb', 0), encoding='utf8')
for r in range(10):
stream.write(u'\u1234'*500)

When run under strace -c does exactly 10 writes: so the performance is predictable. IMO it doesn't make sense to prohibit unbuffered text write I/O. readers may be another matter, but that doesn't suffer the same latency issues.

amauryfa · 2013-03-12T18:58:36Z

The proposed workaround seems to work ("wb" instead of "wt"!), with the following restrictions:

it's not really unbuffered: the encoder has its own buffers (OK, in the stdlib only 'idna' encoding will retain data)
it won't work for reading: TextIOWrapper calls the read1() method, which is only defined by BufferedIO objects.

IMO this explains why it's not a supported combination in io.open().

rbtcollins · 2013-03-12T19:05:19Z

Huh, I didn't realise idna would retain data! But that will still be within the TextIOWrapper itself, right?

And a stream opened 'wt' cannot be read from anyway, so the read1 limitation is irrelevant.

amauryfa · 2013-03-12T19:30:00Z

But that will still be within the TextIOWrapper itself, right?

Yes. And I just noticed that the _io module (the C version) will also buffer encoded bytes, up to f._CHUNK_SIZE.

On the other hand, TextIOWrapper is broken for buffering codecs, encode() is never called with final=True

    >>> import io
    >>> buffer = io.BytesIO()      # <-- not really buffered, right?
    >>> output = io.TextIOWrapper(buffer, encoding='idna')
    >>> output.write("www.somesite.com")
    16
    >>> print(buffer.getvalue())
    b''                            # <-- ok, _CHUNK_SIZE buffering
    >>> output.flush()
    >>> print(buffer.getvalue())
    b'www.somesite.'               # <-- the last word is missing!
    >>> output.close()
    >>> print(buffer.getvalue())
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: I/O operation on closed file.

And it's even worse with python 2.7::

    >>> import io as io
    >>> buffer = io.BytesIO()
    >>> output = io.TextIOWrapper(buffer, encoding='idna')
    >>> output.write("www.somesite.com")
    Traceback (most recent call last):
      File "<stdin>", line 3, in <module>
    TypeError: must be unicode, not str

serhiy-storchaka · 2013-03-13T08:16:20Z

it won't work for reading: TextIOWrapper calls the read1() method, which is only defined by BufferedIO objects.

Since 3.3 TextIOWrapper works with raw IO objects (bpo-12591).

Yes. And I just noticed that the _io module (the C version) will also buffer encoded bytes, up to f._CHUNK_SIZE.

Use write_through=True to disable this.

pitrou · 2013-03-13T09:23:44Z

> - it won't work for reading: TextIOWrapper calls the read1()
> method, which is only defined by BufferedIO objects.

Since 3.3 TextIOWrapper works with raw IO objects (bpo-12591).

It won't be technically unbuffered, though.

vadmium · 2018-12-30T00:00:51Z

It is documented that buffering=0 is not supported in text mode. Look a handful of paragraphs down from <https://docs.python.org/release/3.7.2/library/functions.html#open\>:

“Pass 0 to switch buffering off (only allowed in binary mode)”

Amaury’s problem with the IDNA buffering encoder has now been reported separately in bpo-35611.

Also improves non-blocking support (pythongh-57531) and unbuffered support (pythongh-61606)

rbtcollins added the type-bug An unexpected behavior, bug, or error label Mar 12, 2013

serhiy-storchaka added the topic-IO label Mar 12, 2013

pitrou changed the title ~~ValueError: can't have unbuffered text I/O for io.open(1, 'wt', 0)~~ ValueError: can't have unbuffered text I/O for io.open(1, 'wt', 0) Mar 13, 2013

iritkatriel added 3.8 (EOL) end of life 3.9 only security fixes 3.10 only security fixes labels Nov 6, 2020

ezio-melotti transferred this issue from another repository Apr 10, 2022

cmaloney added a commit to cmaloney/cpython that referenced this issue May 21, 2025

pythongh-85393: Handle partial .write in TextIOWrapper

50027e8

Also improves non-blocking support (pythongh-57531) and unbuffered support (pythongh-61606)

cmaloney added a commit to cmaloney/cpython that referenced this issue May 21, 2025

pythongh-85393: Handle partial .write in TextIOWrapper

d99f750

Also improves non-blocking support (pythongh-57531) and unbuffered support (pythongh-61606)

cmaloney added a commit to cmaloney/cpython that referenced this issue May 21, 2025

pythongh-85393: Handle partial .write in TextIOWrapper

f776a44

Also improves non-blocking support (pythongh-57531) and unbuffered support (pythongh-61606)

cmaloney added a commit to cmaloney/cpython that referenced this issue May 21, 2025

pythongh-85393: Handle partial .write in TextIOWrapper

696abe7

Also improves non-blocking support (pythongh-57531) and unbuffered support (pythongh-61606)

cmaloney mentioned this issue May 21, 2025

gh-85393: Handle partial and blocking .write in TextIOWrapper #134467

Draft

4 tasks

cmaloney added a commit to cmaloney/cpython that referenced this issue May 21, 2025

pythongh-85393: Handle partial .write in TextIOWrapper

d4bfaa8

Also improves non-blocking support (pythongh-57531) and unbuffered support (pythongh-61606)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

ValueError: can't have unbuffered text I/O for io.open(1, 'wt', 0) #61606

ValueError: can't have unbuffered text I/O for io.open(1, 'wt', 0) #61606

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ValueError: can't have unbuffered text I/O for io.open(1, 'wt', 0) #61606

ValueError: can't have unbuffered text I/O for io.open(1, 'wt', 0) #61606

Comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!