bpo-24234: Implement bytes.bytes #27901

corona10 · 2021-08-23T01:37:28Z

https://bugs.python.org/issue24234

corona10 · 2021-08-23T01:39:15Z

Tools/clinic/clinic.py

@@ -2179,7 +2179,6 @@ def __repr__(self):
 __abs__
 __add__
 __and__
-__bytes__


https://docs.python.org/3/c-api/typeobj.html?highlight=nb_int#sub-slots

Same reason as __complex__
#27886

gvanrossum

Should we perhaps combine the two News items for this bpo issue?

corona10 · 2021-08-23T01:48:28Z

Should we perhaps combine the two News items for this bpo issue?

Yes, it looks reasonable to add them as News items :) cc @mdickinson

corona10 · 2021-08-23T02:05:33Z

doctest (doctest) ... 66 tests with zero failures
**********************************************************************
File "/home/runner/work/cpython/cpython/Lib/test/test_doctest.py", line 671, in test.test_doctest.test_DocTestFinder.non_Python_modules
Failed example:
    816 < len(tests) < 836 # approximate number of objects with docstrings
Expected:
    True
Got:
    False
**********

The broken test will be fixed after #27887 is merged and rebased.

mdickinson · 2021-08-23T08:08:33Z

Should we perhaps combine the two News items for this bpo issue?

SGTM

mdickinson

LGTM, modulo the combining news items and the test failure.

corona10 · 2021-08-23T08:43:06Z

Doc/whatsnew/3.11.rst

@@ -169,6 +169,14 @@ Other Language Changes
  (Contributed by Serhiy Storchaka in :issue:`12022`.)


+Other CPython Implementation Changes


@mdickinson Can you please take a look? Sorry, I am not a good English writer.

@corona10 What you have looks fine to me!

bedevere-bot · 2021-08-23T10:09:48Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Fedora Clang 3.x has failed when building commit 24b63c6.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/3/builds/617) and take a look at the build logs.
Check if the failure is related to this commit (24b63c6) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/3/builds/617

Failed tests:

test_multiprocessing_spawn

Failed subtests:

test_process - test.test_multiprocessing_spawn.WithProcessesTestProcess

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

412 tests OK.

10 slowest tests:

test_peg_generator: 2 min 53 sec
test_concurrent_futures: 2 min 30 sec
test_multiprocessing_spawn: 1 min 11 sec
test_multiprocessing_forkserver: 1 min 2 sec
test_multiprocessing_fork: 53.1 sec
test_asyncio: 48.7 sec
test_signal: 46.7 sec
test_io: 36.2 sec
test_socket: 34.9 sec
test_tokenize: 32.8 sec

1 test failed:
test_multiprocessing_spawn

15 tests skipped:
test_devpoll test_ioctl test_kqueue test_msilib test_nis
test_ossaudiodev test_readline test_startfile test_tix test_tk
test_ttk_guionly test_winconsoleio test_winreg test_winsound
test_zipfile64

Total duration: 5 min

Click to see traceback logs

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/Lib/test/_test_multiprocessing.py", line 360, in test_process
    self.assertEqual(q.get(), current.authkey)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b'y\x[24 chars]xf8\xcf\xda' != b'y\x[24 chars]xf8\xcf\xda\x00\xbd\xaf\xa3\xc3IV\xdde|\x8b\xb9\xc0yl\xa9*\x1f'


Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/Lib/test/_test_multiprocessing.py", line 360, in test_process
    self.assertEqual(q.get(), current.authkey)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b'\xd[19 chars]x0f\xb2\xd1Z' != b'\xd[19 chars]x0f\xb2\xd1Z\x00Ly\x8d\r;\xc8c\xf6l}\xbf\x00\xfd\xcf\xc5Hd\x1d'

bedevere-bot · 2021-08-23T10:11:36Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 Fedora Stable 3.x has failed when building commit 24b63c6.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/125/builds/523) and take a look at the build logs.
Check if the failure is related to this commit (24b63c6) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/125/builds/523

Failed tests:

test_concurrent_futures

Failed subtests:

test_ressources_gced_in_workers - test.test_concurrent_futures.ProcessPoolForkProcessPoolExecutorTest

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

414 tests OK.

10 slowest tests:

test_concurrent_futures: 2 min 37 sec
test_peg_generator: 1 min 42 sec
test_multiprocessing_spawn: 1 min 29 sec
test_asyncio: 1 min 24 sec
test_multiprocessing_forkserver: 1 min 9 sec
test_unparse: 1 min 7 sec
test_multiprocessing_fork: 1 min
test_capi: 1 min
test_tokenize: 56.9 sec
test_gdb: 54.6 sec

1 test failed:
test_concurrent_futures

13 tests skipped:
test_devpoll test_ioctl test_kqueue test_msilib test_ossaudiodev
test_startfile test_tix test_tk test_ttk_guionly test_winconsoleio
test_winreg test_winsound test_zipfile64

Total duration: 2 min 45 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_concurrent_futures.py", line 95, in __del__
    self.event.set()
    ^^^^^^^^^^
AttributeError: 'EventfulGCObj' object has no attribute 'event'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_concurrent_futures.py", line 95, in __del__
    self.event.set()
    ^^^^^^^^^^
AttributeError: 'EventfulGCObj' object has no attribute 'event'
Warning -- Unraisable exception
Exception ignored in: <function EventfulGCObj.__del__ at 0xffff8975fb70>
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_concurrent_futures.py", line 95, in __del__
    self.event.set()
Warning -- Unraisable exception
Exception ignored in: <function EventfulGCObj.__del__ at 0xffff8975fb70>
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_concurrent_futures.py", line 95, in __del__
    ^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 1089, in set
    self.event.set()
    ^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 1089, in set
    return self._callmethod('set')
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 814, in _callmethod
    return self._callmethod('set')
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 814, in _callmethod
    self._connect()
    ^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 801, in _connect
    self._connect()
    ^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 801, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/connection.py", line 513, in Client
    conn = self._Client(self._token.address, authkey=self._authkey)
     answer_challenge(c, authkey)
          ^^^^^^^^^^^^^^^^^^^^^^^^^ ^ ^ ^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_concurrent_futures.py", line 95, in __del__
    self.event.set()
    ^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 1089, in set
    return self._callmethod('set')
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 814, in _callmethod
    self._connect()
    ^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 801, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/connection.py", line 507, in Client
    c = SocketClient(address)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/connection.py", line 635, in SocketClient
    s.connect(address)
    ^^^^^^^^^^^^^^^^^^
ConnectionRefusedError: [Errno 111] Connection refused


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/support/hashlib_helper.py", line 36, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_concurrent_futures.py", line 978, in test_ressources_gced_in_workers
    future.result()
    ^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/concurrent/futures/_base.py", line 445, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
    ^^^^^^^^^^^^^^^^^^^^^
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

bedevere-bot · 2021-08-23T10:14:55Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Fedora Stable LTO + PGO 3.x has failed when building commit 24b63c6.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/29/builds/586) and take a look at the build logs.
Check if the failure is related to this commit (24b63c6) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/29/builds/586

Failed tests:

test_multiprocessing_spawn

Failed subtests:

test_process - test.test_multiprocessing_spawn.WithProcessesTestProcess

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

413 tests OK.

10 slowest tests:

test_concurrent_futures: 2 min 26 sec
test_nntplib: 1 min 21 sec
test_multiprocessing_spawn: 1 min 5 sec
test_multiprocessing_forkserver: 57.1 sec
test_asyncio: 54.2 sec
test_multiprocessing_fork: 53.4 sec
test_signal: 47.0 sec
test_io: 37.6 sec
test_pydoc: 31.7 sec
test_xmlrpc: 27.6 sec

1 test failed:
test_multiprocessing_spawn

14 tests skipped:
test_devpoll test_gdb test_ioctl test_kqueue test_msilib
test_ossaudiodev test_startfile test_tix test_tk test_ttk_guionly
test_winconsoleio test_winreg test_winsound test_zipfile64

Total duration: 2 min 52 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/test/_test_multiprocessing.py", line 360, in test_process
    self.assertEqual(q.get(), current.authkey)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b'\xd[27 chars]8d\xaf?<@]\x88o\xc2\xc7\x1aY\xc4' != b'\xd[27 chars]8d\xaf?<@]\x88o\xc2\xc7\x1aY\xc4\x00\xae<\x9d\x04\x0e\xf3\x99k'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/test/_test_multiprocessing.py", line 360, in test_process
    self.assertEqual(q.get(), current.authkey)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b'\xf[45 chars]\xc0\x8a\x97\x14\xfe' != b'\xf[45 chars]\xc0\x8a\x97\x14\xfe\x00\xe1\xdd&\xad\xdcY\xccn\xbd_\x04\xfbTd'

bedevere-bot · 2021-08-23T10:15:20Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL7 3.x has failed when building commit 24b63c6.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/15/builds/668) and take a look at the build logs.
Check if the failure is related to this commit (24b63c6) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/15/builds/668

Failed tests:

test_multiprocessing_forkserver

Failed subtests:

test_process - test.test_multiprocessing_forkserver.WithProcessesTestProcess

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

411 tests OK.

10 slowest tests:

test_mailbox: 7 min 25 sec
test_shelve: 6 min 15 sec
test_concurrent_futures: 2 min 46 sec
test_peg_generator: 2 min 26 sec
test_unparse: 1 min 47 sec
test_multiprocessing_spawn: 1 min 47 sec
test_capi: 1 min 33 sec
test_tokenize: 1 min 24 sec
test_asyncio: 1 min 20 sec
test_multiprocessing_forkserver: 1 min 14 sec

1 test failed:
test_multiprocessing_forkserver

16 tests skipped:
test_devpoll test_gdb test_ioctl test_kqueue test_msilib
test_ossaudiodev test_smtpnet test_ssl test_startfile test_tix
test_tk test_ttk_guionly test_winconsoleio test_winreg
test_winsound test_zipfile64

Total duration: 10 min 14 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64/build/Lib/test/_test_multiprocessing.py", line 360, in test_process
    self.assertEqual(q.get(), current.authkey)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b'\x1[34 chars]x86\x95\xc0o@\x89\x1ao\xa5W\xba\xbd' != b'\x1[34 chars]x86\x95\xc0o@\x89\x1ao\xa5W\xba\xbd\x00=\xb3\xfd\xf1\\\xaf\x90'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64/build/Lib/test/_test_multiprocessing.py", line 360, in test_process
    self.assertEqual(q.get(), current.authkey)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b'\x1[30 chars]M\xb9\xc0\x86{\xbdq9|\xbbH\xed' != b'\x1[30 chars]M\xb9\xc0\x86{\xbdq9|\xbbH\xed\x00X \xe9\xc0\x84S\x99\xec\xaew'

mdickinson · 2021-08-23T10:19:26Z

Hmm. Unfortunately, I think those buildbot failures aren't false positives. By using PyBytes_FromString, we're truncating at the first zero byte. I think we need to use PyBytes_FromStringAndSize instead (and add a regression test).

>>> class B(bytes): pass
... 
>>> x = B(b"\x00\x01\x02")
>>> x.__bytes__()
b''

corona10 requested a review from gvanrossum as a code owner August 23, 2021 01:37

the-knights-who-say-ni added the CLA signed label Aug 23, 2021

bedevere-bot added the awaiting core review label Aug 23, 2021

corona10 requested a review from mdickinson August 23, 2021 01:37

corona10 commented Aug 23, 2021

View reviewed changes

corona10 force-pushed the bpo-24234 branch from 59e0a3a to 01cdf3e Compare August 23, 2021 01:41

corona10 changed the title ~~bpo-34234: Implement bytes.__bytes__~~ bpo-24234: Implement bytes.__bytes__ Aug 23, 2021

gvanrossum approved these changes Aug 23, 2021

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Aug 23, 2021

corona10 force-pushed the bpo-24234 branch from fd8dbc2 to 22009fb Compare August 23, 2021 01:45

mdickinson approved these changes Aug 23, 2021

View reviewed changes

corona10 added 2 commits August 23, 2021 17:17

bpo-24234: Implement bytes.__bytes__

b0638e2

bpo-24234: nit

8f283da

corona10 force-pushed the bpo-24234 branch from 22009fb to 8f283da Compare August 23, 2021 08:18

corona10 commented Aug 23, 2021

View reviewed changes

bpo-24234: Add whatsnew 3.11

a6ba24a

corona10 force-pushed the bpo-24234 branch from 02d838a to a6ba24a Compare August 23, 2021 08:53

corona10 merged commit 24b63c6 into python:main Aug 23, 2021

bedevere-bot removed the awaiting merge label Aug 23, 2021

corona10 deleted the bpo-24234 branch August 23, 2021 10:01

mdickinson mentioned this pull request Aug 23, 2021

bpo-24234: fix bytes.__bytes__ to not truncate at a zero byte #27902

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

bpo-24234: Implement bytes.bytes #27901

bpo-24234: Implement bytes.bytes #27901

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

		@@ -169,6 +169,14 @@ Other Language Changes
		(Contributed by Serhiy Storchaka in :issue:`12022`.)


		Other CPython Implementation Changes

Uh oh!

bpo-24234: Implement bytes.__bytes__ #27901

bpo-24234: Implement bytes.__bytes__ #27901

Uh oh!

Conversation

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Uh oh!

Uh oh!

bpo-24234: Implement bytes.bytes #27901

bpo-24234: Implement bytes.bytes #27901