8000 test_asyncio: test_sendfile failed on ReFS, Windows 11 Dev Drive · Issue #111347 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

test_asyncio: test_sendfile failed on ReFS, Windows 11 Dev Drive #111347

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
zcxsythenew opened this issue Oct 26, 2023 · 2 comments · Fixed by #111377
Closed

test_asyncio: test_sendfile failed on ReFS, Windows 11 Dev Drive #111347

zcxsythenew opened this issue Oct 26, 2023 · 2 comments · Fixed by #111377
Labels
tests Tests in the Lib/test dir topic-asyncio type-bug An unexpected behavior, bug, or error

Comments

@zcxsythenew
Copy link
Contributor
zcxsythenew commented Oct 26, 2023 •
edited by bedevere-app bot
Loading

Bug report

Bug description:

The full name of the failed test case is:

test.test_asyncio.test_sendfile.ProactorEventLoopTests.test_sendfile_close_peer_in_the_middle_of_receiving

Test output:

======================================================================
FAIL: test_sendfile_close_peer_in_the_middle_of_receiving (test.test_asyncio.test_sendfile.ProactorEventLoopTests.test_sendfile_close_peer_in_the_middle_of_receiving)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cpython\Lib\test\test_asyncio\test_sendfile.py", line 473, in test_sendfile_close_peer_in_the_middle_of_receiving
    self.assertTrue(1024 <= self.file.tell() < len(self.DATA),
AssertionError: False is not true : 0

----------------------------------------------------------------------

Based on my comprehension, I re-wrote and simplified the test case, as shown below.

import asyncio
import logging
import socket


class MySendFileProto(asyncio.Protocol):

    def __init__(self, file_size=0, loop=None):
        self.transport = None
        self.nbytes = 0
        self.data = bytearray()
        self.file_size = file_size
        if loop:
            self.done = loop.create_future()

    def connection_made(self, transport):
        self.transport = transport

    def data_received(self, data):
        self.transport.close()
        self.nbytes += len(data)
        self.data.extend(data)
        print('RECV: {}, TOTAL: {}'.format(len(data), self.nbytes))
        super().data_received(data)

    def connection_lost(self, exc):
        if not self.done.done():
            self.done.set_result(self.nbytes)


def main():
    logging.basicConfig(level='DEBUG')

    port = 8075
    size = 1024000
    file_name = 'tmp.txt'

    loop = asyncio.ProactorEventLoop()
    srv_proto = MySendFileProto(file_size=size, loop=loop)

    srv_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    srv_sock.bind(('localhost', port))
    server_creation = loop.create_server(lambda: srv_proto, sock=srv_sock, ssl=None)
    server = loop.run_until_complete(server_creation)

    cli_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    cli_sock.connect(('localhost', port))
    cli_proto = MySendFileProto(loop=loop)
    client_creation = loop.create_connection(lambda: cli_proto, sock=cli_sock, ssl=None, server_hostname=None)
    loop.run_until_complete(client_creation)

    with open(file_name, 'wb') as file:
        file.write(b'x' * size)

    with open(file_name, 'rb') as file:
        sendfile_future = loop.sendfile(cli_proto.transport, file)
        try:
            loop.run_until_complete(sendfile_future)
        except ConnectionError as e:
            print('ConnectionError happy: {}'.format(e))
        loop.run_until_complete(srv_proto.done)
        print(file.tell())

    srv_proto.transport.close()
    cli_proto.transport.close()
    loop.run_until_complete(srv_proto.done)
    loop.run_until_complete(cli_proto.done)
    server.close()


if __name__ == '__main__':
    main()

Output:

  1. NTFS + loop = asyncio.SelectorEventLoop()
DEBUG:asyncio:Using selector: SelectSelector
RECV: 16384, TOTAL: 16384
ConnectionError happy: Connection closed by peer
49152
  1. ReFS + loop = asyncio.SelectorEventLoop()
DEBUG:asyncio:Using selector: SelectSelector
RECV: 16384, TOTAL: 16384
ConnectionError happy: Connection closed by peer
49152
  1. NTFS + loop = asyncio.ProactorEventLoop()
DEBUG:asyncio:Using proactor: IocpProactor
RECV: 32768, TOTAL: 32768
RECV: 65536, TOTAL: 98304
ConnectionError happy: [WinError 64] 指定的网络名不再可用。
32768
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending name='Task-4' coro=<IocpProactor.accept.<locals>.accept_coro() running at C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.240.0_x64__qbz5n2kfra8p0\Lib\asyncio\windows_events.py:561> wait_for=<_OverlappedFuture cancelled>>
  1. ReFS + loop = asyncio.ProactorEventLoop()
DEBUG:asyncio:Using proactor: IocpProactor
RECV: 32768, TOTAL: 32768
RECV: 65536, TOTAL: 98304
ConnectionError happy: [WinError 64] 指定的网络名不再可用。
0
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending name='Task-4' coro=<IocpProactor.accept.<locals>.accept_coro() running at C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.240.0_x64__qbz5n2kfra8p0\Lib\asyncio\windows_events.py:561> wait_for=<_OverlappedFuture cancelled>>

Because file.tell() is 0, the test case fails. However, even though the test case passes on NTFS, it does not mean correctness.

I guess that the main purpose of test_sendfile.py:473 is to check the number of bytes sent by the client. When SelectorEventLoop is used on Windows, Python will firstly try sendfile syscall, which will fail, and then fallback to _sendfile_fallback, which uses POSIX-read function in a loop. After each read, the file offset is sure to increase.

When ProactorEventLoop is used on Windows, Python will call transmitFile function. However, this function does not tell us how it will modify the file offset. In fact, on NTFS, file.tell() is 32768, but at least 98304 bytes have been sent and received.

In conclusion, the statement on test_sendfile.py:473 does not work on Windows. Either another method needs to be found to check the number of bytes sent, or the line needs to be removed (test_sendfile.py:471 may be enough, I think).

Windows Version: 22631.2500 with Windows Feature Experience Pack 1000.22677.1000.0

CPython versions tested on:

3.12, CPython main branch

Operating systems tested on:

Windows

Linked PRs
@zcxsythenew zcxsythenew added the type-bug An unexpected behavior, bug, or error label Oct 26, 2023
@AlexWaygood AlexWaygood added tests Tests in the Lib/test dir topic-asyncio labels Oct 26, 2023
@github-project-automation github-project-automation bot moved this to Todo in asyncio Oct 26, 2023
@gvanrossum
Copy link
Member

Thanks for the bug report. Do you think you can submit a PR to fix it? It should at least be backported to 3.12 and probably also to 3.11, but start on main, please.

@zcxsythenew
Copy link
Contributor Author

Thanks for the bug report. Do you think you can submit a PR to fix it? It should at least be backported to 3.12 and probably also to 3.11, but start on main, please.

Yes, if I submit a PR, I would prefer to remove the line. I will do it later.

zcxsythenew added a commit to zcxsythenew/cpython that referenced this issue Oct 27, 2023
@zcxsythenew
pythongh-111347: Remove wrong assertion in test_sendfile
22a1e87
@zcxsythenew zcxsythenew mentioned this issue Oct 27, 2023
Merged
zcxsythenew added a commit to zcxsythenew/cpython that referenced this issue Oct 27, 2023
@zcxsythenew
pythongh-111347: Remove wrong assertion in test_sendfile
272ab33
@github-project-automation github-project-automation bot moved this from Todo to Done in asyncio Oct 29, 2023
gvanrossum pushed a commit that referenced this issue Oct 29, 2023
@zcxsythenew
gh-111347: Remove wrong assertion in test_sendfile (#111377)
fa35b9e 
Windows is different.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Oct 29, 2023
@zcxsythenew @miss-islington
pythongh-111347: Remove wrong assertion in test_sendfile (pythonGH-11…
c241158 
…1377)

Windows is different.
(cherry picked from commit fa35b9e)

Co-authored-by: zcxsythenew <30565051+zcxsythenew@users.noreply.github.com>
@bedevere-app bedevere-app bot mentioned this issue Oct 29, 2023
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Oct 29, 2023
@zcxsythenew @miss-islington
pythongh-111347: Remove wrong assertion in test_sendfile (pythonGH-11…
30b4207 
…1377)

Windows is different.
(cherry picked from commit fa35b9e)

Co-authored-by: zcxsythenew <30565051+zcxsythenew@users.noreply.github.com>
@bedevere-app bedevere-app bot mentioned this issue Oct 29, 2023
Merged
willingc pushed a commit that referenced this issue Oct 29, 2023
@miss-islington @zcxsythenew
[3.12] gh-111347: Remove wrong assertion in test_sendfile (GH-111377) (…
0681b4c 
…#111461)

gh-111347: Remove wrong assertion in test_sendfile (GH-111377)

Windows is different.
(cherry picked from commit fa35b9e)

Co-authored-by: zcxsythenew <30565051+zcxsythenew@users.noreply.github.com>
willingc pushed a commit that referenced this issue Oct 31, 2023
@miss-islington @zcxsythenew
[3.11] gh-111347: Remove wrong assertion in test_sendfile (GH-111377) (…
a9d5966 
…#111462)

gh-111347: Remove wrong assertion in test_sendfile (GH-111377)

Windows is different.
(cherry picked from commit fa35b9e)

Co-authored-by: zcxsythenew <30565051+zcxsythenew@users.noreply.github.com>
FullteaR pushed a commit to FullteaR/cpython that referenced this issue Nov 3, 2023
@zcxsythenew @FullteaR
pythongh-111347: Remove wrong assertion in test_sendfile (python#111377)
b369a0b 
Windows is different.
aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024
@zcxsythenew @aisk
pythongh-111347: Remove wrong assertion in test_sendfile (python#111377)
20a8924 
Windows is different.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@zcxsythenew @Glyphack
pythongh-111347: Remove wrong assertion in test_sendfile (python#111377)
5EA2
03222b3 
Windows is different.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
tests Tests in the Lib/test dir topic-asyncio type-bug An unexpected behavior, bug, or error
Projects
asyncio
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

gh-111347: Remove wrong assertion in test_sendfile zcxsythenew/cpython
3 participants
@gvanrossum @zcxsythenew @AlexWaygood
0