gh-55454: Add IMAP4 IDLE support to imaplib #122542

foresto · 2024-08-01T03:13:12Z

This extends imaplib with support for the rfc2177 IMAP IDLE command, as requested in #55454. It allows events to be pushed to a client as they occur, rather than having to continually poll for mailbox changes.

The interface is a new idle() method, which returns an iterable context manager. Entering the context starts IDLE mode, during which events (untagged responses) can be retrieved using the iteration protocol. Exiting the context sends DONE to the server, ending IDLE mode.

An optional time limit for the IDLE session is supported, for use with servers that impose an inactivity timeout.

The context manager also offers a burst() method, designed for programs wishing to process events in batch rather than one at a time.

Notable differences from other implementations:

It's an extension to imaplib, rather than a replacement.
It doesn't introduce additional threads.
It doesn't impose new requirements on the use of imaplib's existing methods.
It passes the unit tests in CPython's test/test_imaplib.py module (and adds new ones).
It works on Windows, Linux, and other unix-like systems.
It makes IDLE available on all of imaplib's client variants (including IMAP4_stream).
The interface is pythonic and easy to use.

Caveats:

Due to a Windows limitation, the special case of IMAP4_stream running on Windows lacks a duration/timeout feature. (This is the stdin/stdout pipe connection variant; timeouts work fine for socket-based connections, even on Windows.) I have documented it where appropriate.
The file-like imaplib instance attributes are changed from buffered to unbuffered mode. This could potentially break any client code that uses those objects directly without expecting partial reads/writes. However, they are undocumented. As such, I think (and PEP 8 confirms) that they are fair game for changes. https://peps.python.org/pep-0008/#public-and-internal-interfaces

Usage examples:

#55454 (comment)

Recent discussion:

https://discuss.python.org/t/gauging-interest-in-my-imap4-idle-implementation-for-imaplib/59272

Earlier requests and suggestions:

#55454

https://mail.python.org/archives/list/python-ideas@python.org/thread/C4TVEYL5IBESQQPPS5GBR7WFBXCLQMZ2/

Issue: Add support for IMAP IDLE in imaplib #55454

📚 Documentation preview 📚: https://cpython-previews--122542.org.readthedocs.build/

ghost · 2024-08-01T03:13:16Z

All commit authors signed the Contributor License Agreement.

This extends imaplib with support for the rfc2177 IMAP IDLE command, as requested in python#55454. It allows events to be pushed to a client as they occur, rather than having to continually poll for mailbox changes. The interface is a new idle() method, which returns an iterable context manager. Entering the context starts IDLE mode, during which events (untagged responses) can be retrieved using the iteration protocol. Exiting the context sends DONE to the server, ending IDLE mode. An optional time limit for the IDLE session is supported, for use with servers that impose an inactivity timeout. The context manager also offers a burst() method, designed for programs wishing to process events in batch rather than one at a time. Notable differences from other implementations: - It's an extension to imaplib, rather than a replacement. - It doesn't introduce additional threads. - It doesn't impose new requirements on the use of imaplib's existing methods. - It passes the unit tests in CPython's test/test_imaplib.py module (and adds new ones). - It works on Windows, Linux, and other unix-like systems. - It makes IDLE available on all of imaplib's client variants (including IMAP4_stream). - The interface is pythonic and easy to use. Caveats: - Due to a Windows limitation, the special case of IMAP4_stream running on Windows lacks a duration/timeout feature. (This is the stdin/stdout pipe connection variant; timeouts work fine for socket-based connections, even on Windows.) I have documented it where appropriate. - The file-like imaplib instance attributes are changed from buffered to unbuffered mode. This could potentially break any client code that uses those objects directly without expecting partial reads/writes. However, these attributes are undocumented. As such, I think (and PEP 8 confirms) that they are fair game for changes. https://peps.python.org/pep-0008/#public-and-internal-interfaces Usage examples: python#55454 (comment) Original discussion: https://discuss.python.org/t/gauging-interest-in-my-imap4-idle-implementation-for-imaplib/59272 Earlier requests and suggestions: python#55454 https://mail.python.org/archives/list/python-ideas@python.org/thread/C4TVEYL5IBESQQPPS5GBR7WFBXCLQMZ2/

foresto · 2024-09-01T16:27:30Z

Can someone find time for a code review?

mcepl

LGTM, but I haven’t dived deep enough to the async stuff.

Doc/library/imaplib.rst

- Add example idle response tuples, to make the minor difference from other imaplib response tuples more obvious. - Merge the idle context manager's burst() method docs with the IMAP object's idle() method docs, for easier understanding. - Upgrade the Windows note regarding lack of pipe timeouts to a warning. - Rephrase various things for clarity.

foresto · 2024-09-18T18:06:31Z

@mcepl Did you see my last comment? Have you had a chance to finish reviewing?

mcepl

I think it is very appropriate.

ZeroIntensity

Thank you for contributing, @foresto! With great PRs, comes great requests for changes 😉

Doc/library/imaplib.rst

Lib/test/test_imaplib.py

Lib/imaplib.py

Doc/library/imaplib.rst

Lib/test/test_imaplib.py

Lib/imaplib.py

Co-authored-by: Peter Bierma <zintensitydev@gmail.com>

foresto · 2024-12-17T07:11:25Z

The "Could not symlink lib/libtcl8.6.dylib" CI failure returned. Can someone poke the build bot into trying again?

Lib/test/test_imaplib.py

Doc/library/imaplib.rst

picnixz · 2024-12-17T09:25:29Z

I've rerun the buildbot but the failure is still there. It might be related to runners so we'll just wait for now. You can just ignore the failure for now.

This makes it easier for client code to distinguish a temporary rejection of the IDLE command from a server responding incorrectly to IDLE.

Lib/imaplib.py

Doc/library/imaplib.rst

Lib/imaplib.py

If an IDLE duration or burst() was in use, and an unsolicited response contained a literal string, and crossed a packet boundary, and the subsequent packet was delayed beyond the IDLE feature's time limit, the timeout would leave the incoming protocol stream in a bad state (with the tail of that response appearing where the start of a response is expected). This change moves the IDLE socket timeout to cover only the start of a response, so it can no longer cause that problem.

This ensures that short IDLE durations / burst() intervals won't risk corrupting response lines that span multiple packets.

For resilience if read() or readline() ever complete with more than one bytes object remaining in the buffer. This is not expected to happen, but it seems wise to be prepared for a future change making it possible.

foresto · 2025-02-04T20:49:48Z

@gvanrossum I am now comfortable with this being merged, so here's the ping that we we discussed earlier.

I believe all issues have been addressed, including the last one from vadmium, who appears to be busy now.

You wanted to make sure Serhiy had a chance to weigh in. He posted his thoughts in #55454, and I addressed them there. I don't believe they should hold back this PR, since some are already accounted for, and the rest are beyond the scope of this work (and would not be impeded by it).

I see some tests failed again in the latest auto-build, but they look to me like intermittent CI failures, unrelated to the PR.

The next alpha release is coming up. Can this be merged before then? Is there something more I should do to make it happen?

gvanrossum · 2025-02-04T20:55:54Z

I will make time today for a last scan and then hopefully merge it. Thanks for all your work and patience!

foresto · 2025-02-04T21:00:39Z

Ack... It looks like a conflicting commit was added to the main branch a few days ago. I'll have a look & see if I can get it resolved today.

Our read() implementation designed to support IDLE replaces the one from PR python#119514, fixing the same problem it was addressing. The tests that it added are preserved.

foresto · 2025-02-05T00:56:46Z

@gvanrossum Okay, I've merged the main branch and resolved the conflicting commit.

That commit was an IMAP4.read() replacement that reads in chunks instead of passing server-supplied sizes directly to a single BufferedReader.read() call. My version (needed for IDLE) already did that, so I kept my version along with that commit's new test cases.

The latest CI failure appears once again to have nothing to do with this PR.

Ready to go, I think.

ZeroIntensity · 2025-02-05T11:55:41Z

The latest CI failure appears once again to have nothing to do with this PR.

I've restarted the jobs for you.

gvanrossum

I'm merging this now. Thanks for adding this!

foresto · 2025-02-07T03:36:19Z

Thanks @gvanrossum and everyone else for helping to get this done!

* pythongh-55454: Add IMAP4 IDLE support to imaplib This extends imaplib with support for the rfc2177 IMAP IDLE command, as requested in python#55454. It allows events to be pushed to a client as they occur, rather than having to continually poll for mailbox changes. The interface is a new idle() method, which returns an iterable context manager. Entering the context starts IDLE mode, during which events (untagged responses) can be retrieved using the iteration protocol. Exiting the context sends DONE to the server, ending IDLE mode. An optional time limit for the IDLE session is supported, for use with servers that impose an inactivity timeout. The context manager also offers a burst() method, designed for programs wishing to process events in batch rather than one at a time. Notable differences from other implementations: - It's an extension to imaplib, rather than a replacement. - It doesn't introduce additional threads. - It doesn't impose new requirements on the use of imaplib's existing methods. - It passes the unit tests in CPython's test/test_imaplib.py module (and adds new ones). - It works on Windows, Linux, and other unix-like systems. - It makes IDLE available on all of imaplib's client variants (including IMAP4_stream). - The interface is pythonic and easy to use. Caveats: - Due to a Windows limitation, the special case of IMAP4_stream running on Windows lacks a duration/timeout feature. (This is the stdin/stdout pipe connection variant; timeouts work fine for socket-based connections, even on Windows.) I have documented it where appropriate. - The file-like imaplib instance attributes are changed from buffered to unbuffered mode. This could potentially break any client code that uses those objects directly without expecting partial reads/writes. However, these attributes are undocumented. As such, I think (and PEP 8 confirms) that they are fair game for changes. https://peps.python.org/pep-0008/#public-and-internal-interfaces Usage examples: python#55454 (comment) Original discussion: https://discuss.python.org/t/gauging-interest-in-my-imap4-idle-implementation-for-imaplib/59272 Earlier requests and suggestions: python#55454 https://mail.python.org/archives/list/python-ideas@python.org/thread/C4TVEYL5IBESQQPPS5GBR7WFBXCLQMZ2/ * pythongh-55454: Clarify imaplib idle() docs - Add example idle response tuples, to make the minor difference from other imaplib response tuples more obvious. - Merge the idle context manager's burst() method docs with the IMAP object's idle() method docs, for easier understanding. - Upgrade the Windows note regarding lack of pipe timeouts to a warning. - Rephrase various things for clarity. * docs: words instead of <= Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * docs: improve style in an example Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * docs: grammatical edit Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * docs consistency Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * comment -> docstring Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * docs: refer to imaplib as "this module" Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * imaplib: simplify & clarify idle debug message Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * imaplib: elaborate in idle context manager comment * imaplib: re-raise BaseException instead of bare except Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * imaplib: convert private doc string to comment * docs: correct mistake in imaplib example This is a correction to 8077f2e, which changed a variable name in only one place and broke the subsequent reference to it, departed from the naming convention used in the rest of the module, and shadowed the type() builtin along the way. * imaplib: simplify example code in doc string This is for consistency with the documentation change in 8077f2e and subsequent correction in 013bbf1. * imaplib: rename _Idler to Idler, update its docs * imaplib: add comment in Idler._pop() Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * imaplib: remove unnecessary blank line Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * imaplib: comment on use of unbuffered pipes * docs: imaplib: use the reStructuredText :class: role Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * Revert "docs: imaplib: use the reStructuredText :class: role" This reverts commit f385e44, because it triggers CI failures in the docs by referencing a class that is (deliberately) undocumented. * docs: imaplib: use the reST :class: role, escaped This is a different approach to f385e44, which was reverted for creating dangling link references. By prefixing the reStructuredText role target with a ! we disable conversion to a link, thereby passing continuous integration checks even though the referenced class is deliberately absent from the documentation. * docs: refer to IMAP4 IDLE instead of just IDLE This clarifies that we are referring to the email protocol, not the editor with the same name. Co-authored-by: Guido van Rossum <gvanrossum@gmail.com> * imaplib: IDLE -> IMAP4 IDLE in exception message Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * docs: imaplib idle() phrasing and linking tweaks * docs: imaplib: avoid linking to an invalid target This reverts and rephrases part of a3f21cd which created links to a method on a deliberately undocumented class. The links didn't work consistently, and caused sphinx warnings that broke cpython's continuous integration tests. * imaplib: update test after recent exception change This fixes a test that was broken by changing an exception in b01de95 * imaplib: rename idle() dur argument to duration * imaplib: bytes.index() -> bytes.find() This makes it more obvious which statement triggers the branch. * imaplib: remove no-longer-necessary statement Co-authored-by: Martin Panter <vadmium@users.noreply.github.com> * docs: imaplib: concise & valid method links The burst() method is a little tricky to link in restructuredText, due to quirks of its parent class. This syntax allows sphinx to generate working links without generating warnings (which break continuous integration) and without burdening the reader with unimportant namespace qualifications. It makes the reST source ugly, but few people read the reST source, so it's a tolerable tradeoff. * imaplib: note data types present in IDLE responses * docs: imaplib: add comma to reST changes header Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> * imaplib: sync doc strings with reST docs * docs: imaplib: minor Idler clarifications * imaplib: idle: emit (type, [data, ...]) tuples This allows our iterator to emit untagged responses that contain literal strings in the same way that imaplib's existing methods do, while still emitting exactly one whole response per iteration. * imaplib: while/yield instead of yield from iter() * imaplib: idle: use deadline idiom when iterating This simplifies the code, and avoids idle duration drift from time spent processing each iteration. * docs: imaplib: state duration/interval arg types * docs: imaplib: minor rephrasing of a sentence * docs: imaplib: reposition a paragraph This might improve readability, especially when encountering Idler.burst() for the first time. * docs: imaplib: wrap long lines in idle() section * docs: imaplib: note: Idler objects require 'with' * docs: imaplib: say that 29 minutes is 1740 seconds * docs: imaplib: mark a paragraph as a 'tip' * docs: imaplib: rephrase reference to MS Windows * imaplib: end doc string titles with a period * imaplib: idle: socket timeouts instead of select() IDLE timeouts were originally implemented using select() after checking for the presence of already-buffered data. That allowed timeouts on pipe connetions like IMAP4_stream. However, it seemed possible that SSL data arriving without any IMAP data afterward could cause select() to indicate available application data when there was none, leading to a read() call that would block with no timeout. It was unclear under what conditions this would happen in practice. This change switches to socket timeouts instead of select(), just to be safe. This also reverts IMAP4_stream changes that were made to support IDLE timeouts, since our new implementation only supports socket connections. * imaplib: Idler: rename private state attributes * imaplib: rephrase a comment in example code * docs: imaplib: idle: use Sphinx code-block:: pycon * docs: whatsnew: imaplib: reformat IMAP4.idle entry * imaplib: idle: make doc strings brief Since we generally rely on the reST/html documentation for details, we can keep these doc strings short. This matches the module's existing doc string style and avoids having to sync small changes between two files. * imaplib: Idler: split assert into two statements * imaplib: Idler: move assignment out of try: block * imaplib: Idler: move __exit__() for readability * imaplib: Idler: move __next__() for readability * imaplib: test: make IdleCmdHandler a global class * docs: imaplib: idle: collapse double-spaces * imaplib: warn on use of undocumented 'file' attr * imaplib: revert import reformatting Since we no longer import platform or selectors, the original import statement style can be restored, reducing the footprint of PR python#122542. * imaplib: restore original exception msg formatting This reduces the footprint of PR python#122542. * docs: imaplib: idle: versionadded:: next * imaplib: move import statement to where it's used This import is only needed if external code tries to use an attribute that it shouldn't be using. Making it a local import reduces module loading time in supported cases. * imaplib test: RuntimeWarning on IMAP4.file access * imaplib: use stacklevel=2 in warnings.warn() * imaplib test: simplify IMAP4.file warning test * imaplib test: pre-idle-continuation response * imaplib test: post-done untagged response * imaplib: downgrade idle-denied exception to error This makes it easier for client code to distinguish a temporary rejection of the IDLE command from a server responding incorrectly to IDLE. * imaplib: simplify check for socket object * imaplib: narrow the scope of IDLE socket timeouts If an IDLE duration or burst() was in use, and an unsolicited response contained a literal string, and crossed a packet boundary, and the subsequent packet was delayed beyond the IDLE feature's time limit, the timeout would leave the incoming protocol stream in a bad state (with the tail of that response appearing where the start of a response is expected). This change moves the IDLE socket timeout to cover only the start of a response, so it can no longer cause that problem. * imaplib: preserve partial reads on exception This ensures that short IDLE durations / burst() intervals won't risk corrupting response lines that span multiple packets. * imaplib: read/readline: save multipart buffer tail For resilience if read() or readline() ever complete with more than one bytes object remaining in the buffer. This is not expected to happen, but it seems wise to be prepared for a future change making it possible. * imaplib: use TimeoutError subclass only if needed * doc: imaplib: elaborate on IDLE response delivery * doc: imaplib: elaborate in note re: IMAP4.response * imaplib: comment on benefit of reading in chunks Our read() implementation designed to support IDLE replaces the one from PR python#119514, fixing the same problem it was addressing. The tests that it added are preserved. * imaplib: readline(): treat ConnectionError as EOF --------- Co-authored-by: Gregory P. Smith <greg@krypto.org> Co-authored-by: Peter Bierma <zintensitydev@gmail.com> Co-authored-by: Guido van Rossum <guido@python.org> Co-authored-by: Guido van Rossum <gvanrossum@gmail.com> Co-authored-by: Martin Panter <vadmium@users.noreply.github.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

* pythongh-55454: Add IMAP4 IDLE support to imaplib This extends imaplib with support for the rfc2177 IMAP IDLE command, as requested in python#55454. It allows events to be pushed to a client as they occur, rather than having to continually poll for mailbox changes. The interface is a new idle() method, which returns an iterable context manager. Entering the context starts IDLE mode, during which events (untagged responses) can be retrieved using the iteration protocol. Exiting the context sends DONE to the server, ending IDLE mode. An optional time limit for the IDLE session is supported, for use with servers that impose an inactivity timeout. The context manager also offers a burst() method, designed for programs wishing to process events in batch rather than one at a time. Notable differences from other implementations: - It's an extension to imaplib, rather than a replacement. - It doesn't introduce additional threads. - It doesn't impose new requirements on the use of imaplib's existing methods. - It passes the unit tests in CPython's test/test_imaplib.py module (and adds new ones). - It works on Windows, Linux, and other unix-like systems. - It makes IDLE available on all of imaplib's client variants (including IMAP4_stream). - The interface is pythonic and easy to use. Caveats: - Due to a Windows limitation, the special case of IMAP4_stream running on Windows lacks a duration/timeout feature. (This is the stdin/stdout pipe connection variant; timeouts work fine for socket-based connections, even on Windows.) I have documented it where appropriate. - The file-like imaplib instance attributes are changed from buffered to unbuffered mode. This could potentially break any client code that uses those objects directly without expecting partial reads/writes. However, these attributes are undocumented. As such, I think (and PEP 8 confirms) that they are fair game for changes. https://peps.python.org/pep-0008/#public-and-internal-interfaces Usage examples: python#55454 (comment) Original discussion: https://discuss.python.org/t/gauging-interest-in-my-imap4-idle-implementation-for-imaplib/59272 Earlier requests and suggestions: python#55454 https://mail.python.org/archives/list/python-ideas@python.org/thread/C4TVEYL5IBESQQPPS5GBR7WFBXCLQMZ2/ * pythongh-55454: Clarify imaplib idle() docs - Add example idle response tuples, to make the minor difference from other imaplib response tuples more obvious. - Merge the idle context manager's burst() method docs with the IMAP object's idle() method docs, for easier understanding. - Upgrade the Windows note regarding lack of pipe timeouts to a warning. - Rephrase various things for clarity. * docs: words instead of <= Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * docs: improve style in an example Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * docs: grammatical edit Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * docs consistency Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * comment -> docstring Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * docs: refer to imaplib as "this module" Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * imaplib: simplify & clarify idle debug message Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * imaplib: elaborate in idle context manager comment * imaplib: re-raise BaseException instead of bare except Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * imaplib: convert private doc string to comment * docs: correct mistake in imaplib example This is a correction to 8077f2e, which changed a variable name in only one place and broke the subsequent reference to it, departed from the naming convention used in the rest of the module, and shadowed the type() builtin along the way. * imaplib: simplify example code in doc string This is for consistency with the documentation change in 8077f2e and subsequent correction in 013bbf1. * imaplib: rename _Idler to Idler, update its docs * imaplib: add comment in Idler._pop() Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * imaplib: remove unnecessary blank line Co-authored-by: Pete 1241 r Bierma <zintensitydev@gmail.com> * imaplib: comment on use of unbuffered pipes * docs: imaplib: use the reStructuredText :class: role Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * Revert "docs: imaplib: use the reStructuredText :class: role" This reverts commit f385e44, because it triggers CI failures in the docs by referencing a class that is (deliberately) undocumented. * docs: imaplib: use the reST :class: role, escaped This is a different approach to f385e44, which was reverted for creating dangling link references. By prefixing the reStructuredText role target with a ! we disable conversion to a link, thereby passing continuous integration checks even though the referenced class is deliberately absent from the documentation. * docs: refer to IMAP4 IDLE instead of just IDLE This clarifies that we are referring to the email protocol, not the editor with the same name. Co-authored-by: Guido van Rossum <gvanrossum@gmail.com> * imaplib: IDLE -> IMAP4 IDLE in exception message Co-authored-by: Peter Bierma <zintensitydev@gmail.com> * docs: imaplib idle() phrasing and linking tweaks * docs: imaplib: avoid linking to an invalid target This reverts and rephrases part of a3f21cd which created links to a method on a deliberately undocumented class. The links didn't work consistently, and caused sphinx warnings that broke cpython's continuous integration tests. * imaplib: update test after recent exception change This fixes a test that was broken by changing an exception in b01de95 * imaplib: rename idle() dur argument to duration * imaplib: bytes.index() -> bytes.find() This makes it more obvious which statement triggers the branch. * imaplib: remove no-longer-necessary statement Co-authored-by: Martin Panter <vadmium@users.noreply.github.com> * docs: imaplib: concise & valid method links The burst() method is a little tricky to link in restructuredText, due to quirks of its parent class. This syntax allows sphinx to generate working links without generating warnings (which break continuous integration) and without burdening the reader with unimportant namespace qualifications. It makes the reST source ugly, but few people read the reST source, so it's a tolerable tradeoff. * imaplib: note data types present in IDLE responses * docs: imaplib: add comma to reST changes header Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> * imaplib: sync doc strings with reST docs * docs: imaplib: minor Idler clarifications * imaplib: idle: emit (type, [data, ...]) tuples This allows our iterator to emit untagged responses that contain literal strings in the same way that imaplib's existing methods do, while still emitting exactly one whole response per iteration. * imaplib: while/yield instead of yield from iter() * imaplib: idle: use deadline idiom when iterating This simplifies the code, and avoids idle duration drift from time spent processing each iteration. * docs: imaplib: state duration/interval arg types * docs: imaplib: minor rephrasing of a sentence * docs: imaplib: reposition a paragraph This might improve readability, especially when encountering Idler.burst() for the first time. * docs: imaplib: wrap long lines in idle() section * docs: imaplib: note: Idler objects require 'with' * docs: imaplib: say that 29 minutes is 1740 seconds * docs: imaplib: mark a paragraph as a 'tip' * docs: imaplib: rephrase reference to MS Windows * imaplib: end doc string titles with a period * imaplib: idle: socket timeou A49D ts instead of select() IDLE timeouts were originally implemented using select() after checking for the presence of already-buffered data. That allowed timeouts on pipe connetions like IMAP4_stream. However, it seemed possible that SSL data arriving without any IMAP data afterward could cause select() to indicate available application data when there was none, leading to a read() call that would block with no timeout. It was unclear under what conditions this would happen in practice. This change switches to socket timeouts instead of select(), just to be safe. This also reverts IMAP4_stream changes that were made to support IDLE timeouts, since our new implementation only supports socket connections. * imaplib: Idler: rename private state attributes * imaplib: rephrase a comment in example code * docs: imaplib: idle: use Sphinx code-block:: pycon * docs: whatsnew: imaplib: reformat IMAP4.idle entry * imaplib: idle: make doc strings brief Since we generally rely on the reST/html documentation for details, we can keep these doc strings short. This matches the module's existing doc string style and avoids having to sync small changes between two files. * imaplib: Idler: split assert into two statements * imaplib: Idler: move assignment out of try: block * imaplib: Idler: move __exit__() for readability * imaplib: Idler: move __next__() for readability * imaplib: test: make IdleCmdHandler a global class * docs: imaplib: idle: collapse double-spaces * imaplib: warn on use of undocumented 'file' attr * imaplib: revert import reformatting Since we no longer import platform or selectors, the original import statement style can be restored, reducing the footprint of PR python#122542. * imaplib: restore original exception msg formatting This reduces the footprint of PR python#122542. * docs: imaplib: idle: versionadded:: next * imaplib: move import statement to where it's used This import is only needed if external code tries to use an attribute that it shouldn't be using. Making it a local import reduces module loading time in supported cases. * imaplib test: RuntimeWarning on IMAP4.file access * imaplib: use stacklevel=2 in warnings.warn() * imaplib test: simplify IMAP4.file warning test * imaplib test: pre-idle-continuation response * imaplib test: post-done untagged response * imaplib: downgrade idle-denied exception to error This makes it easier for client code to distinguish a temporary rejection of the IDLE command from a server responding incorrectly to IDLE. * imaplib: simplify check for socket object * imaplib: narrow the scope of IDLE socket timeouts If an IDLE duration or burst() was in use, and an unsolicited response contained a literal string, and crossed a packet boundary, and the subsequent packet was delayed beyond the IDLE feature's time limit, the timeout would leave the incoming protocol stream in a bad state (with the tail of that response appearing where the start of a response is expected). This change moves the IDLE socket timeout to cover only the start of a response, so it can no longer cause that problem. * imaplib: preserve partial reads on exception This ensures that short IDLE durations / burst() intervals won't risk corrupting response lines that span multiple packets. * imaplib: read/readline: save multipart buffer tail For resilience if read() or readline() ever complete with more than one bytes object remaining in the buffer. This is not expected to happen, but it seems wise to be prepared for a future change making it possible. * imaplib: use TimeoutError subclass only if needed * doc: imaplib: elaborate on IDLE response delivery * doc: imaplib: elaborate in note re: IMAP4.response * imaplib: comment on benefit of reading in chunks Our read() implementation designed to support IDLE replaces the one from PR python#119514, fixing the same problem it was addressing. The tests that it added are preserved. * imaplib: readline(): treat ConnectionError as EOF --------- Co-authored-by: Gregory P. Smith <greg@krypto.org> Co-authored-by: Peter Bierma <zintensitydev@gmail.com> Co-authored-by: Guido van Rossum <guido@python.org> Co-authored-by: Guido van Rossum <gvanrossum@gmail.com> Co-authored-by: Martin Panter <vadmium@users.noreply.github.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

foresto requested a review from a team as a code owner August 1, 2024 03:13

bedevere-app bot mentioned this pull request Aug 1, 2024

Add support for IMAP IDLE in imaplib #55454

Closed

bedevere-app bot added the awaiting review label Aug 1, 2024

foresto force-pushed the imaplib-idle branch 7 times, most recently from 44b8c6a to c7c78e4 Compare August 2, 2024 21:59

foresto force-pushed the imaplib-idle branch from c7c78e4 to 4f25877 Compare August 16, 2024 00:57

foresto force-pushed the imaplib-idle branch from 4f25877 to 572d1e5 Compare September 1, 2024 16:02

mcepl reviewed Sep 10, 2024

View reviewed changes

Doc/library/imaplib.rst Outdated Show resolved

foresto force-pushed the imaplib-idle branch from 9dd2562 to dc71241 Compare September 11, 2024 22:32

mcepl approved these changes Sep 19, 2024

View reviewed changes

bedevere-app bot added awaiting core review and removed awaiting review labels Sep 19, 2024

ZeroIntensity requested changes Sep 21, 2024

View reviewed changes

gpshead and others added 5 commits September 21, 2024 10:39

docs: words instead of <=

f41f2bb

Co-authored-by: Peter Bierma <zintensitydev@gmail.com>

docs: improve style in an example

8077f2e

Co-authored-by: Peter Bierma <zintensitydev@gmail.com>

docs: grammatical edit

24fcdbb

Co-authored-by: Peter Bierma <zintensitydev@gmail.com>

docs consistency

2c76b2f

Co-authored-by: Peter Bierma <zintensitydev@gmail.com>

comment -> docstring

5ef5bb2

Co-authored-by: Peter Bierma <zintensitydev@gmail.com>

gpshead self-requested a review September 21, 2024 17:50

docs: refer to imaplib as "this module"

91266e0

Co-authored-by: Peter Bierma <zintensitydev@gmail.com>

imaplib: use stacklevel=2 in warnings.warn()

66d32a0

picnixz reviewed Dec 17, 2024

View reviewed changes

Lib/test/test_imaplib.py Outdated Show resolved Hide resolved

Lib/test/test_imaplib.py Outdated Show resolved Hide resolved

Doc/library/imaplib.rst Show resolved Hide resolved

foresto added 4 commits December 17, 2024 01:44

imaplib test: simplify IMAP4.file warning test

d619580

imaplib test: pre-idle-continuation response

fc8b6f4

imaplib test: post-done untagged response

b767ab6

imaplib: downgrade idle-denied exception to error

dcd0161

This makes it easier for client code to distinguish a temporary rejection of the IDLE command from a server responding incorrectly to IDLE.

vadmium reviewed Dec 22, 2024

View reviewed changes

Lib/imaplib.py Outdated Show resolved Hide resolved

Doc/library/imaplib.rst Show resolved Hide resolved

Doc/library/imaplib.rst Outdated Show resolved Hide resolved

Lib/imaplib.py Show resolved Hide resolved

foresto added 7 commits January 8, 2025 17:47

imaplib: simplify check for socket object

53c7a19

imaplib: preserve partial reads on exception

a47bcb4

This ensures that short IDLE durations / burst() intervals won't risk corrupting response lines that span multiple packets.

imaplib: read/readline: save multipart buffer tail

7fc4b78

For resilience if read() or readline() ever complete with more than one bytes object remaining in the buffer. This is not expected to happen, but it seems wise to be prepared for a future change making it possible.

imaplib: use TimeoutError subclass only if needed

be34141

doc: imaplib: elaborate on IDLE response delivery

F438

e8a8509

doc: imaplib: elaborate in note re: IMAP4.response

8d78010

foresto added 3 commits February 4, 2025 14:11

Merge branch 'main' into imaplib-idle

e8f0532

imaplib: comment on benefit of reading in chunks

f650dfa

Our read() implementation designed to support IDLE replaces the one from PR python#119514, fixing the same problem it was addressing. The tests that it added are preserved.

imaplib: readline(): treat ConnectionError as EOF

3512858

gvanrossum approved these changes Feb 7, 2025

View reviewed changes

gvanrossum merged commit 0fef47e into python:main Feb 7, 2025
39 checks passed

bedevere-app bot removed the awaiting merge label Feb 7, 2025

Uh oh!

gh-55454: Add IMAP4 IDLE support to imaplib #122542

gh-55454: Add IMAP4 IDLE support to imaplib #122542

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!