[pull] main from python:main #308

pull · 2025-05-10T09:01:20Z

See Commits and Changes for more details.

Created by pull[bot] (v2.0.0-alpha.1)

Can you help keep this open source service alive? 💖 Please sponsor : )

Summary by Sourcery

Improve ElementTree deepcopy handling and SQLite CLI with color support

New Features:

Introduced color theming for SQLite CLI output

Bug Fixes:

Fixed potential crashes during ElementTree deepcopy operations
Addressed edge cases in XML element deep copying

Enhancements:

Added color support to SQLite CLI
Improved error handling in SQLite interactive console

Tests:

Added test cases for ElementTree deepcopy edge cases
Added color-related tests for SQLite CLI

…tackbase(f)` (GH-133714)

…133010)

sourcery-ai · 2025-05-10T09:01:23Z

Reviewer's Guide

This pull request primarily enhances the robustness of xml.etree.ElementTree's deepcopy operation by modifying its C implementation to correctly handle mutations to an element's children list during the copy process, preventing potential crashes. It also introduces colorized output to the sqlite3 command-line interface for improved user experience, affecting prompts and error messages through a new theming mechanism. Additionally, it includes minor updates to CPython internal assertions for stack pointer validation and adds corresponding news entries for these changes.

Sequence Diagram: ElementTree `deepcopy` Child Mutation Handling

sequenceDiagram
    title Sequence Diagram: ElementTree `deepcopy` Child Mutation Handling
    participant Caller
    participant Self_deepcopy as "_elementtree_Element___deepcopy___impl(self, memo)"
    participant Helper_deepcopy as "deepcopy(st, obj_to_copy, memo) (C function)"
    participant Obj_To_Copy_Internal_Deepcopy as "obj_to_copy internal __deepcopy__"

    Caller->>Self_deepcopy: Call with self (ElementObject to copy)
    Self_deepcopy->>Self_deepcopy: Store initial self.extra.length as original_child_count
    loop For each child (child_obj) in self.extra.children (iterating up to original_child_count or current length if mutated)
        Self_deepcopy->>Helper_deepcopy: deepcopy(st, child_obj, memo)
        activate Helper_deepcopy
        note over Helper_deepcopy: Handles Py_INCREF(child_obj) before dispatching,<br/>and Py_DECREF(child_obj) after.
        alt child_obj is Element_CheckExact and not overridden
             Helper_deepcopy->>Self_deepcopy: _elementtree_Element___deepcopy___impl(child_obj, memo)
             Self_deepcopy-->>Helper_deepcopy: copied_child_element
        else obj_to_copy uses general st->deepcopy_obj or custom Python __deepcopy__
             Helper_deepcopy->>Obj_To_Copy_Internal_Deepcopy: Invoke internal/custom __deepcopy__(memo) for child_obj
             Obj_To_Copy_Internal_Deepcopy-->>Helper_deepcopy: returns copied_child_part
             note right of Obj_To_Copy_Internal_Deepcopy: This call might have side-effects,<br/>e.g., modifying `self.extra` (parent's children list).
        end
        Helper_deepcopy-->>Self_deepcopy: Returns copied_child
        deactivate Helper_deepcopy
        Self_deepcopy->>Self_deepcopy: Check if self.extra.length (parent's live children list) != original_child_count or expected_count for current element
        alt self.extra.length changed during child's deepcopy
            Self_deepcopy->>Self_deepcopy: Update expected_count = self.extra.length
            Self_deepcopy->>Self_deepcopy: element_resize(new_element_being_built, expected_count)
        end
        Self_deepcopy->>Self_deepcopy: Store copied_child in new_element_being_built.extra.children
    end
    Self_deepcopy->>Self_deepcopy: Set final length of new_element_being_built.extra.length to actual number of items copied.
    Self_deepcopy-->>Caller: Return new_element (copied self)

Sequence Diagram: `sqlite3` CLI Theming and Execution Flow

sequenceDiagram
    title Sequence Diagram: `sqlite3` CLI Theming and Execution Flow
    participant User
    participant main as "main()"
    participant _colorize as "_colorize module"
    participant Console as "SqliteInteractiveConsole"
    participant execute_func as "execute() function"
    participant sqlite3_lib as "sqlite3 library"

    main->>_colorize: get_theme()
    _colorize-->>main: theme_obj
    main->>main: Set sys.ps1, sys.ps2 using theme_obj.syntax
    main->>Console: __init__(connection, use_color=True)
    User->>Console: Types SQL query or CLI command (source)
    Console->>Console: runsource(source, ...)
    activate Console
    Console->>_colorize: get_theme(force_no_color=not self._use_color)
    _colorize-->>Console: current_theme
    alt Input is a CLI command
        alt Command is unknown
            Console->>User: Display colorized error (using current_theme.traceback)
        else Command is known
            Console->>Console: Process command (e.g., .help)
        end
    else Input is SQL
        alt sqlite3.complete_statement(source) is True
            Console->>execute_func: execute(self._cur, source, theme=current_theme)
            activate execute_func
            execute_func->>sqlite3_lib: c.execute(sql)
            alt sqlite3.Error occurs
                sqlite3_lib-->>execute_func: Raises sqlite3.Error
                execute_func->>User: Print colorized error (using current_theme.traceback)
            else Success
                sqlite3_lib-->>execute_func: Returns results
                execute_func->>User: Print results
            end
            deactivate execute_func
        else More input needed (incomplete statement)
            Console-->>User: Display colorized continuation prompt (sys.ps2)
        end
    end
    deactivate Console

Class Diagram: `sqlite3` CLI Enhancements

classDiagram
    title Class Diagram: `sqlite3` CLI Enhancements
    namespace sqlite3.__main__ {
        class SqliteInteractiveConsole {
            -_con: Connection
            -_cur: Cursor
            +_use_color: bool
            +__init__(connection, use_color=False)
            +runsource(source, filename, symbol) bool
        }

        class Theme {
            <<Data Structure>>
            +traceback: object
            +syntax: object
            +reset: str
            note "Represents color theme elements"
        }

        class _colorize {
            <<Module>>
            +get_theme(force_no_color=False) Theme
            +theme_no_color: Theme
        }

        class execute {
            <<Function>>
            +execute(c: Cursor, sql: str, suppress_errors=True, theme: Theme) void
        }
    }

    SqliteInteractiveConsole --|> InteractiveConsole
    SqliteInteractiveConsole ..> _colorize : uses get_theme()
    SqliteInteractiveConsole ..> execute : calls
    execute ..> Theme : uses for error formatting

    note for SqliteInteractiveConsole "Added `_use_color` attribute and `use_color` parameter to `__init__`.
`runsource` now uses `get_theme()` and passes the theme to `execute()` and for its own error messages."
    note for execute "Now accepts a `theme` parameter to colorize error output."

File-Level Changes

Change	Details	Files
Fixed crashes in `ElementTree` `deepcopy` by handling mutations to the children list during C-level iteration and managing refcounts more carefully.	Made C-level `__deepcopy__` iteration safe against child list mutations. Ensured correct ref-counting during C `deepcopy` with potential side effects. Added tests for `deepcopy` with dynamic child list changes (clearing/growing).	`Modules/_elementtree.c` `Lib/test/test_xml_etree.py`
Introduced colorized output to the `sqlite3` command-line interface for prompts and error messages.	Implemented color theming for SQLite3 REPL output using a new `_colorize` helper. Applied themes to REPL prompts and error reporting. Added tests for the new colorized output functionality. Updated existing CLI tests to account for colorization changes using a `force_not_colorized_test_class` decorator where appropriate	`Lib/sqlite3/__main__.py` `Lib/test/test_sqlite3/test_cli.py`
Updated assertions for stack pointer validation in CPython's interpreter frame.	Modified assertions in `_PyFrame_StackPeek` and `_PyFrame_StackPop` to use `_PyFrame_Stackbase(f)` for improved correctness when checking stack bounds.	`Include/internal/pycore_interpframe.h`
Added news files for the `ElementTree` and `sqlite3` CLI enhancements.	Added news entry for `ElementTree` deepcopy fix (UAF: `xml.etree.ElementTree.Element.__deepcopy__` when concurrent mutations happen python/cpython#133009). Added news entry for `sqlite3` CLI colorization (Colorize sqlite3 CLI python/cpython#133447).	`Misc/NEWS.d/next/Library/2025-04-26-15-50-12.gh-issue-133009.etBuz5.rst` `Misc/NEWS.d/next/Library/2025-05-05-18-50-00.gh-issue-133447.ajshdb.rst`

Tips and commands

Interacting with Sourcery

Trigger a new review: Comment @sourcery-ai review on the pull request.
Continue discussions: Reply directly to Sourcery's review comments.
Generate a GitHub issue from a review comment: Ask Sourcery to create an
issue from a review comment by replying to it. You can also reply to a
review comment with @sourcery-ai issue to create an issue from it.
Generate a pull request title: Write @sourcery-ai anywhere in the pull
request title to generate a title at any time. You can also comment
@sourcery-ai title on the pull request to (re-)generate the title at any time.
Generate a pull request summary: Write @sourcery-ai summary anywhere in
the pull request body to generate a PR summary at any time exactly where you
want it. You can also comment @sourcery-ai summary on the pull request to
(re-)generate the summary at any time.
Generate reviewer's guide: Comment @sourcery-ai guide on the pull
request to (re-)generate the reviewer's guide at any time.
Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
pull request to resolve all Sourcery comments. Useful if you've already
addressed all the comments and don't want to see them anymore.
Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
request to dismiss all existing Sourcery reviews. Especially useful if you
want to start fresh with a new review - don't forget to comment
@sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

Enable or disable review features such as the Sourcery-generated pull request
summary, the reviewer's guide, and others.
Change the review language.
Add, remove or edit custom review instructions.
Adjust other review settings.

Getting Help

Contact our support team for questions or feedback.
Visit our documentation for detailed guides and information.
Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

nybblista and others added 4 commits May 10, 2025 11:04

gh 8000 -133713: Compare the f->stackpointer to the result of `_PyFrame_S…

8320582

…tackbase(f)` (GH-133714)

gh-132971: Update shutil.which() docs (GH-133067)

d13d5fd

gh-133009: fix UAF in xml.etree.ElementTree.Element.__deepcopy__ (#…

116a9f9

…133010)

gh-133447: Add basic color to sqlite3 CLI (#133461)

30b1d8f

pull bot added the ⤵️ pull label May 10, 2025

pull bot merged commit 30b1d8f into webfutureiorepo:main May 10, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[pull] main from python:main #308

[pull] main from python:main #308

Uh oh!

Uh oh!

Interacting with Sourcery

Customizing Your Experience

Getting Help

Uh oh!

Uh oh!

[pull] main from python:main #308

[pull] main from python:main #308

Uh oh!

Conversation

Uh oh!

Summary by Sourcery

Uh oh!

Uh oh!

Reviewer's Guide

Sequence Diagram: ElementTree deepcopy Child Mutation Handling

Sequence Diagram: sqlite3 CLI Theming and Execution Flow

Class Diagram: sqlite3 CLI Enhancements

File-Level Changes

Interacting with Sourcery

Customizing Your Experience

Getting Help

Uh oh!

Uh oh!

Sequence Diagram: ElementTree `deepcopy` Child Mutation Handling

Sequence Diagram: `sqlite3` CLI Theming and Execution Flow

Class Diagram: `sqlite3` CLI Enhancements