8000 Partial fix for 1078: Enhance DataFrame Formatter Configuration with Memory and Display Controls by kosiew · Pull Request #1119 · apache/datafusion-python · GitHub
[go: up one dir, main page]
Skip to content

Partial fix for 1078: Enhance DataFrame Formatter Configuration with Memory and Display Controls #1119

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 42 commits into from
May 5, 2025
Merged

Partial fix for 1078: Enhance DataFrame Formatter Configuration with Memory and Display Controls #1119

merged 42 commits into from
May 5, 2025

Conversation

kosiew
Copy link
Contributor
@kosiew kosiew commented Apr 28, 2025

Which issue does this PR close?

partial fix for #1078

Rationale for this change

This change improves the flexibility and performance of DataFrame rendering in notebooks and other environments.
It introduces fine-grained control over memory usage, row display counts, and HTML output optimization, making large data exploration more efficient and user-friendly.
It also cleans up validation logic for formatter settings and supports custom styling providers more robustly.

What changes are included in this PR?

  • Added max_memory_bytes, min_rows_display, and repr_rows parameters to the DataFrame HTML formatter.
  • Updated Python configure_formatter API and documentation to expose new parameters.
  • Improved internal validation for formatter parameters (_validate_positive_int, _validate_bool).
  • Introduced FormatterConfig in Rust to carry display configuration across DataFrame rendering.
  • Updated Rust collect_record_batches_to_display to respect new memory and row limits dynamically.
  • New tests to cover memory limits, row controls, and style provider usage.
  • Documentation updates explaining memory and performance optimizations, including use_shared_styles.

Are these changes tested?

✅ Yes, additional tests have been added:

  • Validation of new parameters in test_html_formatter_memory_and_rows.
  • Verification of custom style provider behavior combined with formatter parameters.
  • Edge case testing for extreme values (e.g., very high/low limits).

Are there any user-facing changes?

✅ Yes:

  • Users can now configure how much memory and how many rows are used when displaying DataFrames.
  • Improved error messages for invalid formatter configurations.
  • Better performance when rendering large numbers of DataFrames in Jupyter notebooks or other rich environments.
  • Documentation updated to reflect the new options available.

kosiew and others added 30 commits April 27, 2025 18:58
@kosiew
feat: add configurable max table bytes and min table rows for DataFra…
f9b78fa 
…me display
@kosiew
Revert "feat: add configurable max table bytes and min table rows for…
4d8fa38 
… DataFrame display"

This reverts commit f9b78fa.
@kosiew
feat: add FormatterConfig for configurable DataFrame display options
a9178fe
@kosiew
refactor: simplify attribute extraction in get_formatter_config function
d0209cf
@kosiew
refactor: remove hardcoded constants and use FormatterConfig for disp…
2ef013f 
…lay options
@kosiew
refactor: simplify record batch collection by using FormatterConfig f…
bea52a3 
…or display options
@kosiew
feat: add max_memory_bytes, min_rows_display, and repr_rows parameter…
ce15f1d 
…s to DataFrameHtmlFormatter
@kosiew
feat: add tests for HTML formatter row display settings and memory limit
e089d7b
@kosiew
refactor: extract Python formatter retrieval into a separate function
a6792c9
@kosiew
Revert "feat: add tests for HTML formatter row display settings and m…
af678b5 
…emory limit"

This reverts commit e089d7b.
@kosiew
feat: add tests for HTML formatter row and memory limit configurations
4090fd2
@kosiew
Revert "feat: add tests for HTML formatter row and memory limit confi…
174374a 
…gurations"

This reverts commit 4090fd2.
@kosiew
feat: add tests for new parameters and validation in DataFrameHtmlFor…
c10d7eb 
…matter
@kosiew
Reorganize tests
620d25e
@kosiew
refactor: rename and restructure formatter functions for clarity and …
82ddd28 
…maintainability
@kosiew
feat: implement PythonFormatter struct and refactor formatter retriev…
302f785 
…al for improved clarity
@kosiew
refactor: improve comments and restructure FormatterConfig usage in P…
2936455 
…yDataFrame
@kosiew @timsaucer
Add DataFrame usage guide with HTML rendering customization options (a…
877226a 
…pache#1108)

* docs: enhance user guide with detailed DataFrame operations and examples

* move /docs/source/api/dataframe.rst into user-guide

* docs: remove  DataFrame API documentation

* docs: fix formatting inconsistencies in DataFrame user guide

* Two minor corrections to documentation rendering

---------

Co-authored-by: Tim Saucer <timsaucer@gmail.com>
@kosiew
Update documentation
ccf2549
@kosiew
refactor: streamline HTML rendering documentation
194ed46
@kosiew
refactor: extract validation logic into separate functions for clarity
8667128
@kosiew
Implement feature X to enhance user experience and optimize performance
15435bf
@kosiew
feat: add validation method for FormatterConfig to ensure positive in…
412ab3a 
…teger values
@kosiew
add comment - ensure minimum rows are collected even if memory or row…
9a1f59f 
… limits are hit
@kosiew
Merge branch 'main' into formatter-options
5e3dd06
@kosiew
Update html_formatter documentation
bfb2b12
@kosiew
update tests
1fdc2c2
@kosiew
remove unused type hints from imports in html_formatter.py
4da231b
@kosiew
remove redundant tests for DataFrameHtmlFormatter and clean up assert…
43a3183 
…ions
@kosiew
refactor get_attr function to support generic default values
b52166b
@kosiew kosiew mentioned this pull request Apr 28, 2025
Closed
@kosiew kosiew force-pushed the formatter-options branch from 6588c8c to d5a3f1f Compare April 28, 2025 08:37
@kosiew kosiew force-pushed the formatter-options branch from 347282c to a9668c7 Compare April 28, 2025 14:02
@kosiew kosiew force-pushed the formatter-options branch from 583bc9b to 714aa70 Compare April 28, 2025 14:11
@timsaucer
Copy link
Contributor

Incredible work. Thank you!

@timsaucer timsaucer merged commit 7d8bcd8 into apache:main May 5, 2025
17 checks passed
@timsaucer timsaucer mentioned this pull request May 5, 2025
Open
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@timsaucer timsaucer timsaucer approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@kosiew @timsaucer
0