8000 test: add a bigquery usage report to notebook test session by milkshakeiii · Pull Request #604 · googleapis/python-bigquery-dataframes · GitHub 10000
[go: up one dir, main page]
Skip to content

test: add a bigquery usage report to notebook test session #604

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
milkshakeiii merged 11 commits into from
Apr 17, 2024
Merged

test: add a bigquery usage report to notebook test session #604

milkshakeiii merged 11 commits into from
Apr 17, 2024

Conversation

@milkshakeiii
Copy link
Contributor
@milkshakeiii milkshakeiii commented Apr 10, 2024

This will enable before/after comparisons of wide notebook-affecting optimizations

@milkshakeiii milkshakeiii requested review from a team as code owners April 10, 2024 22:30
@milkshakeiii milkshakeiii requested a review from ashleyxuu April 10, 2024 22:30
@product-auto-label product-auto-label bot added the size: s Pull request size is small. label Apr 10, 2024
@milkshakeiii milkshakeiii removed the request for review from ashleyxuu April 10, 2024 22:31
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. label Apr 10, 2024
@milkshakeiii milkshakeiii added status: not ready for review and removed api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. size: s Pull request size is small. labels Apr 10, 2024
@product-auto-label product-auto-label bot added the size: m Pull request size is medium. label Apr 11, 2024
@milkshakeiii
Copy link
Contributor Author
milkshakeiii commented Apr 11, 2024 •
edited
Loading

---BIGQUERY USAGE REPORT---
test_model_selection.bytesprocessed - query count: 54, bytes processed sum: 1052284
test_preprocessing.bytesprocessed - query count: 560, bytes processed sum: 2607478
test_progress_bar.bytesprocessed - query count: 14, bytes processed sum: 347364
test_core.bytesprocessed - query count: 127, bytes processed sum: 631994
test_forecasting.bytesprocessed - query count: 83, bytes processed sum: 964700
test_datetimes.bytesprocessed - query count: 82, bytes processed sum: 9472
test_window.bytesprocessed - query count: 112, bytes processed sum: 19712
test_issue355_merge_after_filter.bytesprocessed - query count: 24, bytes processed sum: 13987368
test_series.bytesprocessed - query count: 1096, bytes processed sum: 139709
test_pandas.bytesprocessed - query count: 140, bytes processed sum: 34420
test_scalar.bytesprocessed - query count: 2, bytes processed sum: 128
test_encryption.bytesprocessed - query count: 86, bytes processed sum: 818380
test_index.bytesprocessed - query count: 84, bytes processed sum: 9136
test_llm.bytesprocessed - query count: 169, bytes processed sum: 166403
test_multiindex.bytesprocessed - query count: 260, bytes processed sum: 102168
test_imported.bytesprocessed - query count: 32, bytes processed sum: 66643824
test_pandas_options.bytesprocessed - query count: 48, bytes processed sum: 0
test_dataframe_io.bytesprocessed - query count: 108, bytes processed sum: 68720
test_decomposition.bytesprocessed - query count: 77, bytes processed sum: 268838
test_ipython.bytesprocessed - query count: 8, bytes processed sum: 5560
test_plotting.bytesprocessed - query count: 129, bytes processed sum: 2892850
test_location.bytesprocessed - query count: 184, bytes processed sum: 3588
test_metrics.bytesprocessed - query count: 208, bytes processed sum: 0
test_register.bytesprocessed - query count: 10, bytes processed sum: 0
test_groupby.bytesprocessed - query count: 72, bytes processed sum: 17824
test_metrics_pairwise.bytesprocessed - query count: 24, bytes processed sum: 1200
test_session.bytesprocessed - query count: 332, bytes processed sum: 1325089158
test_remote.bytesprocessed - query count: 12, bytes processed sum: 672
test_numpy.bytesprocessed - query count: 86, bytes processed sum: 133120
test_ensemble.bytesprocessed - query count: 168, bytes processed sum: 80538743044
test_remote_function.bytesprocessed - query count: 144, bytes processed sum: 25524
test_compose.bytesprocessed - query count: 24, bytes processed sum: 59908
test_pipeline.bytesprocessed - query count: 135, bytes processed sum: 13994057751
test_cluster.bytesprocessed - query count: 51, bytes processed sum: 126738
test_dataframe.bytesprocessed - query count: 1258, bytes processed sum: 63632916
test_strings.bytesprocessed - query count: 116, bytes processed sum: 31690
test_linear_model.bytesprocessed - query count: 84, bytes processed sum: 1494414
---total queries: 6203, total bytes: 96014188055---
nox > Session notebook-3.9 was successful.

This is not correct due to dry runs. Updating PR.

@milkshakeiii
Copy link
Contributor Author
milkshakeiii commented Apr 11, 2024

---BIGQUERY USAGE REPORT---
test_metrics_pairwise.py - query count: 24, bytes processed sum: 816
test_window.py - query count: 112, bytes processed sum: 19712
test_multiindex.py - query count: 260, bytes processed sum: 102168
test_series.py - query count: 1095, bytes processed sum: 138253
test_index.py - query count: 84, bytes processed sum: 9064
test_location.py - query count: 184, bytes processed sum: 0
test_core.py - query count: 127, bytes processed sum: 274872
test_preprocessing.py - query count: 560, bytes processed sum: 1444226
test_remote_function.py - query count: 144, bytes processed sum: 20180
test_metrics.py - query count: 208, bytes processed sum: 0
test_dataframe_io.py - query count: 108, bytes processed sum: 58074
test_register.py - query count: 10, bytes processed sum: 0
test_ensemble.py - query count: 176, bytes processed sum: 141910355971
test_datetimes.py - query count: 82, bytes processed sum: 9472
test_ipython.py - query count: 8, bytes processed sum: 3402
test_pandas.py - query count: 140, bytes processed sum: 34420
test_cluster.py - query count: 51, bytes processed sum: 64680
test_llm.py - query count: 164, bytes processed sum: 162720
test_session.py - query count: 332, bytes processed sum: 697506136
test_linear_model.py - query count: 84, bytes processed sum: 748382
test_progress_bar.py - query count: 14, bytes processed sum: 347364
test_numpy.py - query count: 86, bytes processed sum: 133120
test_decomposition.py - query count: 77, bytes processed sum: 134330
test_dataframe.py - query count: 1258, bytes processed sum: 63603613
test_forecasting.py - query count: 83, bytes processed sum: 395368
test_strings.py - query count: 115, bytes processed sum: 30890
test_scalar.py - query count: 2, bytes processed sum: 128
test_encryption.py - query count: 86, bytes processed sum: 690146
test_issue355_merge_after_filter.py - query count: 24, bytes processed sum: 13987368
test_groupby.py - query count: 72, bytes processed sum: 17824
test_plotting.py - query count: 134, bytes processed sum: 2896287
test_compose.py - query count: 24, bytes processed sum: 30182
test_model_selection.py - query count: 54, bytes processed sum: 1052284
test_pipeline.py - query count: 135, bytes processed sum: 58349854394
test_remote.py - query count: 12, bytes processed sum: 672
test_imported.py - query count: 32, bytes processed sum: 33322578
test_pandas_options.py - query count: 48, bytes processed sum: 0
---total queries: 6209, total bytes: 201077449096---
nox > Session notebook-3.9 was successful.

@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. label Apr 11, 2024
@milkshakeiii
Copy link
Contributor Author
milkshakeiii commented Apr 11, 2024

---BIGQUERY USAGE REPORT---
remote_function.ipynb - query count: 16, bytes processed sum: 86086638598
easy_linear_regression.ipynb - query count: 24, bytes processed sum: 160225
sklearn_linear_regression.ipynb - query count: 25, bytes processed sum: 139053
getting_started_bq_dataframes.ipynb - query count: 24, bytes processed sum: 204521
bq_dataframes_llm_kmeans.ipynb - query count: 29, bytes processed sum: 4133565995
large_language_models.ipynb - query count: 6, bytes processed sum: 5567
dataframe.ipynb - query count: 57, bytes processed sum: 4036591
integrations.ipynb - query count: 7, bytes processed sum: 270
bq_dataframes_covid_line_graphs.ipynb - query count: 2, bytes processed sum: 12948886977
regionalized.ipynb - query count: 270, bytes processed sum: 1933764
---total queries: 460, total bytes: 103175571561---

@milkshakeiii
Copy link
Contributor Author
milkshakeiii commented Apr 12, 2024

---BIGQUERY USAGE REPORT---
regionalized.ipynb - query count: 270, bytes processed sum: 1933764
bq_dataframes_llm_kmeans.ipynb - query count: 29, bytes processed sum: 4135762331
large_language_models.ipynb - query count: 6, bytes processed sum: 6381
integrations.ipynb - query count: 7, bytes processed sum: 270
dataframe.ipynb - query count: 57, bytes processed sum: 4036591
remote_function.ipynb - query count: 16, bytes processed sum: 86086638598
easy_linear_regression.ipynb - query count: 24, bytes processed sum: 160225
sklearn_linear_regression.ipynb - query count: 25, bytes processed sum: 139053
bq_dataframes_covid_line_graphs.ipynb - query count: 2, bytes processed sum: 12948886977
getting_started_bq_dataframes.ipynb - query count: 24, bytes processed sum: 204521
---total queries: 460, total bytes: 103177768711---

chelsea-lin
chelsea-lin requested changes Apr 15, 2024
View reviewed changes
chelsea-lin
chelsea-lin approved these changes Apr 15, 2024
View reviewed changes
Copy link
Contributor
@chelsea-lin chelsea-lin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed offline.

@milkshakeiii milkshakeiii merged commit 250548c into main Apr 17, 2024
@milkshakeiii milkshakeiii deleted the b329655946-cost-report branch April 17, 2024 00:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chelsea-lin chelsea-lin chelsea-lin approved these changes

@TrevorBergeron TrevorBergeron Awaiting requested review from TrevorBergeron

Assignees

No one assigned

Labels

api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. size: m Pull request size is medium. status: ready for review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@milkshakeiii @chelsea-lin
0