BigQuery Storage API sample for reading pandas dataframe #1994
Conversation
0eaca4e to
a3a48c0
Compare
| # [START bigquerystorage_pandas_read_query_results] | ||
| import uuid | ||
|
|
||
| # Due to a known issue in the BigQuery Storage API (TODO: link to |
There was a problem hiding this comment.
Should we consider simply running a large query that emits enough data to avoid the inline? Pros: better demonstrates the perf of the new API, and avoids us having to revisit the sample. Cons: test time overhead and potential pitfalls for people kicking tires with small results. Part of this is dependent on how the team will be maintaining their KI list.
There was a problem hiding this comment.
Cons: test time overhead.
Long test time is my main reason for avoiding queries that return big results. I guess it's not so bad since this repo can test the different directories independently.
Cons: potential pitfalls for people kicking tires with small results.
The current failure case is rather bad: it returns a successful response, but gives you an empty result set. That's a pretty big pitfall, because catching it requires noticing that you didn't get the data you thought you were going to get.
There was a problem hiding this comment.
Works for me. Let's confirm how the KIs will be maintained so you can link them.
How to get a pandas DataFrame, fast! The first two examples use the existing BigQuery client. These examples create a thread pool and read in parallel. The final example shows using just the new BigQuery Storage client, but only shows how to read with a single thread.
Blacken sample.
* Move imports inside region tags. * Adjust query indentation to match region tags.
Move duplicate imports out of region tags. Add region tag for the whole sample.
to just above the sample where it is used. This makes the complete source code for the sample make more sense (bigquerystorage_pandas_tutorial_all)
f064277 to
925fe3b
Compare
…ogleCloudPlatform/python-docs-samples#1994) * BigQuery Storage API sample for reading pandas dataframe How to get a pandas DataFrame, fast! The first two examples use the existing BigQuery client. These examples create a thread pool and read in parallel. The final example shows using just the new BigQuery Storage client, but only shows how to read with a single thread.
…ogleCloudPlatform/python-docs-samples#1994) * BigQuery Storage API sample for reading pandas dataframe How to get a pandas DataFrame, fast! The first two examples use the existing BigQuery client. These examples create a thread pool and read in parallel. The final example shows using just the new BigQuery Storage client, but only shows how to read with a single thread.
How to get a pandas DataFrame, fast!
The first two examples use the existing BigQuery client. These examples
create a thread pool and read in parallel. The final example shows using
just the new BigQuery Storage client, but only shows how to read with a
single thread.