DataFusion Python Examples

Some examples rely on data which can be downloaded from the following site:

https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page

Here is a direct link to the file used in the examples:

https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2021-01.parquet

Creating a SessionContext

Creating a SessionContext

Executing Queries with DataFusion

Running User-Defined Python Code

Substrait Support

Serialize query plans using Substrait

Executing SQL against DataFrame Libraries (Experimental)

TPC-H Examples

Within the subdirectory tpch there are 22 examples that reproduce queries in the TPC-H specification. These include realistic data that can be generated at arbitrary scale and allow the user to see use cases for a variety of data frame operations.

In the list below we describe which new operations can be found in the examples. The queries are designed to be of increasing complexity, so it is recommended to review them in order. For brevity, the following list does not include operations found in previous examples.