Reuse the Tokio Runtime #341
Conversation
|
Thanks @kylebrooks-8451. Can you share any info on how much difference this makes to performance? I am wondering if we should delay the 23.0.0 release to get this merged in? |
|
I wanted to get some numbers on this but I didn't have a great way to benchmark it. I know it is significant for our use case which is running an Arrow Flight Server using Datafusion as as engine but I don't have any hard numbers. Is there an easy way to benchmark the Python bindings? I see a benchmark suite for Datafusion proper. |
|
I'm mostly testing with TPC-H using code here: https://github.com/sql-benchmarks/sqlbench-runners/tree/main/datafusion-python I doubt it will impact this benchmark all that much though. This PR is fixing an ugly hack, so I think we should go ahead and merge this. cc @jdye64 |
|
@andygrove - I ran that benchmark you linked on my MacBook Pro 6 Core i7 2.6 GHz. Using the TPCH Parquet Data with a Scale Factor of
|
Which issue does this PR close?
Closes #340
Rationale for this change
Currently, we create a new Tokio Runtime and associated threads often which is not good for performance. This PR uses a module level attribute to create this once and reuse it.
Are there any user-facing changes?
No.