8000 Tracking Upgrade to Datafusion 37 · Issue #663 · apache/datafusion-python · GitHub
[go: up one dir, main page]
Skip to content
Tracking Upgrade to Datafusion 37 #663
New issue
New issue
Closed
#669
Closed
Tracking Upgrade to Datafusion 37#663
#669
@Michael-J-Ward

Description

@Michael-J-Ward
Michael-J-Ward
opened on May 1, 2024

I sketched out some of the upgrade in #662 and wanted to share what I encountered.

Some major changes in between datafusion 36 and 37.

Needs Completing

  • 1) Porting functions.rs BuiltInFunctons to datafusion-functions
  • 2) Replacing SectionContext::tables or updating the tests to not rely on it
  • 3) new trait methods for ExecutionPlan and ExecutionPlanProperties for DatasetExec need implementing

1) Porting BuiltinFunctions

See this epic apache/datafusion#9285.

That means much of datafusion-python::functions.rs needs an update. An incomplete first pass is 83249fe

2) SectionContext::tables

It was removed in apache/datafusion#9627

Many of the tests rely on this method. Does each test get updated to a new API or do we implement our own version to start and then update the tests until it's not necessary?

3) New Trait Methods

I just marked these as TODO to get the upgrade to compile. See 8cd4fd4

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions
      0