8000 Add async iterator support to `RecordBatchStream` · Issue #974 · apache/datafusion-python · GitHub
[go: up one dir, main page]

Skip to content
Add async iterator support to RecordBatchStream #974
Closed
@kylebarron

Description

@kylebarron

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

RecordBatchStream represents an async stream of record batches, but currently it's only possible to iterate synchronously through each batch.

Describe the solution you'd like

Implement __anext__ as well as __next__ here, so it can be used as an asynchronous iterator from Python:

fn next(&mut self, py: Python) -> PyResult<Option<PyRecordBatch>> {
let result = self.stream.next();
match wait_for_future(py, result) {
None => Ok(None),
Some(Ok(b)) => Ok(Some(b.into())),
Some(Err(e)) => Err(e.into()),
}
}

Describe alternatives you've considered

Additional context

This is similar to what I already have here wrapping ObjectStore::get

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      0