8000 Map and apply with multiprocessing · Issue #1173 · docarray/docarray · GitHub
[go: up one dir, main page]

Skip to content 8000

Map and apply with multiprocessing #1173

@samsja

Description

@samsja

Context

we want to

da.apply(lambda x : x, num_worker=4 )

This will be different from doing

for doc in da:
    doc = f(doc)

map(f,a)

Is that we will leverage multiprocessing. (Need to be extra careful and add good test to check that nothing is going wrong)

  • benchmarking

We need to choose between Map or Apply but we need only of them.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    0