8000 add google summer of code 2023 changes by adi611 · Pull Request #48 · nipype/pydra-tutorial · GitHub
[go: up one dir, main page]

Skip to content

add google summer of code 2023 changes #48

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 16, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 24 additions & 23 deletions jupyterbook/_toc.yml
Original file line number Diff line number Diff line change
@@ -1,23 +1,24 @@
# Table of contents
# Learn more at https://jupyterbook.org/customize/toc.html

format: jb-book
root: welcome
parts:
- caption: Basic Concepts
numbered: True
chapters:
- file: notebooks/1_intro_pydra
- file: notebooks/2_intro_functiontask
- file: notebooks/3_intro_functiontask_state
- file: notebooks/4_intro_workflow
- file: notebooks/5_intro_shelltask
- caption: General Linear Model
numbered: True
chapters:
- file: notebooks/6_firstlevel_glm_nilearn
- file: notebooks/7_twolevel_glm_nilearn
- caption: About Pydra
chapters:
- file: about/team
- file: about/cite_pydra
# Table of contents
# Learn more at https://jupyterbook.org/customize/toc.html

format: jb-book
root: welcome
parts:
- caption: Basic Concepts
numbered: True
chapters:
- file: notebooks/1_intro_pydra
- file: notebooks/2_intro_functiontask
- file: notebooks/3_intro_functiontask_state
- file: notebooks/4_intro_workflow
- file: notebooks/5_intro_shelltask
- file: notebooks/8_intro_workers
- caption: General Linear Model
numbered: True
chapters:
- file: notebooks/6_firstlevel_glm_nilearn
- file: notebooks/7_twolevel_glm_nilearn
- caption: About Pydra
chapters:
- file: about/team
- file: about/cite_pydra
3 changes: 2 additions & 1 deletion notebooks/1_intro_pydra.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,8 @@ There are two main types of objects in *pydra*: `Task` and `Workflow`, that is a
Pydra supports multiple workers to execute `Tasks` and `Workflows`:
- `ConcurrentFutures`
- `SLURM`
- `Dask` (experimental)
- `Dask`
- `PSI/J`

+++

Expand Down
74 changes: 74 additions & 0 deletions notebooks/8_intro_workers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
---
jupytext:
text_representation:
extension: .md
format_name: myst
format_version: 0.13
jupytext_version: 1.14.0
kernelspec:
display_name: Python 3
language: python
name: python3
---

# Workers in Pydra

Pydra workers are classes that are used to execute `Tasks` and `Workflows`. Pydra currently supports the following workers:
- `SerialWorker`
- `ConcurrentFuturesWorker`
- `SlurmWorker`
- `DaskWorker`
- `SGEWorker`
- `PsijWorker`

## 1. SerialWorker

A worker to execute linearly.
```
with pydra.Submitter(plugin='serial') as sub:
sub(wf)
```

## 2. ConcurrentFuturesWorker

A worker to execute in parallel using Python's concurrent futures.
```
with pydra.Submitter(plugin='cf') as sub:
sub(wf)
```

## 3. SlurmWorker

A worker to execute tasks on SLURM systems.
```
with pydra.Submitter(plugin='slurm') as sub:
sub(wf)
```

## 4. DaskWorker

A worker to execute in parallel using Dask.distributed.
```
with pydra.Submitter(plugin='dask') as sub:
sub(wf)
```

## 5. SGEWorker

A worker to execute tasks on SLURM systems.
```
with pydra.Submitter(plugin='sge') as sub:
sub(wf)
```

## 6. PsijWorker

A worker to execute tasks using PSI/J executors. Currently supported executors are: `local` and `slurm`.
```
with pydra.Submitter(plugin='psij-local') as sub:
sub(wf)
```
```
with pydra.Submitter(plugin='psij-slurm') as sub:
sub(wf)
```
0