From 832697fe620420877c975d9162d5f0e3722916c0 Mon Sep 17 00:00:00 2001 From: Aditya Agarwal <50960175+adi611@users.noreply.github.com> Date: Thu, 21 Sep 2023 13:32:24 +0530 Subject: [PATCH 1/2] add google summer of code 2023 changes --- notebooks/1_intro_pydra.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/notebooks/1_intro_pydra.md b/notebooks/1_intro_pydra.md index 5d2c8aa..fe65a2d 100644 --- a/notebooks/1_intro_pydra.md +++ b/notebooks/1_intro_pydra.md @@ -48,7 +48,8 @@ There are two main types of objects in *pydra*: `Task` and `Workflow`, that is a Pydra supports multiple workers to execute `Tasks` and `Workflows`: - `ConcurrentFutures` - `SLURM` -- `Dask` (experimental) +- `Dask` +- `PSI/J` +++ From c2035b06feaba4f1bb406f33676184f71e66d85f Mon Sep 17 00:00:00 2001 From: Aditya Agarwal <50960175+adi611@users.noreply.github.com> Date: Sat, 23 Sep 2023 00:41:15 +0530 Subject: [PATCH 2/2] add workers section --- jupyterbook/_toc.yml | 47 ++++++++++++----------- notebooks/8_intro_workers.md | 74 ++++++++++++++++++++++++++++++++++++ 2 files changed, 98 insertions(+), 23 deletions(-) create mode 100644 notebooks/8_intro_workers.md diff --git a/jupyterbook/_toc.yml b/jupyterbook/_toc.yml index 7323fba..7228142 100644 --- a/jupyterbook/_toc.yml +++ b/jupyterbook/_toc.yml @@ -1,23 +1,24 @@ -# Table of contents -# Learn more at https://jupyterbook.org/customize/toc.html - -format: jb-book -root: welcome -parts: - - caption: Basic Concepts - numbered: True - chapters: - - file: notebooks/1_intro_pydra - - file: notebooks/2_intro_functiontask - - file: notebooks/3_intro_functiontask_state - - file: notebooks/4_intro_workflow - - file: notebooks/5_intro_shelltask - - caption: General Linear Model - numbered: True - chapters: - - file: notebooks/6_firstlevel_glm_nilearn - - file: notebooks/7_twolevel_glm_nilearn - - caption: About Pydra - chapters: - - file: about/team - - file: about/cite_pydra +# Table of contents +# Learn more at https://jupyterbook.org/customize/toc.html + +format: jb-book +root: welcome +parts: + - caption: Basic Concepts + numbered: True + chapters: + - file: notebooks/1_intro_pydra + - file: notebooks/2_intro_functiontask + - file: notebooks/3_intro_functiontask_state + - file: notebooks/4_intro_workflow + - file: notebooks/5_intro_shelltask + - file: notebooks/8_intro_workers + - caption: General Linear Model + numbered: True + chapters: + - file: notebooks/6_firstlevel_glm_nilearn + - file: notebooks/7_twolevel_glm_nilearn + - caption: About Pydra + chapters: + - file: about/team + - file: about/cite_pydra diff --git a/notebooks/8_intro_workers.md b/notebooks/8_intro_workers.md new file mode 100644 index 0000000..f6eed28 --- /dev/null +++ b/notebooks/8_intro_workers.md @@ -0,0 +1,74 @@ +--- +jupytext: + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.14.0 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +# Workers in Pydra + +Pydra workers are classes that are used to execute `Tasks` and `Workflows`. Pydra currently supports the following workers: +- `SerialWorker` +- `ConcurrentFuturesWorker` +- `SlurmWorker` +- `DaskWorker` +- `SGEWorker` +- `PsijWorker` + +## 1. SerialWorker + +A worker to execute linearly. +``` +with pydra.Submitter(plugin='serial') as sub: + sub(wf) +``` + +## 2. ConcurrentFuturesWorker + +A worker to execute in parallel using Python's concurrent futures. +``` +with pydra.Submitter(plugin='cf') as sub: + sub(wf) +``` + +## 3. SlurmWorker + +A worker to execute tasks on SLURM systems. +``` +with pydra.Submitter(plugin='slurm') as sub: + sub(wf) +``` + +## 4. DaskWorker + +A worker to execute in parallel using Dask.distributed. +``` +with pydra.Submitter(plugin='dask') as sub: + sub(wf) +``` + +## 5. SGEWorker + +A worker to execute tasks on SLURM systems. +``` +with pydra.Submitter(plugin='sge') as sub: + sub(wf) +``` + +## 6. PsijWorker + +A worker to execute tasks using PSI/J executors. Currently supported executors are: `local` and `slurm`. +``` +with pydra.Submitter(plugin='psij-local') as sub: + sub(wf) +``` +``` +with pydra.Submitter(plugin='psij-slurm') as sub: + sub(wf) +``` \ No newline at end of file