8000 NF: Add Environment class, with initial Native/Docker implementations by effigies · Pull Request #516 · nipype/pydra · GitHub
[go: up one dir, main page]

Skip to content

NF: Add Environment class, with initial Native/Docker implementations #516

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 39 commits into from
Jan 29, 2024
Merged
Changes from 1 commit
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
9406a9c
NF: Add Environment class, with initial Native/Docker implementations
effigies Mar 22, 2022
9f11b85
TEST: Add basic test for native environment
effigies Mar 31, 2022
58b4a34
FIX: import and arg
effigies Mar 31, 2022
39874d4
removing resetting self.output_ to None (not sure why it was needed)
djarecka Jun 15, 2022
b6b8130
implementing execute for the Docker env; changes to the ShellCommandT…
djarecka Jan 19, 2023
5922c47
updating docker tests; fixing issue with working directory for docker
djarecka Jan 30, 2023
dcac97e
adding need docker to the tests
djarecka Feb 27, 2023
4f21181
FIX: Bad rebase
effigies Aug 29, 2023
42a0588
RF: Rewrite _check_input with FileSets
effigies Aug 29, 2023
3dbc054
FIX: Missing argument
effigies Aug 29, 2023
81abb09
FIX: Get mode
effigies Aug 29, 2023
f495366
TEST: Update tests to be typing-friendly
effigies Aug 29, 2023
377692a
Merge branch 'rf/environments' of https://github.com/nipype/pydra int…
djarecka Sep 16, 2023
2eb88da
Update pydra/engine/task.py
djarecka Sep 17, 2023
ff3f5d0
8000 Update pydra/engine/task.py
djarecka Sep 17, 2023
6889577
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 17, 2023
25f14a2
Merge branch 'rf/environments' of https://github.com/nipype/pydra int…
djarecka Sep 17, 2023
e4644a5
removing temporary old env, some renaming
djarecka Sep 17, 2023
fa240d9
Merge pull request #705 from djarecka/env
djarecka Sep 17, 2023
aaa399f
adding environment to run_el for slurm worker
djarecka Sep 18, 2023
7d39b4d
cleaning DockerTask remains
djarecka Sep 19, 2023
4d2098b
Merge pull request #706 from djarecka/env
djarecka Sep 19, 2023
bd76c3d
adding the Singularity environment class
djarecka Oct 5, 2023
9cc0904
Merge pull request #711 from djarecka/env
djarecka Oct 5, 2023
979bb60
removing psi plugin from one test
djarecka Oct 28, 2023
2dd5603
creating Container class
djarecka Nov 2, 2023
161635b
cleaning: removing ContainerTask and ContainerSpec
djarecka Nov 3, 2023
8d60dd1
adding xarg to env command, changing output_cpath to root used in the…
djarecka Nov 3, 2023
58038f5
Merge pull request #718 from djarecka/env
djarecka Nov 3, 2023
9673908
Update pydra/engine/environments.py
djarecka Nov 15, 2023
89de9e6
Update pydra/engine/environments.py
djarecka Nov 15, 2023
bfaa681
Update pydra/engine/environments.py
djarecka Nov 15, 2023
575785e
Update pydra/engine/tests/test_specs.py
djarecka Nov 15, 2023
3d54253
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 15, 2023
73e6856
Update pydra/engine/environments.py
djarecka Nov 15, 2023
d1793cb
small fix
djarecka Nov 15, 2023
89c1e27
Merge pull request #721 from djarecka/env
djarecka Nov 15, 2023
be5a870
small edits to the core
djarecka Nov 15, 2023
0b0c71b
Merge pull request #722 from djarecka/env
djarecka Nov 15, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
RF: Rewrite _check_input with FileSets
  • Loading branch information
effigies committed Sep 7, 2023
commit 42a0588b7866492f3d51413eff689eacf94d639b
56 changes: 20 additions & 36 deletions pydra/engine/task.py
Original file line number Diff line number Diff line change
Expand Up 10000 @@ -599,42 +599,26 @@ def _run_task(self, environment=None):
self.output_ = environment.execute(self)

def _check_inputs(self, root):
fields = attr_fields(self.inputs)
for fld in fields:
if (
fld.type in [File, Directory]
or "pydra.engine.specs.File" in str(fld.type)
or "pydra.engine.specs.Directory" in str(fld.type)
):
if fld.name == "image":
continue
file = Path(getattr(self.inputs, fld.name))
if fld.metadata.get("container_path"): # TODO: this should go..
# if the path is in a container the input should be treated as a str (hash as a str)
# field.type = "str"
# setattr(self, field.name, str(file))
pass
# if this is a local path, checking if the path exists
# TODO: if copyfile, ro -> rw
# TODO: what if it's a directory? add tests
elif file.exists(): # is it ok if two inputs have the same parent?
# todo: probably need only keys
if fld.metadata.get("mandatory"):
mod = "rw"
else:
mod = "ro"
self.bindings[Path(file.parent)] = (
Path(f"{root}{file.parent}"),
mod,
)
self.inputs_mod_root[fld.name] = f"{root}{Path(file).absolute()}"
# error should be raised only if the type is strictly File or Directory
elif fld.type in [File, Directory]:
raise FileNotFoundError(
f"the file {file} from {fld.name} input does not exist, "
f"if the file comes from the container, "
f"use field.metadata['container_path']=True"
)
for fld in attr_fields(self.inputs):
if TypeParser.contains_type(FileSet, fld.type):
# Is container_path necessary? Container paths should just be typed PurePath
assert not fld.metadata.get("container_path")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is presumably failing in test_docker_inputspec_3. I think we need a clearer use-case and figure out the correct behavior. In any case, we've been putting up with a failing test for a while. I'm inclined to say let's re-add this feature when someone actually needs it.

# Should no longer happen with environments; assertion for testing purposes
# XXX: Remove before merge, so "image" can become a valid input file
assert not fld.name == "image"
fileset = getattr(self.inputs, fld.name)
copy = parse_copyfile(fld)[0] == FileSet.CopyMode.copy

host_path, env_path = fileset.parent, Path(f"{root}{fileset.parent}")

# Default to mounting paths as read-only, but respect existing modes
old_mode = self.bindings.get(host_path, ("", "ro"))
self.bindings[host_path] = (env_path, "rw" if copy else old_mode)

# Provide in-container paths without type-checking
self.inputs_mod_root[fld.name] = tuple(
env_path / rel for rel in fileset.relative_fspaths
)

DEFAULT_COPY_COLLATION = FileSet.CopyCollation.adjacent

Expand Down
0