-
Notifications
You must be signed in to change notification settings - Fork 60
NF: Add Environment class, with initial Native/Docker implementations #516
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
39 commits
Select commit
Hold shift + click to select a range
9406a9c
NF: Add Environment class, with initial Native/Docker implementations
effigies 9f11b85
TEST: Add basic test for native environment
effigies 58b4a34
FIX: import and arg
effigies 39874d4
removing resetting self.output_ to None (not sure why it was needed)
djarecka b6b8130
implementing execute for the Docker env; changes to the ShellCommandT…
djarecka 5922c47
updating docker tests; fixing issue with working directory for docker
djarecka dcac97e
adding need docker to the tests
djarecka 4f21181
FIX: Bad rebase
effigies 42a0588
RF: Rewrite _check_input with FileSets
effigies 3dbc054
FIX: Missing argument
effigies 81abb09
FIX: Get mode
effigies f495366
TEST: Update tests to be typing-friendly
effigies 377692a
Merge branch 'rf/environments' of https://github.com/nipype/pydra int…
djarecka 2eb88da
Update pydra/engine/task.py
djarecka ff3f5d0
8000
Update pydra/engine/task.py
djarecka 6889577
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 25f14a2
Merge branch 'rf/environments' of https://github.com/nipype/pydra int…
djarecka e4644a5
removing temporary old env, some renaming
djarecka fa240d9
Merge pull request #705 from djarecka/env
djarecka aaa399f
adding environment to run_el for slurm worker
djarecka 7d39b4d
cleaning DockerTask remains
djarecka 4d2098b
Merge pull request #706 from djarecka/env
djarecka bd76c3d
adding the Singularity environment class
djarecka 9cc0904
Merge pull request #711 from djarecka/env
djarecka 979bb60
removing psi plugin from one test
djarecka 2dd5603
creating Container class
djarecka 161635b
cleaning: removing ContainerTask and ContainerSpec
djarecka 8d60dd1
adding xarg to env command, changing output_cpath to root used in the…
djarecka 58038f5
Merge pull request #718 from djarecka/env
djarecka 9673908
Update pydra/engine/environments.py
djarecka 89de9e6
Update pydra/engine/environments.py
djarecka bfaa681
Update pydra/engine/environments.py
djarecka 575785e
Update pydra/engine/tests/test_specs.py
djarecka 3d54253
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 73e6856
Update pydra/engine/environments.py
djarecka d1793cb
small fix
djarecka 89c1e27
Merge pull request #721 from djarecka/env
djarecka be5a870
small edits to the core
djarecka 0b0c71b
Merge pull request #722 from djarecka/env
djarecka File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
RF: Rewrite _check_input with FileSets
- Loading branch information
commit 42a0588b7866492f3d51413eff689eacf94d639b
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -599,42 +599,26 @@ def _run_task(self, environment=None): | |
self.output_ = environment.execute(self) | ||
|
||
def _check_inputs(self, root): | ||
djarecka marked this conversation as resolved.
Show resolved
Hide resolved
|
||
fields = attr_fields(self.inputs) | ||
for fld in fields: | ||
if ( | ||
fld.type in [File, Directory] | ||
or "pydra.engine.specs.File" in str(fld.type) | ||
or "pydra.engine.specs.Directory" in str(fld.type) | ||
): | ||
if fld.name == "image": | ||
continue | ||
file = Path(getattr(self.inputs, fld.name)) | ||
if fld.metadata.get("container_path"): # TODO: this should go.. | ||
# if the path is in a container the input should be treated as a str (hash as a str) | ||
# field.type = "str" | ||
# setattr(self, field.name, str(file)) | ||
pass | ||
# if this is a local path, checking if the path exists | ||
# TODO: if copyfile, ro -> rw | ||
# TODO: what if it's a directory? add tests | ||
elif file.exists(): # is it ok if two inputs have the same parent? | ||
# todo: probably need only keys | ||
if fld.metadata.get("mandatory"): | ||
mod = "rw" | ||
else: | ||
mod = "ro" | ||
self.bindings[Path(file.parent)] = ( | ||
Path(f"{root}{file.parent}"), | ||
mod, | ||
) | ||
self.inputs_mod_root[fld.name] = f"{root}{Path(file).absolute()}" | ||
# error should be raised only if the type is strictly File or Directory | ||
elif fld.type in [File, Directory]: | ||
raise FileNotFoundError( | ||
f"the file {file} from {fld.name} input does not exist, " | ||
f"if the file comes from the container, " | ||
f"use field.metadata['container_path']=True" | ||
) | ||
for fld in attr_fields(self.inputs): | ||
if TypeParser.contains_type(FileSet, fld.type): | ||
# Is container_path necessary? Container paths should just be typed PurePath | ||
assert not fld.metadata.get("container_path") | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is presumably failing in |
||
# Should no longer happen with environments; assertion for testing purposes | ||
# XXX: Remove before merge, so "image" can become a valid input file | ||
assert not fld.name == "image" | ||
djarecka marked this conversation as resolved.
Show resolved
Hide resolved
|
||
fileset = getattr(self.inputs, fld.name) | ||
copy = parse_copyfile(fld)[0] == FileSet.CopyMode.copy | ||
|
||
host_path, env_path = fileset.parent, Path(f"{root}{fileset.parent}") | ||
|
||
# Default to mounting paths as read-only, but respect existing modes | ||
old_mode = self.bindings.get(host_path, ("", "ro")) | ||
self.bindings[host_path] = (env_path, "rw" if copy else old_mode) | ||
|
||
# Provide in-container paths without type-checking | ||
self.inputs_mod_root[fld.name] = tuple( | ||
env_path / rel for rel in fileset.relative_fspaths | ||
) | ||
|
||
DEFAULT_COPY_COLLATION = FileSet.CopyCollation.adjacent | ||
|
||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.