8000 Type checking and coercion by tclose · Pull Request #662 · nipype/pydra · GitHub
[go: up one dir, main page]
Skip to content

Type checking and coercion #662

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 142 commits into from
Aug 4, 2023
Merged

Type checking and coercion #662

Changes from 1 commit
Commits
Show all changes
142 commits
Select commit Hold shift + click to select a range
2d7142a
ENH: Add generic object hasher
effigies May 2, 2023
1e36d1d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 2, 2023
f52bc66
RF: Alternate hashing and serialization
effigies May 2, 2023
da4a6c7
TEST: Hash dispatcher, dogfood register
effigies May 2, 2023
dffc10f
FIX: Use typing.Dict until Python 3.9 is minimum
effigies May 2, 2023
f211d0b
ENH: Support classes registering using a __bytes_repr__ method
effigies May 3, 2023
189dadf
FIX: Pull Iterator from typing for now
effigies May 4, 2023
4dda520
ENH: Hash frozensets
effigies May 5, 2023
29de5aa
ENH: Register byte serializer for None
effigies May 9, 2023
94eb216
RF: Implement hash_function() with hash_object()
effigies May 9, 2023
b4d6125
TEST: Update expected hashes
effigies May 9, 2023
7c25bf8
XXX: Temporarily mark test_wf_with_blocked_tasks xfail
effigies May 9, 2023
4076ff2
WIP: Start serializing pathlikes
effigies May 9, 2023
76f83d1
PY37: Remove walrus operator
effigies May 9, 2023
abe77f6
RF: Move file_hash to use dispatched hash
effigies May 9, 2023
ae6a15c
ENH: Flesh out more builtin types, make HasBytesRepr protocol
effigies May 10, 2023
1bb05c2
ENH: Add directory path hashing, test pathlikes
effigies May 10, 2023
a6fd5ca
TEST: Untested types, files with contents, ignore coverage in protocol
effigies May 10, 2023
e6bf4e9
RF: Always hash pathlike paths, not contents
effigies May 11, 2023
6344a98
RF: Make File a PathLike that serializes to file contents
effigies May 11, 2023
cb01648
ENH: Add MtimeCachingHash to cache hashes for PathLikes with mtime in…
effigies May 11, 2023
2ca41e3
implemented TypeCoercer callable class (to be passed to attrs convert…
tclose May 30, 2023
dc619ff
replaced type validation with TypeCoercer converter, which coerces an…
tclose May 31, 2023
4bb8b90
type checking unittests pass
tclose Jun 1, 2023
3ccb372
activated converter/type-checking on setattr
tclose Jun 1, 2023
c267b72
renamed gathered to array
tclose Jun 2, 2023
feabd1a
renamed array back to gathered
tclose Jun 4, 2023
bcf9aea
added bytes_repr for types & typing special forms
tclose Jun 4, 2023
0155960
changed initial values set to output class attrs in generated_output_…
tclose Jun 5, 2023
c146370
fixed setting of allowed values validator
tclose Jun 5, 2023
2964ad0
handle case of typing special forms in LazyOut _get_type
tclose Jun 5, 2023
c62d728
switched lzoutall unittests to use list version of add2_sub2_res
tclose Jun 5, 2023
a15a661
split up type-checking unittests
tclose Jun 5, 2023
b655a7c
switched not returned outputs back to None instead of attrs.NOTHING
tclose Jun 5, 2023
1523a5d
reworked MultiInputObj so that it can be coerced to (i.e. it doesn't …
tclose Jun 5, 2023
8d57077
replaced hash_value with hash_function
tclose Jun 6, 2023
007a862
implemented load_and_run functionality for SerialWorker
tclose Jun 7, 2023
f967234
removed file_hash reference in checksum_states
tclose Jun 7, 2023
42c8868
moved type_checking to typing in utils
tclose Jun 7, 2023
00e604c
added fileformats to pyproject deps
tclose Jun 8, 2023
291f144
added import from typing_extensions for Python < 3.10
tclose Jun 8, 2023
3740b98
removed Python 3.10 union syntax (i.e. '|')
tclose Jun 8, 2023
d163cd9
enclose runtime_checkable in import error catch
tclose Jun 8, 2023
b295f96
replaced list|tuple|dict generic types with ty.List, ty.Tuple etc
tclose Jun 8, 2023
e667a71
enclosed get_origin, get_args from typing in import error catch
tclose Jun 8, 2023
21f60e9
Added workarounds for Python 3.7 support
tclose Jun 8, 2023
132dc61
replaced deprecated py.path tmpdir with pathlib.Path tmp_path in sing…
tclose Jun 8, 2023
7265a37
added explicit check to see that the return type isn't of type file, …
tclose Jun 8, 2023
79225db
updated python in slurm docker image
tclose Jun 9, 2023
9130c9b
set python version to 3.7.12 in slurm test build
tclose Jun 9, 2023 8000
079deb1
added more descriptive error message in exec_as_coro
tclose Jun 9, 2023
c9ac936
encapsulated inputs to workflow in gathered object in test_wf_lzoutal…
tclose Jun 9, 2023
1ca65c7
upped python version in slurm test to 3.7.16
tclose Jun 10, 2023
f0a57cf
marked wf_lzoutall_st_2a as xfail on slurm tests
tclose Jun 12, 2023
d5979ac
changed audit_shellcommandtask_file expected output to be the filepat…
tclose Jun 13, 2023
79fdea8
replacing copyfile functionality with fileformats
tclose Jun 13, 2023
7aee412
renamed TypeChecker to TypeParser
tclose Jun 13, 2023
f8c7007
moved MountIndentifier class in from fileformats (used to be a collec…
tclose Jun 14, 2023
b6e0ad7
remove commented code
tclose Jun 14, 2023
15524ab
added ability to type inputs and outputs of workflows
tclose Jun 14, 2023
dc4bb8d
adding in support for copy collation
tclose Jun 18, 2023
d8dbf8a
renamed collation default from separated to any
tclose Jun 19, 2023
18abe7f
reverted inputs with "output_file_template" to type str|Path from Fil…
tclose Jun 19, 2023
06427c7
reverted inputs with output_file_template needed to be of type File
tclose Jun 19, 2023
d2bde99
added "sed" fixture to check for "gsed" and use if installed
tclose Jun 19, 2023
cb7dbad
fixed up in-place sed on mac
tclose Jun 19, 2023
93942af
debugged stdout coercion to File
tclose Jun 19, 2023
faa0543
commented out incomplete mtime caching to boost coverage
tclose Jun 19, 2023
a30a559
added new tests to increase coverage, removed some unused code
tclose Jun 20, 2023
156bb8d
renamed gathered to StateArray
tclose Jun 20, 2023
680d2c3
implementing splitter syntax
tclose Jun 20, 2023
3c8d095
added doc strings and typing
tclose Jun 21, 2023
0cf3b50
reworking wrapping of lazy-field types in StateArrays
tclose Jun 22, 2023
0fd5ef3
debugging typing of split/combine workflows
tclose Jun 22, 2023
db55a29
renamed StateArray to SplitArray
tclose Jun 22, 2023
76dc337
renamed SplitArray to Split
tclose Jun 22, 2023
b7d1169
finally debugged state_arrays test
tclose Jun 22, 2023
5448e5a
Yet more debugging (probably going around in circles)
tclose Jun 25, 2023
7312b3f
starting to refactor split-combine in lazy-field
tclose Jun 25, 2023
f80a89a
another go at debugging splitting/combining with types
tclose Jun 26, 2023
5223677
almost got it working
tclose Jun 26, 2023
f052a68
implemented combination over scalar upstream splits
tclose Jun 26, 2023
1d95204
added check for empty states
tclose Jun 26, 2023
611c505
added __bytes_repr__ to lazy field, debugged all unittests I think
tclose Jun 26, 2023
b4254a7
supported basic input specs with just names and types
tclose Jun 26, 2023
0775b3f
renamed Split to StateArray
tclose Jun 26, 2023
9f89c7b
removed references to StateArray from unittests
tclose Jun 26, 2023
96e721f
debugged singularity test and reinstated test decorators that were c…
tclose Jun 27, 2023
9a831a6
refactored LazyField into LazyInField and LazyOutField, removed unuse…
tclose Jun 27, 2023
c3cb603
added a couple more tests
tclose Jun 27, 2023
1f6fe6c
[skip ci] touched up docs section relating to splitting
tclose Jun 27, 2023
ec2b681
removed relative path conversion that is no longer necessary
tclose Jun 27, 2023
5256e26
deleted dummy scripts added to play around with splitting/combining
tclose Jun 27, 2023
c225454
reinstated etelemetry
tclose Jun 27, 2023
9849b77
changed submitted from cf to serial in problematic test
tclose Jun 28, 2023
a5482cf
switched failing tests to serial plugin
tclose Jun 28, 2023
8ded823
switched serial plugin to the tests that were actually failing
tclose Jun 28, 2023
df92bd1
[skip ci] deleted comment that is no longer valid
tclose Jun 29, 2023
0d34855
incorporates Ghisvail's change in https://github.com/nipype/pydra/pul…
tclose Jun 29, 2023
23c33de
deleted exception handling util I didn't end up using
tclose Jun 29, 2023
200c520
Added lazy field cast test, from specific -> general -> specific (wit…
tclose Jun 30, 2023
074ae3b
added check that annotations are types in function task
tclose Jul 10, 2023
7e15285
reverted type assertion in FunctionTask
tclose Jul 10, 2023
e4755b7
removed commented out code in combine()
tclose Jul 12, 2023
415d5bd
added test_hash_file back in
tclose Jul 12, 2023
73e4c68
changed test_task_state back to having values set after split
tclose Jul 12, 2023
99bed52
made _unwrap_splitter doc string more explicit
tclose Jul 12, 2023
9b98935
added tests for copy_nested_files
tclose Jul 13, 2023
64347ad
added support for parsing fields of type ty.Type[*]
tclose Jul 17, 2023
7c254b6
added check for unrecognised types with args
tclose Jul 17, 2023
e4041a7
fixed up bug I just introduced
tclose Jul 17, 2023
9c1a0d5
fixed up case where type to coerce to is iterable
tclose Jul 17, 2023
e53b2ee
fixed up handling of iterables
tclose Jul 17, 2023
0dd8693
[skip ci] touch up exception string
tclose Jul 17, 2023
e58edac
fixed up a couple of mypy errors
tclose Jul 19, 2023
72f9a6b
moved validation of output_file_template from application of the temp…
tclose Aug 1, 2023
e4e7cb0
More robust coercion of output paths to FileSets handles nested types…
tclose Aug 1, 2023
b0c51c1
relaxed restriction on fields with output_file_template to allow str …
tclose Aug 1, 2023
81eded5
applying changes from Chris' review
tclose Aug 1, 2023
62e9523
applied remaining suggestions from Chris' review
tclose Aug 1, 2023
23d5980
actually (I think this time) finished applying all of Chris' suggesti…
tclose Aug 1, 2023
131b18a
applied Chris's suggestions to core.py
tclose Aug 2, 2023
b637b33
implemented more of Chris' suggestions
tclose Aug 2, 2023
6009924
reverted submitter plugin
tclose Aug 2, 2023
f10bfdb
added back in return statement that was inadvertently removed
tclose Aug 2, 2023
51872bb
fixed up unittests to match slightly more expressive error messages
tclose Aug 2, 2023
01353af
reverted test_shell_cmd_inputspec_11 and test_shell_cmd_outputspec_7a…
tclose Aug 2, 2023
1dc29da
[skip ci] added link to issue for XXX comment
tclose Aug 2, 2023
ed8fe28
applied Ghisvail's review suggestions
tclose Aug 3, 2023
9710906
made lzin a cached_property
tclose Aug 3, 2023
ef3f7e0
fixed up typing bug that was showing up for Enum types
tclose Aug 3, 2023
4adac59
added handling for hashing attrs slots classes
tclose Aug 3, 2023
8200918
added upper bound on fileformats package
tclose Aug 3, 2023
2a831dc
removed upper bound on fileformats version
tclose Aug 4, 2023
b4f49ff
added type var to state array instantiations
tclose Aug 4, 2023
0b8e3e5
added notimplementederror to catch typevar templates in function sign…
tclose Aug 4, 2023
d2ed9a0
relaxed type-checking to allow parent->child matching when connecting…
tclose Aug 4, 2023
6397e85
reverted relaxation of parent->child type checking as it will be a bi…
tclose Aug 4, 2023
8ec38dd
attempting to upgrade the python version within the slurm container t…
tclose Aug 4, 2023
03c1286
added a couple of hashing tests
tclose Aug 4, 2023
9a28917
[skip ci] froze sphinx version
tclose Aug 4, 2023
292fd3f
[skip ci] pin sphinx to 6.2.1
tclose Aug 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
removed upper bound on fileformats version
  • Loading branch information
@tclose
tclose committed Aug 4, 2023
commit 2a831dccd2f88cc84ebbea46a77eb6a1efadf689
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ dependencies = [
"cloudpickle >=2.0.0",
"etelemetry >=0.2.2",
"filelock >=3.0.0",
"fileformats >=0.8,<0.9",
"fileformats >=0.8",
"importlib_resources >=5.7; python_version < '3.11'",
"typing_extensions >=4.6.3; python_version < '3.10'",
"typing_utils >=0.1.0; python_version < '3.10'",
Expand Down
0