10000 Allow finer grain specification of requirements. · jmchilton/galaxy@cd2cf49 · GitHub
[go: up one dir, main page]
Skip to content

Commit cd2cf49

Browse files
jmchiltonjmchilton
committed
Allow finer grain specification of requirements.
This adapts CWL-style ``SoftwareRequirement`` ``specs`` to solve galaxyproject#1927. Here I'm trying to implement the CWL specification in a way that helps enable the feasibility of Conda packaging in Galaxy. It is a delicate balancing act aimed to upset as many interested parties as I can. To understand the problem - consider the ``blast+`` requirement found in the Galaxy wrappers. It looks something like this: ``` <requirement type="package" version="2.2.31" name="blast+"> ``` Great, that works for Galaxy and Tool Shed packages. It doesn't work for bioconda at all. I think this problem is fairly uncommon - most packages have simple names shared across Debian, Brew, Conda, etc... - but it does happen in some cases that there are inconsistencies. Some people have taken to duplicating the requirement - this is bad and should not be done since they are mutually exclusive and Galaxy will attempt to resolve both. This introduces the following syntax for tools with profile >= 16.10: ``` <requirement type="package" version="2.2.31" name="blast+"> <specification uri="https://anaconda.org/bioconda/blast" /> <specification uri="https://packages.debian.org/sid/ncbi-blast+" version="2.2.31-3" /> </requirement> ``` This allows finer grain resolution of the requirement without sacrificing the abstract name at the top. It allows the name and the version to be adapted by resolvers as needed (hopefully rarely so). This syntax is the future facing one, but obviously this tool would not work on older Galaxy versions. To remedy this - an alternative syntax can be used for tools targetting Galaxy verions pre-16.10: ``` <requirement type="package" version="2.2" specification_uris="https://anaconda.org/bioconda/blast@2.2.31,https://packages.debian.org/jessie/ncbi-blast+@2.2.29-3">blast+</requirement> ``` This syntax sucks - but it does give newer Galaxies the ability to resolve the specifications without breaking the more simple functionality for older Galaxies. For more information on the CWL side of this - checkout the discussion on common-workflow-language/cwltool#214. The CWL specification information is defined at http://www.commonwl.org/v1.0/CommandLineTool.html#SoftwarePackage. Conflicts: lib/galaxy/tools/deps/__init__.py test/functional/tools/samples_tool_conf.xml
1 parent 4eade31 commit cd2cf49

File tree

7 files changed

+171
-24
lines changed

7 files changed

+171
-24
lines changed

‎lib/galaxy/tools/deps/__init__.py

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,7 @@ def _requirements_to_dependencies_dict(self, requirements, **kwds):
141141
# Check requirements all at once
142142
all_unmet = len(requirement_to_dependency) == 0
143143
if all_unmet and hasattr(resolver, "resolve_all"):
144+
# TODO: Handle specs.
144145
dependencies = resolver.resolve_all(resolvable_requirements, **kwds)
145146
if dependencies:
146147
assert len(dependencies) == len(resolvable_requirements)
@@ -155,7 +156,17 @@ def _requirements_to_dependencies_dict(self, requirements, **kwds):
155156
if requirement in requirement_to_dependency:
156157
continue
157158

158-
dependency = resolver.resolve( requirement.name, requirement.version, requirement.type, **kwds )
159+
name = requirement.name
160+
version = requirement.version
161+
specs = requirement.specs
162+
163+
if hasattr(resolver, "find_specification"):
164+
spec = resolver.find_specification(specs)
165+
if spec is not None:
166+
name = spec.short_name
167+
version = spec.version or version
168+
169+
dependency = resolver.resolve( name, version, requirement.type, **kwds )
159170
if require_exact and not dependency.exact:
160171
continue
161172

‎lib/galaxy/tools/deps/requirements.py

Lines changed: 69 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -14,29 +14,66 @@ class ToolRequirement( object ):
1414
run (for example, a program, package, or library). Requirements can
1515
optionally assert a specific version.
1616
"""
17-
def __init__( self, name=None, type=None, version=None ):
17+
def __init__( self, name=None, type=None, version=None, specs=[] ):
1818
self.name = name
1919
self.type = type
2020
self.version = version
21+
self.specs = specs
2122

2223
def to_dict( self ):
23-
return dict(name=self.name, type=self.type, version=self.version)
24+
specs = [s.to_dict() for s in self.specs]
25+
return dict(name=self.name, type=self.type, version=self.version, specs=specs)
2426

2527
@staticmethod
2628
def from_dict( dict ):
2729
version = dict.get( "version", None )
2830
name = dict.get("name", None)
2931
type = dict.get("type", None)
30-
return ToolRequirement( name=name, type=type, version=version )
32+
specs = [RequirementSpecification.from_dict(s) for s in dict.get("specs", [])]
33+
return ToolRequirement( name=name, type=type, version=version, specs=specs )
3134

3235
def __eq__(self, other):
33-
return self.name == other.name and self.type == other.type and self.version == other.version
36+
return self.name == other.name and self.type == other.type and self.version == other.version and self.specs == other.specs
3437

3538
def __ne__(self, other):
3639
return not self.__eq__(other)
3740

3841
def __hash__(self):
39-
return hash((self.name, self.type, self.version))
42+
return hash((self.name, self.type, self.version, frozenset(self.specs)))
43+
44+
45+
class RequirementSpecification(object):
46+
"""Refine a requirement using a URI."""
47+
48+
def __init__(self, uri, version=None):
49+
self.uri = uri
50+
self.version = version
51+
52+
@property
53+
def specifies_version(self):
54+
return self.version is not None
55+
56+
@property
57+
def short_name(self):
58+
return self.uri.split("/")[-1]
59+
60+
def to_dict(self):
61+
return dict(uri=self.uri, version=self.version)
62+
63+
@staticmethod
64+
def from_dict(dict):
65+
uri = dict.get["uri"]
66+
version = dict.get("version", None)
67+
return RequirementSpecification(uri=uri, version=version)
68+
69+
def __eq__(self, other):
70+
return self.uri == other.uri and self.version == other.version
71+
72+
def __ne__(self, other):
73+
return not self.__eq__(other)
74+
75+
def __hash__(self):
76+
return hash((self.uri, self.version))
4077

4178

4279
class ToolRequirements(object):
@@ -173,10 +210,29 @@ def parse_requirements_from_xml( xml_root ):
173210

174211
requirements = ToolRequirements()
175212
for requirement_elem in requirement_elems:
176-
name = xml_text( requirement_elem )
213+
if "name" in requirement_elem.attrib:
214+
name = requirement_elem.get( "name" )
215+
spec_elems = requirement_elem.findall("specification")
216+
specs = map(specification_from_element, spec_elems)
217+
else:
218+
name = xml_text( requirement_elem )
219+
spec_uris_raw = requirement_elem.attrib.get("specification_uris", "")
220+
specs = []
221+
for spec_uri in spec_uris_raw.split(","):
222+
if not spec_uri:
223+
continue
224+
version = None
225+
if "@" in spec_uri:
226+
uri, version = spec_uri.split("@", 1)
227+
else:
228+
uri = spec_uri
229+
uri = uri.strip()
230+
if version:
231+
version = version.strip()
232+
specs.append(RequirementSpecification(uri, version))
177233
type = requirement_elem.get( "type", DEFAULT_REQUIREMENT_TYPE )
178234
version = requirement_elem.get( "version", DEFAULT_REQUIREMENT_VERSION )
179-
requirement = ToolRequirement( name=name, type=type, version=version )
235+
requirement = ToolRequirement( name=name, type=type, version=version, specs=specs )
180236
requirements.append( requirement )
181237

182238
container_elems = []
@@ -188,6 +244,12 @@ def parse_requirements_from_xml( xml_root ):
188244
return requirements, containers
189245

190246

247+
def specification_from_element(specification_elem):
248+
uri = specification_elem.get("uri", None)
249+
version = specification_elem.get("version", None)
250+
return RequirementSpecification(uri, version)
251+
252+
191253
def container_from_element(container_elem):
192254
identifier = xml_text(container_elem)
193255
type = container_elem.get("type", DEFAULT_CONTAINER_TYPE)

‎lib/galaxy/tools/deps/resolvers/__init__.py

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,34 @@ def _to_requirement(self, name, version=None):
6767
return ToolRequirement(name=name, type="package", version=version)
6868

6969

70+
class SpecificationAwareDependencyResolver:
71+
"""Mix this into a :class:`DependencyResolver` to implement URI specification matching.
72+
73+
Allows adapting generic requirements to more specific URIs - to tailor name
74+
or version to specified resolution system.
75+
"""
76+
__metaclass__ = ABCMeta
77+
78+
@abstractmethod
79+
def find_specification(self, specs):
80+
"""Find closest matching specification for discovered resolver."""
81+
82+
83+
class SpecificationPatternDependencyResolver:
84+
"""Implement the :class:`SpecificationAwareDependencyResolver` with a regex pattern."""
85+
86+
@abstractproperty
87+
def _specification_pattern(self):
88+
"""Pattern of URI to match against."""
89+
90+
def find_specification(self, specs):
91+
pattern = self._specification_pattern
92+
for spec in specs:
93+
if pattern.match(spec.uri):
94+
return spec
95+
return None
96+
97+
7098
class InstallableDependencyResolver:
7199
""" Mix this into a ``DependencyResolver`` and implement to indicate
72100
the dependency resolver can attempt to install new dependencies.

‎lib/galaxy/tools/deps/resolvers/conda.py

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55

66
import logging
77
import os
8+
import re
89

910
import galaxy.tools.deps.installable
1011

@@ -29,6 +30,7 @@
2930
InstallableDependencyResolver,
3031
ListableDependencyResolver,
3132
NullDependency,
33+
SpecificationPatternDependencyResolver,
3234
)
3335

3436

@@ -39,9 +41,10 @@
3941
log = logging.getLogger(__name__)
4042

4143

42-
class CondaDependencyResolver(DependencyResolver, ListableDependencyResolver, InstallableDependencyResolver):
44+
class CondaDependencyResolver(DependencyResolver, ListableDependencyResolver, InstallableDependencyResolver, SpecificationPatternDependencyResolver):
4345
dict_collection_visible_keys = DependencyResolver.dict_collection_visible_keys + ['conda_prefix', 'versionless', 'ensure_channels', 'auto_install']
4446
resolver_type = "conda"
47+
_specification_pattern = re.compile(r"https\:\/\/anaconda.org\/\w+\/\w+")
4548

4649
def __init__(self, dependency_manager, **kwds):
4750
self.versionless = _string_as_bool(kwds.get('versionless', 'false'))

‎lib/galaxy/tools/xsd/galaxy.xsd

Lines changed: 24 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -227,7 +227,7 @@ complete descriptions of the runtime of a tool.
227227
</xs:sequence>
228228
</xs:complexType>
229229

230-
<xs:complexType name="Requirement">
230+
<xs:complexType name="Requirement" mixed="true">
231231
<xs:annotation>
232232
<xs:documentation xml:lang="en"><![CDATA[
233233
@@ -276,20 +276,29 @@ resolver.
276276
277277
]]></xs:documentation>
278278
</xs:annotation>
279-
<xs:simpleContent>
280-
<xs:extension base="xs:string">
281-
<xs:attribute name="type" type="RequirementType" use="required">
282-
<xs:annotation>
283-
<xs:documentation xml:lang="en"> This value defines the which type of the 3rd party module required by this tool. </xs:documentation>
284-
</xs:annotation>
285-
</xs:attribute>
286-
<xs:attribute name="version" type="xs:string">
287-
<xs:annotation>
288-
<xs:documentation xml:lang="en"> For package type requirements this value defines a specific version of the tool dependency. </xs:documentation>
289-
</xs:annotation>
290-
</xs:attribute>
291-
</xs:extension>
292-
</xs:simpleContent>
279+
<xs:sequence>
280+
<xs:element name="specification" minOccurs="0" maxOccurs="unbounded" type="xs:anyType" />
281+
</xs:sequence>
282+
<xs:attribute name="type" type="RequirementType" use="required">
283+
<xs:annotation>
284+
<xs:documentation xml:lang="en"> This value defines the which type of the 3rd party module required by this tool. </xs:documentation>
285+
</xs:annotation>
286+
</xs:attribute>
287+
<xs:attribute name="version" type="xs:string">
288+
<xs:annotation>
289+
<xs:documentation xml:lang="en"> For package type requirements this value defines a specific version of the tool dependency. </xs:documentation>
290+
</xs:annotation>
291+
</xs:attribute>
292+
<xs:attribute name="name" type="xs:string">
293+
<xs:annotation>
294+
<xs:documentation xml:lang="en">Name of requirement (if body of ``requirement`` element contains specification URIs).</xs:documentation>
295+
</xs:annotation>
296+
</xs:attribute>
297+
<xs:attribute name="specification_uris" type="xs:string">
298+
<xs:annotation>
299+
<xs:documentation xml:lang="en">URIs and versions of requirement specification.</xs:documentation>
300+
</xs:annotation>
301+
</xs:attribute>
293302
</xs:complexType>
294303
<xs:complexType name="Container">
295304
<xs:annotation>

‎test/functional/tools/requirement_specification_1.xml

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
<tool id="requirement_specification_1" name="requirement_specification_1" version="0.1.0" profile="16.10">
2+
<command><![CDATA[
3+
blastn -help > $out_file1 ;
4+
echo "Moo" >> $out_file1 ;
5+
]]></command>
6+
<requirements>
7+
<requirement type="package" version="2.2.31" name="blast+">
8+
<specification uri="https://anaconda.org/bioconda/blast" />
9+
<specification uri="https://packages.debian.org/sid/ncbi-blast+" version="2.2.31-3" />
10+
</requirement>
11+
</requirements>
12+
<inputs>
13+
<param name="input1" type="data" optional="true" />
14+
</inputs>
15+
<outputs>
16+
<data name="out_file1" format="txt" />
17+
</outputs>
18+
</tool>

‎test/functional/tools/requirement_specification_2.xml

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
<tool id="requirement_specification_2" name="requirement_specification_2" version="0.1.0" profile="16.01">
2+
<command><![CDATA[
3+
blastn -help > $out_file1 ;
4+
echo "Moo" >> $out_file1 ;
5+
]]></command>
6+
<requirements>
7+
<!-- Demonstrate backward-compatible-ish specification_uri syntax. -->
8+
<requirement type="package" version="2.2" specification_uris="https://anaconda.org/bioconda/blast@2.2.31,https://packages.debian.org/jessie/ncbi-blast+@2.2.29-3">blast+</requirement>
9+
</requirements>
10+
<inputs>
11+
<param name="input1" type="data" optional="true" />
12+
</inputs>
13+
<outputs>
14+
<data name="out_file1" format="txt" />
15+
</outputs>
16+
</tool>

0 commit comments

Comments
 (0)
0