ref(sdk/python): migrate python code analysis from runtime to ast-based #11655
+2,414
−5
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The Python SDK's code analysis has been migrated from runtime-based to AST-based analysis. This allows types to be available before dependencies are fetched, enabling static analysis without executing user code. Files Created | File | Purpose | |--------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------| | sdk/python/src/dagger/mod/_ir.py | Immutable dataclasses for intermediate representation (TypeAnnotation, ParameterIR, FunctionIR, FieldIR, EnumMemberIR, ObjectTypeIR, ModuleIR) | | sdk/python/src/dagger/mod/_ast_visitor.py | Core AST parsing with DaggerModuleVisitor class that extracts decorated types without executing code | | sdk/python/src/dagger/mod/_ast_resolver.py | TypeResolver and IRToTypeDefConverter classes to convert IR to Dagger TypeDefs | | sdk/python/src/dagger/mod/_analyzer.py | Main entry points: PythonModuleAnalyzer, analyze_package(), analyze_source() | | sdk/python/tests/mod/test_ast_analyzer.py | 38 comprehensive tests covering all components | Files Modified | File | Changes | |----- B03E ---------------------------------|----------------------------------------------------------------------------------------------------------------| | sdk/python/src/dagger/mod/cli.py | Added analyze parameter to app(), analyze_standalone() function for CLI, analyze_and_register() async function | | sdk/python/src/dagger/mod/_module.py | Added use_ast parameter to register(), _typedefs_from_ast() method | | sdk/python/ruff.toml | Added per-file ignores for AST complexity rules | Key Features 1. Static Analysis: Parses Python source using ast module without importing/executing code 2. Full Decorator Support: @object_type, @function, @field, @enum_type, @interface, @check 3. Metadata Extraction: Handles Annotated[] with Doc, Name, DefaultPath, Ignore, Deprecated 4. Name Conversion: Automatic snake_case → camelCase, trailing underscore normalization 5. Type Resolution: Maps Python types to Dagger TypeDef (builtins, Dagger core types, module types) 6. Constructor Support: Default constructors from dataclass fields, explicit constructors via name="" 7. Package Analysis: Supports single files and full package directories with recursive scanning Usage # Programmatic API from dagger.mod._analyzer import analyze_package, analyze_source # Analyze a package directory module_ir = analyze_package(Path("./src"), main_object="MyModule") # Analyze source code string module_ir = analyze_source(source_code, "test.py") # CLI standalone analysis python -m dagger --analyze # Uses DAGGER_SOURCE_DIR env var # Integration with Module.register() await mod.register(use_ast=True) Architecture The implementation follows a clean separation of concerns: Source Code → AST Visitor → IR Dataclasses → TypeDef Converter → Dagger API This allows the parsing and type resolution to be tested independently and makes future extensions straightforward. Signed-off-by: Yves Brissaud <yves.brissaud@gmail.com>
19466e8 to
fd2b913
Compare
Contributor
|
This PR is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 7 days. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The Python SDK's code analysis has been migrated from runtime-based to AST-based analysis. This allows types to be available before dependencies are fetched, enabling static analysis without executing user code.
Files Created
Files Modified
Key Features
Usage
Architecture
The implementation follows a clean separation of concerns:
Source Code → AST Visitor → IR Dataclasses → TypeDef Converter → Dagger API
This allows the parsing and type resolution to be tested independently and makes future extensions straightforward.