Add type stub generation to cython #3818

ax487 · 2020-09-07T21:15:46Z

This was brought up in #3150 by @scoder:

If generating .pyi stubs is not too complex, Cython could do it automatically, yes, similar to how it now generates the embedded signatures in AutoDocTransform. PR welcome.

I have written a very simple prototype to generate stub files, based on a TreeVisitor. Based on this input:

#cython: language_level=3

import typing

import typing as tipi

from typing import List

import package.subpackage

import test.fancy

from .other import Other
from ..other import Test

cdef class TypingTestClass:

    cdef should_not_appear(self, int x):
        pass

    cpdef int cpdef_typed(self, int l, int u, int s = 4):
        return 0

    cpdef cpdef_untyped(self, l):
        pass

    def func_test(self, i: int) -> int:
        return 0

    def func_test_pos(self, *pos):
        return 0

    def func_test_kwds(self, **kwargs):
        return 0

    def func_test_pos_kwds(self, *pos, **kwargs):
        return 0

    def func_with_default(self, value=1) -> int:
        return 0

cdef class Argument:
    pass

cdef class ArgumentUser:
    cpdef func(self, Argument argument):
        pass

def global_pfunc(i: int) -> int:
    return 1

cpdef str global_cpfunc(str i):
    return "asd"

class Outer:

    def outer_method(self, i: int):
        pass

    class Inner:
        def inner_method(self, j):
            return 1

    def __dealloc__(self):
        print("asd")

    def __cinit__(self):
        print("cinit")


class TypingClass:

    def method(self, v: typing.Dict[str, str]) -> typing.List[int]:
        return []

    def other_method(self, v: typing.Tuple[str, int, List[int]]):
        pass


global_typed: int

global_untyped = 0

it generates the following stub file:

# Python stub file generated by Cython 3.0a6

import typing
import typing as tipi
from typing import List
import package.subpackage
import test.fancy
from .other import Other
from ..other import Test
class TypingTestClass:
  def cpdef_typed(self, l: int, u: int, s: int = ...) -> int: ...
  def cpdef_untyped(self, l: object) -> object: ...
  def func_test(self, i: int) -> int: ...
  def func_test_pos(self, *pos): ...
  def func_test_kwds(self, **kwargs): ...
  def func_test_pos_kwds(self, *pos, **kwargs): ...
  def func_with_default(self, value = ...) -> int: ...
  def __reduce_cython__(self): ...
  def __setstate_cython__(self, __pyx_state): ...

class Argument:
  def __reduce_cython__(self): ...
  def __setstate_cython__(self, __pyx_state): ...

class ArgumentUser:
  def func(self, argument: Argument) -> Argument: ...
  def __reduce_cython__(self): ...
  def __setstate_cython__(self, __pyx_state): ...

def global_pfunc(i: int) -> int: ...
def global_cpfunc(i: unicode) -> unicode: ...
class Outer
  def outer_method(self, i: int): ...
  class Inner
    def inner_method(self, j): ...

  def __dealloc__(self): ...
  def __cinit__(self): ...

class TypingClass
  def method(self, v: typing.Dict[str, str]) -> typing.List[int]: ...
  def other_method(self, v: typing.Tuple[str, int, List[int]]): ...

def __pyx_unpickle_TypingTestClass(__pyx_type, __pyx_checksum, __pyx_state): ...
def __pyx_unpickle_Argument(__pyx_type, __pyx_checksum, __pyx_state): ...
def __pyx_unpickle_ArgumentUser(__pyx_type, __pyx_checksum, __pyx_state): ...

The following problems need to be resolved:

I will continue to work on the PR, tracking the changes. I would be happy about any feedback...

scoder · 2020-09-08T03:38:25Z

Ah, cool, thanks for volunteering to work on this.
However, it seems that you didn't see the Cython.CodeWriter module. It already generates Cython code from a tree, and would be a much better basis for this than a plain TreeVisitor. The DeclarationWriter seems the best starting point. Maybe you can extract some more functionality that you need from its subclasses into reusable mixin classes, e.g. the import handling of the StatementWriter.

scoder · 2020-09-08T04:10:36Z

The right place in the pipeline is not really clear

I was considering to run it right after type analysis, but I now think that it would really be best to run it at the very end. Multiple code checks are executed shortly before the C code generation phase, and I don't know if users would expect a stub file to be generated (or overwritten) if the translation run fails in the end. We might even consider doing it after the C code generation, because that is the point where we know that everything really went smoothly.

cython adds some methods to the classes and the global scope. I don't think those should be part of the stub. Same for __cinit__ and friends.

Those are added somewhat early in the pipeline, so you wouldn't find a place to run the .pyi generation before them. Rather, exclude everything that starts with a double underscore. Or only __pyx_…, if you want.

Declarations such as global_typed: int without a value are swallowed by the parser, this probably needs to be rectified.

Not sure if it's really the parser. Probably some later transform, such as OptimizeBuiltinCalls. Look visit_ExprStatNode, in case there are other places. It might be ok to keep only those variable references that carry annotations.

Retain type variables (of the form T = TypeVar("T"))

Oh, yes. Tricky one. You might start by adding only assignments to the visitor and tracking names imported from the typing module. We already do that for the cython module in the InterpretCompilerDirectives transform. Might be a good time to extract that functionality into a separate transform (or mixin class) because we'd also generally need it for tracking usages of the typing module at some point (which is exactly your use case here).

Maybe add a directive to enable / disable generation of type stubs?!

Sounds more like an option to me, but should be handled in the same way as annotate, i.e. an option that is internally treated (and usable) as a directive.

Add some test code

Sure. You can base that on the CodeWriter tests as well, see Cython.Tests.TestCodeWriter.

da-woods · 2020-09-08T07:29:32Z

Determining types for cpdef correctly: I am using a combination of the py_type_name of the PyrexType and the type.name of the node method so far. Unfortunately, for an int, the py_type_name returns '(int, long)' see here. What is more, an object type is probably a little useless...

I ended up having to solve a very similar problem for my dataclasses PR. My approach was GetTypeNode. It's currently unreviewed (so may have issues) and also may not match exactly what you want, but it might be worth a look seeing if you can borrow that code?

Also: added support for async functions and limited decorator support

ax487 · 2020-09-08T20:55:29Z

Thank you for the feedback and support. The first draft was more of a prototype to convey the ideas behind the stub generator. I am glad you agree with the based architecture, so I will just go on until I have a working prototype 😄

scoder · 2020-09-09T19:12:01Z

agree with the based architecture

Well, most of this should still reuse the existing DeclarationWriter, or a common base class. Just reusing the indentation part still leads to a lot of code duplication that can be avoided. Stub files aren't inherently different from .pxd files. Improvements for one should ideally benefit the other as well.

ax487 · 2020-09-09T20:48:27Z

Well, there is definitely some overlap between writing pxd and pxi files, but there are also some differences. For example, the declaration of a cdef class corresponds to an ordinary class in the stub file. Also the number of node types visited by both DeclarationWriter and the TypeStubGenerator is fairly small. But I will try my best to avoid duplication nonetheless.

@da-woods: Thank you for the suggestions. Regarding tests, I think I will try to rig something up using round trips via the python ast...

Cython/Compiler/Pipeline.py

scoder · 2020-09-10T05:32:08Z

Cython/Compiler/TypeStubGenerator.py

+        "__releasebuffer__",
+        "__getreadbuffer__",
+        "__getwritebuffer__",
+        "___getsegcount__",


___ – typo or intended?

scoder · 2020-09-10T05:43:07Z

the number of node types visited by both DeclarationWriter and the TypeStubGenerator is fairly small.

Since the CodeWriter module has a pretty complete Python code generator, the total overlap is probably quite large, in fact. I think most of the differences can be handled by adding suitable small hook methods that the stub file writer can overwrite when it needs to generate slightly different output (ellipses etc.).

tobiasdiez · 2021-11-28T21:42:46Z

What's the status of this PR? Automatically generating typing stubs would be a very nice feature!

scoder · 2021-11-29T08:26:47Z

I think the original author stopped working on it. Help is welcome.

Look through my previous comments to see what needs to be changed.

Vizonex · 2023-06-24T03:50:24Z

@scoder I like this change a lot and I think I could resume this branch and fix that typo if you want me to even though it's been almost two full years since the author abandoned the project so some other fixes that might be needed such as variable name changes and other things. My reasons on finishing this are simple. I'm a VS Code user, I found it very annoying to have to write large amounts of stub files for things. I think that this will be beneficial to projects such as PyImgui and Pyducktape2 which lack stub files. Having these tools available may help those projects out tremendously. But hey, this isn't my first rodeo making a large solution to a problem and it won't be my last either.

scoder · 2023-06-24T04:12:56Z

I'd be happy to see this resumed. Several problems that existed at the time have been solved now, for features like the imported pure Python mode or dataclasses. The Cython code writer was also definitely improved since then. I haven't looked into this PR for years, so I can't judge the details right now, but it should be much easier to get something working these days. I'm still very much against the code duplication that this PR added at the time. That is even more redundant now and needs to be cleaned up. In general, we should try to output what Cython itself uses/accepts as annotations and decorators. See Shadow.py and the pure Python docs on this. We have a stub file for that, so analysers should understand it.

Vizonex · 2023-06-24T04:39:26Z

I agree with that as well. having conversions from this example I wrote below is what I've always wanted.

# myexample.pyx
cpdef int add(int a, int b): 
    return a + b

# myexample.pyi
def add(a: int , b: int ) -> int:..

I also just took a look at Shadow.pyi as well and I think it would be a good idea to use those variables to help with type-hinting simillar to how mypy imports things from typing...

My only question is this, should I try and make a new fork off of the old branch or make an entirely new fork and bring in some of the things this author worked on as a template?

RaubCamaioni · 2023-08-06T02:47:37Z

Compilation of PYX (cython) -> PYI (stub) file converters.

CyStub: uses cython compiler to generate PYX -> PYI
cythonbuilder: custom parser PYX -> PYI
CythonPeg: pyparsing custom definitions PYX -> PYI
stubgen: uses python runtime objects to generate PYI files

I should have done a more complete search earlier. I would have considered using the cython compiler as the parser.
I started a conversation in cython-users mailing list to compile additional converter options (if it gets approved).
If you know about additional PYX -> PYI code/methods post it there.

Vizonex · 2023-08-07T17:58:53Z

On Sat, Aug 5, 2023 at 9:47 PM Raub Camaioni ***@***.***> wrote: Compilation of PYX (cython) -> PYI (stub) file converters. - CyStub <https://github.com/Vizonex/CyStub>: uses cython compiler to generate PYX -> PYI - cythonbuilder <https://github.com/mike-huls/cythonbuilder>: custom parser PYX -> PYI - CythonPeg <https://github.com/RaubCamaioni/CythonPEG>: pyparsing custom definitions PYX -> PYI - stubgen <https://mypy.readthedocs.io/en/stable/stubgen.html>: uses python runtime objects to generate PYI files I should have done a more complete search earlier. I would have considered using the cython compiler as the parser. I started a conversation in cython-users <https://groups.google.com/g/cython-users> mailing list to compile additional converter options (if it gets approved). If you know about additional PYX -> PYI code/methods post it there. — Reply to this email directly, view it on GitHub <#3818 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3K7GGRC5IYA65YNYMH2IG3XT4AVNANCNFSM4Q65BJHQ> . You are receiving this because you commented.Message ID: ***@***.***>

Sure thing I will do that as the maintainer of cystub my experimentation with the nodes has died off since 3.0 since there is no easy way to get them immediate but I would love to resume what I had going for it. I just wanted to make sure the compilation was perfect such as type hinting the return values or other things even if the user didn’t have it in place. There are many ideas I had on improving this but it’s just been difficult to extract all the information needed from them such as conversions from C type to python object type and keeping initialized classes for later annotations and what not.

moi90 · 2023-09-03T09:31:08Z

Just from glancing at the linked pages, it seems that CyStub by @Vizonex is the most complete solution at the moment, right? However, it does not work anymore with Cython 3...

Stubgen is a mostly stable solution, but is unable to infer most types from Cython code. (They just become Any which is not very helpful.)

As the author of this pull request stopped working on it, maybe it should be closed?

Vizonex · 2023-09-06T03:45:54Z

On Sun, Sep 3, 2023 at 4:31 AM Simon-Martin Schröder < ***@***.***> wrote: Just from glancing at the linked pages, it seems that CyStub by @Vizonex <https://github.com/Vizonex> is the most complete solution at the moment, right? However, it does not work anymore with Cython 3... Stubgen is a mostly stable solution, but is unable to infer most types from Cython code. (They just become Any which is not very helpful.) As the author of this pull request stopped working on it, maybe it should be closed? — Reply to this email directly, view it on GitHub <#3818 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3K7GGSDBBJXEQ6CGZPGF73XYRE6TANCNFSM4Q65BJHQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

I can resume this project but I need to find a clean solution to extract the node-tree nobody tells me anything including big changes like this one. But if someone could just provide me the steps I could resume what I currently have.

Vizonex · 2023-09-22T21:57:55Z

@moi90 I think the only reason why I have been currently stuck on writing something good for my current .pyx to .pyi compiler and doing a pull request is because I'm trying to implement a feature similar to what PyLance is able to do where it has the ability to resolve return type annotations even when the programmer doesn't have anything put in. The tricky thing about that is that Pyrex Types like to hide certain variable types that are things such as built-in types which would commonly return with object which may not be very useful if the programmer has declared that it's going to be something completely different. However I have had been stuck on that idea of resolving these annotations for months with no success. But I'll just start working on the fork for Cython now without the feature of resolving these missing return type annotations until I have figured it out.

moi90 · 2023-09-23T03:46:55Z

But I'll just start working on the fork for Cython now without the feature of resolving these missing return type annotations until I have figured it out.

That sounds totally reasonable! :)

(I'm sorry, I'm not of much help. I have very limited time and no prior knowledge on this issue. I just would like to use it ^^)

Vizonex · 2023-09-28T20:26:33Z

Update: my PyiWriter Class object has been added to my fork of Cython but it still requires a bit of bug fixing and I need to find a place to add it to the pipeline and there needs to be a way to output the file so more research will be needed, so I won't be doing the pull request just yet but it's now being set in motion. I thank all of you for inspiring me to get this feature in and implemented. https://github.com/Vizonex/cython/commit/e4234121210321a00ee64aaec5fc307ece256fee

leo-hstone · 2025-03-21T13:53:36Z

Hi! I just started looking into this topic and want to add another item to the list of @RaubCamaioni. The vscode cython extension has an experimental feature to generate stub files: https://github.com/ktnrg45/vs-code-cython

Unfortunatly I do not have the skills to check if it is based one one of the items listed. Maybe it is of help for this feature.

< 7E39 /div>

ax487 added 3 commits September 6, 2020 21:53

Started working on type stub generator prototype

e37f056

Added watermark, handled imports

629d4ed

Improved cpdef type names

637a5ac

ax487 added 4 commits September 8, 2020 20:41

Bypassed some constructs, moved generator up the pipeline

05235a3

Added missing property documentation

0941925

Added IndentationWriter mixin class

9682127

Moved type stub generator to use indentation writer

d94d684

Also: added support for async functions and limited decorator support

ax487 added 4 commits September 9, 2020 17:40

Fixed empty classes problem

6d338e0

Added support for module-level global variables

cdf7b18

Improved empty class code generation

fcfe5d0

Added support for class variables, hiding cython private functions

00a6cdf

scoder reviewed Sep 10, 2020

View reviewed changes

Cython/Compiler/Pipeline.py Outdated Show resolved Hide resolved

scoder reviewed Sep 10, 2020

View reviewed changes

da-woods mentioned this pull request Nov 23, 2020

Create Stub File for Type Annotation #3915

Closed

This was referenced Jan 12, 2021

Typing Stubs and PEP 561 compatibility pandas-dev/pandas#28142

Closed

TYPING: add skeleton stub for Timestamp pandas-dev/pandas#28195

Closed

kmaehashi mentioned this pull request Mar 31, 2021

First step of mypy introduction cupy/cupy#4955

Merged

jakebailey mentioned this pull request Jul 9, 2021

Add stub files for main cython classes explosion/spaCy#8427

Merged

3 tasks

da-woods mentioned this pull request Jul 17, 2021

Can Cython use Python type hints? #2587

Closed

Undo accidental removal

ae872fc

scoder added feature Python Semantics General Python (3) language semantics labels Nov 29, 2021

scoder added the Type Analysis label Nov 29, 2021

rchiodo mentioned this pull request Jul 27, 2023

intellisesne isn't working with .pyc modules microsoft/pylance-release#242

Closed

Vizonex mentioned this pull request Oct 4, 2023

Add PyiWriter to Cython and new Cython directive 'write_stub_file' #5744

Closed

tobiasdiez mentioned this pull request Oct 28, 2023

src/doc/en/developer: Describe static typing workflow sagemath/sage#36557

Merged

5 tasks

FeldrinH mentioned this pull request Dec 16, 2023

Generate/add stubs for PEP561 and add py.typed segevfiner/cypcap#8

Open

PeterJCLaw mentioned this pull request Jan 3, 2024

Publish type annotations Ezibenroc/PyRoaringBitMap#105

Closed

vyasr mentioned this pull request Nov 6, 2024

[FEA] Provide type stubs for pylibcudf package rapidsai/cudf#15190

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add type stub generation to cython #3818

Add type stub generation to cython #3818

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Add type stub generation to cython #3818

Are you sure you want to change the base?

Add type stub generation to cython #3818

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!