Not sure what else I need to add, except for better translations for Pyrex types to python types and a brand new test suite.

I got the PyiWriter to finally work with the rest of the transforms by adding in a function that was missing from the transform. Turns out that visit_StatListNode was missing from the functions so I had to add it in and now everything seems to compile as normal. I will have to turn off the debug argument when it is writing but the only thing we need now is to find a clean way to test it.

I have tested everything using the cythonize command since I am working with the edited version made inside of a virtual environment. I guess the next part of my plan now is to create a test class that uses both __cinit__ and __init__ functions since it may cause problems in the future. Beyond that will be to make a unit test in the Tests folder where the Compiler is located. Not sure how exactly I will be writing it, but I'm currently am trying to make it.

Below is a file that I recently did to be sure the PyiWriter works and that all the other underlying visitors work with it without a problem.

# cython: language_level = 3
# cython: embedsignature = True 
# cython: embedsignature.python = True
# cython: write_stub_file = True

# This function doesn't make sense but I put it here for demonstration purposes only...
def test_data(int a):
    if a == 1:
        return float(a)
    else:
        return str(a)

cdef class RoundRobin:
    cdef Py_ssize_t current , limit
    def __cinit__(self, Py_ssize_t limit):
        self.limit = limit
        self.current = 0
    cpdef Py_ssize_t next(self):
        cdef Py_ssize_t result = self.current
        self.current += 1 
        if self.current >= self.limit:
            self.current = 0
        return result 
    # This should not be added in with the pyi stub because it's C
    cdef Py_ssize_t get_limit(self):
        return self.limit   

# Python stub file generated by Cython 3.0.2

def test_data(a: int): ...
class RoundRobin:
    def __init__(self, limit: int): ...
    def next(self) -> int: ...

A good start to the testing would just be to enable the stub generator generally on the Cython test suite (like we do with the annotated html). That wouldn't actually test any of the functionality, but it would confirm that it can process a lot of Cython code without crashing. Changes would probably need to be in runtest.py.

I'd probably put TypeStubGenerator.py through something like black to standardize the formatting. It's a lot better now but not perfect. We don't generally do this for existing files, but it's a good idea for new files I think.

Again, I haven't have a chance to have a detailed look yet so I'm sure there's more comments, but that bit of testing is definitely a good place to start

We should just run black selectively on all newly added files, as part of the code checks. It has a mode that fails on necessary format changes.

Sounds like a great idea actually. That would be a good way to verify that it works. I'll see what I can come up with.

Did not realize that the imports were accidently deleted but I have that fixed now so this should all work as normal now.

are we ok to merge this pull request or do I still need to write in a unittest for it?

are we ok to merge this pull request or do I still need to write in a unittest for it?

My view is that we definitely need some testing before we consider merging it. The testing doesn't have to be comprehensive but right now we have no evidence it works at all.

Like I said before, I think it would be worth enabling the directive for every CythonCompileTestCase in the entire existing test-suite. We don't even have to look at the output, but it'll at least prove that it doesn't crash given existing valid Cython code.

I think setting up proper tests for this may be quite hard, and maybe we shouldn't leave it to you. I'd think we want to either look for specific generated code in the pyi files with a regex search, or we'd want to re-parse the pyi files into Cython and look for specific nodes using something similar to test_assert_path_exists

The other thing to say with testing: if you have files that you've used to test it manually yourself it might still be useful to add them as part of the review process so we can think about how to automate it

The other thing to say with testing: if you have files that you've used to test it manually yourself it might still be useful to add them as part of the review process so we can think about how to automate it

@da-woods Sounds good sorry I didn't get back sooner but sounds like a great plan I will have to figure out a way to start verifying off pyi files since I cannot seem to find any python libraries that specifically check stub files yet but I will keep looking for an implementation that does. I will also find a way to test it as well since I'm currently working on a unittest for it.

I cannot seem to find any python libraries that specifically check stub files yet but I will keep looking for an implementation that does

I think a stub file does end up being a valid Python file. So either putting it into Python or Cython itself would at least show that you've written a valid file.

Let me make sure I get the file formatted using black real fast...

cc @dalcinl for vis

@Vizonex You should test your work against mpi4py

git clone https://github.com/mpi4py/mpi4py.git
cd mpi4py
python conf/cythonize.py

Your code currently crashes Cython:

Error compiling Cython file:
------------------------------------------------------------
...
    bint      recv_mprobe
    MPI_Count irecv_bufsz
    int       errors

cdef Options options
options.initialize = 1
                     ^
------------------------------------------------------------

src/mpi4py/MPI/atimport.pxi:95:21: Compiler crash in PyiWriter

ModuleNode.body = StatListNode(MPI.pyx:15:0)
StatListNode.stats[1] = StatListNode(MPI.pyx:3:0)
StatListNode.stats[3] = StatListNode(atimport.pxi:3:0)
StatListNode.stats[8] = SingleAssignmentNode(atimport.pxi:95:21)

Compiler crash traceback from this point on:
  File "/tmp/cython/Cython/Compiler/Visitor.py", line 182, in _visit
    return handler_method(obj)
           ^^^^^^^^^^^^^^^^^^^
  File "/tmp/cython/Cython/Compiler/TypeStubGenerator.py", line 387, in visit_SingleAssignmentNode
    name = node.lhs.name
           ^^^^^^^^^^^^^
AttributeError: 'AttributeNode' object has no attribute 'name'

@dalcinl I will be sure to use the repository you mentioned for testing. And I also think some of the visitors in my code should use a little more recursion which is what scoder suggested I do a bit earlier. The attributes themselves I think need better attention than what they have currently been given.

I had an Idea for writing decorators by making a new class object that could chain attributes together for the decorators but I am not sure how I am going to approach this problem. Here is an example of what the problem is...

@very.long.wrapper
def func():

Oh I see there is an example in the Expression Writer class Nevermind....

Decorators:

https://typing.readthedocs.io/en/latest/source/stubs.html#decorators

Type stubs may only use decorators defined in the typing module, plus a fixed set of additional ones:

classmethod

staticmethod

property (including .setter)

abc.abstractmethod

dataclasses.dataclass

asyncio.coroutine (although async should be used instead)

To me that says you can be fairly eager to drop anything unrecognised. I'd probably do something like:

1. If it's a CallNode, drop the CallNode and just take node.function to step 2.
2. If it isn't a NameNode or AttributeNode then drop the decorator completely.
3. Maybe just write out the name/attribute? (I think filtering to the fixed list of names could be hard)

Decorators:

https://typing.readthedocs.io/en/latest/source/stubs.html#decorators

Type stubs may only use decorators defined in the typing module, plus a fixed set of additional ones:
classmethod
staticmethod
property (including .setter)
abc.abstractmethod
dataclasses.dataclass
asyncio.coroutine (although async should be used instead)

To me that says you can be fairly eager to drop anything unrecognised. I'd probably do something like:

1. If it's a CallNode, drop the CallNode and just take node.function to step 2.
2. If it isn't a NameNode or AttributeNode then drop the decorator completely.
3. Maybe just write out the name/attribute? (I think filtering to the fixed list of names could be hard)

So should I make the visitor visit call nodes as well because I am also worried about it accidently being mixed with cython directives. But I will figure out if there's already a check inplace for that.

So should I make the visitor visit call nodes as well because I am also worried about it accidently being mixed with cython directives. But I will figure out if there's already a check inplace for that.

You should be able to identify cython directives with node.as_cython_attribute(). You should probably ignore any cython directives.

So should I make the visitor visit call nodes as well because I am also worried about it accidently being mixed with cython directives. But I will figure out if there's already a check inplace for that.

You should be able to identify cython directives with node.as_cython_attribute(). You should probably ignore any cython directives.

I just looked at it and it seems that there is a better solution. So would something like this be a bit better?

def write_decorator(self, decorator):
    attribte = decorator.as_cython_attribute()
    if attribte:
         self.putline("@%s" % attribte)

Probably

    attribte = decorator.as_cython_attribute()
    if not attribte:

(i.e. you don't want to write out Cython directives)

I'm gonna be sure I get to fixing visit_SingleAssignmentNode since there seems to be some nodes that do not have a name attribute as seen in the error above. Might need to re-work on that function when I can get to it.

I think the reason why I stopped trying was because everything lacked readability (I blame VS code's intellisense/pyright tools for that). I may resume my attempts on this project soon but I think that writing stub files or maybe even programming external tools for writing tree visitors and finding a way for to figure out node variables and their different class members might be helpful. I'll probably close this Pull Request soon and come up with a new one in the future. For those depending on me to program stub file generation into cython please don't worry, I haven't given up on this. I've just been busy with other stuff that I completely forgot about this project.