[BUG] Building namespace packages does not work with glob patterns #5237
Description
Describe the bug
The cythonize function gets a glob pattern or list of Extension objects with glob patters as module_list. When using a namespace package, the source glob patterns still match the correct files (thanks to #2946?), but the Extension.name attributes returned by cythonize are missing the namespace package names. This is due to the implementation of the fully_qualified_name function which only regards folders containing an __init__ file as packages.
For example, in the code to reproduce, nsp is a namespace package and nsp.m1 is a subpackage with files __init__.py and a.pyx. The cythonize call converts Extension("nsp.*", ["nsp/**/*.pyx"]) to Extension("m1.a", ["nsp/m1/a.c"]) (missing nsp.).
Code to reproduce the behaviour:
The following test is a slight modification of the existing build/cythonize_pep420_namespace.srctree test. Only the argument of cythonize is changed to have glob patterns. By doing this, the test fails.
# tests/build/cythonize_pep420_namespace_glob.srctree
PYTHON setup.py build_ext --inplace
PYTHON -c "import runner"
######## setup.py ########
from Cython.Build.Dependencies import cythonize
from distutils.core import setup, Extension
setup(
ext_modules=cythonize([
Extension("nsp.*", ["nsp/**/*.pyx"]),
]),
)
######## nsp/m1/__init__.py ########
######## nsp/m1/a.pyx ########
cdef class A:
pass
######## nsp/m1/a.pxd ########
cdef class A:
pass
######## nsp/m2/__init__.py ########
######## nsp/m2/b.pyx ########
from nsp.m1.a cimport A
from nsp.m3.c.d cimport D
cdef class B(A):
pass
######## nsp/m3/__init__.py ########
######## nsp/m3/c/d.pyx ########
cdef class D:
pass
######## nsp/m3/c/d.pxd ########
cdef class D:
pass
######## runner.py ########
from nsp.m1.a import A
from nsp.m2.b import B
from nsp.m3.c.d import D
a = A()
b = B()
Output:
$ python runtests.py -vv build.cythonize_pep420_namespace_glob
Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0]
Running tests against Cython 3.0.0a12.dev0 d888842ed8efec0d859dc5978900bb08161d7835 + uncommitted changes
Using Cython language level 2.
Test dependency not found: 'numpy'
Test dependency not found: 'pythran'
Test dependency found: 'setuptools.sandbox' version 65.5.1
Test dependency found: 'asyncio' version 3.10.6
Test dependency found: 'pstats' version 3.10.6
Test dependency found: 'posix' version 3.10.6
Test dependency found: 'array' version 3.10.6
Test dependency not found: 'Cython.Coverage'
Test dependency not found: 'Cython.Coverage'
Test dependency not found: 'IPython.testing.globalipapp'
Test dependency not found: 'jedi_BROKEN_AND_DISABLED'
Test dependency found: 'test.support' version 3.10.6
Backends: c,cpp
runTest (__main__.EndToEndTest)
[-1] End-to-end cythonize_pep420_namespace_glob ... [-1] ['/.../cython/env/bin/python', 'setup.py', 'build_ext', '--inplace']
Compiling nsp/m1/a.pyx because it changed.
Compiling nsp/m2/b.pyx because it changed.
Compiling nsp/m3/c/d.pyx because it changed.
[1/3] Cythonizing nsp/m1/a.pyx
[2/3] Cythonizing nsp/m2/b.pyx
[3/3] Cythonizing nsp/m3/c/d.pyx
running build_ext
building 'm1.a' extension
creating build
creating build/temp.linux-x86_64-cpython-310
creating build/temp.linux-x86_64-cpython-310/nsp
creating build/temp.linux-x86_64-cpython-310/nsp/m1
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/.../cython/env/include -I/usr/include/python3.10 -c nsp/m1/a.c -o build/temp.linux-x86_64-cpython-310/nsp/m1/a.o
creating build/lib.linux-x86_64-cpython-310
creating build/lib.linux-x86_64-cpython-310/m1
x86_64-linux-gnu-gcc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 build/temp.linux-x86_64-cpython-310/nsp/m1/a.o -L/usr/lib/x86_64-linux-gnu -o build/lib.linux-x86_64-cpython-310/m1/a.cpython-310-x86_64-linux-gnu.so
building 'm2.b' extension
creating build/temp.linux-x86_64-cpython-310/nsp/m2
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/.../cython/env/include -I/usr/include/python3.10 -c nsp/m2/b.c -o build/temp.linux-x86_64-cpython-310/nsp/m2/b.o
creating build/lib.linux-x86_64-cpython-310/m2
x86_64-linux-gnu-gcc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 build/temp.linux-x86_64-cpython-310/nsp/m2/b.o -L/usr/lib/x86_64-linux-gnu -o build/lib.linux-x86_64-cpython-310/m2/b.cpython-310-x86_64-linux-gnu.so
building 'd' extension
creating build/temp.linux-x86_64-cpython-310/nsp/m3
creating build/temp.linux-x86_64-cpython-310/nsp/m3/c
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/.../cython/env/include -I/usr/include/python3.10 -c nsp/m3/c/d.c -o build/temp.linux-x86_64-cpython-310/nsp/m3/c/d.o
x86_64-linux-gnu-gcc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 build/temp.linux-x86_64-cpython-310/nsp/m3/c/d.o -L/usr/lib/x86_64-linux-gnu -o build/lib.linux-x86_64-cpython-310/d.cpython-310-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-cpython-310/m1/a.cpython-310-x86_64-linux-gnu.so -> m1
/.../cython/Cython/Compiler/Main.py:370: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /.../cython/TEST_TMP/build/cythonize_pep420_namespace_glob/nsp/m1/a.pxd
tree = Parsing.p_module(s, pxd, full_module_name)
/.../cython/Cython/Compiler/Main.py:370: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /.../cython/TEST_TMP/build/cythonize_pep420_namespace_glob/nsp/m2/b.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
/.../cython/Cython/Compiler/Main.py:370: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /.../cython/TEST_TMP/build/cythonize_pep420_namespace_glob/nsp/m3/c/d.pxd
tree = Parsing.p_module(s, pxd, full_module_name)
error: could not create 'm1/a.cpython-310-x86_64-linux-gnu.so': No such file or directory
Final directory layout of 'cythonize_pep420_namespace_glob':
./setup.py
./runner.py
./nsp/m3/__init__.py
./nsp/m3/c/d.c
./nsp/m3/c/d.pyx
./nsp/m3/c/d.pxd
./nsp/m2/b.pyx
./nsp/m2/__init__.py
./nsp/m2/b.c
./nsp/m1/a.pyx
./nsp/m1/a.pxd
./nsp/m1/__init__.py
./nsp/m1/a.c
./build/temp.linux-x86_64-cpython-310/nsp/m3/c/d.o
./build/temp.linux-x86_64-cpython-310/nsp/m2/b.o
./build/temp.linux-x86_64-cpython-310/nsp/m1/a.o
./build/lib.linux-x86_64-cpython-310/d.cpython-310-x86_64-linux-gnu.so
./build/lib.linux-x86_64-cpython-310/m2/b.cpython-310-x86_64-linux-gnu.so
./build/lib.linux-x86_64-cpython-310/m1/a.cpython-310-x86_64-linux-gnu.so
FAIL
======================================================================
FAIL: runTest (__main__.EndToEndTest)
[-1] End-to-end cythonize_pep420_namespace_glob
----------------------------------------------------------------------
Traceback (most recent call last):
File "/.../cython/runtests.py", line 1994, in runTest
self.assertEqual(0, res, "non-zero exit status, last output was:\n%r\n-- stdout:%s\n-- stderr:%s\n" % (
AssertionError: 0 != 1 : non-zero exit status, last output was:
'/.../cython/env/bin/python setup.py build_ext --inplace'
-- stdout:Compiling nsp/m1/a.pyx because it changed.
Compiling nsp/m2/b.pyx because it changed.
Compiling nsp/m3/c/d.pyx because it changed.
[1/3] Cythonizing nsp/m1/a.pyx
[2/3] Cythonizing nsp/m2/b.pyx
[3/3] Cythonizing nsp/m3/c/d.pyx
running build_ext
building 'm1.a' extension
creating build
creating build/temp.linux-x86_64-cpython-310
creating build/temp.linux-x86_64-cpython-310/nsp
creating build/temp.linux-x86_64-cpython-310/nsp/m1
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/.../cython/env/include -I/usr/include/python3.10 -c nsp/m1/a.c -o build/temp.linux-x86_64-cpython-310/nsp/m1/a.o
creating build/lib.linux-x86_64-cpython-310
creating build/lib.linux-x86_64-cpython-310/m1
x86_64-linux-gnu-gcc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 build/temp.linux-x86_64-cpython-310/nsp/m1/a.o -L/usr/lib/x86_64-linux-gnu -o build/lib.linux-x86_64-cpython-310/m1/a.cpython-310-x86_64-linux-gnu.so
building 'm2.b' extension
creating build/temp.linux-x86_64-cpython-310/nsp/m2
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/.../cython/env/include -I/usr/include/python3.10 -c nsp/m2/b.c -o build/temp.linux-x86_64-cpython-310/nsp/m2/b.o
creating build/lib.linux-x86_64-cpython-310/m2
x86_64-linux-gnu-gcc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 build/temp.linux-x86_64-cpython-310/nsp/m2/b.o -L/usr/lib/x86_64-linux-gnu -o build/lib.linux-x86_64-cpython-310/m2/b.cpython-310-x86_64-linux-gnu.so
building 'd' extension
creating build/temp.linux-x86_64-cpython-310/nsp/m3
creating build/temp.linux-x86_64-cpython-310/nsp/m3/c
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/.../cython/env/include -I/usr/include/python3.10 -c nsp/m3/c/d.c -o build/temp.linux-x86_64-cpython-310/nsp/m3/c/d.o
x86_64-linux-gnu-gcc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 build/temp.linux-x86_64-cpython-310/nsp/m3/c/d.o -L/usr/lib/x86_64-linux-gnu -o build/lib.linux-x86_64-cpython-310/d.cpython-310-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-cpython-310/m1/a.cpython-310-x86_64-linux-gnu.so -> m1
-- stderr:/.../cython/Cython/Compiler/Main.py:370: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /.../cython/TEST_TMP/build/cythonize_pep420_namespace_glob/nsp/m1/a.pxd
tree = Parsing.p_module(s, pxd, full_module_name)
/.../cython/Cython/Compiler/Main.py:370: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /.../cython/TEST_TMP/build/cythonize_pep420_namespace_glob/nsp/m2/b.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
/.../cython/Cython/Compiler/Main.py:370: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /.../cython/TEST_TMP/build/cythonize_pep420_namespace_glob/nsp/m3/c/d.pxd
tree = Parsing.p_module(s, pxd, full_module_name)
error: could not create 'm1/a.cpython-310-x86_64-linux-gnu.so': No such file or directory
----------------------------------------------------------------------
Ran 1 test in 5.415s
FAILED (failures=1)
Most expensive pipeline stages:
Times:
etoe-build : 5.40 sec ( 1, 5.402 / run) - slowest: 'c:cythonize_pep420_namespace_glob(1)' (5.40s)
ALL DONE
Expected behaviour
The name of Extension objects return by cythonize should include namespace packages.
I'm not sure how a better implementation of fully_qualified_name should look like since every parent directory could be a namespace package.
Environment
OS: Linux
Python 3.10.6
Cython 3.0.0a11
Additional context
No response