Migrate CI build-doc to Meson #39973

tobiasdiez · 2025-04-19T09:53:04Z

Make it possible to build the documentation with meson via

meson compile -C builddir doc-html

For this a couple of changes was necessary in the docbuilder:

Make sage_docbuild independent of sage so that meson can use sage_docbuild during config time to construct all the docbuild targets (otherwise one needs to first install sage, and then could configure the docbuild)
Properly handle input and output dirs for the docbuild, without relying on magic sage env variables

📝 Checklist

The title is concise and informative.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation and checked the documentation preview.

⌛ Dependencies

user202729 · 2025-04-19T10:16:41Z

Looks like some file permission goes wrong? (symbolic link get converted to normal file?)

user202729 · 2025-04-19T10:18:41Z

src/sage_docbuild/builders.py

+    docs = [doc[1] for doc in sorted(documents)]
+    # Put the bibliography first, because it needs to be built first:
+    docs.remove(Path('reference/references'))
+    docs.insert(0, Path('reference/references'))


Why is such a large change necessary in this file?

To be honest, I got carried a way a bit: the strictly necessary change is to remove mentions of import sage, then I used the chance to use pathlib, and to catch all those paths vs strings issues added typing info. Oh, and there were a few old deprecated stuff in there which was easier to remove than to migrate to pathlib 🦊

…tory-reference

github-actions · 2025-05-17T03:51:02Z

Documentation preview for this PR (built with commit 3408887; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

tobiasdiez · 2025-05-17T10:28:32Z

Doc build is working now, and the changes relative to the last develop build seem to be minimal and mostly due to using a newer sphinx version now.

Marking it as blocker since currently the docbuild CI for other PRs is completely failing.

dimpase · 2025-05-17T15:56:44Z

https://github.com/sagemath/sage/actions/runs/15081532937/job/42398821050#step:9:1

fails while building docs the old way - is it normal?

tobiasdiez · 2025-05-17T16:40:21Z

It fails with

ImportError: libSingular-4.4.0.so: cannot open shared object file: No such file or directory

https://github.com/sagemath/sage/actions/runs/15081532937/job/42398821050#step:9:3982 so this is the same problem as for the other PRs.

dimpase · 2025-05-17T23:56:18Z

no, my question was - why does that CI run build docs the old way?

tobiasdiez · 2025-06-26T11:44:39Z

Thanks for the review @dimpase. I've created new issues for the problems you mentioned.

@user202729 This is fixed in #40071

sagemathgh-39973: Migrate CI build-doc to Meson       Make it possible to build the documentation with meson via ``` meson compile -C builddir doc-html ``` For this a couple of changes was necessary in the docbuilder: - Make `sage_docbuild` independent of `sage` so that meson can use `sage_docbuild` during config time to construct all the docbuild targets (otherwise one needs to first install sage, and then could configure the docbuild) - Properly handle input and output dirs for the docbuild, without relying on magic sage env variables ### 📝 Checklist  - [ ] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    URL: sagemath#39973 Reported by: Tobias Diez Reviewer(s): Dima Pasechnik, Tobias Diez, user202729

antonio-rojas · 2025-07-07T18:15:39Z

I can't figure out how to build the documentation now. I used to do

python -m sage_docbuild --no-pdf-links --mathjax all html

which no longer works since the all option was removed here. If I do

meson setup sage build
meson compile -C build doc-html

then I get errors such as

FAILED: src/doc/inventoryreferences 
/usr/bin/python /build/sagemath-doc/src/sage/src/build-docs.py --no-pdf-links reference/references inventory -o src/doc --source /build/sagemath-doc/src/sage/src/doc
Error building the documentation.
Traceback (most recent call last):
  File "/build/sagemath-doc/src/sage/src/build-docs.py", line 4, in <module>
    main()
    ~~~~^^
  File "/build/sagemath-doc/src/sage/src/sage_docbuild/__main__.py", line 545, in main
    import sage.all  # TODO: Remove once all modules can be imported independently  # noqa: F401
    ^^^^^^^^^^^^^^^
  File "/build/sagemath-doc/src/sage/src/sage/all.py", line 64, in <module>
    from sage.all__sagemath_repl import *
  File "/build/sagemath-doc/src/sage/src/sage/all__sagemath_repl.py", line 108, in <module>
    from sage.all__sagemath_objects import *
  File "/build/sagemath-doc/src/sage/src/sage/all__sagemath_objects.py", line 17, in <module>
    from sage.misc.all__sagemath_objects import *
  File "/build/sagemath-doc/src/sage/src/sage/misc/all__sagemath_objects.py", line 4, in <module>
    import sage.structure.all   # to break a cyclic import
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/build/sagemath-doc/src/sage/src/sage/structure/__init__.py", line 3, in <module>
    import sage.structure.element
ModuleNotFoundError: No module named 'sage.structure.element'

since sage hasn't been compiled. Even if I compile it with

meson setup sage build
meson compile -C build
meson compile -C build doc-html

I get the same error, since the compiled modules aren't available in the python path. The closest I get is if I delete the sage source dir and replace it with a symlink to the installed sage python module, but then I get cryptic errors such as

FAILED: src/doc/inventorymisc 
/usr/bin/python /build/sagemath-doc/src/sage/src/build-docs.py --no-pdf-links reference/misc inventory -o src/doc --source /build/sagemath-doc/src/sage/src/doc
[misc     ] building [inventory]: targets for 100 source files that are out of date
[misc     ] updating environment: [new config] 100 added, 0 changed, 0 removed
[misc     ] /usr/lib/python3.13/site-packages/sphinx/events.py:394: DeprecationWarning:
[misc     ] Importing benchmark from here is deprecated; please use "from sage.misc.benchmark import benchmark" instead.
[misc     ] See https://github.com/sagemath/sage/issues/34259 for details.
[misc     ]   repr_args = repr(args)
[misc     ] WARNING: error while formatting arguments for sage.misc.c3_controlled.HierarchyElement.bases:  [autodoc]
[misc     ] WARNING: error while formatting arguments for sage.misc.c3_controlled.HierarchyElement.cls:  [autodoc]
[misc     ] WARNING: error while formatting arguments for sage.misc.c3_controlled.HierarchyElement.mro:  [autodoc]
[misc     ] WARNING: error while formatting arguments for sage.misc.c3_controlled.HierarchyElement.mro_controlled:  [autodoc]
[misc     ] WARNING: error while formatting arguments for sage.misc.c3_controlled.HierarchyElement.mro_standard:  [autodoc]
[misc     ] The inventory file is in src/doc/inventory/en/reference/misc.
Error building the documentation.
Traceback (most recent call last):
  File "/build/sagemath-doc/src/sage/src/build-docs.py", line 4, in <module>
    main()
    ~~~~^^
  File "/build/sagemath-doc/src/sage/src/sage_docbuild/__main__.py", line 548, in main
    build()
    ~~~~~^^
  File "/build/sagemath-doc/src/sage/src/sage_docbuild/builders.py", line 645, in _wrapper
    getattr(DocBuilder, build_type)(self, *args, **kwds)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/build/sagemath-doc/src/sage/src/sage_docbuild/builders.py", line 136, in f
    runsphinx()
    ~~~~~~~~~^^
  File "/build/sagemath-doc/src/sage/src/sage_docbuild/sphinxbuild.py", line 324, in runsphinx
    sys.stderr.raise_errors()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/build/sagemath-doc/src/sage/src/sage_docbuild/sphinxbuild.py", line 255, in raise_errors
    raise OSError(self._error)
OSError: WARNING: error while formatting arguments for sage.misc.c3_controlled.HierarchyElement.bases:  [autodoc]

So how do I do it?

dimpase · 2025-07-08T04:58:37Z

perhaps it's better to look at #39030, which is basically done, and converted sagelib build and docs build to using meson

tobiasdiez · 2025-07-08T08:46:26Z

Sorry for the inconvenience!
The easiest solution is to really install sage in the active python prefix (i.e. pip install - but meson install should work as well). Then I still got the same error as you in certain situations because the src/sage folder was shadowing the installed sage. This is fixed in #40379 by a bit of sys.path shenanigan.

Looking forward, I was wondering if it's better to move the src/sage_docbuild folder somewhere (tools?) and/or src/doc to <root>/doc (which seems to be the most common convention). Any ideas on that?

antonio-rojas · 2025-07-08T14:44:28Z

Sorry for the inconvenience! The easiest solution is to really install sage in the active python prefix (i.e. pip install - but meson install should work as well). Then I still got the same error as you in certain situations because the src/sage folder was shadowing the installed sage. This is fixed in #40379 by a bit of sys.path shenanigan.

Thanks, that helps. The path shuffing from #40379 allows the build to proceed, but then I hit the second error:

FAILED: src/doc/inventorysets 
/usr/bin/python /build/sagemath-doc-git/src/sage/src/build-docs.py --no-pdf-links reference/sets inventory -o src/doc --source /build/sagemath-doc-git/src/sage/src/doc
[sets     ] building [inventory]: targets for 19 source files that are out of date
[sets     ] updating environment: [new config] 19 added, 0 changed, 0 removed
[sets     ] WARNING: error while formatting arguments for sage.sets.recursively_enumerated_set.RecursivelyEnumeratedSet_forest.children:  [autodoc]
[sets     ] The inventory file is in src/doc/inventory/en/reference/sets.
Error building the documentation.
Traceback (most recent call last):
  File "/build/sagemath-doc-git/src/sage/src/build-docs.py", line 11, in <module>
    main()
    ~~~~^^
  File "/build/sagemath-doc-git/src/sage/src/sage_docbuild/__main__.py", line 548, in main
    build()
    ~~~~~^^
  File "/build/sagemath-doc-git/src/sage/src/sage_docbuild/builders.py", line 645, in _wrapper
    getattr(DocBuilder, build_type)(self, *args, **kwds)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/build/sagemath-doc-git/src/sage/src/sage_docbuild/builders.py", line 136, in f
    runsphinx()
    ~~~~~~~~~^^
  File "/build/sagemath-doc-git/src/sage/src/sage_docbuild/sphinxbuild.py", line 324, in runsphinx
    sys.stderr.raise_errors()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/build/sagemath-doc-git/src/sage/src/sage_docbuild/sphinxbuild.py", line 255, in raise_errors
    raise OSError(self._error)
OSError: WARNING: error while formatting arguments for sage.sets.recursively_enumerated_set.RecursivelyEnumeratedSet_forest.children:  [autodoc]

and the same for a few other files. Adding that warning to the ignore list in sphinxbuild.py allows the build to finish successfully. But I did not get that warning with the previous build procedure (and the same sphinx version), and I don't know if it has any consequence for the doc output.

Another issue is that the compiled docs are not using mathjax (not even the online version), they look like this:

antonio-rojas · 2025-07-08T18:14:07Z

The mathjax issue is not caused by this change, but by the fact that MATHJAX_DIR is broken in meson outside sage-the-distro because of clashes between env.py and the meson-generated config.py.

#40388 should fix this and other potential future issues. In the long term we should fix the env.py/config.py duplication and centralize the variable settings and fallbacks in a single place.

tobiasdiez · 2025-07-10T08:38:48Z

but then I hit the second error:

Sorry, I've never seen that error before and it also looks a bit like an internal sphinx error.

antonio-rojas · 2025-07-10T14:30:07Z

but then I hit the second error:

Sorry, I've never seen that error before and it also looks a bit like an internal sphinx error.

I figured out the issue: this is (again) caused by missing pyx sources. Specifically, if the cachefunc.pyx source file where the @cached_method decorator is defined is not accessible, then autodoc chokes when parsing any method decorated with it.

I guess this is another point in favor of installing pyx files.

sagemathgh-39030: Use meson in sage-the-distro       Replace the old setuptools-based build by the new meson-based one in sage-the-distro. Delete most of the old stuff that is no longer needed now. ### 📝 Checklist  - [ ] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies - sagemath#40133 - sagemath#39973    URL: sagemath#39030 Reported by: Tobias Diez Reviewer(s): Dima Pasechnik

sagemathgh-40379: Fix docbuilding on release tag       After sagemath#39973, some necessary steps during the docbuild on release tags are no longer run (see https://github.com/sagemath/sage/actions/runs/161 00707903/job/45429461356). Sorry - that was my fault. This is fixed here. Test run: https://github.com/tobiasdiez/sage/actions/runs/16323033505/jo b/46105684396 ### 📝 Checklist  - [ ] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    - sagemath#40383 URL: sagemath#40379 Reported by: Tobias Diez Reviewer(s): Dima Pasechnik

user202729 reviewed Apr 19, 2025

View reviewed changes

Meson: add building of documentation as target

10fea7f

tobiasdiez force-pushed the meson-docs branch from c90e95d to 10fea7f Compare April 19, 2025 10:27

tobiasdiez added 11 commits April 20, 2025 16:09

Fix meson doc build

d7c3b6e

Fix various issues

a805a97

Fix ci

e881cf1

Fix path to sage_docbuild

af4cb32

Add shell specification for storing old documentation

0470c2f

Try with editable install

199be4e

Fix directory creation

dd44ea7

add copybutton conda pkg

5ee3c51

Work with absolute paths everywhere

7550538

Add dependency for 'reference_top' on other reference docs

230fd8f

Exclude 'reference_top' from the list of other documents in doc-inven…

ee77f03

…tory-reference

tobiasdiez mentioned this pull request May 10, 2025

instaling sage cschwan/sage-on-gentoo#804

Open

tobiasdiez added 4 commits May 16, 2025 14:19

Merge remote-tracking branch 'origin/develop' into meson-docs

b07c0a6

Improve dependency handling

f71e282

Add dependency of 'top' docs on all other references

327a7bb

Use ccache to build sage

f6d186b

Don't create neasted doc directories

5a3c014

tobiasdiez marked this pull request as ready for review May 17, 2025 10:23

github-actions bot added the s: needs review label May 17, 2025

tobiasdiez requested a review from dimpase May 17, 2025 10:28

tobiasdiez added the p: blocker / 1 label May 17, 2025

This was referenced Jun 26, 2025

Meson: update of dependency results in linker error #40307

Open

meson compile doc-pdf fails in all-pdf #40308

Open

Merge branch 'develop' into meson-docs

3408887

github-actions bot added s: needs review and removed s: positive review labels Jun 26, 2025

tobiasdiez added s: positive review and removed s: needs review labels Jun 26, 2025

tobiasdiez mentioned this pull request Jul 6, 2025

Use meson in sage-the-distro #39030

Open

5 tasks

vbraun merged commit 7b79f0e into sagemath:develop Jul 6, 2025
22 of 27 checks passed

github-actions bot removed the s: positive review label Jul 6, 2025

tobiasdiez mentioned this pull request Jul 6, 2025

Fix docbuilding on release tag #40379

Merged

5 tasks

antonio-rojas mentioned this pull request Jul 7, 2025

rebase sage_autodoc with sphinx-8.2.3 and update sphinx to same. #39737

Closed

2 tasks

tobiasdiez deleted the meson-docs branch July 8, 2025 08:46

Uh oh!

Migrate CI build-doc to Meson #39973

Migrate CI build-doc to Meson #39973

Uh oh!

Conversation

tobiasdiez commented Apr 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Checklist

⌛ Dependencies

Uh oh!

user202729 commented Apr 19, 2025

Uh oh!

user202729 Apr 19, 2025

Choose a reason for hiding this comment

Uh oh!

tobiasdiez Apr 19, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tobiasdiez commented May 17, 2025

Uh oh!

dimpase commented May 17, 2025

Uh oh!

tobiasdiez commented May 17, 2025

Uh oh!

dimpase commented May 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tobiasdiez commented Jun 26, 2025

Uh oh!

Uh oh!

antonio-rojas commented Jul 7, 2025

Uh oh!

dimpase commented Jul 8, 2025

Uh oh!

tobiasdiez commented Jul 8, 2025

Uh oh!

antonio-rojas commented Jul 8, 2025

Uh oh!

antonio-rojas commented Jul 8, 2025

Uh oh!

tobiasdiez commented Jul 10, 2025

Uh oh!

antonio-rojas commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

tobiasdiez commented Apr 19, 2025 •

edited

Loading

github-actions bot commented May 17, 2025 •

edited

Loading

dimpase commented May 17, 2025 •

edited

Loading

antonio-rojas commented Jul 10, 2025 •

edited

Loading