Skip to content

DOC: Set default as -j 1 for spin docs and move -W to SPHINXOPTS #26478

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 20, 2024

Conversation

bmwoodruff
Copy link
Member

This commit tries to solve two problems.

  • The pydata_sphinx_theme extension warning can be avoided by setting the default --jobs to 1. This matches spin test.
  • The -W option is currently hardcoded in ALLSPHINXOPTS and impossible to override. This commit moves -W to SPHINXOPTS which allows a local machine to remove -W if needed, as described in the documentation. This adds to the discussion in PR DOC: Bump pydata-sphinx-theme version #26125.

[skip actions] [skip azp] [skip cirrus]

Context

While trying to build a workflow that incorporates spin docs, I encountered two issues. First, I noticed that the the following suggestion in the spin documentation was invalid.

numpy/.spin/cmds.py

Lines 142 to 145 in fba4fc1

By default, SPHINXOPTS="-W", raising errors on warnings.
To build without raising on warnings:
SPHINXOPTS="" spin docs

The fact that -W is hard coded in ALLSPHINXOPTS makes it impossible to remove without changing the makefile.

The second issue is the known error message

WARNING: the pydata_sphinx_theme extension is not safe for parallel writing
WARNING: doing serial write

This was fixed for those using make by removing -j2 from SPHINXOPTS. The circle build uses make but new users are directed to use spin docs. When someone runs SPHINXOPTS="-n" make -e html everything works fine. When someone runs spin docs, the generated doc build command is

LANG=C sphinx-build -b html -WT --keep-going -d build/doctrees  -j auto source build/html

The problem is -j auto. Running SPHINXOPTS="-j 1" spin docs generates

LANG=C sphinx-build -b html -WT --keep-going -d build/doctrees  -j 1 -j auto source build/html

The problem remains. Using spin docs -j 1 instead of SPHINXOPTS="-j 1" spin docs works. Discovering this fix was difficult for a newcomer, as the rest of the examples in the codebase model changing SPHINXOPTS.

I set up 7 different codespace environments, and sometimes the docs would build without errors, while other times they would fail. My best guess now is that this may have to do with how many cores -j auto was able to access on each different codespace.

Further Discussion

I'd love to change something to help newcomers so they don't have to question why their doc build gives an error message (that they can ignore). Here's some options.

  • Swapping the default to 1, and later updating it back to auto when an upstream fix is found, would simplify things for newcomers. This PR follows this option. I purposefully matched the default and help from spin test.
  • Alternately, we could add a note in the docs wherever spin docs appears that adds a comment to use spin docs -j 1 if they encounter a warning. IMHO, this makes it harder for newcomers to contribute. If anything, this PR will provide a record for people to find a fix.
  • We could test if the pydata_sphinx_theme extension is safe for parallel writing (they fixed things upstream), and then set the default based on that. This muddies up cmds.py, but does mean the change to auto will auto happen.

As for the -W option change, I understand why we want the doc build to throw an error on numpy/numpy. Moving -W to SPHINXOPTS allows both upstream and local repos to set the option as needed.

  • I left doc/neps/makefile alone. The -W command is still in ALLSPHINXOPTS. I can update that file as well, if wanted.

This commit tries to solve two problems.
- The pydata_sphinx_theme extension warning can be avoided by setting
the default `--jobs` to `1`. This matches `spin test`.
- The -W option is currently hardcoded in ALLSPHINXOPTS and
impossible to override. This commit moves `-W` to SPHINXOPTS which
allows a local machine to remove -W if needed, as described in the
documentation. This adds to the discussion in PR numpy#26125.

[skip actions] [skip azp] [skip cirrus]
@eagunn
Copy link
Contributor

eagunn commented May 20, 2024

As a newcomer to building the docs, I strongly endorse this PR, both in its general aims and in its specific fixes.

I just spent several hours across three days trying to get spin docs to run to completion without an error. I determined fairly early that the pydata_sphinx_theme extension warning was "just a warning" and that all of the doc files were being correctly generated. But I was trying to play by the rules and either a) eliminate the warning or b) "correctly" alter the treat-warnings-as-errors option.

I never hit upon spin docs -j 1 (to eliminate the warnings). I did try SPHINXOPTS="" spin docs (to eliminate the terminal error message) which, as @bmwoodruff documents, did not work as indicated. In the end, I was able to pull the make command out of the spin build log and run it from the commandline, outside of spin, where it worked fine. I finally decided, reluctantly, to simply ignore the warnings and the terminal error and start making my changes to docs.

Please accept the default -j 1 to fix the build process so it works consistently across all environments. (Mine is WSL Ubuntu where it generated warnings always. Having it generate warnings randomly in codespaces is worse; that's the dependable environment recommended for newbies so they don't have to worry about setup.)

I also agree completely with:

As for the -W option change, I understand why we want the doc build to throw an error on numpy/numpy. Moving -W to SPHINXOPTS allows both upstream and local repos to set the option as needed.

I'm completely new to this project but I've been on teams where warnings were ignored and we always paid the price eventually. Warnings-as-errors is the right default, but individuals should be able to successfully override the -W setting as needed during their local development.

@charris charris merged commit 59d92bf into numpy:main May 20, 2024
5 checks passed
@charris
Copy link
Member

charris commented May 20, 2024

Thanks @bmwoodruff .

@bmwoodruff
Copy link
Member Author

@charris That was fast!

My only worry with this PR is that it could have unintended consequences for some local projects that don't realize the -W flag got moved to SPHINXOPTS. Workflows won't "break", but could start accepting warnings in the future. As it's just a docs build, hopefully that won't be too problematic.

Thanks for looking at this! It was a fun couple days of hunting, and I did use AI to help. I'll make sure my writeup about how I used AI appears below, once I'm finished writing it up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy