Skip to content

feat(gazelle): Gazelle plugin generates py_proto_library #3057

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 45 commits into from
Jul 15, 2025
Merged

feat(gazelle): Gazelle plugin generates py_proto_library #3057

merged 45 commits into from
Jul 15, 2025

Conversation

shaldengeki
Copy link
Contributor

@shaldengeki shaldengeki commented Jul 5, 2025 •
edited
Loading

Fixes #2994.

Please go over this with a fine-toothed comb! This is my first contribution to rules_python / the gazelle plugin, and while I've worked in Gazelle before, I'm pretty unfamiliar with the Python plugin's architecture.

This adds support in the Gazelle plugin for generating py_proto_library rules automatically, if there are any proto_library rules detected in a given package. We do this via a new Gazelle directive, python_generate_proto, which defaults to true, and controls whether these rules are generated.

See the tests in testdata/directive_python_generate_proto for examples.

By default, we source the py_proto_library rule from the @protobuf repository. I think this the intended long-term home of the rule? Users are expected to use gazelle:map_kind to change this if need be.

I haven't done anything here to support resolution of imports of py_proto_library. I think this is worth landing first, to save folks from having to maintain these by hand. But this should lay the foundation for resolving that in #1703.

shaldengeki added 13 commits July 2, 2025 23:18
@shaldengeki
[wip] Start work on generating py_proto_library, by creating a bunch …
c85c56f 
…of scaffolding & tests
@shaldengeki
Remove test cases that don't make any sense; py_proto_library runs on…
8b5a91b 
… the singular input proto
@shaldengeki
Add additional test fixtures required to run tests
d8f9b1a
@shaldengeki
[wip] Start working on generate.go
caf5d6e
@shaldengeki
Fix visibility in tests
b712696
@shaldengeki
Load py_proto_library from protobuf repository, since rules_proto's c…
3186761 
…opy is deprecated
@shaldengeki
Remove unneessary code
e073055
@shaldengeki
Delete some more unused stuff
aa9f4aa
@shaldengeki
Update CHANGELOG.md
b6df17b
@shaldengeki
Have a sudden change-of-heart and call it python_generate_proto, to a…
d794ac1 
…lign with the other directives
@shaldengeki
Update README.md
cad6849
@shaldengeki
Don't repeatedly initialize a variable
a0e4f5f
@shaldengeki
Add test enforcing that unnecessary rules are removed, and change log…
161cc51 
…ic to implement this
dougthor42
dougthor42 reviewed Jul 5, 2025
View reviewed changes
@shaldengeki shaldengeki mentioned this pull request Jul 5, 2025
Closed
@shaldengeki
Copy link
Contributor Author

While writing a test covering the bzlmod integration I ran into a small blocker, so I PRed a fix for that separately: #3060

@shaldengeki
Copy link
Contributor Author

Looks like the build failure is unrelated:

(16:04:21) ERROR: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/external/abseil-cpp~/absl/hash/BUILD.bazel:35:11: Compiling absl/hash/internal/hash.cc [for tool] failed: (Exit 1): gcc failed: error executing CppCompile command (from target @@abseil-cpp~//absl/hash:hash)
# ...
external/abseil-cpp~/absl/hash/internal/hash.h:66:10: fatal error: filesystem: No such file or directory

Happening, for example, in this PR, too: #3075

dougthor42
dougthor42 reviewed Jul 12, 2025
View reviewed changes
Copy link
Collaborator

@dougthor42 dougthor42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the details in README.md!

I wanted to add some tests that cover the bzlmod use-cases. I can't do that until gazelle gets upgraded, and it sounds like that's quite thorny right now. My inclination is to delete the bzlmod tests (they do pass, fwiw) and land this PR without 'em, to let you plug away at the upgrade at your own pace. Then after the upgrade's landed I can reinstate the tests. Does that sound reasonable to you?

Yes, the gazelle and rules_go upgrades are a PITA...

Can you elaborate on why you you can't add the bzlmod tests without bumping gazelle / rules_go / go version? From what I see in cdeafde, I don't see any reason why the bzlmod tests won't work.

@shaldengeki
Copy link
Contributor Author

shaldengeki commented Jul 12, 2025 •
edited
Loading

Can you elaborate on why you you can't add the bzlmod tests without bumping gazelle / rules_go / go version? From what I see in cdeafde, I don't see any reason why the bzlmod tests won't work.

Yep! Should've made this clear earlier. You can reproduce the errors I ran into by checking out 086c6aa (the last commit before I undid the upgrade & removed the tests), and undoing the upgrade:

$ bazel test //...
# ...
FAIL: //python:python_test_directive_python_generate_proto_bzlmod_protobuf_renamed (Exit 1) (see /home/charles/.cache/bazel/_bazel_charles/25da5a3f5a375d8b77b073daf0c17156/execroot/_main/bazel-out/k8-fastbuild/testlogs/python/python_test_directive_python_generate_proto_bzlmod_protobuf_renamed/test.log)
FAIL: //python:python_test_directive_python_generate_proto_bzlmod_protobuf (Exit 1) (see /home/charles/.cache/bazel/_bazel_charles/25da5a3f5a375d8b77b073daf0c17156/execroot/_main/bazel-out/k8-fastbuild/testlogs/python/python_test_directive_python_generate_proto_bzlmod_protobuf/test.log)
INFO: From Testing //python:python_test_directive_python_generate_proto_bzlmod_protobuf:
==================== Test output for //python:python_test_directive_python_generate_proto_bzlmod_protobuf:
--- FAIL: TestGazelleBinary (0.00s)
    --- FAIL: TestGazelleBinary/directive_python_generate_proto_bzlmod_protobuf (0.00s)
        python_test.go:169: expected gazelle exit code: 0
            got: 1
        python_test.go:179: expected gazelle stderr: 
            got: gazelle: -repo_root not specified, and WORKSPACE cannot be found: file does not exist
        python_test.go:144: "" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf/BUILD" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf/MODULE.bazel" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf/README.md" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf/foo.proto" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf/test.yaml" exists
FAIL
================================================================================
INFO: From Testing //python:python_test_directive_python_generate_proto_bzlmod_protobuf_renamed:
==================== Test output for //python:python_test_directive_python_generate_proto_bzlmod_protobuf_renamed:
--- FAIL: TestGazelleBinary (0.00s)
    --- FAIL: TestGazelleBinary/directive_python_generate_proto_bzlmod_protobuf_renamed (0.01s)
        python_test.go:169: expected gazelle exit code: 0
            got: 1
        python_test.go:179: expected gazelle stderr: 
            got: gazelle: -repo_root not specified, and WORKSPACE cannot be found: file does not exist
        python_test.go:144: "" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf_renamed" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf_renamed/BUILD" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf_renamed/MODULE.bazel" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf_renamed/README.md" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf_renamed/foo.proto" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf_renamed/test.yaml" exists
FAIL
================================================================================

which reads to me like Gazelle doesn't know about bzlmod, and is just looking for the WORKSPACE.

@shaldengeki
Copy link
Contributor Author

@dougthor42 one change I just made -- I double-checked how the golang & java extensions work, and they both generate e.g. foo_proto -> foo_(go|java)_proto, so I made our default here foo_py_proto, instead of foo_py_pb2 (which came from my own idiosyncratic naming in my personal repo).

I think that makes sense? And it'll make it just a tiny bit easier to resolve imports down the line.

@shaldengeki shaldengeki mentioned this pull request Jul 12, 2025
Open
@dougthor42
Copy link
Collaborator

dougthor42 commented Jul 12, 2025 via email •
edited
Loading

@shaldengeki
Copy link
Contributor Author

shaldengeki commented Jul 12, 2025 •
edited
Loading

Ah, dang it. I knew that came from somewhere.

It is... an unfortunate naming convention, I'll venture? I can put it back, but it's weird because a) it doesn't match other languages, and b) it doesn't even fit the actual naming of the generated file, which is like foo_pb2.py (while the convention is foo_py_pb2).

Let me know if you'd prefer I revert it.

dougthor42
dougthor42 reviewed Jul 12, 2025
View reviewed changes
Copy link
Collaborator

@dougthor42 dougthor42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me know if you'd prefer I revert it.

Yeah let's revert 3e0fdcc and we'll use #3081 to support naming things like foo_py_proto

Otherwise this LGTM. We just have to figure out that CI failure...

@dougthor42
Copy link
Collaborator

which reads to me like Gazelle doesn't know about bzlmod, and is just looking for the WORKSPACE.

Ah yes I see the error and repro'd it locally. Turns out you still need a dummy (empty) WORKSPACE file in the test dir. You can re-add those test cases if you'd like, as long as you're not bumping versions.

With the empty WORKSPACE file, things pass for me locally so hopefully they pass CI too.

@shaldengeki shaldengeki requested a review from dougthor42 July 13, 2025 02:22
dougthor42
dougthor42 approved these changes Jul 13, 2025
View reviewed changes
Copy link
Collaborator

@dougthor42 dougthor42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love it, thank you so much! 🎉

@dougthor42 dougthor42 mentioned this pull request Jul 13, 2025
Closed
@shaldengeki
Copy link
Contributor Author

shaldengeki commented Jul 13, 2025 •
edited
Loading

@dougthor42 Thanks for the review -- really excited to get this in!

I just noticed, as part of doing #3081, that I'm technically not following the convention you linked. I'm generating foo_proto_py_pb2 instead of foo_py_pb2. I just pushed a change that fixes this, both in the plugin and also in a bunch of examples where I noticed this as well. Hope that's OK?

@dougthor42 dougthor42 enabled auto-merge July 14, 2025 23:52
@dougthor42 dougthor42 added this pull request to the merge queue Jul 14, 2025
Merged via the queue into bazel-contrib:main with commit dd6550f Jul 15, 2025
3 checks passed
@shaldengeki shaldengeki mentioned this pull request Jul 15, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dougthor42 dougthor42 dougthor42 approved these changes

@aignas aignas Awaiting requested review from aignas aignas is a code owner

@rickeylev rickeylev Awaiting requested review from rickeylev rickeylev is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Have Gazelle generate py_proto_library targets
3 participants
@shaldengeki @dougthor42 @aignas
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy