-
-
Notifications
You must be signed in to change notification settings - Fork 610
fix(gazelle) Delete python targets with invalid srcs #3046
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
f0f7a1b
to
c573eb3
Compare
IMO this should be handled by properly supporting Specifically (emphasis mine):
We should not be deleting rules when running in I know it's a PITA - my users complain about it too - but there are too many legitimate cases where a |
@linzhp tagged me on this.
I don't know Python well enough to say whether that same behavior makes sense here. What does a The purpose of These days, rule sets are a lot more stable, so |
Ah, well I stand corrected then! Thanks for the background and info - I misunderstood the difference between
It's valid. For example, we use it to handle circular dependencies: # ########## START Autogenerated cycle targets and directives ##########
# See go/qos-doc-2024-24 for more info.
# cycle_c63218a5 (2 targets):
# gazelle:resolve py labrad.client //src:cycle_c63218a5
# gazelle:resolve py labrad.proto_over_labrad //src:cycle_c63218a5
py_library(
name = "cycle_c63218a5",
srcs = [],
imports = ["."],
tags = ["cycle"],
visibility = ["//visibility:public"],
deps = [
"//src/labrad:client",
"//src/labrad:proto_over_labrad",
],
) In the above case, Why do we put the cycles in So Gazelle deleting |
Currently, Gazelle is already cleaning up rules_python/gazelle/python/generate.go Lines 269 to 271 in 4e22d25
The situation we are trying to fix is the per-file There are two ways to move forward with this PR:
Let us know which approach you prefer. |
@dougthor42 friendly ping on this ☝️ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This causes nontrivial issues with our QuanutmAI repositories. Sorry, I'll have to block this until I can figure out a solution.
Can you try down-scoping to just py_binary
as mentioned in #3046 (comment)? That might to a long way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, I ran the "binary only" code on our codebase and no unexpected changes were made, yay!
Please update CHANGELOG.md with a short description of the PR.
gazelle/python/generate.go
Outdated
@@ -32,6 +32,7 @@ import ( | |||
"github.com/emirpasic/gods/sets/treeset" | |||
godsutils "github.com/emirpasic/gods/utils" | |||
|
|||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: remove
gazelle/python/generate.go
Outdated
@@ -485,6 +488,44 @@ func (py *Python) GenerateRules(args language.GenerateArgs) language.GenerateRes | |||
return result | |||
} | |||
|
|||
// getRulesWithInvalidSrcs checks existing Python rules in the BUILD file and return the rules with invalid srcs. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: please describe what "invalid srcs" means in the comment.
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: please remove copyright header - they're no longer needed.
@@ -0,0 +1,5 @@ | |||
py_binary( | |||
name = "keep_target_binary", | |||
srcs = ["//test/binary:__main__.py"], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: please refer to a binary found in the local dir (eg make keep_binary/foo.py
with and if __name__ == "__main__":
block in it)
@@ -0,0 +1 @@ | |||
workspace(name = "remove_invalid_binary") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
optional nit: this file can (should?) be empty.
gazelle/python/generate.go
Outdated
if existingRule.Kind() != pyBinaryKind { | ||
continue | ||
} | ||
allInvalidSrcs := true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would avoid double negative in the code. So instead of "allInvalidSrcs == false", it's easier to understand if we say "hasValidSrcs == true"
gazelle/python/generate.go
Outdated
for _, file := range args.RegularFiles { | ||
filesMap[file] = struct{}{} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, I should have caught this earlier, but I don't think we should read args.RegularFiles
here. There is a lot of filtering logic at the beginning of GenerateRules
that should also be applied here:
rules_python/gazelle/python/generate.go
Lines 116 to 132 in cab415d
if cfg.IgnoresFile(filepath.Base(f)) { | |
continue | |
} | |
ext := filepath.Ext(f) | |
if ext == ".py" { | |
pyFileNames.Add(f) | |
if !hasPyBinaryEntryPointFile && f == pyBinaryEntrypointFilename { | |
hasPyBinaryEntryPointFile = true | |
} else if !hasPyTestEntryPointFile && f == pyTestEntrypointFilename { | |
hasPyTestEntryPointFile = true | |
} else if f == conftestFilename { | |
hasConftestFile = true | |
} else if matchesAnyGlob(f, testFileGlobs) { | |
pyTestFilenames.Add(f) | |
} else { | |
pyLibraryFilenames.Add(f) | |
} |
For example, if a py_binary
only has excluded srcs, it should be cleaned up too.
There is also logic to collect py files from subdirs that is needed by project mode:
rules_python/gazelle/python/generate.go
Line 148 in cab415d
for _, d := range args.Subdirs { |
Since we scope this PR down to py_binary
only, the only "regular files" we care about are __main__.py
and mainModules
here:
rules_python/gazelle/python/generate.go
Line 236 in cab415d
allDeps, mainModules, annotations, err := parser.parse(srcs) |
The "GenFiles" and "isTarget` logic can stay, because they are not handled anywhere else
hasValidSrcs := true | ||
for _, src := range existingRule.AttrStrings("srcs") { | ||
if isTarget(src) { | ||
continue | ||
} | ||
if _, ok := filesMap[src]; ok { | ||
continue | ||
} | ||
hasValidSrcs = false | ||
break | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hasValidSrcs := true | |
for _, src := range existingRule.AttrStrings("srcs") { | |
if isTarget(src) { | |
continue | |
} | |
if _, ok := filesMap[src]; ok { | |
continue | |
} | |
hasValidSrcs = false | |
break | |
} | |
var hasValidSrcs bool | |
for _, src := range existingRule.AttrStrings("srcs") { | |
if isTarget(src) { | |
hasValidSrcs = true | |
break | |
} | |
if _, ok := filesMap[src]; ok { | |
hasValidSrcs = true | |
break | |
} | |
} |
It should be something like this.
return | ||
} | ||
filesMap := make(map[string]struct{}) | ||
for _, file := range args.RegularFiles { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about those python files in subdirs? Thoughts on my previous comment about reading mainModules
instead of args.RegularFiles
?
When running Gazelle, it generated the following target:
After
__main__.py
was deleted and the change committed, re-running Gazelle did not remove the file from the srcs list.This change introduces logic to check whether all entries in a Python target’s srcs attribute correspond to valid files. If none of them exist, the target is added to result.Empty to signal that it should be cleaned up. This cleanup behavior applies to when python_generation mode is package or file, as all
srcs
are expected to reside directly within the current directory.