Test each minor version of the CodeQL CLI in PR checks #932
Conversation
4ef2013 to
49c1e2f
Compare
Test the latest patch version of each minor version of the CodeQL CLI we support.
b1e12fa to
bdd647c
Compare
|
👍 from me, I like the direction of this a lot. |
|
Thanks for putting this up. A point of discussion is that we now have 250 checks that run for each PR. This change added 60 new checks. Each check is fairly small and runs reasonably quickly. Every new PR check we add will add 24 new checks (previously, they would add 18). I don't actually know if there are any real limits that we will hit if we keep on increasing the number of jobs at the rate. I do know that an external contributor who ran the actions checks in a fork reached a limit and had quite a few failing jobs in their fork. |
|
That's a really good point. There is an org-wide limit of 500 actions jobs running concurrently I believe, so maybe we need to refactor/throttle how we do these checks before merging this. |
|
OK...so we're not at the 500 checks yet, but we're inching towards it. I wonder if we can combine some of our PR checks into a single job. We'd still need to matrix them, but at least, in theory, we could run all the checks for actions version XYZ on windows in a single job. We'd need to change the workflow generator to produce that. |
|
Another option is reconsidering whether we have some checks that it'd be sufficient to run using a single bundle or a limited set of bundles, for instance the "Analyze: 'ref' and 'sha' from inputs" check. |
There was a problem hiding this comment.
OK...let's get this one in and then we can pursue one or both possibilities:
- run multiple checks in each job and keep the matrix
- avoid running the matrix for each check. Some checks will only run once.
I would definitely like the freedom to ensure that we can add new checks and integration tests without hitting any limits.
|
I would like to get this in, but I think it's worth doing the other cleanups first so we don't make merging PRs any more painful. I'm on vacation Friday and next week but will return to this when I get back. |
|
Closing as stale, will pick up when there's capacity. |
Opening this for discussion: this PR ensures that we test the latest patch version of each minor version of the CodeQL CLI we support. This gives us better assurance that the versions of the CodeQL CLI we claim to support function correctly with the Action. However, it does lead to an increase in CI jobs on a repo where the number of CI jobs is already becoming blocking due to concurrency limits.
Let's discuss whether we want this or not, and if we don't want it, let's come up with a principled way of deciding what versions of the CodeQL Bundle we test in PR checks.
Merge / deployment checklist