Skip to content

ZJIT: Start testing against /test/ruby/ and update all ZJIT related testing targets/docs #13791

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jul 15, 2025
Merged

ZJIT: Start testing against /test/ruby/ and update all ZJIT related testing targets/docs #13791

merged 2 commits into from
Jul 15, 2025

Conversation

st0012
Copy link
Member

@st0012 st0012 commented Jul 4, 2025 •
edited
Loading

  • make zjit-check has replaced the old make zjit-test-all, which runs ZJIT unit tests (Rust) and make test-all TESTS=test/ruby/test_zjit.rb
  • make zjit-test-all now runs ZJIT with tests under /test/ruby/ with some test cases excluded under test/.excludes-zjit/ due to known crash/panic/failure

There are 3 ways tests fail to execute with ZJIT:

  1. The behaviour is incorrect so the test failed
  2. Ruby crashes (due to assertion failure...etc.)
  3. ZJIT panicked

We can avoid 1 and 2 by excluding those specific cases with exclude(:test_method_name, "reason"). But this doesn't avoid ZJIT panics.
"I think" this is because the exclusion is done through removing methods, so if ZJIT panics due to compiling certain iseqs, it'd panic before those methods even get removed?

@matzbot matzbot requested a review from a team July 4, 2025 18:34
@st0012 st0012 mentioned this pull request Jul 4, 2025
Closed
@st0012 st0012 marked this pull request as draft July 5, 2025 14:13
@k0kubun
Copy link
Member

k0kubun commented Jul 7, 2025 •
edited
Loading

zjit-test | zjit-test-rust

I'm reluctant to make the name make zjit-test longer. Unlike YJIT, we can't use the bare cargo test for ZJIT, so we have to type and run the command a lot. It seems pretty fair that cargo test for ZJIT is called zjit-test because every other test is just part of regular CRuby tests.

zjit-test-ruby

Running a single test file in test-all is something you have to do a lot regardless of ZJIT, so I'm not sure if it's worth a new make target. But if you want to give a name to test-all TESTS='test/ruby/test_zjit.rb', maybe repurpose make zjit-test-all to it because it's a test case in test-all for ZJIT?

zjit-test-all | zjit-test-suite

I admit that zjit-test-all was not a good name, but if we're renaming it, how about make zjit-check? make check runs all make targets for testing CRuby, so make zjit-check can be a ZJIT version of it.

zjit-test-ruby-all

It's confusing to insert "ruby" in between "test-all". test-all is a very well-known make target name (probably one of the most frequently used names), so calling it "test-ruby-all" makes it hard to recognize what it is.

I don't come up with a good name for this other than make zjit-test-all, but I think you're just making too many make targets. I'd suggest dropping either "zjit-test-ruby" or "zjit-test-ruby-all". At some point we'll just be able to use the actual make test-all for this, so maybe drop zjit-test-ruby-all (or both)?

If you need a place to maintain a set of skipped test cases for ZJIT, you might want to try "excludes" directory. Look at test/.excludes* for example. We could have test/.excludes-zjit and use that.

@st0012
Copy link
Member Author

st0012 commented Jul 9, 2025

@k0kubun Thanks for the suggestions! I didn't know about .excludes* folders but it's indeed better.
I'll get back to this after Shopify#620 is resolved as it fails a lot tests right now.

@st0012 st0012 force-pushed the zjit-test-against-ruby-tests branch 2 times, most recently from 3bad873 to cf88fc9 Compare July 14, 2025 23:48
@launchable-app Launchable app

This comment has been minimized.

Sign in to view
@st0012 st0012 force-pushed the zjit-test-against-ruby-tests branch 4 times, most recently from 794fc5f to d790fc7 Compare July 15, 2025 21:03
st0012
st0012 commented Jul 15, 2025
View reviewed changes
.github/workflows/zjit-macos.yml
@@ -38,9 +38,8 @@ jobs:
- test_task: 'ruby' # build test for combo build
configure: '--enable-yjit --enable-zjit'

- test_task: 'test-all'
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if it's worth testing test_zjit by itself on CI when we'll be running it in zjit-test-all anyway. Also the test-all name will be confusing along with zjit-test-all.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As long as it is running in CI somehow.

Although it may give more informative and early warnings than other test suites because it's more targeted

test/.excludes-zjit/TestProc.rb
# exclude(:test_proc_args_pos_rest_block, 'Test crashes with ZJIT')
# exclude(:test_proc_args_rest_post_block, 'Test crashes with ZJIT')

# exclude(:test_binding_receiver, 'Test fails with ZJIT')
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As explained in description, I need to exclude all test methods to avoid the ZJIT panic. But I also identified these tests that'd also fail/crash even without the panic.

@st0012 st0012 marked this pull request as ready for review July 15, 2025 21:25
@st0012 st0012 force-pushed the zjit-test-against-ruby-tests branch from d790fc7 to 953f709 Compare July 15, 2025 21:27
k0kubun
k0kubun reviewed Jul 15, 2025
View reviewed changes
k0kubun
k0kubun approved these changes Jul 15, 2025
View reviewed changes
Copy link
Member

@k0kubun k0kubun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job 👍 Let's merge it once the CI passes.

@st0012
ZJIT: Update test targets and documentation
80495ea 
- Rename `zjit-test-all` to `zjit-check`
- Add `zjit-test-all` target to zjit.mk to run all Ruby tests with ZJIT enabled
  excluding known failing tests
- Update documentation and CI workflow to reflect the new targets
@st0012 st0012 force-pushed the zjit-test-against-ruby-tests branch from 953f709 to 80495ea Compare July 15, 2025 21:32
@k0kubun k0kubun merged commit 5723945 into ruby:master Jul 15, 2025
79 of 82 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tekknolagi tekknolagi tekknolagi left review comments

@k0kubun k0kubun k0kubun approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@st0012 @k0kubun @tekknolagi
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy