Skip to content

Add xdsl benchmark #403

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Add xdsl benchmark #403

wants to merge 7 commits into from

Conversation

EdmundGoodman
Copy link

@EdmundGoodman EdmundGoodman commented Jul 31, 2025 •
edited
Loading

As discussed in the xDSL Zulip with @Fidget-Spinner, this PR adds a benchmark for constant folding with xDSL.

The new benchmark runs as expected locally:

$ pyperformance run --benchmark xdsl
Python benchmark suite 1.11.0


==================================================
( 1/1) creating venv for benchmark (xdsl)

(trying common venv first)
Installing requirements into the virtual environment /Users/edjg/.../pyperformance/venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b
# /Users/edjg/.../pyperformance/venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/bin/python -m pip install xdsl==0.46.0 pyperf==2.9.0
Requirement already satisfied: xdsl==0.46.0 in ./venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/lib/python3.12/site-packages (0.46.0)
Requirement already satisfied: pyperf==2.9.0 in ./venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/lib/python3.12/site-packages (2.9.0)
Requirement already satisfied: immutabledict<4.2.2 in ./venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/lib/python3.12/site-packages (from xdsl==0.46.0) (4.2.1)
Requirement already satisfied: typing-extensions<5,>=4.7 in ./venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/lib/python3.12/site-packages (from xdsl==0.46.0) (4.14.1)
Requirement already satisfied: ordered-set==4.1.0 in ./venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/lib/python3.12/site-packages (from xdsl==0.46.0) (4.1.0)
Requirement already satisfied: psutil>=5.9.0 in ./venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/lib/python3.12/site-packages (from pyperf==2.9.0) (7.0.0)

[notice] A new release of pip is available: 24.3.1 -> 25.1.1
[notice] To update, run: python -m pip install --upgrade pip
Installing requirements into the virtual environment /Users/edjg/.../pyperformance/venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b
# /Users/edjg/.../pyperformance/venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/bin/python -m pip install psutil
Requirement already satisfied: psutil in ./venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/lib/python3.12/site-packages (7.0.0)

[notice] A new release of pip is available: 24.3.1 -> 25.1.1
[notice] To update, run: python -m pip install --upgrade pip
# /Users/edjg/.../pyperformance/venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/bin/python -m pip freeze
immutabledict==4.2.1
ordered-set==4.1.0
psutil==7.0.0
pyperf==2.9.0
setuptools==80.9.0
typing_extensions==4.14.1
wheel==0.45.1
xdsl==0.46.0
# /Users/edjg/.../pyperformance/venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/bin/python -m pip freeze
immutabledict==4.2.1
ordered-set==4.1.0
psutil==7.0.0
pyperf==2.9.0
setuptools==80.9.0
typing_extensions==4.14.1
wheel==0.45.1
xdsl==0.46.0

[1/1] xdsl...
# /Users/edjg/.../pyperformance/venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/bin/python -u /Users/edjg/.../pyperformance/pyperformance/data-files/benchmarks/bm_xdsl/run_benchmark.py --output /var/folders/dw/qzkqk1s50cd9stxcw22_ymj80000gn/T/tmp4lrfn0vn --inherit-environ PYPERFORMANCE_RUNID
.....................
WARNING: the benchmark result may be unstable
* Not enough samples to get a stable result (95% certainly of less than 1% variation)

Try to rerun the benchmark with more runs, values and/or loops.
Run 'python -m pyperf system tune' command to reduce the system jitter.
Use pyperf stats, pyperf dump and pyperf hist to analyze results.
Use --quiet option to hide these warnings.

xdsl_constant_fold: Mean +- std dev: 24.3 ms +- 1.0 ms

Performance version: 1.11.0
Python version: 3.12.9 (64-bit)
Report on macOS-15.5-arm64-arm-64bit
Number of logical CPUs: 8
Start date: 2025-07-23 12:32:57.663942
End date: 2025-07-23 12:33:18.527806

### xdsl_constant_fold ###
Mean +- std dev: 24.3 ms +- 1.0 ms

@Fidget-Spinner
Copy link
Member

I talked to Edmund and the other xDSL authors and I want to upstream an xDSL benchmark to pyperformance.

The reason is that xDSL is an MLIR DSL framework written in pure Python. This represents something in the compiler-like space and is an interesting workload. We previously had a mypy benchmark relying in the pyston macrobenchmarks but that did a lot of IO and was too unstable so we took it out.

This in general is also a worst-case for the JIT, as it uses a lot of Python's dynamic features.

I will review and merge this assuming no objections.

@corona10 corona10 self-requested a review July 31, 2025 12:12
@EdmundGoodman EdmundGoodman mentioned this pull request Jul 31, 2025
Closed
Fidget-Spinner
Fidget-Spinner approved these changes Aug 5, 2025
View reviewed changes
Copy link
Member

@Fidget-Spinner Fidget-Spinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Will wait for a few more days before merging. I assume you might be busy soon, so please don't mind if I push changes to your branch. Sorry!

@EdmundGoodman
Copy link
Author

LGTM. Will wait for a few more days before merging. I assume you might be busy soon, so please don't mind if I push changes to your branch. Sorry!

Sounds good -- I won't be able to do any work more on this after August 10th, so feel free to make any changes required. Thanks for your time discussing and reviewing this change!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Fidget-Spinner Fidget-Spinner Fidget-Spinner approved these changes

@corona10 corona10 Awaiting requested review from corona10

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@EdmundGoodman @Fidget-Spinner
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy