Skip to content

Add xdsl benchmark #403

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Add xdsl benchmark #403

wants to merge 7 commits into from

Conversation

EdmundGoodman
Copy link

@EdmundGoodman EdmundGoodman commented Jul 31, 2025 •
edited
Loading

As discussed in the xDSL Zulip with @Fidget-Spinner, this PR adds a benchmark for constant folding with xDSL.

The new benchmark runs as expected locally:

$ pyperformance run --benchmark xdsl
Python benchmark suite 1.11.0


==================================================
( 1/1) creating venv for benchmark (xdsl)

(trying common venv first)
Installing requirements into the virtual environment /Users/edjg/.../pyperformance/venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b
# /Users/edjg/.../pyperformance/venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/bin/python -m pip install xdsl==0.46.0 pyperf==2.9.0
Requirement already satisfied: xdsl==0.46.0 in ./venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/lib/python3.12/site-packages (0.46.0)
Requirement already satisfied: pyperf==2.9.0 in ./venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/lib/python3.12/site-packages (2.9.0)
Requirement already satisfied: immutabledict<4.2.2 in ./venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/lib/python3.12/site-packages (from xdsl==0.46.0) (4.2.1)
Requirement already satisfied: typing-extensions<5,>=4.7 in ./venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/lib/python3.12/site-packages (from xdsl==0.46.0) (4.14.1)
Requirement already satisfied: ordered-set==4.1.0 in ./venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/lib/python3.12/site-packages (from xdsl==0.46.0) (4.1.0)
Requirement already satisfied: psutil>=5.9.0 in ./venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/lib/python3.12/site-packages (from pyperf==2.9.0) (7.0.0)

[notice] A new release of pip is available: 24.3.1 -> 25.1.1
[notice] To update, run: python -m pip install --upgrade pip
Installing requirements into the virtual environment /Users/edjg/.../pyperformance/venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b
# /Users/edjg/.../pyperformance/venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/bin/python -m pip install psutil
Requirement already satisfied: psutil in ./venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/lib/python3.12/site-packages (7.0.0)

[notice] A new release of pip is available: 24.3.1 -> 25.1.1
[notice] To update, run: python -m pip install --upgrade pip
# /Users/edjg/.../pyperformance/venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/bin/python -m pip freeze
immutabledict==4.2.1
ordered-set==4.1.0
psutil==7.0.0
pyperf==2.9.0
setuptools==80.9.0
typing_extensions==4.14.1
wheel==0.45.1
xdsl==0.46.0
# /Users/edjg/.../pyperformance/venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/bin/python -m pip freeze
immutabledict==4.2.1
ordered-set==4.1.0
psutil==7.0.0
pyperf==2.9.0
setuptools==80.9.0
typing_extensions==4.14.1
wheel==0.45.1
xdsl==0.46.0

[1/1] xdsl...
# /Users/edjg/.../pyperformance/venv/cpython3.12-30629e8c4b43-compat-64aaec271e6b/bin/python -u /Users/edjg/.../pyperformance/pyperformance/data-files/benchmarks/bm_xdsl/run_benchmark.py --output /var/folders/dw/qzkqk1s50cd9stxcw22_ymj80000gn/T/tmp4lrfn0vn --inherit-environ PYPERFORMANCE_RUNID
.....................
WARNING: the benchmark result may be unstable
* Not enough samples to get a stable result (95% certainly of less than 1% variation)

Try to rerun the benchmark with more runs, values and/or loops.
Run 'python -m pyperf system tune' command to reduce the system jitter.
Use pyperf stats, pyperf dump and pyperf hist to analyze results.
Use --quiet option to hide these warnings.

xdsl_constant_fold: Mean +- std dev: 24.3 ms +- 1.0 ms

Performance version: 1.11.0
Python version: 3.12.9 (64-bit)
Report on macOS-15.5-arm64-arm-64bit
Number of logical CPUs: 8
Start date: 2025-07-23 12:32:57.663942
End date: 2025-07-23 12:33:18.527806

### xdsl_constant_fold ###
Mean +- std dev: 24.3 ms +- 1.0 ms

@Fidget-Spinner
Copy link
Member

I talked to Edmund and the other xDSL authors and I want to upstream an xDSL benchmark to pyperformance.

The reason is that xDSL is an MLIR DSL framework written in pure Python. This represents something in the compiler-like space and is an interesting workload. We previously had a mypy benchmark relying in the pyston macrobenchmarks but that did a lot of IO and was too unstable so we took it out.

This in general is also a worst-case for the JIT, as it uses a lot of Python's dynamic features.

I will review and merge this assuming no objections.

@corona10 corona10 self-requested a review July 31, 2025 12:12
@EdmundGoodman EdmundGoodman mentioned this pull request Jul 31, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@corona10 corona10 Awaiting requested review from corona10

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@EdmundGoodman @Fidget-Spinner
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy