Skip to content

gh-135953: Profile a module or script with sampling profiler #136777

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
+466 −39
Open

gh-135953: Profile a module or script with sampling profiler #136777

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1bf048a
Profile a module or script with sampling profiler
lkollar Jul 15, 2025
6ad8ad7
Wait for interpreter to initialize in subprocess
lkollar Aug 3, 2025
3e325e8
Remove unnecessary argument check
lkollar Aug 3, 2025
dc9aad3
Handle the process runs shorter than expected
lkollar Aug 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 106 additions & 27 deletions Lib/profile/sample.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
import _remote_debugging
import os
import pstats
import subprocess
import statistics
import sys
import sysconfig
Expand All @@ -13,6 +14,9 @@
from .stack_collector import CollapsedStackCollector

FREE_THREADED_BUILD = sysconfig.get_config_var("Py_GIL_DISABLED") is not None
MAX_STARTUP_ATTEMPTS = 5
STARTUP_RETRY_DELAY_SECONDS = 0.1
Comment on lines 16 to +18
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps make these private constants (leading underscore)?



class SampleProfiler:
def __init__(self, pid, sample_interval_usec, all_threads):
Expand Down Expand Up @@ -53,6 +57,14 @@ def sample(self, collector, duration_sec=10):
break
except (RuntimeError, UnicodeDecodeError, MemoryError, OSError):
errors += 1
except ValueError as e:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should add an extra condition here and only do this if we are the ones who spawned the subprocess. Maybe this is fine as-is. What do you think @pablogsal?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uf, I am not a fun of this as this may be hiding errors in collector.collect(stack_frames). We need some better signal here

# Process is waiting to be reaped by the parent (us)
if self._is_process_running():
duration_sec = (
current_time - start_time
)
Comment on lines +62 to +65
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line length is fine here:

Suggested change
if self._is_process_running():
duration_sec = (
current_time - start_time
)
if self._is_process_running():
duration_sec = current_time - start_time

break
raise e from None
except Exception as e:
if not self._is_process_running():
break
Expand Down Expand Up @@ -537,51 +549,95 @@ def _validate_collapsed_format_args(args, parser):
args.outfile = f"collapsed.{args.pid}.txt"


def wait_for_process_and_sample(process_pid, sort_value, args):
for attempt in range(MAX_STARTUP_ATTEMPTS):
try:
sample(
process_pid,
sort=sort_value,
sample_interval_usec=args.interval,
duration_sec=args.duration,
filename=args.outfile,
all_threads=args.all_threads,
limit=args.limit,
show_summary=not args.no_summary,
output_format=args.format,
realtime_stats=args.realtime_stats,
)
break
except RuntimeError:
if attempt < MAX_STARTUP_ATTEMPTS - 1:
print("Waiting for process to start...")
time.sleep(STARTUP_RETRY_DELAY_SECONDS)
else:
raise RuntimeError("Process failed to start after maximum retries")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
raise RuntimeError("Process failed to start after maximum retries")
raise RuntimeError("Process failed to start after maximum retries") from None



def main():
# Create the main parser
parser = argparse.ArgumentParser(
description=(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be easier to put this description in a single module-level multi-line string?

"Sample a process's stack frames and generate profiling data.\n"
"Supports two output formats:\n"
" - pstats: Detailed profiling statistics with sorting options\n"
" - collapsed: Stack traces for generating flamegraphs\n"
"Supports the following target modes:\n"
" - -p PID: Profile an existing process by PID\n"
" - -m MODULE [ARGS...]: Profile a module as python -m module ... \n"
" - filename [ARGS...]: Profile the specified script by running it in a subprocess\n"
"\n"
"Examples:\n"
" # Profile process 1234 for 10 seconds with default settings\n"
" python -m profile.sample 1234\n"
" python -m profile.sample -p 1234\n"
"\n"
" # Profile a script by running it in a subprocess\n"
" python -m profile.sample myscript.py arg1 arg2\n"
"\n"
" # Profile a module by running it as python -m module in a subprocess\n"
" python -m profile.sample -m mymodule arg1 arg2\n"
"\n"
" # Profile with custom interval and duration, save to file\n"
" python -m profile.sample -i 50 -d 30 -o profile.stats 1234\n"
" python -m profile.sample -i 50 -d 30 -o profile.stats -p 1234\n"
"\n"
" # Generate collapsed stacks for flamegraph\n"
" python -m profile.sample --collapsed 1234\n"
" python -m profile.sample --collapsed -p 1234\n"
"\n"
" # Profile all threads, sort by total time\n"
" python -m profile.sample -a --sort-tottime 1234\n"
" python -m profile.sample -a --sort-tottime -p 1234\n"
"\n"
" # Profile for 1 minute with 1ms sampling interval\n"
" python -m profile.sample -i 1000 -d 60 1234\n"
" python -m profile.sample -i 1000 -d 60 -p 1234\n"
"\n"
" # Show only top 20 functions sorted by direct samples\n"
" python -m profile.sample --sort-nsamples -l 20 1234\n"
" python -m profile.sample --sort-nsamples -l 20 -p 1234\n"
"\n"
" # Profile all threads and save collapsed stacks\n"
" python -m profile.sample -a --collapsed -o stacks.txt 1234\n"
" python -m profile.sample -a --collapsed -o stacks.txt -p 1234\n"
"\n"
" # Profile with real-time sampling statistics\n"
" python -m profile.sample --realtime-stats 1234\n"
" python -m profile.sample --realtime-stats -p 1234\n"
"\n"
" # Sort by sample percentage to find most sampled functions\n"
" python -m profile.sample --sort-sample-pct 1234\n"
" python -m profile.sample --sort-sample-pct -p 1234\n"
"\n"
" # Sort by cumulative samples to find functions most on call stack\n"
" python -m profile.sample --sort-nsamples-cumul 1234"
" python -m profile.sample --sort-nsamples-cumul -p 1234\n"
),
formatter_class=argparse.RawDescriptionHelpFormatter,
)

# Required arguments
parser.add_argument("pid", type=int, help="Process ID to sample")
# Target selection
target_group = parser.add_mutually_exclusive_group(required=True)
target_group.add_argument(
"-p", "--pid", type=int, help="Process ID to sample"
)
target_group.add_argument(
"-m", "--module",
nargs=argparse.REMAINDER,
help="Run and profile a module as python -m module [ARGS...]"
)
target_group.add_argument(
"script",
nargs=argparse.REMAINDER,
help="Script to run and profile, with optional arguments"
)

# Sampling options
sampling_group = parser.add_argument_group("Sampling configuration")
Expand Down Expand Up @@ -712,19 +768,42 @@ def main():

sort_value = args.sort if args.sort is not None else 2

sample(
args.pid,
sample_interval_usec=args.interval,
duration_sec=args.duration,
filename=args.outfile,
all_threads=args.all_threads,
limit=args.limit,
sort=sort_value,
show_summary=not args.no_summary,
output_format=args.format,
realtime_stats=args.realtime_stats,
)
if args.module is not None and not args.module:
parser.error("the following arguments are required: module name")
Comment on lines +771 to +772
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would argparse.PARSER work here for nargs?


if args.pid:
sample(
args.pid,
sample_interval_usec=args.interval,
duration_sec=args.duration,
filename=args.outfile,
all_threads=args.all_threads,
limit=args.limit,
sort=sort_value,
show_summary=not args.no_summary,
output_format=args.format,
realtime_stats=args.realtime_stats,
)
elif args.module or args.script:
if args.module:
cmd = [sys.executable, "-m", *args.module]
else:
cmd = [sys.executable, *args.script]
Comment on lines +788 to +791
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if args.module:
cmd = [sys.executable, "-m", *args.module]
else:
cmd = [sys.executable, *args.script]
if args.module:
cmd = (sys.executable, "-m", *args.module)
else:
cmd = (sys.executable, *args.script)


process = subprocess.Popen(cmd)

# If we are the ones starting the process, we need to wait until the
# runtime state is initialized
try:
wait_for_process_and_sample(process.pid, sort_value, args)
finally:
if process.poll() is None:
process.terminate()
try:
process.wait(timeout=2)

This comment was marked as outdated.

Sign in to view

except subprocess.TimeoutExpired:
process.kill()
process.wait()

if __name__ == "__main__":
main()
Loading
Loading
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy