4
4
The Python Profilers
5
5
********************
6
6
7
- **Source code: ** :source: `Lib/profile.py ` and :source: `Lib/pstats .py `
7
+ **Source code: ** :source: `Lib/profile.py `, :source: ` Lib/pstats.py `, and :source: `Lib/profile/sample .py `
8
8
9
9
--------------
10
10
@@ -14,23 +14,32 @@ Introduction to the profilers
14
14
=============================
15
15
16
16
.. index ::
17
+ single: statistical profiling
18
+ single: profiling, statistical
17
19
single: deterministic profiling
18
20
single: profiling, deterministic
19
21
20
- :mod: ` cProfile ` and :mod: ` profile ` provide :dfn: `deterministic profiling ` of
22
+ Python provides both :dfn: ` statistical profiling ` and :dfn: `deterministic profiling ` of
21
23
Python programs. A :dfn: `profile ` is a set of statistics that describes how
22
24
often and for how long various parts of the program executed. These statistics
23
25
can be formatted into reports via the :mod: `pstats ` module.
24
26
25
- The Python standard library provides two different implementations of the same
26
- profiling interface:
27
+ The Python standard library provides three different profiling implementations:
27
28
28
- 1. :mod: `cProfile ` is recommended for most users; it's a C extension with
29
+ **Statistical Profiler: **
30
+
31
+ 1. :mod: `profile.sample ` provides statistical profiling of running Python processes
32
+ using periodic stack sampling. It can attach to any running Python process without
33
+ requiring code modification or restart, making it ideal for production debugging.
34
+
35
+ **Deterministic Profilers: **
36
+
37
+ 2. :mod: `cProfile ` is recommended for development and testing; it's a C extension with
29
38
reasonable overhead that makes it suitable for profiling long-running
30
39
programs. Based on :mod: `lsprof `, contributed by Brett Rosen and Ted
31
40
Czotter.
32
41
33
- 2 . :mod: `profile `, a pure Python module whose interface is imitated by
42
+ 3 . :mod: `profile `, a pure Python module whose interface is imitated by
34
43
:mod: `cProfile `, but which adds significant overhead to profiled programs.
35
44
If you're trying to extend the profiler in some way, the task might be easier
36
45
with this module. Originally designed and written by Jim Roskind.
@@ -44,6 +53,77 @@ profiling interface:
44
53
but not for C-level functions, and so the C code would seem faster than any
45
54
Python one.
46
55
56
+ **Profiler Comparison: **
57
+
58
+ +-------------------+----------------------+----------------------+----------------------+
59
+ | Feature | Statistical | Deterministic | Deterministic |
60
+ | | (``profile.sample ``) | (``cProfile ``) | (``profile ``) |
61
+ +===================+======================+======================+======================+
62
+ | **Target ** | Running process | Code you run | Code you run |
63
+ +-------------------+----------------------+----------------------+----------------------+
64
+ | **Overhead ** | Virtually none | Moderate | High |
65
+ +-------------------+----------------------+----------------------+----------------------+
66
+ | **Accuracy ** | Statistical approx. | Exact call counts | Exact call counts |
67
+ +-------------------+----------------------+----------------------+----------------------+
68
+ | **Setup ** | Attach to any PID | Instrument code | Instrument code |
69
+ +-------------------+----------------------+----------------------+----------------------+
70
+ | **Use Case ** | Production debugging | Development/testing | Profiler extension |
71
+ +-------------------+----------------------+----------------------+----------------------+
72
+ | **Implementation**| C extension | C extension | Pure Python |
73
+ +-------------------+----------------------+----------------------+----------------------+
74
+
75
+ .. note ::
76
+
77
+ The statistical profiler (:mod: `profile.sample `) is recommended for most production
78
+ use cases due to its extremely low overhead and ability to profile running processes
79
+ without modification. It can attach to any Python process and collect performance
80
+ data with minimal impact on execution speed, making it ideal for debugging
81
+ performance issues in live applications.
82
+
83
+
84
+ .. _statistical-profiling :
85
+
86
+ What Is Statistical Profiling?
87
+ ==============================
88
+
89
+ :dfn: `Statistical profiling ` works by periodically interrupting a running
90
+ program to capture its current call stack. Rather than monitoring every
91
+ function entry and exit like deterministic profilers, it takes snapshots at
92
+ regular intervals to build a statistical picture of where the program spends
93
+ its time.
94
+
95
+ The sampling profiler uses process memory reading (via system calls like
96
+ ``process_vm_readv `` on Linux, ``vm_read `` on macOS, and ``ReadProcessMemory `` on
97
+ Windows) to attach to a running Python process and extract stack trace
98
+ information without requiring any code modification or restart of the target
99
+ process. This approach provides several key advantages over traditional
100
+ profiling methods.
101
+
102
+ The fundamental principle is that if a function appears frequently in the
103
+ collected stack samples, it is likely consuming significant CPU time. By
104
+ analyzing thousands of samples, the profiler can accurately estimate the
105
+ relative time spent in different parts of the program. The statistical nature
106
+ means that while individual measurements may vary, the aggregate results
107
+ converge to represent the true performance characteristics of the application.
108
+
109
+ Since statistical profiling operates externally to the target process, it
110
+ introduces virtually no overhead to the running program. The profiler process
111
+ runs separately and reads the target process memory without interrupting its
112
+ execution. This makes it suitable for profiling production systems where
113
+ performance impact must be minimized.
114
+
115
+ The accuracy of statistical profiling improves with the number of samples
116
+ collected. Short-lived functions may be missed or underrepresented, while
117
+ long-running functions will be captured proportionally to their execution time.
118
+ This characteristic makes statistical profiling particularly effective for
119
+ identifying the most significant performance bottlenecks rather than providing
120
+ exhaustive coverage of all function calls.
121
+
122
+ Statistical profiling excels at answering questions like "which functions
123
+ consume the most CPU time?" and "where should I focus optimization efforts?"
124
+ rather than "exactly how many times was this function called?" The trade-off
125
+ between precision and practicality makes it an invaluable tool for performance
126
+ analysis in real-world applications.
47
127
48
128
.. _profile-instant :
49
129
@@ -54,6 +134,18 @@ This section is provided for users that "don't want to read the manual." It
54
134
provides a very brief overview, and allows a user to rapidly perform profiling
55
135
on an existing application.
56
136
137
+ **Statistical Profiling (Recommended for Production): **
138
+
139
+ To profile an existing running process::
140
+
141
+ python -m profile.sample 1234
142
+
143
+ To profile with custom settings::
144
+
145
+ python -m profile.sample -i 50 -d 30 1234
146
+
147
+ **Deterministic Profiling (Development/Testing): **
148
+
57
149
To profile a function that takes a single argument, you can do::
58
150
59
151
import cProfile
@@ -121,8 +213,208 @@ results to a file by specifying a filename to the :func:`run` function::
121
213
The :class: `pstats.Stats ` class reads profile results from a file and formats
122
214
them in various ways.
123
215
216
+ .. _sampling-profiler-cli :
217
+
218
+ Statistical Profiler Command Line Interface
219
+ ===========================================
220
+
221
+ .. program :: profile.sample
222
+
223
+ The :mod: `profile.sample ` module can be invoked as a script to profile running processes::
224
+
225
+ python -m profile.sample [options] PID
226
+
227
+ **Basic Usage Examples: **
228
+
229
+ Profile process 1234 for 10 seconds with default settings::
230
+
231
+ python -m profile.sample 1234
232
+
233
+ Profile with custom interval and duration, save to file::
234
+
235
+ python -m profile.sample -i 50 -d 30 -o profile.stats 1234
236
+
237
+ Generate collapsed stacks to use with tools like `flamegraph.pl
238
+ <https://github.com/brendangregg/FlameGraph> `_::
239
+
240
+ python -m profile.sample --collapsed 1234
241
+
242
+ Profile all threads, sort by total time::
243
+
244
+ python -m profile.sample -a --sort-tottime 1234
245
+
246
+ Profile with real-time sampling statistics::
247
+
248
+ python -m profile.sample --realtime-stats 1234
249
+
250
+ **Command Line Options: **
251
+
252
+ .. option :: PID
253
+
254
+ Process ID of the Python process to profile (required)
255
+
256
+ .. option :: -i , --interval INTERVAL
257
+
258
+ Sampling interval in microseconds (default: 100)
259
+
260
+ .. option :: -d , --duration DURATION
261
+
262
+ Sampling duration in seconds (default: 10)
263
+
264
+ .. option :: -a , --all-threads
265
+
266
+ Sample all threads in the process instead of just the main thread
267
+
268
+ .. option :: --realtime-stats
269
+
270
+ Print real-time sampling statistics during profiling
271
+
272
+ .. option :: --pstats
273
+
274
+ Generate pstats output (default)
275
+
276
+ .. option :: --collapsed
277
+
278
+ Generate collapsed stack traces for flamegraphs
279
+
280
+ .. option :: -o , --outfile OUTFILE
281
+
282
+ Save output to a file
283
+
284
+ **Sorting Options (pstats format only): **
285
+
286
+ .. option :: --sort-nsamples
287
+
288
+ Sort by number of direct samples
289
+
290
+ .. option :: --sort-tottime
291
+
292
+ Sort by total time
293
+
294
+ .. option :: --sort-cumtime
295
+
296
+ Sort by cumulative time (default)
297
+
298
+ .. option :: --sort-sample-pct
299
+
300
+ Sort by sample percentage
301
+
302
+ .. option :: --sort-cumul-pct
303
+
304
+ Sort by cumulative sample percentage
305
+
306
+ .. option :: --sort-nsamples-cumul
307
+
308
+ Sort by cumulative samples
309
+
310
+ .. option :: --sort-name
311
+
312
+ Sort by function name
313
+
314
+ .. option :: -l , --limit LIMIT
315
+
316
+ Limit the number of rows in the output (default: 15)
317
+
318
+ .. option :: --no-summary
319
+
320
+ Disable the summary section in the output
321
+
322
+ **Understanding Statistical Profile Output: **
323
+
324
+ The statistical profiler produces output similar to deterministic profilers but with different column meanings::
325
+
326
+ Profile Stats:
327
+ nsamples sample% tottime (ms) cumul% cumtime (ms) filename:lineno(function)
328
+ 45/67 12.5 23.450 18.6 56.780 mymodule.py:42(process_data)
329
+ 23/23 6.4 15.230 6.4 15.230 <built-in>:0(len)
330
+
331
+ **Column Meanings: **
332
+
333
+ - **nsamples **: ``direct/cumulative `` - Times function was directly executing / on call stack
334
+ - **sample% **: Percentage of total samples where function was directly executing
335
+ - **tottime **: Estimated time spent directly in this function
336
+ - **cumul% **: Percentage of samples where function was anywhere on call stack
337
+ - **cumtime **: Estimated cumulative time including called functions
338
+ - **filename:lineno(function) **: Location and name of the function
339
+
124
340
.. _profile-cli :
125
341
342
+ :mod: `profile.sample ` Module Reference
343
+ =======================================================
344
+
345
+ .. module :: profile.sample
346
+ :synopsis: Python statistical profiler.
347
+
348
+ This section documents the programmatic interface for the :mod: `profile.sample ` module.
349
+ For command-line usage, see :ref: `sampling-profiler-cli `. For conceptual information
350
+ about statistical profiling, see :ref: `statistical-profiling `
351
+
352
+ .. function :: sample(pid, *, sort=2, sample_interval_usec=100, duration_sec=10, filename=None, all_threads=False, limit=None, show_summary=True, output_format="pstats", realtime_stats=False)
353
+
354
+ Sample a Python process and generate profiling data.
355
+
356
+ This is the main entry point for statistical profiling. It creates a
357
+ :class: `SampleProfiler `, collects stack traces from the target process, and
358
+ outputs the results in the specified format.
359
+
360
+ :param int pid: Process ID of the target Python process
361
+ :param int sort: Sort order for pstats output (default: 2 for cumulative time)
362
+ :param int sample_interval_usec: Sampling interval in microseconds (default: 100)
10000
363
+ :param int duration_sec: Duration to sample in seconds (default: 10)
364
+ :param str filename: Output filename (None for stdout/default naming)
365
+ :param bool all_threads: Whether to sample all threads (default: False)
366
+ :param int limit: Maximum number of functions to display (default: None)
367
+ :param bool show_summary: Whether to show summary statistics (default: True)
368
+ :param str output_format: Output format - 'pstats' or 'collapsed' (default: 'pstats')
369
+ :param bool realtime_stats: Whether to display real-time statistics (default: False)
370
+
371
+ :raises ValueError: If output_format is not 'pstats' or 'collapsed'
372
+
373
+ Examples::
374
+
375
+ # Basic usage - profile process 1234 for 10 seconds
376
+ import profile.sample
377
+ profile.sample.sample(1234)
378
+
379
+ # Profile with custom settings
380
+ profile.sample.sample(1234, duration_sec=30, sample_interval_usec=50, all_threads=True)
381
+
382
+ # Generate collapsed stack traces for flamegraph.pl
383
+ profile.sample.sample(1234, output_format='collapsed', filename='profile.collapsed')
384
+
385
+ .. class :: SampleProfiler(pid, sample_interval_usec, all_threads)
386
+
387
+ Low-level API for the statistical profiler.
388
+
389
+ This profiler uses periodic stack sampling to collect performance data
390
+ from running Python processes with minimal overhead. It can attach to
391
+ any Python process by PID and collect stack traces at regular intervals.
392
+
393
+ :param int pid: Process ID of the target Python process
394
+ :param int sample_interval_usec: Sampling interval in microseconds
395
+ :param bool all_threads: Whether to sample all threads or just the main thread
396
+
397
+ .. method :: sample(collector, duration_sec=10)
398
+
399
+ Sample the target process for the specified duration.
400
+
401
+ Collects stack traces from the target process at regular intervals
402
+ and passes them to the provided collector for processing.
403
+
404
+ :param collector: Object that implements ``collect() `` method to process stack traces
405
+ :param int duration_sec: Duration to sample in seconds (default: 10)
406
+
407
+ The method tracks sampling statistics and can display real-time
408
+ information if realtime_stats is enabled.
409
+
410
+ .. seealso ::
411
+
412
+ :ref: `sampling-profiler-cli `
413
+ Command-line interface documentation for the statistical profiler.
414
+
415
+ Deterministic Profiler Command Line Interface
416
+ =============================================
417
+
126
418
.. program :: cProfile
127
419
128
420
The files :mod: `cProfile ` and :mod: `profile ` can also be invoked as a script to
@@ -564,7 +856,7 @@ What Is Deterministic Profiling?
564
856
call *, *function return *, and *exception * events are monitored, and precise
565
857
timings are made for the intervals between these events (during which time the
566
858
user's code is executing). In contrast, :dfn: `statistical profiling ` (which is
567
- not done by this module) randomly samples the effective instruction pointer, and
859
+ provided by the :mod: ` profile.sample ` module) periodically samples the effective instruction pointer, and
568
860
deduces where time is being spent. The latter technique traditionally involves
569
861
less overhead (as the code does not need to be instrumented), but provides only
570
862
relative indications of where time is being spent.
0 commit comments