How can I profile Python code line-by-line?
I believe that's what Robert Kern's line_profiler is intended for. From the link:
File: pystone.py
Function: Proc2 at line 149
Total time: 0.606656 s
Line # Hits Time Per Hit % Time Line Contents
==============================================================
149 @profile
150 def Proc2(IntParIO):
151 50000 82003 1.6 13.5 IntLoc = IntParIO + 10
152 50000 63162 1.3 10.4 while 1:
153 50000 69065 1.4 11.4 if Char1Glob == 'A':
154 50000 66354 1.3 10.9 IntLoc = IntLoc - 1
155 50000 67263 1.3 11.1 IntParIO = IntLoc - IntGlob
156 50000 65494 1.3 10.8 EnumLoc = Ident1
157 50000 68001 1.4 11.2 if EnumLoc == Ident1:
158 50000 63739 1.3 10.5 break
159 50000 61575 1.2 10.1 return IntParIO
You could also use pprofile(pypi). If you want to profile the entire execution, it does not require source code modification. You can also profile a subset of a larger program in two ways:
toggle profiling when reaching a specific point in the code, such as:
import pprofile profiler = pprofile.Profile() with profiler: some_code # Process profile content: generate a cachegrind file and send it to user. # You can also write the result to the console: profiler.print_stats() # Or to a file: profiler.dump_stats("/tmp/profiler_stats.txt")toggle profiling asynchronously from call stack (requires a way to trigger this code in considered application, for example a signal handler or an available worker thread) by using statistical profiling:
import pprofile profiler = pprofile.StatisticalProfile() statistical_profiler_thread = pprofile.StatisticalThread( profiler=profiler, ) with statistical_profiler_thread: sleep(n) # Likewise, process profile content
Code annotation output format is much like line profiler:
$ pprofile --threads 0 demo/threads.py
Command line: ['demo/threads.py']
Total duration: 1.00573s
File: demo/threads.py
File duration: 1.00168s (99.60%)
Line #| Hits| Time| Time per hit| %|Source code
------+----------+-------------+-------------+-------+-----------
1| 2| 3.21865e-05| 1.60933e-05| 0.00%|import threading
2| 1| 5.96046e-06| 5.96046e-06| 0.00%|import time
3| 0| 0| 0| 0.00%|
4| 2| 1.5974e-05| 7.98702e-06| 0.00%|def func():
5| 1| 1.00111| 1.00111| 99.54%| time.sleep(1)
6| 0| 0| 0| 0.00%|
7| 2| 2.00272e-05| 1.00136e-05| 0.00%|def func2():
8| 1| 1.69277e-05| 1.69277e-05| 0.00%| pass
9| 0| 0| 0| 0.00%|
10| 1| 1.81198e-05| 1.81198e-05| 0.00%|t1 = threading.Thread(target=func)
(call)| 1| 0.000610828| 0.000610828| 0.06%|# /usr/lib/python2.7/threading.py:436 __init__
11| 1| 1.52588e-05| 1.52588e-05| 0.00%|t2 = threading.Thread(target=func)
(call)| 1| 0.000438929| 0.000438929| 0.04%|# /usr/lib/python2.7/threading.py:436 __init__
12| 1| 4.79221e-05| 4.79221e-05| 0.00%|t1.start()
(call)| 1| 0.000843048| 0.000843048| 0.08%|# /usr/lib/python2.7/threading.py:485 start
13| 1| 6.48499e-05| 6.48499e-05| 0.01%|t2.start()
(call)| 1| 0.00115609| 0.00115609| 0.11%|# /usr/lib/python2.7/threading.py:485 start
14| 1| 0.000205994| 0.000205994| 0.02%|(func(), func2())
(call)| 1| 1.00112| 1.00112| 99.54%|# demo/threads.py:4 func
(call)| 1| 3.09944e-05| 3.09944e-05| 0.00%|# demo/threads.py:7 func2
15| 1| 7.62939e-05| 7.62939e-05| 0.01%|t1.join()
(call)| 1| 0.000423908| 0.000423908| 0.04%|# /usr/lib/python2.7/threading.py:653 join
16| 1| 5.26905e-05| 5.26905e-05| 0.01%|t2.join()
(call)| 1| 0.000320196| 0.000320196| 0.03%|# /usr/lib/python2.7/threading.py:653 join
Note that because pprofile does not rely on code modification it can profile top-level module statements, allowing to profile program startup time (how long it takes to import modules, initialise globals, ...).
It can generate cachegrind-formatted output, so you can use kcachegrind to browse large results easily.
Disclosure: I am pprofile author.