Line Profilers for Gentlemen in a Hurry

Silktrader (2550 points)
1 6 16
asked Oct 19, 2020 in Programming in Python by Silktrader (2,550 points)

The first homework has already, worryingly, set a trend to (needlessly) micro-optimise the simplest of statements. In a bid to dissect and extinguish the lamest parts of my scripts I've resorted to using a line profiler, which describes the number of calls and execution time of every line of code. The common cProfiler instead presents aggregate information about function calls.

The "Software and Commands" page mentions a line profiler that works within Spyder, although I had multiple issues with it. For starters I was prompted to update or install 239**34 other required packages, change my family name and convert to Buddhism. It doesn't seem to be possible to run Spyder's line profiler outside of the IDE (Shift-F10) and without selecting each script to be profiled from the GUI. Now, I wouldn't touch Anaconda's stock IDE with a barge pole, so I find the command line based kernprof line profiler more convenient.

The steps to set it up were the following:

  • conda install -c anaconda line_profiler in "Anaconda's Prompt"

  • decorate the functions to be profiled with @profile

  • run kernprof -l -v

One can also run "pytest" files with "kernprof", which will analyse all the decorated functions called by the tests. Unfortunately, running test files on their own, without relying on "kernprof" and while the decorator's enabled, throws an error as "pytest" complains that the "profile" module can't be found. I comment out the offending line when needed.

This is the kind of result one can expect. The line above the function's definition includes the decorator:

My gripes with the line profiler are two:

  1. it doesn't allow one to run multiple tests and average results (as with timeit, for instance) — one must manually set up multiple calls to decorated functions and look at aggregates
  2. pytest and both line profilers don't play well together whilst pytest's own profiler doesn't output execution statistics line by line

It'd be nice if anybody came up with ways to address either point.


2 Answers

Best answer
andrea.sterbini (167420 points)
487 897 1726
answered Oct 19, 2020 by andrea.sterbini (167,420 points)
selected Oct 21, 2020 by Silktrader
See my post on the Italian forum for a solution to leave @profile when doing pytest
Silktrader (2550 points)
1 6 16
commented Oct 21, 2020 by Silktrader (2,550 points)

I assume that this is the answer you're referring to. You suggested to define a @profile decorator at the start of the module when the line profiler doesn't inject its own:

if 'profile' not in __builtins__:
    def profile(x): return x 

Works. Thanks!

Additionally, this comment will prevent PyCharm from complaining that @profile is potentially undefined:

# noinspection PyUnboundLocalVariable
angelo.spognardi (4120 points)
31 49 79
answered Oct 19, 2020 by angelo.spognardi (4,120 points)
Silktrader (2550 points)
1 6 16
commented Oct 21, 2020 by Silktrader (2,550 points)

I did follow the guide when I first set up my environment. I linked to it in the post above.