pytest inconsistency

Question

More than a problem this is more a curiosity. Sometimes when i run the test on my laptop i get some inconsistency in the time it takes to run the tests.

I am talking about going from 30/40 ms to 2/3 or even more seconds. i checked if i had some weird process running in the background but nothing caught my attention.

So my question is what does influence on this? is it just the processes running in the background or there are some other major factors in play ?

Answer 1

There are so many things at play one couldn't list them all in a couple of paragraphs. These are some off the top of my head:

• the OS power-saving measures — your CPU may be throttled depending on your power profiles (battery saving modes, etc.)
• unexpected concurrent processes — on Windows boxes the "Antimalware Service Executable" has the irritating tendency to turn on out of the blue when CPU load is low for instance
• processors and motherboards boosting modes — some default configurations allow processors to run at faster clocks when their thermal state allows it, so performance is on average higher but more volatile
• gremlins, etc.

Given our use cases and memory allocations I doubt the garbage collector can trigger some slow downs in limited memory scenarios, when writing to a pagefile or swap partition.

When you profile your code you can always pay close attention to relative times — or how long a function or line of code took in relation to the rest. While absolute values will change, relative ones will be stable and indicative.

Comments

or else you could try to use the timeit module to average over more runs