Scalene - accumulative memory vs instant memory

[sjvdm]

Scalene is really a super useful tool - thanks to the author!

I wanted to discuss clarity around accumulative memory vs instant memory. See Sample code below. When looped once, scalene profiles the functions using 4GB. When looped 5 times, scalene profiles the function as using 20GB (which is accumulative). is there a way to get the max instant (aka 4GB) from the profile without only running the loop once?

import pandas as pd
import numpy as np
import gc

def f():
    print('called f')
    #Uses around 4GB of memory when looped once
    df = np.ones(500000000)
    
#Uses around 20GB of memory when looped 5 times
for i in range(0,5):
    f()

[emeryberger]

It's not currently possible in Scalene. Here's why. The only approach I am aware of is counting lines via https://docs.python.org/3/library/sys.html#sys.settrace. In my tests, this can slow down pure Python execution by almost 10x, which makes it a dealbreaker as a default. On the other hand, for programs like the one above that make extensive use of native modules like numpy and Pandas, the overhead is negligible, so it could be optionally activated.

[sjvdm]

Thanks for the swift response! Makes sense. Perhaps an option to profile this would be useful, as one would possibly only want to profile this on-demand. Can I log this as a feature request?

[emeryberger]

I have good news! :)

#266 (comment)

[emeryberger]

TL;DR Scalene now reports average memory consumption. We figured out a new approach that lets us gather this information with little added cost!