Speeding up volAttenForecastScaleCap

[PurpleHazeIan]

Hi Rob,

In the volAttenForecastScaleCap class in rob_system I saw this comment "this is a little slow so suggestions for speeding up are welcome". I have had to deal with a similar problem before, but not in Python, which gave me an idea for an improvement. I have little python expertise, but I offer the following, which seemed much faster when I tested it against a sample forecast series.

from sortedcontainers import SortedList
def quantile_of_points_in_data_series(data_series):
    sl = SortedList()
    results = []
    for current_value in data_series:
            
        sl.add(current_value)
        sorted_position = sl.index(current_value)

        results.append(sorted_position/len(sl))
    
    results_series = pd.Series(results, index = data_series.index)
    return results_series

[robcarver17]

Thanks, thats' great but I don't really want to add another third party dependency.

[PurpleHazeIan]

Thanks, a fair point that I hadn't considered.

On a 10,000 point GOLD forecast that code ran in <1% of the time. The following suggestion, which realises that it's not necessary to build the sorted list, uses only numpy and runs in about 2-3% of the time.

def quantile_of_points_in_data_series(data_series):
    
    numpy_series = np.array(data_series)
    results = []
    
    for irow in range(len(data_series)):
        
        current_value = numpy_series[irow]
        count_less_than = (numpy_series < current_value)[:irow].sum()
        results.append(count_less_than/(irow+1))
    
    results_series = pd.Series(results, index = data_series.index)
    return results_series

[robcarver17]

There are some slight differences in the calculation particularly in the first few years, but I think that is because the old implementation is doing some kind of interpolation... I like it.

[robcarver17]

Committed and credited

[PurpleHazeIan]

Thanks!

Marginally faster still is to slice before compare i.e. count_less_than = (numpy_series[:irow] < current_value).sum()

Yes, I noted slight differences - I used as my test an ewmac forecast with some NaNs at the front, which raises a question of what to do with them - but I felt it's in the nature of using a quantile position on a reasonable number of data points that the exact quantile value shouldn't matter.