Performance: Compiled sub package for index generation

[DillonJ]

Discussions on the use of package_compiler (e.g. #885) seem to indicate that it's of limited use due to the dependence on the underlying data.

Also - investigations and discussions so far are suggesting that the main culprit for the poor model-building time (even the second run time as in Yi's tests vs PyPSA) is due to indexing that is needed to support flexibile temporal resolution and stochastic structure.

However, the stochastic and temporal structures depend on a relatively small amount of data. I wonder would there be a benefit in developing a compiled subpackage that generates all the fundamental indices that are needed for SpineOpt. This could be RUST or C or compiled Julia - but in the case of Julia - this would need to be done in a way that a reusable system image could be compiled that is minimally dependent on the data - if this is possible at all).

The best tradeoff between effort and utility would probably be to have it sit between SpineInterface / SpinedbAPI and SpineOpt. I.e. we use Spineinterface and/or SpineDBAPI to get the data we need to construct the temporal and stochastic structures and fundamental sets... but do all the heavy lifting in a subpackage that creates some sets that are then used by SpineOpt.

Any thoughts @manuelma @abelsiqueira @jkiviluo @Tasqu @suvayu @datejada ?

[Tasqu]

TLDR Version

@DillonJ In principle, I like the idea of clearer separation of handling the model structure as opposed to the model generation. However, I'm not sure if doing such a sub-module for SpineOpt would be worth it if the only objective is speed. As far as I understand, Julia can be fast enough, we just need to learn to use it properly, and fix our code.

Extended thoughts on the topic

This is related to something I've pondered occasionally as an outlet for frustration when dealing with SpineOpt and Backbone (a lot of which I imagine applies to other large-scale energy system modelling frameworks as well). There's a lot of similar "gruntwork" that goes into building and maintaining these types of models (mostly points 1 & 2), and I can't help but feel that it could be better modularised to allow for reuse. To me, there are at least 5 distinct stages of the modelling process that could ideally be modularised:

1. Data: Essentially, all the code dealing with different types of scenario-forecast-time-dependent data and their interoperability. For SpineOpt, this is basically handled in SpineInterface, where we have the TimeSeries, TimePattern, Map, etc. arithmetic and aggregation functionality, allowing the constraint code to not bother with all the parameter-data-structure-related details. (For Backbone this part is a waking nightmare thanks to GAMS not being a proper programming language...)

   • I feel SpineInterface isn't really the proper place for this, as SpineInterface to me is essentially the Julia-side of Spine DB API, and the data-convenience should be separated into a dedicated module.
   • Ideally, this kind of a low-level library should probably be implemented with a fast compiled language (e.g. Rust or C?), as it could then be used more easily from different higher-level languages? Users shouldn't basically ever need to tweak the functionality in this module, except when implementing completely new functionality.

2. Model Structure: Basically the part you're talking about with all the indexing etc., figuring out the stochastic and temporal structures. However, multi-model structures could also be implemented here: rolling a model, rolling multiple models in an arbitrary sequence, and possibly decomposition algorithms?

   • While implementing this in a low-level compiled language would improve performance, I'm worried that this part can never be written to be flexible enough for all use-cases. Users might need to customise this module more often for their specific modelling purposes, and using low-level languages could hinder reuseability.

3. Model Generation: The part that actually generates the optimization problem for the solver, essentially all the variable and constraint code. Relies on the above Data and Structure modules.

   • Users would likely need to customize this module quite a bit for different use cases (or even implement their own models on top of 1. and 2.), so high-level language probably makes that easier.

4. Solver: Already a clearly separate part in modelling workflows, but a crucial part nonetheless.

5. Results Processing: Not sure if this is separable from Model Generation to be honest. Still, it's a clearly separate step from the Structure-Generation-Solver loop that is required for complicated multi-model workflows.

[manuelma]

This sounds like a good idea, however I'd like to point out that we haven't yet gotten to the bottom of the performance problems I don't feel. We blame the indexing functions for all our problems - based on an initial profiling - but I have experienced that there is much to gain in other parts of the code too.

But the idea of isolating performance critical code in a separate package that is easy to compile is a good idea - whatever that part of the code is.

[DillonJ]

@manuelma that for this. In your opinion, what would the next steps be to indentify what that code is and indentify the code most ripe for optimisation?

[manuelma]

Just keep trying with @profile and even @time certain functions. For example I just found another 'problem'. The way we call Parameter, we pass a combination of 'entity-level indices' (unit, node, etc.) and 'map level indices' (stochastic_scenario, analysis_time, t) in the same call without any distinction. That means SpineInterface needs to split those indices into the two categories to first find the 'map' using the entity-level indices, and then find the value within the map using the map-level indices.

So you can imagine, if we have a lot of time-slices, there will be a lot of those calls where the entity-level indices are the same and only the map-level indices (namely the t) change... but SpineInterface still needs to split the indices and find the map for every call. Instead, I think we should find the map in a separate call (passing only the entity-level indices), and then access the map with the map-level indices. Otherwise there is a lot of repeated work.

But that means changing the code in the caller side I think, and that's a lot of replacing in many files.

So it's all a combination of trying out things and then implementing potentially substantial changes (although maybe not substantial in terms of design changes, but just a lot of typing so to say).