[DESIGN] Refactoring the extension module system

[jougs]

The SLI Interpreter encapsulates a certain portion of its extended functionality in modules, which consist of some C++ code, implementing new commands, and (optionally) some bundled .sli file(s) for type checking and handling function overloading. All such modules derive from SLIModule and are added to the interpreter at compile-time by calling SLIInterpreter::addmodule().

At some point in time, the DynamicLoaderModule was introduced, which let users load modules dynamically at run-time. The main use case for dynamically loaded modules is adding new neuron and synapse models, connection builders, or recording and (soon, see #1456) stimulation backends to NEST. The DynamicLoaderModule as well as the extension modules themselves are derived from SLIModule and thus are, at least technically, a part of SLI rather than of NEST.

As SLI's modules were originally not meant to be loaded dynamically, but rather for encapsulation and to provide new commands to the SLI Interpreter, the API they (and by extension also the DynamicLoaderModule) rely on are lacking some functionality like the possibility to reset a module and read/pass parameters from/to them.

NEST's mechanism for encapsulation are managers. A manager is responsible for a certain aspect of the simulation kernel (e.g. Models, MPI, Threading, RNGs, ...). In a sense, managers serve a similar purpose for NEST (encapsulation) as the modules originally did for SLI, however the implementations are somewhat orthogonal and the managers provide a much more complete API when it comes to dynamic parameter changes at run-time.

A closer look at the existing modules reveals the following situation:

1. Some modules (e.g. the modelsmodule providing the standard neuron, device and synapse models to NEST) just use the module mechanism for encapsulation, while they are actually not interacting with the SLI Interpreter, but rather only with NEST.
2. Other modules (e.g. the SLI language modules) are mainly providing new commands to the SLI Interpreter (that may still call functions of the NEST kernel internally)

This basically means that we have two distinct classes of modules:

1. modules that extend the SLI interpreter (think Python modules, but linked statically at compile-time)
2. modules that extend NEST (dynamically loaded at run-time or statically linked at compile-time)

The current use of modules derived from SLIModule to extend both SLI and NEST statically (i.e. at compile-time) and dynamically (i.e. at run-time) constitutes a breach of abstraction and a stronger than necessary dependency of the NEST simulation kernel on the SLI Interpreter. This leads to a number of downsides:

• All modules have full access to both the SLI Interpreter and the NEST simulation kernel. Providing a restricted (and stable) API to modules is thus quite hard.
• Solving the problem of un-/re-loading an extension module (as requested by Unload or reload module in NEST Kernel #1642) would require quite massive changes to the module mechanism in SLI, while the required interface is for most parts already existing for the managers in NEST.
• Using NEST without the SLI interpreter is currently not possible, as core functionality is implemented as SLI Modules. This hinders the development of more efficient and lightweight Python binding, or a C++ API that allows to embed the simulation kernel in third-party tools.

In order to address these issues, I propose a refactoring of the module mechanisms for NEST and SLI along the following list of concrete action items:

• Get the number of existing modules in NEST down:
  • integrate the PreciseModule into the ModelsModule (done in Merge precisemodule into modelsmodule #1705)
  • remove the SLI neuron; rapid prototyping of neuron models is now available via NESTML (done in Remove SLI neuron #1750)
  • dissolve the ModelsModule and register built-in models from the KernelManager instead (prototype in Refactor ModelsModule to not be a SLIModule anymore #1853; then concluded that this should rather be done during the second phase)
  • refactor the conngen module to become a ConnBuilder and auxiliary SLI commands in NestModule (done in Refactor conngen module to be a ConnBuilder #1830)
  • integrate the functionality of the TopologyModule into the NEST kernel proper (done in Dissolve Topology module and move files to nestkernel/spatial #1748).
• Refactor the dynamic module mechanism for NEST to be independent of SLI
  • convert the DynamicLoaderModule to an ExtensionManager for NEST
  • create a new base class NESTExtension from which future extension modules (also from NESTML) can be derived
  • refactor the ModelsModule to become a first proof-of-concept NESTExtension
  • update the nest-extension-module to the new extension module base class

Along the way, this might lead to a more complete C++ API in order to provide hooks for the new NEST extension modules.

[gtrensch]

👍 I think that reducing the SLI dependencies and unifying the architecture is very good.

[terhorstd]

👍 yes, the dependencies are currently the wrong way around, which -as you mention- leads to unintuitive difficulties again and again. Thanks for taking a look at this and tidying up the long grown thicket.

[heplesser]

@jougs Thank you for describing this issue in detail and proposing a clean solution. I think converting, e.g., ModelsModule to a NESTExtension seems very sensible. One point I am wondering a little about is that of granularity: Should we collect, e.g., all neuron, device and synapse models that ship with NEST in a single extension, or should we create more focused extensions?

Concerning names, how about NESTComponent rather than NESTExtension, since NEST without any extensions would be a naked kernel, rather useless.

[jougs]

@heplesser: These are very good points. I like the name NESTComponent quite a lot. It also fits perfectly with an idea that I am currently pondering with @clinssen: What, if we used a much more fine-grained approach for these components, namely if we eliminated collective modules altogether and instead made each and every component (a neuron model, a connection rule, a recording backend) by itself a loadable entity (linked either statically or dynamically)?

This would open the future to a highly customized NEST kernel that would just have exactly the components you need for a given simulation, or load the required components at run-time.

[heplesser]

@heplesser: What, if we used a much more fine-grained approach for these components, namely if we eliminated collective modules altogether and instead made each and every component (a neuron model, a connection rule, a recording backend) by itself a loadable entity (linked either statically or dynamically)?

This would open the future to a highly customized NEST kernel that would just have exactly the components you need for a given simulation, or load the required components at run-time.

This is a very interesting idea. In common current models, you probably have 1-2 neuron, 1-2 synapse and 2-5 device models, so in all 5-10 different entities that would require importing. In more complex models the number may be greater. I wonder a little about the overhead of importing, say 20-50 shared libraries in a very large distributed simulation of a complex brain model. And how would the user tell NEST to load the neuron model you need. If a model could only be used after the user ran an "install" command for the model, this would make interactive work cumbersome.

[jougs]

I guess the answer would be that the user does not tell NEST to load the component at all, but it would be inferred from the function calls themselves. That is if NEST encounters a call to Create and does not know the neuron model it shall instantiate, it could automatically try to load the corresponding library. The same would be true for connection rules, recording backends and all other types of components we might have in the future. There's always a point in the script where the type and name of the component is known, after all.

I don't think loading 50 something libraries is a problem, but if it turns out to be one, the required (or desired, in case of an interactive working scenario) components could always be statically linked.

[github-actions]

Issue automatically marked stale!