Link helper libraries statically

[jougs]

The NEST source code is split into several directories (conngen, libnestutil, librandom, models, nestkernel, precise, sli, topology), each built into a dynamic helper library. The nest executable (residing in the nest subdirectory) comes into existence by a bit of glue code and dynamically linking all of the helper libraries.

This issue demands that the setup is changed so that the nest executable is statically linking its helper libraries, while still allowing to link all external libraries dynamically.

@gtrensch and @heplesser might have additional ideas and thoughts on this.

[Silmathoron]

Out of curiosity, what is the advantage of linking statically compared to the current dynamical linking?

[jougs]

The main advantage is that static linking eliminates dynamic linking as a possible problem source ;-)

An example would be an issue we are currently seeing on JURON, where the system scheduler (LSF) also installs a library called libtopology.so. Due to the specific setup of the library search paths on the system, that is found before NEST's library with the same name and a pethora of undefined references is thrown at the user. Debugging this took one of us almost a full day and it would be nicer to avoid such problems altogether.

[ikitayama]

gem5 requires apps be statically linked, for example.

…

On Thu, Apr 18, 2019 at 0:35 Tanguy Fardet ***@***.***> wrote: Out of curiosity, what is the advantage of linking statically compared to the current dynamical linking? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1182 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA6w5_L2WuUIZCC9WXj22BzQ9AfYjS5aks5vhz9FgaJpZM4c1PXi> .

[Silmathoron]

Ok, thanks!

[AndiH]

I want to note somewhere how we solved the issue for JURON, which @jougs referred to in a previous reply.

As @jougs said, the issue arises due to LSF coming with its own liptopology.so.

CMake builds the Nest executable using RUNPATH to locate Nest's libtology.so and use the symbols defined in there. RUNPATH is searched after LD_LIBRARY_PATH. And here lies the issue: JURON's LSF installation is taught to the environment via an environment module, appending LD_LIBRARY_PATH by the LSF location. When now Nest searches for libtology.so, it will first look for the file in all locations of LD_LIBRARY_PATH, and only then consider the paths defined in its RUNPATH. This fails, as the Nest executable finds LSF's libtopology.so.

A possible fix is to convert RUNPATH into RPATH as RPATH is searched before LD_LIBRARY_PATH and not after (like RUNPATH) is.

I converted RUNPATH to RPATH with this little snippet I found on a Github Gist, but there are probably more suitable solutions out there.

[github-actions]

Issue automatically marked stale!

[terhorstd]

Is this related to #1630? Should this be handled consistently?

[jougs]

@terhorstd: No, it's not. This one is purely about the internal helper libraries that are linked together to form the nest and sli executables and the pynestkernel.so.

As I'm anyway here, I might as well give a quick update on the status:

• topology has been integrated into the kernel in Dissolve Topology module and move files to nestkernel/spatial #1748
• precise has been merged into models in Merge precisemodule into modelsmodule #1705
• librandom has been integrated into the kernel Modernize random number generator setup #1549
• conngen is available as a ConnBuilder since Refactor conngen module to be a ConnBuilder #1830
• models will be refactored in [DESIGN] Refactoring the extension module system #1717, but dynamic loading will still be supported

[jougs]

It would be good to investigate, if it makes sense to change the default so that static linking becomes the default for all modules and enable dynamic linking only on demand.

Also see #1717.

[github-actions]

Issue automatically marked stale!