Consistent Link-Time Optimization (LTO) usage across Pop!_OS repos

[zamazan4ik]

Hi!

I noticed that in the Cargo.toml file Link-Time Optimization (LTO) for the project is not enabled. I suggest switching it on since it will reduce the binary size (always a good thing to have) and will likely improve the application's performance a bit.

I suggest enabling LTO only for the Release builds so as not to sacrifice the developers' experience while working on the project since LTO consumes an additional amount of time to finish the compilation routine. If you think that a regular Release build should not be affected by such a change as well, then I suggest adding an additional dist or release-lto profile where additionally to regular release optimizations LTO will also be added. Such a change simplifies life for maintainers and others interested in the project persons who want to build the most performant version of the application. Using ThinLTO should also help to reduce the build-time overhead with LTO.

Basically, it can be enabled with the following lines:

[profile.release]
lto = true

I checked multiple pop_os projects and found that LTO is not consistently used across projects. For some of them (like cosmic-applibrary or cosmic-panel) LTO is enabled. For others like system76-power - it isn't. I propose enabling LTO in all pop_os projects for the Release (or a dedicated) profile. And only for projects where it has huge limitations (like triggering a compiler error) disable it. I expect that only for a few projects we can meet such problems.

Users who have build issues with LTO like increased memory consumption also will be able to easily switch off LTO in their specific cases (or just reduce the amount of parallel compilation jobs).

Thank you.

[jacobgkau]

If it's already enabled in some repos, you're welcome to open PRs in the repos where it's not, and engineering and QA can take a look at if it's alright to enable. system76-power is far older than any of the COSMIC repos, so if this is a case of best practices having changed, then older repos may simply not have been updated with it yet.

[zamazan4ik]

I just want to share a bit of my thoughts regarding these changes:

• I expect that for the most projects the increased build times in the Release mode is acceptable even with Fat LTO since Release binaries are meant to be delivered to the end-users ("build once heavily-optimized and then deliver to many users").
• For cases where Fat LTO build time increase is not acceptable but we still want to get Fat LTO optimizations - we can create a dedicated profile like profile.release-lto. I saw multiple projects that do things in this way. Then, this profile can be triggered on CI systems only for Releases or from time-to-time (multiple options are available)
• Even if this way is not good enough, we can switch from Fat LTO to Thin LTO in places, where build time is critical. Yeah, Thin LTO is less efficient compared to Fat LTO from the performed optimizations perspective (my own experience and several publications from Google about ThinLTO) but ThinLTO is still better than no LTO at all.

I hope my comments will be helpful.

[mmstick]

Fat is too much for many distribution's build systems, and will most certainly slow down our packaging CI significantly. We enabled this for cosmic-applets only because it had a significant impact there, given that it's a multicall binary with a lot of applets bundled together. And similarly cosmic-comp is a critical component of the desktop. In most projects, there isn't much difference between thin and fat, and any performance benefit is imperceptible.

[zamazan4ik]

If you say so, am I right that you are ok with enabling ThinLTO by default for pop!_os projects?

[JohnRTitor]

ThinLTO actually performs better in many cases AFAIK.

[zamazan4ik]

ThinLTO actually performs better in many cases AFAIK.

It depends on the definition of "better". If you mean "better" from the compilation speed - yes, you are right, Thin LTO compiles MUCH faster than Fat LTO.

If you mean "better" from the performed optimizations perspectives - nope, it's not true. It comes from the ThinLTO nature. I will explain, why. Fat LTO gives to the compiler visibility of the whole application, and the compiler, having more information, can perform more optimizations.

ThinLTO splits the program into N parts, and each part is LTO-optimized. In this case, we can achieve parallelization in the price of a bit less aggressive optimizations across these N splitted parts. It's not only theoretical thing - it's written in the original Thin LTO paper: https://research.google/pubs/thinlto-scalable-and-incremental-lto/ (however, this benchmark is kinda limited only to SPEC). More mentions of this detail can be found in various places like this

I can also confirm such a difference for Rust applications - in all my tests Fat LTO results in better optimizations compared to Thin LTO. Just a few examples: one, two, three (I have more of them).

So, if you have evidence that Thin LTO in general works better than Fat LTO from the performed optimizations perspective, I would be happy to see such benchmarks (especially, if they test various software not only specific cases).

Knowing all of these, the next question can be - is Fat LTO worth it? it completely depends on a project. As @mmstick said earlier, even if Fat LTO performs better than Thin LTO, such an overhead is unacceptable for PopOS due to limited CI resources, and a difference in efficiency between Fat and Thin is not worth it in this case - that's a valid point, and I can agree with it. In this case, I would be happy to see at least ThinLTO enabled by default for all PopOS repos. Unfortunately, I don't see any activity in this area after my last comment in this thread - hopefully it will change soon.