Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch build system to ninja #47

Merged
merged 9 commits into from
Dec 12, 2023
Merged

Switch build system to ninja #47

merged 9 commits into from
Dec 12, 2023

Conversation

roblabla
Copy link
Member

Nmake was fun, but it was very annoying to use. It suffered from multiple problems:

  • It didn't do header dependency tracking, meaning that if we changed a header, it wouldn't automatically recompile the source files that include it. It turns out that only ninja really supports this for cl - raw makefiles can't do it easily since cl lacks a way to output makefile dependency info.
  • It didn't do cmdline tracking either - meaning switching between a DIFFBUILD and a NORMAL build could cause some very weird thing to happen as it would try to link objfiles from different types of builds.
  • It was very annoying to maintain, as nmake documentation and examples are very sparse.

Because of all this, I want to switch to another system. This system needs to be:

  1. Very dumb. I don't want the build system to do things behind my back - I want to tell it explicitly what the cmdline of the program I'm launching it, what its dependencies are, etc...
  2. Able to do dependency tracking at least for the cmdline, but even better if it can do header tracking with cl.exe
  3. Ideally, it should work on Windows XP, for continued memes.

So the first thing people will think of is "cmake". Thing is, we tried cmake, and it was miserable. CMake as a language is terrible, and CMake as a tool does a ton of stuff behind your back - it was hard to get it to invoke the compiler with exactly the flags we wanted - something we really need here. We need something both simpler and more flexible.

So my thoughts went to ninja. Ninja is a build tool that's supposed to be very fast, very dumb (in the good way), and very robust. We should be able to write a build.ninja file for our build that works well, right? Well, that does work if we only had a single build type. But ninja lacks the ability to do conditionals, complicating the ability to have both a diffbuild and a normal build. We may be able to work around this by creating different targets, but it's not entirely straightforward how that would work.

Instead, the way we're supposed to use ninja is by wrapping it in a generator, and ninja is kind enough to give us a very nice tool to make our own wrapper: ninja_syntax.py. So the plan is this: We create a new python script, build.py, which will automatically generate the appropriate build.ninja, and run ninja on it.

This works great! It does both cmdline tracking and header tracking, meaning when we switch from a normal build to a diffbuild, or when we edit a header, ninja will automatically know to recompile some of the files!

@roblabla roblabla force-pushed the ninja branch 9 times, most recently from b867a84 to a5dd1cf Compare December 12, 2023 18:32
roblabla and others added 8 commits December 12, 2023 19:59
Currently, it's very slow to iterate on MRs that cause changes to the
devenv, as rebuilding it can take a long time (currently around 19
minutes) because of the very slow download speeds of archive.org.

Let's work around this by always saving the cache, even if the job
failed, so long as the install_prefix step succeeded.
msiexec has its own very peculiar cmdline parser that does not obey the
usual quotation rules. In particular, if a PROPERTY=X requires the value
to have spaces, it should be quoted as PROPERTY="X WITH SPACES". Trying
to quote the entire property, as "PROPERTY=X WITH SPACES", will result
in the msiexec call failing.
One of the files we compute the sha256 for is 1.6GiB in size. Some
machines may not have that much memory available, so let's just stream
the file chunk by chunk.
@roblabla roblabla force-pushed the ninja branch 3 times, most recently from eb55d61 to 574b521 Compare December 12, 2023 21:36
@roblabla roblabla merged commit 060c02b into happyhavoc:master Dec 12, 2023
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant