Build system using vcpkg

[dpogue]

Currently, our build instructions rely on downloading a zip file of pre-compiled dependencies. While this is much easier than finding the right versions of dependencies and compiling them in the right configurations yourself, it's also less than ideal for testing changes to those dependencies or non-standard build configurations.

vcpkg is a Microsoft-developed package manager for C++ libraries that supports (almost) all of our dependencies. It has a feature to declare dependencies in a JSON-based manifest file, and also integrates as a CMake toolchain to automatically install those dependencies as part of the build step (Docs).

Many of us have been using vcpkg manually for over a year to compile dependencies for local development, and @Hoikas has been leading efforts to get it set up as part of our standard build system.

The goal is to simplify our entire build process to the following for both development, and continuous integration:

git clone --recurse-submodules https://github.com/H-uru/Plasma.git
cd Plasma
cmake -B builddir -S .
cmake --build builddir

Both CMake and vcpkg have some built-in support in Visual Studio, but as of December 2020 it's not quite at a point where it works smoothly.

Prerequisites

• Upgrade to PhysX 4.1 (which is available in vcpkg) (PR)
• Remove dependency on specific DirectX SDK version (PR)
• Fix the issues with the vcpkg package for Python 3.8 (PR, PR & more)
• Add a vcpkg.json manifest file
• Add vcpkg as a git submodule (to discuss)
• Reference the vcpkg toolchain in CMake

Known Limitations

• The 3D Studio Max SDK is not available through vcpkg, nor is it redistributable. We will not be able to compile PlasmaMax plugin as part of our CI pipeline.

[dpogue]

Note: I was trying to use vcpkg to experiment with a 64-bit build yesterday, and found microsoft/vcpkg#14545 is also necessary for libvpx to compile properly from the manifest file

[Hoikas]

Was this on Linux or Windows?

[dpogue]

This was on Windows, it wanted the 32-bit version of yasm-tools for some reason, despite compiling a x64 triplet

[dpogue]

Question: vcpkg supports Windows, macOS, and Linux. If we integrate it in our CMake process, will it be used by default on those platforms even ahead of system libraries?

Most of Plasma's dependencies are available through package managers on Linux, and the preference there (for development anyways) is to dynamically link against those libraries rather than compiling and statically linking them yourself. It would be great if we could use the system library if it exists, but still rely on vcpkg for stuff that's not commonly in package managers (string-theory, physX, etc.)

[Hoikas]

vcpkg will only be used if cmake is invoked with the vcpkg toolchain: cmake -DCMAKE_TOOLCHAIN_FILE=../vcpkg/scripts/buildsystems.vcpkg.cmake ... Ideally, you either use vcpkg for everything or you use it for nothing. But, on Linux and macOS, it sometimes does require things be installed from the system package manager.

[dpogue]

yeah, I was sort of imagining that we would add the toolchain reference to our top-level CMakeLists.txt, pointing to the local vcpkg directory:

set(CMAKE_TOOLCHAIN_FILE ${CMAKE_CURRENT_SOURCE_DIR}/vcpkg/scripts/buildsystems/vcpkg.cmake
  CACHE STRING "Vcpkg toolchain file")

[Hoikas]

That could work, but we'd probably want an escape hatch, just in case. I've been thinking that we would want to modify prepare_env.ps1 - vcpkg can use NuGet as a cache for libraries, and GitHub operates a NuGet package server. I think this would be helpful for keeping build times down for both CI and folks who are compiling the code at home.

[dpogue]

Question: Including vcpkg as a git submodule is currently recommended as the best way to "lock" dependencies to certain versions. It sounds like there are future plans to allow specifying version requirements in the manifest file.

The upside of a local vcpkg copy is that we can be sure the CMake toolchain file is in a known location.

The downside of a local vcpkg copy is that you have to clone the git submodule and all of vcpkg's history.

[Hoikas]

Even with the versioning support in vcpkg (which is coming fairly soon, it seems), there are still the issues of vcpkg ports having bugs or missing features. Case in point, I recently added support for macOS and Linux to the physx port and the lingering issue with needing to manually specify python artifacts. Having a submodule will allow us to lock in a definite set of known good vcpkg ports.

[Deledrius]

It appears that submodules support shallow cloning, so we can mitigate the problem of pulling down the entire vcpkg history. If we're including it for the purpose of locking to specific versions, than a --depth 1 should be entirely sufficient.

[Hoikas]

#733 is also a requirement for this, I think.

[Hoikas]

Related... I did a significant amount of tinkering to get a fairly "working" OOB experience with the client from CI on Windows/Visual Studio with vcpkg. Here is a build that should "just-work" if you drop in the assets from moul-assets. One sticking point that we have there and in the instructions included in README.md is that there is either a lot of manual moving of files or hackery at the cmake level required to get to this point.

To fix this, I propose that we create an install target using cmake's built-in install command. Given that we have in this repository the client, many tools, scripts, and max plugin, I think we should not necessarily install using the standard GNU directories. The goal is to create a coherent distribution such that if you download a ZIP from CI, it's not just a jumble of files. So, I propose our structure look something like this:

- client
|    - dat
|    - python
|    |    - ki
|    |    - system
|    - SDL
|    - sfx
|    - plClient.exe
|    - plUruLauncher.exe
- tools_cli
|    - plShaderAssembler.exe
|    - [...]
- tools_gui
|    - plResBrowser.exe
|    - Qt5[*].dll

We may also want to consider adding a submodule to moul-assets to further this goal, but that is less urgent, I think.

[Deledrius]

That seems like a good idea to me.

[Hoikas]

Initial work available in #745.

[Hoikas]

The first half of the vcpkg Python work got merged today, and I'll be submitting the second half in the next few days. PhysX 4.1 is in master, the DirectX SDK is dead, and we have a nice install target now. Yay progress!

On the topic of concerns... One desired workflow was brought up in the OpenUru Discord that may be worth considering. Namely, that in Visual Studio 2019, there is baked-in support for git and GitHub. So, it should be very easy to open Visual Studio 2019 and clone our repository. Ideally, we could use this and bake in some vcpkg bootstrapping into our default CMake on Windows to provide a rather seamless experience. This would make prepare_env.ps1 rather redundant in that everything now happens in the IDE. So the workflow would look something like:

• Open Visual Studio 2019
• File > Clone Repository
• Clone Plasma
• Install Project
• Profit

As a reminder, the current workflow is to download the repository's zip file, run prepare_env.bat, build the solution's install project, and download the resource.dat.

Our documentation all currently points to Visual Studio 2017, which does not have such seamless git integration from what I can tell. It does support CMake projects, though, I believe. If we were to bump to requiring Visual Studio 2019, then we would be irrevocably dropping support for Windows XP, which we constantly hear whining about. This means that anyone using Visual Studio 2017, the current recommended compiler, would need to:

• Use a git client (eg gitkraken or GitHub Desktop) to clone the repository -- the repository zip file does not include submodules and omits the .git directory, so cloning the vcpkg submodule would be challenging
• Figure out how to open the project in VS 2017 where CMake integration is less than optimal OR run something similar to prepare_env.pa1
• Build the install target.
• Profit

Assuming an official adoption of vcpkg and removal of the "devlibs" concept... Another issue we may see, assuming we use our CI system to provide cached libraries, is the libraries built on CI under Visual Studio 2017 may not restore properly on systems with Visual Studio 2019 as the most recent compiler. IIRC the compiler hash is part of each cached library's hash in vcpkg. This isn't a critical error, it just means that it's more likely that folks could wind up building the dependencies in vcpkg -- the only hardship this represents is time, however.

I suppose all this is to just say that I think we should consider "officially" bumping to Visual Studio 2019 if we want to really adopt vcpkg on Windows with a submodule.

[Hoikas]

I'm worried that using multiple compilers (but the same vcpkg triplet) would potentially introduce nondeterminant behavior in our CI caching. I think we should make a decision one way or another.

AFAIK the Windows XP issue remains on MOULa. At this point, it's a 19 year old OS that's been out of support for 6.5 years. At this point, I think it's unreasonable to support XP... Nevermind that Windows 7 has gone the way of the dodo since then. But, still, I expect someone will bring it up.

[dpogue]

I realize this would significantly increase build times, but it is possible to tell (only) the VS2017 build not to cache vcpkg libraries?

[Hoikas]

Yes, but IMO the increased time makes it really not worth pursuing. The gain we receive from avoiding the AppVeyor queue would be effectively eliminated. Not to mention the extra CI script complexity. 😢

[Hoikas]

Something I just realized, btw, regarding Windows XP... Python 3.9 dropped support for Windows 7, so I think it highly unlikely that 3.8 supports XP. Thus, making that entire point moot.

Edit: Yep. Python 3.4 was the last version to support XP. Luckily, it looks like the Windows 7 support drop in Python 3.9 was just a block added to the installer.

[Deledrius]

On the other hand, we know for a fact that the official MOULa client accidentally dropped Windows XP support in August. Was that ever fixed, or is status quo on MOULa that XP is no longer supported?

AFAIK the Windows XP issue remains on MOULa.

To be honest, it's disappointing and frustrating. We've been told for years that this was a mandatory support requirement, so while we've dropped direct support in our builds we've been mindful of not actively harming the ability to build for it in deference to the utility of our code for MOULa. Now that MOULa dropped support for XP without warning or discussion (and the fact that it was immediately noticed and reported but not fixed), it seems safe to assume that it was either intentional or an accident that has been embraced as the new baseline going forward. Considering we have changes in ready to go that will finally preclude support for XP substantially due to third-party libraries no longer supporting it whatsoever, I feel its time has come.

If the libraries the project depends on cannot be built for a specific OS, it makes no sense to block on upgrading the build tools for failing to support said OSes.

[ghost]

Just fired up an old VM, and it appears MOULa does not support Windows XP. Attempting to run the MOULInstaller.exe from mystonline.com on WinXP 32-bit gives the common "this is not a valid Win32 application" error.

Copying an existing installation (version Uru.Live.1.918.OpenUru#173) over also does not work. It throws "The procedure entry point WakeAllConditionVariable could not be located in the dynamic link library KERNEL32.DLL" when UruLauncher tries to launch UruExplorer.exe. According to Microsoft, that function is only in Windows 7 and above.

[Hoikas]

While I suspect the installer not working on XP was an oversight, it could be interpreted as an official "Windows XP is no longer supported" from Cyan. As time moves forward, more and more 3rd parties either intentionally drop support for or unintentionally break Windows XP. It seems that, given these circumstances, the argument we had for sticking with VS 2017 is rather moot. I'd be curious to know @zrax's take.

[zrax]

If our only reason for sticking with VS 2017 was the XP toolchain, I see no reason not to bump our requirement up to VS 2019. I believe most of us (the active developers/contributors) are using VS 2019 by now anyway.

[Hoikas]

OK, I have a working prototype of this ready for experimenting. You can see the code here and the working CI output here. The CI now outputs clients and tools that work once you supply the game assets. For reasons enumerated below, we need to look at this here before I open a pull request.

So, what this does... By default on Windows, we now use an in-repo vcpkg submodule to manage dependencies. This way we are locked into a known working set of dependencies. Further, these dependencies are now built in the Plasma repo's CI and our cmake+vcpkg are responsible for fetching the "devlibs", effectively obsoleting the H-uru/PlasmaPrefix repo. This uses the GitHub Packages to manage everything.

This approach has some drawbacks. Foremost is that GitHub Packages has no anonymous access. Meaning that in order for this to work, we will need to generate a token on the H-uru organization with the scope read:packages and include it in-repo. Right now, everything is pointed to my fork and account. For the actual PR, this will need to be changed to a H-uru token. Also, because of the security scoping, testing this in PR form once that is done may be challenging... Pull requests won't be able to write to the H-uru packages store, but everyone will be reading from the H-uru package store. In orther words, test it now ^_^.

On the plus side... We are now using GitHub Actions instead of AppVeyor, so we get plenty of CI parallelism. No more need to wait 1 hour for each Windows CI run 😄

If you insist on not using the vcpkg submodule, you can avoid it by passing -DUSE_VCPKG=OFF on the cmake command line. This is only needed on Windows - vcpkg is both opt-in and untested on Linux/macOS. If you want to use this, then you will probably need to regenerate your build directory - this is a vcpkg limitation that cannot be overcome.

[dpogue]

Another option:

• Git Command Line
  • Submodules are off by default... some flag can be passed with clone (varies by git version)
  • Updating submodules is arcane sorcery (don't bother with the docs, they are largely incomprehensible)
  • What even is UI?

[zrax]

One thing to consider here: If you recommend a specific client, you should be prepared to support it. I personally only use the git command line client, and have no experience with the various GUIs listed above (and last time I tried GitKraken, I hated it -- though I admit it was a long time ago and it might have gotten better since then). Is anyone here knowledgeable enough with one of the clients to support it when other contributors inevitably have questions or issues with it?

For the command line client, submodules are not nearly as arcane or difficult to work with in modern Git versions as they were years ago (and on Windows, where we actually care, it's not difficult to get a relatively up-to-date version). A quick example:

# clone the repo with all submodules
$ git clone --recurse-submodules <URL>

# Pull with submodule updates
$ git pull --recurse-submodules

# Make "--recurse-submodules" the default
$ git config --global submodule.recurse true

[Hoikas]

That's a good point. In the instructions I drafted last night, I explicitly called out git clone --recurse-submodules <URL>. Perhaps it would be sufficient to remove the specific recommendation of gitkraken and leave a note to be sure that submodules are fetched.

[Deledrius]

My other concern with GitKraken is that, last time I checked, it requires a user account to even use the software. That's an extra (unnecessary) hurdle. It's worth it if you're going to be using it regularly and don't mind doing so, but for a single checkout it's just a user funnel into their ecosystem and they know it.

[Deledrius]

A detailed walkthrough How to check out the repository that covers three or four major clients would make an excellent candidate for a wiki page. The README should stick to the most efficient route, possibly linking to such a page for more information.

[Hoikas]

💥