Improve build efficiency by building library subprojects in parallel #630
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Each library variant build consists of three subprojects, which require sequential configuring and building. While the build steps can be parallelized by using multiple processes to compile the sources, the configuration step is largely single threaded. The multilib project has to build many variants now, and so a significant amount of time is spent waiting for the configuration of various projects to finish. This patch attempts to make the build process more efficient by running all the configuration steps for every variant subproject in parallel. The configuration/build steps in the underlying runtimes project are exposed, so that the higher level multilib project can invoke them as a single target. So instead of building each variant in turn, all the versions of compiler-rt are configured, then all versions are built, then all the C libraries are configured, and so on. This maximizes the amount of work that can be done at any given time. The build steps already benefit from parallelization: although built in series, each has the full number of available processes to use. However, at higher CPU counts there is an observable limit on how many processes each build can effectively use. So it may also be more efficient to instead run the build steps in parallel with a smaller number of proceses available to each. The option to control it (ENABLE_PARALLEL_LIB_BUILD) is OFF by default as which strategy is faster may come down to your available hardware, whereas ENABLE_PARALLEL_LIB_CONFIG should always be beneficial, so I've defaulted that to ON.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Each library variant build consists of three subprojects, which require sequential configuring and building. While the build steps can be parallelized by using multiple processes to compile the sources, the configuration step is largely single threaded.
The multilib project has to build many variants now, and so a significant amount of time is spent waiting for the configuration of various projects to finish.
This patch attempts to make the build process more efficient by running all the configuration steps for every variant subproject in parallel. The configuration/build steps in the underlying runtimes project are exposed, so that the higher level multilib project can invoke them as a single target. So instead of building each variant in turn, all the versions of compiler-rt are configured, then all versions are built, then all the C libraries are configured, and so on. This maximizes the amount of work that can be done at any given time.
The build steps already benefit from parallelization: although built in series, each has the full number of available processes to use. However, at higher CPU counts there is an observable limit on how many processes each build can effectively use. So it may also be more efficient to instead run the build steps in parallel with a smaller number of proceses available to each. The option to control it (ENABLE_PARALLEL_LIB_BUILD) is OFF by default as which strategy is faster may come down to your available hardware, whereas ENABLE_PARALLEL_LIB_CONFIG should always be beneficial, so I've defaulted that to ON.