dmno build - flatten services config for use within docker (or similar) scenarios
#118
Conversation
🦋 Changeset detectedLatest commit: 4cae8c3 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
✅ Deploy Preview for dmno ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
✅ Deploy Preview for signup-api canceled.
|
| // TODO: will need to figure out how to generalize this | ||
| // we could copy everything that is not git-ignored? | ||
| // we could let plugins specify a list of patterns to copy | ||
| await exec(`cp ${workspaceService.path}/.dmno/*.vault.json ${buildPackageDirPath}/.dmno`); |
There was a problem hiding this comment.
there are other items in the .dmno folder which may be needed after flattening - for example our vault file(s) from the encrypted vault plugin. For now this is hardcoded, but we'll either need to rely on git to know which files to copy, or let plugins define some glob pattern of files they care about.
| await exec(`cp ${workspaceService.path}/.dmno/*.vault.json ${buildPackageDirPath}/.dmno`); | ||
|
|
||
| // copy package.json file | ||
| await copyFile(`${workspaceService.path}/package.json`, `${buildPackageDirPath}/package.json`); |
There was a problem hiding this comment.
I'm copying the package.json files. Currently it's only used to get the package name, but we could skip this and store it elsewhere. We also probably want to remove the strict need for having a package.json file anyway, to start supporting more polyglot use-cases.
| await copyFile(`${workspaceService.path}/package.json`, `${buildPackageDirPath}/package.json`); | ||
| } | ||
|
|
||
| await writeFile(`${buildDirPath}/dmno-build-info.json`, JSON.stringify(buildMetadata, null, 2), 'utf8'); |
There was a problem hiding this comment.
Not much in this file yet (see above) but it at least gives us somewhere to put extra data.
| addCacheFlags(program); | ||
| addServiceSelection(program); | ||
|
|
||
| program.action(async (opts: { |
There was a problem hiding this comment.
most of the logic in here should probably move somewhere else, but just wanted to get it working as a POC.
| @@ -540,6 +540,13 @@ export class DmnoWorkspace { | |||
| } | |||
| throw new Error(`unable to find service - ${descriptor}`); | |||
| } | |||
| getServiceMetaForBuild(serviceName: string) { | |||
There was a problem hiding this comment.
we'll probably want to include more metadata here, like which plugins are required, what their dependency versions are, which config items in the related services are actually being used, etc
| @@ -99,10 +99,18 @@ export class ConfigLoader { | |||
| } | |||
|
|
|||
| async reload() { | |||
| // make sure everything is initialized | |||
| await this.isReady; | |||
| if (!this.workspaceInfo) await this.finishInit(); | |||
There was a problem hiding this comment.
had to rework the init flow, since we have the new -b/--built flag which affects the initial load. This isn't quite right but seems ok for now.
| // make sure everything is initialized | ||
| // await this.isReady; | ||
|
|
||
| if (!this.viteNodeRunner) { |
There was a problem hiding this comment.
moving the vite server init is definitely going to make things slower, so we should try to move it and make sure it's happening as early as possible.
theoephraim
force-pushed
the
flatten-build
branch
from
6126b87 to
4cae8c3
Compare
|
commit: |
| checkForSchemaErrors(workspace); | ||
|
|
||
| if (!ctx.configLoader.workspaceInfo.isMonorepo) { | ||
| throw new CliExitError('`dmno build` only works in monorepos'); |
There was a problem hiding this comment.
maybe something slightly more explicit here, like "dmno build is only necessary in monorepos, please use dmno run"
There was a problem hiding this comment.
maybe just link to docs page
|
What's the status of this? |
|
@rawkode - was a working proof of concept, but I need to come back to it. Do you have the rest of your stuff in a state to test it out (one I rebase and get it working again?) |
DMNO allows you to reuse config across a monorepo, picking config items from other services (both parents and siblings). In a scenario where we do not have access to the entire monorepo (like in a docker image), we no longer have access to the config files of these other services (the ones being picked from) or their dependencies (
node_modules). So we must expose these config files and required dependencies somehow. To make this simpler for end users, we will provide admno buildcommand which flattens the config and dependencies into something that can be included in the final build artifact without needing to preserve the entire monorepo.The strategy here is to create a new
.dmno-builtdirectory which has a directory for each service (predecessors in the services DAG only). We'll perform a build (powered by rollup) on each config file, and can inline required dependencies. One complicating factor is that we do want to preserve a single version ofdmnoitself, and any plugins that are being injected, since we useinstanceofchecks in some places, and plugin authors may be making use of the module context or the plugin class for singletons. Eventually we will need to deal with warning users about version mismatches between services - but this should be doable since we have everything available at build time.One complicating factor is that some dependencies could install additional binaries or have build artifacts from installation that would need to be made available in the child. The simple workaround is to explicitly install those dependencies in the child service as well, even if it is not explicitly used in the child's dmno config. We need to think through if there is any better way to deal with this...
This first iteration is not perfect, but it works and it seems the strategy is sound. This should be usable to test out the DX as long as users are aware of the versioning and binary caveats mentioned above.
Also note that currently, I'm naively just including the entire config, but since we understand the full graph, we should be able to remove unrelated config items in the predecessor services, which should help enable further tree-shaking of dependencies.