Semver imports

[dselman]

Feature Request 🛍️

Support imports across semver ranges.

Use Case

Allows less strict management of imports, meaning that when an imported namespace is referenced, if backwards compatible changes to the imported namespace are made, then the version number of the importing namespace does not have to be incremented.

Possible Solution

Support some form of semver range for import statements:

namespace [email protected]

concept Person {
   o String name
}

With strict import (as today):

namespace [email protected]

import [email protected].{Person}

concept Vehicle {
   o Person owner
}

With semver import:

namespace [email protected]

import [email protected].{Person}

concept Vehicle {
   o Person owner
}

Context

Today you can run Concerto in two modes:

• Default: namespaces and imports are unversioned
• Strict: namespaces and imports are fully versioned

Strict mode is very useful in that it makes the dependency graph between concepts completely deterministic, given a single CTO file. I.e. you can take a single CTO file and follow all its versioned imports and build a deterministic graph of every element the model contains. The downside is that changes to namespaces that are "deep" in the dependency hierarchy cause a ripple of changes through out the entire graph. As the graph gets large this is untenable.

Detailed Description

By supporting semver range imports these ripple updates of import versions are minimised, imports only need to be updated when the semver range constraint no longer matches. The downside is that the dependency graph from a CTO file is no longer purely defined by the CTO file itself, it depends on what other CTO files are present, and their versions. The so-called "package-lock.json" problem - there may need to be another mechanism to ensure deterministic versions of imported packages are used, to ensure fully reproducible builds.

Ensuring Reproducible Builds

Below I illustrate an approach of writing back the resolved namespace version into the AST for the model, allowing semver imports to be "pinned" while maintaining them in the CTO or AST.

With semver import, with ability to specify a "resolved version" for an import:

namespace [email protected]

@ResolvedVersion("1.0.1")
import [email protected].{Person}

concept Vehicle {
   o Person owner
}

Note: today we do not support decorators on imports, we should have to make changes to the meta model and the runtime to support this syntax.

1. Create CTO files [email protected] and [email protected]
2. Create a semver import of [email protected] into [email protected]
3. Create a ModelManager with options strict:true and semverImports:true set (do we want a flag for this? Perhaps it should just be supported by default?)
4. Add Parent and Child to the ModelManager
5. Parent resolves imports from [email protected] and models validate
6. ModelManager.resolveMetamodel is used to create an AST that has the ResolvedVersion decorator added to all semver imports, capturing the version of the namespace that was used
7. AST can be stored and converted back to CTO for debugging or audit

When a reproducible ModelManager is required:

1. Create a ModelManager with options strict:true and useResolvedImportVersions:true set
2. Add the ModelFiles for Parent and Child
3. The import of [email protected] into Parent is ignored (because it has the ResolvedVersion decorator) and an import to version 1.0.1 is used. An exception is thrown if this exact version of the model file is not in the model manager.

When the ModelManager should pick up new namespace versions:

1. Create a ModelManager with options strict:true set and semverImports:true
2. Add the updated ModelFiles to the ModelManager, for example [email protected]
3. Parent resolves imports from [email protected] and models validate
4. Call ModelManager.resolveMetamodel to store this configuration, if necessary

Code Generation

When we generate code we will need to use the resolved imports, rather than imports than contain semver ranges.

Tasks

• Decide on semantics of semver imports (.x, ranges etc)
• Update the meta model to support semver imports
• Update the parser to support semver imports for CTO
• Convert AST back to CTO
• Update ModelFile to resolve imports with semver
• Update VSCode extension
• Update the meta model to support decorators on imports
• Update the runtime to support decorators on imports

[mttrbrts]

Is there consensus on format of ranges we should support? Here are some alternatives

• 1.x X-ranges
• ^1.0.0 Caret ranges
• >=1.1.0 <1.2.0-0 desugared comparators
• 1 Partial ranges

Further, would we only support major version ranges (as above), or would we support more complex expressions, e.g.

• 1.2.x
• >=1 <=3.2

As for the package-lock.json problem ...

We'll need to fully resolve versions to do validation and introspection inside the modelManager. It seems reasonable that we could dump the static versions for auditing purposes quite easily. However, the reproducibility problem seems harder, especially that we don't have a mechanism to pull model versions from a repository (we don't even have a standard repository protocol / index!). We could conclude that reproducibility is a client-concern for now! 😨

[dselman]

Semver imports also complicates caching / immutability of models. I.e. if I request namespace [email protected] (along with its dependencies) the response payload cannot be cached using [email protected] as the cache key.

[dselman]

Another consideration is that today the ModelFile has a map (importShortNames) from local short name to FQN that is populated from the AST. This approach would not work, because the FQN to use could change whenever a new model file is added to the ModelManager that matches semver range for the import. We'd have to use logic similar to how strict:false wildcard imports are used, which are resolved on-demand (some performance impact), or introduce some sort of lifecycle control on the ModelManager whereby it would be declared frozen.

Or, we need an eventing mechanism to allow the ModelFiles to dump their importShortNames map when a new ModelFile is added to the ModelManager.

[dselman]

Is there consensus on format of ranges we should support? Here are some alternatives

• 1.x X-ranges
• ^1.0.0 Caret ranges
• >=1.1.0 <1.2.0-0 desugared comparators
• 1 Partial ranges

I'm leaning towards just supporting x-ranges. E.g. 1.x or 1.2.x. I think that gives us what we know we need and should not complicate the parser too much. x-ranges are also simple to explain and to understand.

Do you think we should give people "the rope" and support x-ranges like x which allow for breaking changes?

[mttrbrts]

I'm leaning towards just supporting x-ranges. E.g. 1.x or 1.2.x. I think that gives us what we know we need and should not complicate the parser too much. x-ranges are also simple to explain and to understand.

👍🏻

Do you think we should give people "the rope" and support x-ranges like x which allow for breaking changes?

Yes, I can imagine scenarios where version upgrades are forced despite the dependency not changing.

Here's the spec for x-ranges from npmjs semver package, https://www.npmjs.com/package/semver.

Any of X, x, or * may be used to "stand in" for one of the numeric values in the [major, minor, patch] tuple.

• := >=0.0.0 (Any non-prerelease version satisfies, unless includePrerelease is specified, in which case any version at all satisfies)
  1.x := >=1.0.0 <2.0.0-0 (Matching major version)
  1.2.x := >=1.2.0 <1.3.0-0 (Matching major and minor versions)
  A partial version range is treated as an X-Range, so the special character is in fact optional.

"" (empty string) := * := >=0.0.0
1 := 1.x.x := >=1.0.0 <2.0.0-0
1.2 := 1.2.x := >=1.2.0 <1.3.0-0

[dselman]

What do we do for pre-release imports?

[dselman]

Name collisions are possible - that could be breaking changes for specialised types. E.g. when I add a new property to a super type (semver minor) that conflicts with an existing property in a specialised type.

[mttrbrts]

Let's release this behind a feature flag.