Rework Coursier Integration and dependency handling (RFC)

[lefou]

Our current coursier integration has some flaws.

• Most prominently, we don't have full control over the used version, as we inherit it transitively through Ammonite.
• We still use the older low level Coursier API
• We haven't found a clean way to handle mismatching cross versions in resolution. The fact that we mix the coursiers dependency class with our own adaptions isn't making this easier.

I think the cleanest solution would be to let our CoursierModule use a Worker which encapsulates coursier, and refactor Mill API to no longer re-use any coursier API. This would give the greatest flexibility, but it's probably the most work. An alternative is to use newer Coursier high level API, but I'm still not sure whether this will fix all our issues. Especially, as this API is considered unstable upstream.

[DamianReeves]

It would be great if there was something in the Coursier implementation which would allow us to mirror directly the handling in sbt. For example when you are behind a proxy like I am for work we use sbt proxy repository configuration; however, it isn't clear how to do this for mill since it seems to be a mismatch of vanilla Coursier configuration that doesn't seem to always carry over (i.e. in workers or forked JVMs).

[jackcviers]

The courier 2.1.8 implementation, beyond other things, by default reads the scala cli global configuration file.

This configuration file specifies

• repository mirrors
• credentials with secret evaluation integratio
• httpProxy handling
• Java properties
• signing keys, gh tokens and other publishing information

And provides keyparsing out of the box. Unfortunately, the PlatformResolve#finalRepositories implementation is broken - it reads the repositories from the proper location, but flattens the iterators in such a way that the information is thrown away. It applies the mirrors correctly, but does not apply the credentials to the configured repositories.

It also pulls in the values of Coursier's environment and jvm property values.

Resolve can take many more config files than the default CoursierPaths and Mirror config files that are already provided, and the ConfigDB abstraction provides ways of parsing values from these various formats.

Given that other RFCs are discussing additional config file considerations, I think it would be wise not to throw out the Coursier dependency.

What may need to happen instead is to fix the known issues upstream via PR, and/or override the behaviors we want to override in CoursierModule, treating it as the boundary adapter to the Coursier integration.

These customizations should be protected or public in scope, so users may override them when necessary.

I would suggest an extension trait for CoursierModule to preserve compatibility, but this introduces an ergonomic problem: users would have to mix it in themselves as the modules that extend CoursierModule wouldn't get the corrected behavior.

Abandoning coursier dependencies would also require users to update all their builds - IvyRepository and MavenRepository models are defined in the courier library.

I think, if we want to change the resolution, that it would be better to provide an extension module that changes the behavior to preserve compatibility. The new extension can be as narrow of a scope as needed, and won't throw the useful parts of coursier out with anything we want to fix.