Remove Manifests from reference tests

[gdalle]

Following the discussion in #415, the reference test suite needs weakening.
As soon as PkgTemplates.jl installs dependencies that fall outside the standard library, it is impossible (or very cumbersome) to control that a specific version of every subdependency is selected. As a result, I only include Project.toml files in the reference tests, and no longer Manifest.toml files. In fact, the latter are dropped from the repo altogether thanks to .gitignore.

Ping @hannahilea, this is an easy review

[nickrobinson251]

do you know why we were ever including these?

[gdalle]

do you know why we were ever including these?

I wasn't there when the ancient code was written, but I can't think of a compelling reason

[nickrobinson251]

could you please take a quick look in the history to check there wasn't a good reason

otherwise, i've no objection to this

[gdalle]

For one thing there is #143: it's useful to check that the parent package is indeed dev-ed in the docs environment, and this cannot be seen from the Project alone

[nickrobinson251]

nice find! i think that's maybe an important test -- can we keep that test somehow? e.g. keep the docs/Manifest.toml (perhaps with a more limited test that just checks for the presence of the package we expect to be dev'ed)?

[codecov]

Codecov Report

Merging #422 (3632a6a) into master (2c1b526) will decrease coverage by 38.79%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master     #422       +/-   ##
===========================================
- Coverage   93.59%   54.81%   -38.79%     
===========================================
  Files          23       23               
  Lines         687      686        -1     
===========================================
- Hits          643      376      -267     
- Misses         44      310      +266     

see 17 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[gdalle]

I went for the easy solution of only discarding test/Manifest.toml for now, so that this and #415 can be merged. Testing the dev-ing of the package into docs/Manifest.toml is a lot of boilerplate, and might have to be repeated for benchmark/Manifest.toml or the like, so I'd rather do incremental progress.

[nickrobinson251]

thanks!

[hannahilea]

Nice!

Might be worth filing issue so that adding back coverage of #422 (comment) isn't lost

[nickrobinson251]

don't we still have that coverage, because we kept the docs/Manifest.toml? Or are we filtering them out too? As far as i understand, we should be able to guarantee the docs/Manifest.toml versions (for a fixed Documenter version), no?

[hannahilea]

You're right---we're only filtering out joinpath("test", "Manifest.toml")), not all "Manifest.toml". Good!

[cossio]

Any reason to keep WackyOptions/Manifest.toml around? Causes a CI failure at #426, which is otherwise green.

[gdalle]

I think it is related to #423