Verify: Validation test case enumeration

[asraa]

Hi all!

I was curious about starting a list of test-cases for validation failure modes during verification. Some of these are difficult because it requires tampering with the Rekor responses, which won't be solved until #25 is done.

I wanted to take a stab at creating a list of the failure modes... I'm sure I am missing some.

Failure Mode	Testcase(s)
Signature Validation Failure	Mismatched artifact against a given signing certificate and signature. Mismatched signature against a given artifact and signing certificate. Mismatch signing certificate against a given artifact and certificate. Empty signature. Empty signing certificate. Empty artifact.
Certificate Validation Failure	Invalid Certificate chain: not issued by configured root CA Certificate issued after Fulcio Root CA expiration [MOCK]
Short-lived certificate signature validation failure	Bypass transparency log upload: no transparency log entry returned. Invalid transparency log entry returned: mismatch with entry. [MOCK] Invalid transparency log entry returned: incorrectly signed entry timestamp [MOCK] Invalid transparency log entry returning: invalid inclusion proof [MOCK] Invalid transparency log entry returning: invalid signed root hash [MOCK] Integrated time of log entry was past certificate validity period.
Offline Rekor proofs or Bundles	Invalid bundle: Invalid signed entry timestamp Invalid bundle: does not match provided signature, artifact, and certificate. Integrated time of offline log entry was past certificate validity

[woodruffw]

That's a great starting list, thanks for enumerating and collecting it!

I think some of these will be captured in the conformance suite, but it'll be great to have them as standalone tests within sigstore-python as well.

[tetsuo-cpp]

I've started filling in some of this test coverage that doesn't require mocks in the linked PR. The way I see it, 1 and 6 from "Short-lived certificate signature validation failure" probably also require mocks (though I might just be lacking imagination haha).

• 1: Clients won't write certificates and signatures to disk until the log entry is published to Rekor. Therefore, I can't reproduce this situation where we have a legitimate signature and certificate written to disk for a given artifact and Rekor somehow doesn't know about it.
• 6: In order to have an log entry with an integrated time after the certificate expiry, I'll need to somehow make the Sigstore client sleep for 10 min after retrieving the certificate from Fulcio. I think it'll be easier to get the mock Rekor to just forge the integrated time.

[asraa]

Instead of mocking whole Rekor instances, could you use some candidate canned Rekor responses?

1: Clients won't write certificates and signatures to disk until the log entry is published to Rekor. Therefore, I can't reproduce this situation where we have a legitimate signature and certificate written to disk for a given artifact and Rekor somehow doesn't know about it.

Can you create a malicious library code that comments out the upload to Rekor, or upload to a different instance (e.g. upload to staging, try to verify against prod)?

6: In order to have an log entry with an integrated time after the certificate expiry, I'll need to somehow make the Sigstore client sleep for 10 min after retrieving the certificate from Fulcio. I think it'll be easier to get the mock Rekor to just forge the integrated time.

OH I think the issue is that you're trying to use the same clients to produce the bad entries? These will have to be canned and only on the verification path: as in, in sigstore-python or your client of choice, create this artifact, and feed it in as a test-case to the verifiers in each langauge during conformence test.
Some of these cases intentionally require tampering with the client.

[tetsuo-cpp]

Instead of mocking whole Rekor instances, could you use some candidate canned Rekor responses?

Thanks @asraa, that's a good idea.

I was initially imagining a Rekor-like mock that can pass some basic cases and for each test, I'd be able to override the handler and make it return something different (bad SET, etc).

Your idea makes a lot more sense though. I can get away with something simple that replays canned responses and I think we'll be able to test most of what we're interested in. For some of the more tricky cases, I can edit the JSON by hand too.

Can you create a malicious library code that comments out the upload to Rekor, or upload to a different instance (e.g. upload to staging, try to verify against prod)?

I think if I upload to staging, verification will fail at the certificate check since we'll be using the production Fulcio root certificate to verify. But commenting out the Rekor upload will work.

[woodruffw]

Here's another test case we'll want to include: we'll want to make sure that clients can successfully verify old signatures, even when their corresponding Fulcio cert chain/CT log key have been rotated out. This would ensure that clients aren't just vendoring the newest roots of trust, but are instead using TUF to fetch both the active and historic roots.

[steiza]

Today in the Sigstore clients meeting we talked about updating the CLI protocol to have the conformance tests additionally supply a trust root file, a TUF root URL, or both. This will allow us to extend our test cases, like @woodruffw mentions above, and also test that supplying the wrong trust root (or a malformed trust root) fails.

We should add these new tests with a feature flag, like we did for #81, so clients can opt-in to the updated CLI protocol on their own timeline without breaking existing users (although eventually we will require using the updated CLI protocol).

Longer-term, we might update conformance testing to make use of its own trust roots, so we can construct more complicated test scenarios.

[woodruffw]

Something that @jleightcap and I noticed today: the Protobuf JSON mapping allows both camelCase and snake_case names, so we should probably add a testcase that ensures that clients accept both.

[tnytown]

Another case that falls under Certificate Validation Failure: Certificate that isn't included in the CT log, xref sigstore/sigstore-python#886.