Feat: Create immutable builder for incremental Cardano DB

[sfauvel]

Content

The CardanoDatabaseArtifactBuilder can build and upload immutables files and digests file

Implement the artifact builder for the Incremental Cardano Database with the target JSON structure:

{
  "merkle_root": "0506e5e1f7df9314ebdbb532768a5addc2931004c5cd8fd8cd464090f95e46e4",
  "beacon": {
    "network": "mainnet",
    "epoch": 525,
    "immutable_file_number": 6550
  },
  "certificate_hash": "04219eb7a66c37f0d917160b8bdc617dc4ff911344f2e72bdb22c1316cbe8d71",
  "total_db_size_uncompressed": 52470081414,
  "created_at": "2024-12-02T04:36:40.811764301Z",
  "locations": {
    "digests": [
      {
        "type": "aggregator",
        "uri": "https://aggregator.release-mainnet.api.mithril.network/aggregator/artifact/cardano-database/digests"
      },
      {
        "type": "cloud_storage",
        "uri": "https://storage.googleapis.com/cdn.aggregator.release-mainnet.api.mithril.network/mainnet-e525-i6550.digests.tar.zst"
      }
    ],
    "immutables": [
      {
        "type": "cloud_storage",
        "uri": {
                "Template": "https://storage.googleapis.com/cdn.aggregator.release-mainnet.api.mithril.network/mainnet-e525-i6550.{immutable_file_number}.tar.zst"
        }
      },
      {
        "type": "bit_torrent",
        "uri": {
                "Template": "https://storage.googleapis.com/cdn.aggregator.release-mainnet.api.mithril.network/mainnet-e525-i6550.{immutable_file_number}.tar.zst"
        }
      }
    ],
    "ancillary": [
      {
        "type": "cloud_storage",
        "uri": "https://storage.googleapis.com/cdn.aggregator.release-mainnet.api.mithril.network/mainnet-e525.ancillary.tar.zst"
      }
    ]
  },
  "compression_algorithm": "zstandard",
  "cardano_node_version": "10.1.2"
}

And the target design:

Pre-submit checklist

• Branch
  • Tests are provided (if possible)
  • Crates versions are updated (if relevant)
  • CHANGELOG file is updated (if relevant)
  • Commit sequence broadly makes sense
  • Key commits have useful messages
• PR
  • No clippy warnings in the CI
  • Self-reviewed the diff
  • Useful pull request description
  • Reviewer requested
• Documentation
  • Update README file (if relevant)
  • Update documentation website (if relevant)
  • Add dev blog post (if relevant)

Issue(s)

Closes #2151

[github-actions]

Test Results

    4 files  ± 0     52 suites  ±0   10m 18s ⏱️ -28s
1 524 tests +31  1 524 ✅ +31  0 💤 ±0  0 ❌ ±0 
1 780 runs  +31  1 780 ✅ +31  0 💤 ±0  0 ❌ ±0 

Results for commit 1b1913b. ± Comparison against base commit 90f44cd.

This pull request removes 6 and adds 37 tests. Note that renamed tests count towards both.

mithril-aggregator ‑ snapshotter::tests::test_dumb_snapshotter_snasphot_return_archive_name_with_size_0
mithril-common ‑ entities::cardano_database::tests::compute_hash_returns_different_hash_with_different_beacon
mithril-common ‑ entities::cardano_database::tests::compute_hash_returns_different_hash_with_different_merkle_root
mithril-common ‑ entities::cardano_database::tests::compute_hash_returns_different_hash_with_same_epoch_in_beacon
mithril-common ‑ entities::cardano_database::tests::compute_hash_returns_same_hash_with_same_cardano_database_snapshot
mithril-common ‑ entities::cardano_database::tests::test_cardano_database_snapshot_compute_hash

mithril-aggregator ‑ artifact_builder::cardano_database_artifacts::digest::tests::create_digest_file_should_create_json_file_with_all_digests
mithril-aggregator ‑ artifact_builder::cardano_database_artifacts::digest::tests::digest_artifact_builder_return_digests_route_on_aggregator
mithril-aggregator ‑ artifact_builder::cardano_database_artifacts::digest::tests::upload_digest_file_should_log_upload_errors
mithril-aggregator ‑ artifact_builder::cardano_database_artifacts::digest::tests::upload_digest_file_should_not_error_even_if_no_location_returned_from_uploaders
mithril-aggregator ‑ artifact_builder::cardano_database_artifacts::digest::tests::upload_digest_file_should_return_all_uploaders_returned_locations
mithril-aggregator ‑ artifact_builder::cardano_database_artifacts::digest::tests::upload_digest_file_should_return_location_even_with_uploaders_errors
mithril-aggregator ‑ artifact_builder::cardano_database_artifacts::digest::tests::upload_should_call_upload_with_created_digest_file_and_delete_the_file
mithril-aggregator ‑ artifact_builder::cardano_database_artifacts::immutable::tests::batch_upload::extract_archive_name_to_deduce_template_location
mithril-aggregator ‑ artifact_builder::cardano_database_artifacts::immutable::tests::batch_upload::returns_error_when_uploaded_filename_not_templatable_without_5_digits
mithril-aggregator ‑ artifact_builder::cardano_database_artifacts::immutable::tests::create_archive::return_all_archives_but_not_rebuild_archives_already_compressed
…

♻️ This comment has been updated with latest results.

[jpraynaud]

LGTM 👍

[dlachaume]

LGTM 🚀

Just a quick note, I think the location examples in the openapi.yaml file should be updated with the example provided in the description of this PR.
The examples for CardanoDatabaseArtifactsLocationsMessagePart and CardanoDatabaseSnapshotMessage should be updated with the values from the JSON above.

[Alenar]

LGTM 🚀

I've some doubt on the new methods added to the snapshotter trait but this can be addressed in another PR.

[Alenar]

If I'm not wrong there's a little grammar error here, it should be does instead of is

Suggested change

	fn is_snapshot_exist(&self, filepath: &Path) -> bool;
	fn does_snapshot_exist(&self, filepath: &Path) -> bool;

[Alenar]

Those new methods are difficult to understand and use.

They operate using relative path from a cardano database directory:

• This provide no abstraction for callers as they need to know exactly the structure of a cardano database meaning that the knowledge of it is spread out on all callers.
• This allow incorrect use such as giving an absolute path.

We could refactor the whole trait the following way:

• make public the TarAppender trait (maybe rename it) and change it's implementers to be domain based:
  • ImmutableFileFullAppender instead of AppenderDirAll
  • AncilaryAppender and ImmutableTrioAppender instead of AppenderEntries
• replace the snapshot_all and snapshot_subset methods of the snapshotter with a unique snapshot method that take those TarAppender implementation as a parameter.

Note

An alternative would be to introduce a new type instead, maybe an enum, and keep the existing appender system private with a mechanism that construct the appropriate appender for a given new type.

This refactor would allow to remove the needs for snapshotter user to have knowledge of the cardano database inner structure. Simplified usage example:

• before:

if !snapshot_all.is_snapshot_exist(Path::from('immutables_trio-i4378.tar.zst')) {
  snapshotter.snapshot_subset(
    vec![
      PathBuff::from(IMMUTABLE_DIR).join("04378.chunk"),
      PathBuff::from(IMMUTABLE_DIR).join("04378.primary"),
      PathBuff::from(IMMUTABLE_DIR).join("04378.secondary")
    ]
  );
}

• after:

let appender = ImmutableTrioAppender::immutable(4378);
if !snapshot_all.is_snapshot_exist(&appender) {
  snapshotter.snapshot(appender);
}