Pruning skeleton #1229

rakanalh · 2024-09-25T11:39:04Z

Description

This PR implements pruning.

Here we introduce the Pruner structure which runs for as long as the node is running to prune blocks every X number of blocks.
At each L2 block production, we calculate the trigger block number which would invoke pruning X - distance blocks. This happens by spawning blocking tasks (since we use sync rocksdb) to delete specific state based on up_to_block.

The implementation of the actual pruning is implemented in later tickets referenced in:
#1190

Linked Issues

Fixes Pruning: Implement pruning crate & config #1191

MERGE AFTER

crates/pruning/src/lib.rs

crates/prover/src/runner.rs

crates/pruning/src/lib.rs

ercecan

So I thought the distance thing was supposed to work like the window size of a sliding window, and prune after each new l2 to keep the distance same, but after we talked with @rakanalh it made sense to at least wait another distance amount of blocks to prune so that it does not have to deal with extra (blocking) reads and writes every new block

yaziciahmet

Great work overall, just have some nitpicks and questions.

crates/pruning/src/lib.rs

crates/pruning/src/pruners/evm.rs

crates/pruning/src/lib.rs

bin/citrea/provers/risc0/guest-mock/Cargo.lock

resources/configs/mock/sequencer_config.toml

crates/pruning/src/lib.rs

* Structured concurrency: Part 1 (#1214) * Implement managed tasks for SubscriptionManager * Fix clippy * Add comment about trace task * Use broadcast instead of mpsc for l2 blocks events * Fix typo (#1246) * Structured Concurrency: Part 2 (#1216) * Implement managed tasks for SubscriptionManager * Fix clippy * Add comment about trace task * Use broadcast instead of mpsc for l2 blocks events * Fix spawns in DaService * Add TaskManager * Use task manager in sequencer * Document TaskManager * Handle shutdown event * Use TaskTracker * Add comment about using a cancellation token * Use JoinHandles instead of TaskTracker * Use TaskManager in fullnode and prover * Improve bitcoin-da service * Force spawned tasks to accept a cancellation token * Use biased polling * Satisfy clippy * Address PR feedback * Fix checks * Pin foundry (#1253) * Pin foundary * Add comment * Remove puclihs mock da block script * Use default block time on error (#1244) * Use default block time on error * Constant target_block_time * Remove import * Dump logs and cleanup on assert failures (#1252) * Fix estiamte gas l1 fee issue when metamask max amount is selected (#1261) * Fix estiamte gas l1 fee issue when metamask max amount is selected * Fix tests * Remove unnecessary comment * Fix hive (#1263) * Fix configs * Remove publish mock block from docker * Update port for hive * Rename config in hive * Refactor eth estimate gas l1 fee issue (#1264) * Refactor eth estimate gas l1 fee issue * Nits * Fix bug * Refactor * Do not ignore resources * Fix --------- Co-authored-by: Roman Proskuryakoff <[email protected]> * Pruning skeleton (#1229) * Implement managed tasks for SubscriptionManager * Fix clippy * Add comment about trace task * Use broadcast instead of mpsc for l2 blocks events * Fix spawns in DaService * Add TaskManager * Use task manager in sequencer * Document TaskManager * Handle shutdown event * Use TaskTracker * Add comment about using a cancellation token * Use JoinHandles instead of TaskTracker * Use TaskManager in fullnode and prover * Improve bitcoin-da service * Force spawned tasks to accept a cancellation token * Use biased polling * Satisfy clippy * WIP * Add pruning tables * Pruning skeleton implementation * Use pruner in nodes * Use biased polling based on order * WIP * Fix how config is done * Derive default * Add logs * Let the tasks finish without panicing * Use pruning config in fullnode and prover * Add simple run test * Use option instead of PruningMode * Unneccessary changes * l2_receiver * Cleanup prints * Use last pruned block in calculation * Implement pruning criteria * Lint and add comment * Set the last_pruned_block to up_to_block value * Don't store config internally * Remove from constructor * Should not change * Move config value * Remove pruning from sequencer / prover * Derive SequencerClient (#1269) * Derive SequencerClient * Renames * Lint * Improve estimate gas and create access list rpcs (#1265) * Remove unnecessary transact_to conversion * unwrap_or_default * More concise block_env initialization * Actually get highest gas limit from request and block env * One liner set * Allow unused * Cleanup prepare_call_env * Fix tests & lint * Replace unwrap * Set state to block in eth_call * Consume request in prepare_call_env * Lint * Set state before reading config * Remove unnecessary clone * Replace allow unused with feature native gate * Add sys txs to evm tests (#1255) * modify config_push_contracts * modify call_multiple_test * modify tests in call_tests.rs * moving common functions to utils.rs * minor fixes * updated lock files * modify tests --------- Co-authored-by: Esad Yusuf Atik <[email protected]> * make prover config arg not optional (#1278) * Pin to 27.1 (#1279) * E2E tests using citrea-e2e (#1277) * E2E tests using citrea-e2e * Lint * Set CITREA path * With github.workspace * Target citrea-e2e main * Update citrea-e2e rev * Use debug build * Update citrea-e2e * Prover generate input rpc (#1280) * WIP Implement prover generate rpcg * It compiles but stf is modified * Context * Fix Context: Send * Merge fix * Move function from common to prover * Remove code duplicate and unnecessary log * Add optional parameter to break commitments into groups * Return input as string --------- Co-authored-by: Roman Proskuryakoff <[email protected]> * Move node configs to citrea-common (#1286) * move node configs to a seperate crate * fix udeps * remove native feature * move SequencerConfig * move config to citrea-common * fix use statements * Update e2e test framework and fix tests (#1305) * Update e2e test framework and fix tests * Fix bug * Update ci binary env key * Test if new fix works * Update bitcoincore-rpc version * Dprint * Try against fix prover config rev * Target main HEAD rev --------- Co-authored-by: jfldde <[email protected]> * build and publish a new image for every commit to the nightly branch (#1309) * Get fee recommendation from mempool space (#1302) * Get fee recommendation from mempool space * Fix bug * Construct mempool space endpoint by network --------- Co-authored-by: Esad Yusuf Atik <[email protected]> * new path fix for nightly (#1310) * Enable pending block tag in simulation endpoints (#1303) * treat pending tag same as latest * fix lint * return new sealed block for pending * handle pending tag externally * revert enabling pending in some endpoints * get blockenv instead of sealed block * address review comments * implement tests for eth_call, eth_estimateGas, eth_createAccessList pending blocks * rename tests --------- Co-authored-by: eyusufatik <[email protected]> * Implement state and block overrides (#1270) * Implement state and block overrides * Update comment * apply_block_overrides func * apply_state_overrides func * Use a single func for replacing account storage * Clippy * Address feedback Use Reth's BlockOverrides with saturating_to to convert back to u64 * Test for block overrides * Test for state overrides * Comment tests * Create a fresh working set * Remove box * remove alloy-serde --------- Co-authored-by: eyusufatik <[email protected]> * Use spawn_blocking for da queue (#1311) * Fetch smart fee only if none (#1312) * update run doc and changelog (#1315) --------- Co-authored-by: Rakan Al-Huneiti <[email protected]> Co-authored-by: Ahmet Yazıcı <[email protected]> Co-authored-by: Erce Can Bektüre <[email protected]> Co-authored-by: jfldde <[email protected]> Co-authored-by: Erce Can Bektüre <[email protected]> Co-authored-by: Roman Proskuryakoff <[email protected]> Co-authored-by: Ege Okan Ünaldı <[email protected]> Co-authored-by: Çetin <[email protected]>

rakanalh added 16 commits September 23, 2024 13:20

Implement managed tasks for SubscriptionManager

c15282c

Fix clippy

4ccca66

Add comment about trace task

16c555a

Merge branch 'nightly' into rakanalh/structured-concurrency-pt1

a3e5e83

Use broadcast instead of mpsc for l2 blocks events

ff7bdda

Fix spawns in DaService

09db1dc

Add TaskManager

5977760

Use task manager in sequencer

e9fac49

Document TaskManager

b4c07bb

Handle shutdown event

7ca4d31

Use TaskTracker

6044fa1

Add comment about using a cancellation token

a3e6801

Use JoinHandles instead of TaskTracker

232bd06

Use TaskManager in fullnode and prover

97b810d

Improve bitcoin-da service

d0a4eaa

Force spawned tasks to accept a cancellation token

f6b7eef

auto-assign bot requested review from ercecan, eyusufatik, jfldde, kpp and yaziciahmet September 25, 2024 11:39

auto-assign bot assigned rakanalh Sep 25, 2024

rakanalh changed the base branch from nightly to rakanalh/structured-concurrency-pt2 September 25, 2024 11:39

rakanalh added the HOLD-MERGE PR is not draft but should not be merged yet label Sep 25, 2024

Use biased polling

b155097

rakanalh force-pushed the rakanalh/pruning branch from 13e05e8 to 7bf55af Compare September 25, 2024 11:45

rakanalh added 4 commits September 25, 2024 14:50

Satisfy clippy

f79aade

WIP

c1b5077

Add pruning tables

0e9350e

Pruning skeleton implementation

f1fc7cb

rakanalh added 4 commits October 1, 2024 14:54

Unneccessary changes

796f5cc

l2_receiver

0783efb

Cleanup prints

7f3586a

Merge remote-tracking branch 'origin/nightly' into rakanalh/pruning

a51033f

yaziciahmet requested changes Oct 1, 2024

View reviewed changes

crates/pruning/src/lib.rs Outdated Show resolved Hide resolved

crates/pruning/src/lib.rs Outdated Show resolved Hide resolved

Use last pruned block in calculation

3f7bbcc

ercecan reviewed Oct 1, 2024

View reviewed changes

crates/prover/src/runner.rs Outdated Show resolved Hide resolved

ercecan reviewed Oct 1, 2024

View reviewed changes

crates/pruning/src/lib.rs Outdated Show resolved Hide resolved

ercecan reviewed Oct 1, 2024

View reviewed changes

crates/pruning/src/lib.rs Outdated Show resolved Hide resolved

kpp reviewed Oct 1, 2024

View reviewed changes

crates/pruning/src/lib.rs Outdated Show resolved Hide resolved

rakanalh added 2 commits October 2, 2024 15:10

Implement pruning criteria

2c4ceca

Lint and add comment

04b10c9

ercecan approved these changes Oct 2, 2024

View reviewed changes

yaziciahmet requested changes Oct 2, 2024

View reviewed changes

crates/pruning/src/lib.rs Outdated Show resolved Hide resolved

crates/pruning/src/pruners/evm.rs Show resolved Hide resolved

rakanalh commented Oct 2, 2024

View reviewed changes

crates/pruning/src/lib.rs Outdated Show resolved Hide resolved

Set the last_pruned_block to up_to_block value

dae85de

yaziciahmet approved these changes Oct 2, 2024

View reviewed changes

rakanalh added 2 commits October 2, 2024 16:46

Don't store config internally

acb87db

Remove from constructor

a9d50b3

eyusufatik requested changes Oct 3, 2024

View reviewed changes

bin/citrea/provers/risc0/guest-mock/Cargo.lock Outdated Show resolved Hide resolved

resources/configs/mock/sequencer_config.toml Outdated Show resolved Hide resolved

crates/pruning/src/lib.rs Show resolved Hide resolved

rakanalh added 3 commits October 3, 2024 15:45

Should not change

4b7982f

Move config value

f352117

Remove pruning from sequencer / prover

9eb691e

eyusufatik approved these changes Oct 3, 2024

View reviewed changes

eyusufatik removed the HOLD-MERGE PR is not draft but should not be merged yet label Oct 3, 2024

eyusufatik merged commit 7c3440a into nightly Oct 4, 2024
14 checks passed

eyusufatik deleted the rakanalh/pruning branch October 4, 2024 07:58

eyusufatik mentioned this pull request Oct 17, 2024

Extend RollupConfig with pruning config #729

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pruning skeleton #1229

Pruning skeleton #1229

rakanalh commented Sep 25, 2024 •

edited

Loading

ercecan left a comment

yaziciahmet left a comment

Pruning skeleton #1229

Pruning skeleton #1229

Conversation

rakanalh commented Sep 25, 2024 • edited Loading

Description

Linked Issues

MERGE AFTER

ercecan left a comment

Choose a reason for hiding this comment

yaziciahmet left a comment

Choose a reason for hiding this comment

rakanalh commented Sep 25, 2024 •

edited

Loading