fix: prevent panic on chain replay

[karlem]

Close #1196

Removing the dependency on the CometBFT client in favor of caching. When CometBFT is catching up—replaying from the beginning of the chain to synchronize with the ABCI app—it does not start the RPC API. Unfortunately, our ABCI app relied on the API during consensus events, which made it impossible to replay the chain.

Solved by relying on the exec and app states instead of making calls to CometBFT.

[karlem]

@LePremierHomme could you have another look please?

[cryptoAtwill]

@karlem A quick question, how will the cache be recovered after crashes? Seems not queried when booting?

[karlem]

@karlem A quick question, how will the cache be recovered after crashes? Seems not queried when booting?

Yeah good point. We would loose it after the crash. Do you think maybe saving it to the app state might be a way to go?

[cryptoAtwill]

@karlem The rabbit hole gets deeper. But I think it's possible to read the state during boot up. The validators and gas limits are all stored in the contract or actor, so technically we can call the getters from the store directly.

[raulk]

@cryptoAtwill that's a good point; it would also take us one tiny step closer to our desired end state of having as much logic as possible in on-chain actors.

[karlem]

@cryptoAtwill Yeah, I think working with the contract makes more sense than this. I would ditch this in favor of using the contracts, as they are part of the app state—much more solid than this.

[karlem]

The validators and gas limits are all stored in the contract or actor, so technically we can call the getters from the store directly.

@cryptoAtwill , I believe the issue is that validators are only stored in the actor after top-down finality has been finalized, whereas we need them available earlier.

We have two potential approaches to resolve this:

Store the initial set of validators from genesis in the top-down finality actor/contract.
This approach might introduce unintended side effects, so it may not be ideal.

Create a new actor specifically for storing these validators before top-down finality is achieved.

What are your thoughts on this? Also looping in @raulk for input.

[karlem]

Related: #1166

[karlem]

@cryptoAtwill Changed the implementation to rely on app and exec states instead.

[raulk]

I'm wrapping my head around the problem and the proposed solution to help push things forward. Here are some notes.

If I'm following correctly, this attempts to fix an edge case whereby during a CometBFT startup, the latter may attempts to perform a chain replay either from the WAL or from the network, by feeding blocks to the ABCI app. The issue arises because we recently introduced logic to fetch the block proposer's public key so we could credit gas premiums to their on-chain account. This happens in begin_block and was introduced in #1173. Because our local cache is empty, we fall back to CometBFT, whose RPC API has not started yet, therefore making us fail.

I think all of this can be greatly simplified if instead of querying CometBFT, we query our power table from the gateway actor and match on the block proposer's identity. The gateway already has public keys. Such a call would be entirely inside the state tree, so it has no dependencies on CometBFT, and would break the problematic cycle.

In fact, we already do this in end_block in order to calculate power table updates.

1. end_block calls interpreter.end():

   ipc/fendermint/app/src/app.rs

   Line 802 in c74a21c

   .modify_exec_state(|s| self.interpreter.end(s))

2. When we create a checkpoint, it may come with power table updates:

   ipc/fendermint/vm/interpreter/src/fvm/exec.rs

   Line 222 in 3ad2c0f

   checkpoint::maybe_create_checkpoint(&self.gateway, &mut state, &mut block_end_events)

3. We query the current power table:

   ipc/fendermint/vm/interpreter/src/fvm/checkpoint.rs

   Line 71 in 3ad2c0f

   ipc_power_table(gateway, state).context("failed to get the current power table")?;

4. We already have a helper that queries the gateway actor in the state tree and returns domain objects, containing validator pubkeys:

   ipc/fendermint/vm/interpreter/src/fvm/state/ipc.rs

   Line 178 in 03e3ebd

   pub fn current_power_table(

In a nutshell, I don't think we need one more cache here, nor to manage its lifecycle, nor anything like that. Assuming I'm right, we can kill the ValidatorTracker entirely and simply replace it with the above.

@cryptoAtwill does this sound right to you?

[karlem]

@cryptoAtwill and @raulk

After the last review and team decision, I have updated the PR again. The following changes are included:

• Rename ValidatorTracker to ValidatorCache: This emphasizes that it serves as a cache and is not the source of truth.
• Remove dependency on the CometBFT client:
  • Instead of relying on the CometBFT client, the system now queries the gateway through FvmExecState, leveraging the current_power_table from the cache.
• Make the cache immutable:
  • The cache now offers two methods for interaction:
    1. new_from_state: Constructs and populates a new cache based on the current state.
    2. get_validator: Retrieves a validator from the cache.
• Integrate the cache into the App state wrapped in an Option:
  • Initially, the cache is None.
  • In begin_block, the cache is initialized if it is currently None.
  • In end_block, the existing cache is replaced with a freshly populated one if the validator table has been updated.

Hopefully, this aligns closely with what we want.

[raulk]

A few nits; other than that, LGTM. Glad we're landing this after the several twists and bends this has taken.

[raulk]

Good cleanup here!

Stale