[1.0.4] Fix nodeos_startup_catchup test

[linh2931]

The test is broken in Rel 1.0.3 and flaky in main. The counts the number of unlinkable_block occurrences in logging files and make sure it is not greater than 500.

But in Rel 1.0.3, unlinkable_block (part of unlinkable_block_exception) shows up 3 times per unlinkable block, for example:

warn  2024-11-14T13:48:06.038 nodeos    controller.cpp:4334           push_block           ] 3030001 unlinkable_block_exception: Unlinkable block
info  2024-11-14T13:48:06.038 nodeos    producer_plugin.cpp:938       operator()           ] Exception on block 130: 3030001 unlinkable_block_exception: Unlinkable block
info  2024-11-14T13:48:06.038 nodeos    net_plugin.cpp:3870           process_signed_block ] unlinkable_block_exception connection - 1: #130 1d74c43582d10251...: Unlinkable block (3030001)

This PR searches for unlinkable_block_exception connection - \d+: #(\d+), extracts the unlinkable block number, and verifies that

1. unlinkable blocks are consecutive
2. the number of unlinkable blocks is less than sync fetch span

Resolves #1015

[greg7mdp]

Can you explain why the number of unlinkable blocks is bounded by sync fetch span in order not to be an error? It doesn't seem obvious to me.

[linh2931]

Can you explain why the number of unlinkable blocks is bounded by sync fetch span in order not to be an error? It doesn't seem obvious to me.

The idea is, it is inevitable to have some unlinkable blocks; the worst case is all blocks in a batch are temporarily unlinkable.

[ericpassmore]

Note:start
category: Tests
component: Internal
summary: Fix nodeos_startup_catchup tests.
Note:end