Skip to content

Latest commit

 

History

History
57 lines (42 loc) · 5.03 KB

2019-11-06-#15845.md

File metadata and controls

57 lines (42 loc) · 5.03 KB

PR#15845 - Fast rescan with BIP157 block filters 👀

A PR by Marco Falke to use BIP157 block filters (if available) in order to speed up rescans by fetching only the blocks matched by the computed Golomb-Coded Set of all the wallet’s scripts.

Notes

  • This PR was eventually Closed.
  • Other takes on faster wallet rescan:

Questions

How many scripts will a newly created wallet have? How many items will be added to the GCS returned by GetAllRelevantScriptPubKeys()? 🔗

  • The keypool is used to implement a 'gap limit'. The keypool maintains a set of keys (by default 1000) ahead of the last used key and scans for the addresses of those keys.
  • There are 2 keypools (external and change) and each has 1000 keys. Each key has 3 scriptPubKeys (p2pkh, p2sh-p2wpkh, native p2wpkh).
  • Therefore with an empty, newly created wallet we are initially looking out for 6000 different scripts.

In the current (pre-PR) code, what happens when we find a matching scriptPubKey for a key that is in our keypool in a block transaction? 🔗

  • ScanForWalletTransactions is where this happens and specifically at SyncTransaction.
  • We add the UTXO to the wallet db AddToWalletIfInvolvingMe, mark the keypool keys as used (and every keypool entry before it) and before continuing, top up the keypool back to the 'gap limit' (as long as the wallet is unlocked (not encrypted)).
  • The gap limit of 1000 means that if there are any gaps, or the keys are used out-of-order, then as long as the gap is less than 1000, we won't miss any transactions.
    • A gap in your external addresses can occur when you give out an address and someone just not pay you.
    • A gap in change addresses is less likely. If you RBF bump, then you use a new key for change (#TOVALIDATE).
    • Addresses can be seen out-of-order if you create txns at different fee rates that get mined at different times.
    • This limit started with 100 and after HD-wallets bumped to 1000.
  • We DO NOT update our balance, that is done dynamically whenever it's needed.

bonus: Has anyone spotted the bug in this PR yet? 🔗

  • It's important to TopUp the keypool in the rescan loop to make sure that if keys are used in order then we won't miss any transactions.
  • We are building the Golomb-Coded Set (GCS) filter from the response of GetAllRelevantScriptPubKeys() at the beginning, outside the rescan loop. That means as we advance through the blockchain, it doesn't get updated.
  • What happens if you create a wallet, make a backup, then receive 1001 payments, and then try to restore from that backup?
  • Only the initial 1000 keys are in the backup, which means that the GCS filter will not include the topup keys.
  • From this comment is seems that an earlier version of the PR might have recalculated the GCS for each block, which fixes this bug but is slow.

What is the false positive rate for Basic compact block filters? 🔗