sweep: start tracking input spending status in the fee bumper #9447

yyforyongyu · 2025-01-26T02:18:12Z

There are two places tracking the spending status of a given input - one in the sweeper, the other in the fee bumper. We now move the tracking to be handled in the fee bumper so we always have a single source of truth. By the end of this fix, we should see that,

the fee func will be kept on the original line when retrying sweeps.
both the sweeper and the fee bumper can recover their state from a restart.
for the neutrino backend, the initial sweeping tx is now always RBF-compliant.

The fix is made of two PRs to keep the size small - the first PR will enable tracking the spending status of inputs in the fee bumper, and the second will fix the rest.

Depends on,

sweeper: rename Failed to Fatal and minor refactor #9446

coderabbitai · 2025-01-26T02:18:41Z

Important

Review skipped

Auto reviews are limited to specific labels.

🏷️ Labels to auto review (1)

llm-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

Roasbeef

First pass, one thing that wasn't immediately obvious to me is: where do we fix the issue that the the state of the fee function is properly carried over into the new batch (instead of reset) when one of the inputs in a cohort is spent?

itest/lnd_sweep_test.go

Roasbeef · 2025-02-04T00:11:03Z

itest/lnd_sweep_test.go

+	// Carol should have numPayments incoming HTLCs on channel Bob -> Carol.
+	ht.AssertNumActiveHtlcs(carol, numPayments)
+
+	// Suspen Bob so he won't get the preimage from Carol.


Suspen -> Suspend

Roasbeef · 2025-02-04T00:18:56Z

sweep/fee_bumper.go

 				continue
 			}

-			log.Warnf("Detected third party spent of output=%v "+
-				"in tx=%v", op, spend.SpendingTx.TxHash())
+			spendingTx := spend.SpendingTx


I wonder if we should actually block here, even if just for a moment, to allow the scheduler to run the goroutine that does the dispatch.

Spent a bit of time to re-familiarize myself with the notifier after the latest set of refactors, and I don't see an area where we'll insta dispatch the response before exiting the initial method call.

you mean we perform a tiny sleep (sth likecase <- time.After(100ms)) instead?

We did refactor this area a bit here 1200b75, which makes sure the block is always sent before the tx, but we cannot guarantee the order is maintained since pipeline is a bit deep we cannot be sure they are read in this order.

A previous attempt was to implement a method HasOutpointSpent on Blockbeat - the idea is that, whenever we are notified of a block, we can easily access the block data to see if the watched outputs are spent or not, hence eliminating the race, which guarantees we won't miss a spending event. However there was some difficult involved when implementing it in neutrino, as discussed here. Now that you mention it I think it's still worthy to keep it as a TODO, since we can just register spend when it's neutrino to avoid fetching full blocks, and read the block data when it's a full node.

yyforyongyu · 2025-02-05T13:02:59Z

Note to reviewers - the itest is disabled here and the CI should pass in #9448. I think once approved we can merge #9448 to this branch, and merge this PR to master.

To track the input and its spending tx, which will be used later to detect unknown spends.

This commit refactors the `processRecords` to always handle the inputs spent when processing the records. We now make sure to handle unknown spends for all backends (previously only neutrino), and rely solely on the spending notification to give us the onchain status of inputs.

We now rename "third party" to "unknown" as the inputs can be spent via an older sweeping tx, a third party (anchor), or a remote party (pin). In fee bumper we don't have the info to distinguish the above cases, and leave them to be further handled by the sweeper as it has more context.

yyforyongyu added utxo sweeping no-itest no-changelog size/micro small bug fix or feature, less than 15 mins of review, less than 250 labels Jan 26, 2025

yyforyongyu added this to the v0.19.0 milestone Jan 26, 2025

yyforyongyu mentioned this pull request Jan 26, 2025

sweep: properly handle failed sweeping txns #9448

Open

yyforyongyu force-pushed the yy-prepare-fee-replace branch from e8cb0c7 to a738e7f Compare January 27, 2025 09:51

yyforyongyu force-pushed the yy-sweeper-fix branch from 48d4631 to bd0f218 Compare January 27, 2025 09:51

Roasbeef reviewed Feb 4, 2025

View reviewed changes

yyforyongyu force-pushed the yy-prepare-fee-replace branch from a738e7f to b98542b Compare February 5, 2025 11:53

yyforyongyu force-pushed the yy-sweeper-fix branch 2 times, most recently from 0ca8914 to 18df4fb Compare February 5, 2025 12:49

yyforyongyu requested review from Roasbeef and morehouse February 5, 2025 13:15

yyforyongyu added 6 commits February 5, 2025 22:49

itest: add fee replacement test

dcd5119

sweep: add method hasInputSpent

149e073

To track the input and its spending tx, which will be used later to detect unknown spends.

sweep: add a new event TxUnknownSpend

100a10e

sweep: remove dead code and add better logging

58008d5

yyforyongyu changed the base branch from yy-prepare-fee-replace to master February 5, 2025 14:49

yyforyongyu force-pushed the yy-sweeper-fix branch from 18df4fb to af48039 Compare February 5, 2025 14:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sweep: start tracking input spending status in the fee bumper #9447

sweep: start tracking input spending status in the fee bumper #9447

yyforyongyu commented Jan 26, 2025

coderabbitai bot commented Jan 26, 2025

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

Roasbeef left a comment

Roasbeef Feb 4, 2025

yyforyongyu Feb 5, 2025

Roasbeef Feb 4, 2025

yyforyongyu Feb 5, 2025

yyforyongyu commented Feb 5, 2025

sweep: start tracking input spending status in the fee bumper #9447

Are you sure you want to change the base?

sweep: start tracking input spending status in the fee bumper #9447

Conversation

yyforyongyu commented Jan 26, 2025

coderabbitai bot commented Jan 26, 2025

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

Roasbeef left a comment

Choose a reason for hiding this comment

Roasbeef Feb 4, 2025

Choose a reason for hiding this comment

yyforyongyu Feb 5, 2025

Choose a reason for hiding this comment

Roasbeef Feb 4, 2025

Choose a reason for hiding this comment

yyforyongyu Feb 5, 2025

Choose a reason for hiding this comment

yyforyongyu commented Feb 5, 2025

CodeRabbit Configuration File (`.coderabbit.yaml`)