blockchain, netsync: implement a complete headers-first download during ibd #2428

kcalvinalvin · 2025-09-22T16:18:14Z

Change Description

Right now the headers-first download is only based off of the checkpoints and is thus limited to the last checkpoint.
The newly implemented headers-first download will always download headers-first and will validate them to see if the headers connect and have proper proof of work.

Then the block download will be based off of the verified headers. This now eliminates any potential downloading of txs or orphan blocks during ibd. It also makes future parallel block download much better as a parallel block download can only happen for blocks we already have headers for.

~~It's not yet put into the code yet but this allows the node to also receive block headers instead of invs during block propagation.~~ I'll do this in a follow up later.

Steps to Test

Perform ibd on any of the networks and see that the node syncs up fine.
Check that the newly added function tests are correct.

Pull Request Checklist

Testing

Your PR passes all CI checks.
Tests covering the positive and negative (error paths) are included.
Bug fixes contain tests triggering the bug to prevent regressions.

Code Style and Documentation

The change is not insubstantial. Typo fixes are not accepted to fight bot spam.
The change obeys the Code Documentation and Commenting guidelines, and lines wrap at 80.
Commits follow the Ideal Git Commit Structure.
Any new logging statements use an appropriate subsystem and logging level.

📝 Please see our Contribution Guidelines for further guidance.

We add a chainview of bestHeaders so that we'll be able to keep track of headers separately from the bestChain. This is needed as we're getting headers for new block annoucements instead of invs.

Since we may now have blockNodes with just the block header stored without the data, we add a new status to account for this.

coveralls · 2025-09-22T16:22:33Z

Pull Request Test Coverage Report for Build 18000176133

Details

140 of 253 (55.34%) changed or added relevant lines in 6 files are covered.
69 unchanged lines in 4 files lost coverage.
Overall coverage increased (+0.6%) to 55.424%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
blockchain/chainio.go	1	3	33.33%
blockchain/chain.go	12	22	54.55%
netsync/manager.go	40	141	28.37%

Files with Coverage Reduction	New Missed Lines	%
btcutil/gcs/gcs.go	2	80.95%
rpcclient/infrastructure.go	14	47.94%
netsync/manager.go	18	7.06%
blockchain/chainio.go	35	69.75%

Totals
Change from base Build 17994535365:	0.6%
Covered Lines:	31403
Relevant Lines:	56660

💛 - Coveralls

maybeAcceptHeader performs checks to accept block headers into the header chain. This function allows for a true headers-first download where we only accpet block headers for downloading new blocks.

ProcessBlockHeader performs chain selection and context-free & contextual validation for the given block header. The function allows a header-first downloading of blocks even without checkpoints.

On flushes to the database, we check that the blockNodes we have for the download block headers are not flushed to the disk unless the block data is stored as well for backwards compatibility. With older btcd clients, they rely on the fact that the blockNode is present to check if the block data is also present. Since we now store blockNodes for just the block headers, this is no longer true. Because of this, we don't flush the blockNodes if there's no accompanying block data for it. This results in the downloading and verifying the headers again if the node were to restart but since the header data is small and the verification is quick, it's not a big downside.

In block processing and block downloading, HaveBlock is used to check if the block data already exists. It was ok to just check for the existence of the blockNode but we now we also need to check if the data exists as the blockNode may be present for just the block header.

The added exported methods on BlockChain provide access to the block header tip like fetching block hashes and heights from the headers chain.

checkHeadersList takes in a blockhash and returns if it's a checkpointed block and the correct behavior flags for the verification of the block.

fetchHigherPeers provides a convenient function to get peers that are sync candidates and are at a higher advertised height than the passed in height.

isInIBDMode returns if the SyncManager needs to download blocks and sync to the latest chain tip. It determines if it's in ibd mode by checking if the blockchain thinks we're current and if we don't have peers that are at higher advertised blocks.

fetchHeaders picks a random peer at a higher advertised block and requests headers from them.

Instead of the old headerList based header processing, we make use of the new ProcessBlockHeader function.

headers Since we now utilize ProcessBlockHeader, we change the fetchHeaderBlocks to be based off of the block index instead of the headerList in SyncManager.

ince we now utilize ProcessBlockHeaders, we change the startSync function to utilize the block index for downloading blocks/headers instead of using the headerList in SyncManager.

handleBlockMsg is updated to not be based off of the headerList and sm.nextCheckpoint when doing operations related to checkpoints and headers.

Since we no longer utilize the headerList for doing headers-first download and checkpoint tracking, we remove related code.

ibdMode is a more fitting name than headersFirstMode since all blocks are downloaded headers-first during the initial block download.

saubyk · 2025-10-21T16:51:07Z

cc: @gijswijs for review

saubyk · 2025-10-21T16:51:50Z

cc: @mohamedawnallah for review

Roasbeef

Great set of changes! I found the commit structure very easy to follow as well.

Have performed an initial review pass, but will start to run some active tests on one of my nodes to get a better feel for the changes.

Roasbeef · 2025-10-31T23:26:59Z

blockchain/blockindex.go

 }

+// HaveHeader returns whether the header data is stored in the database.
+func (status blockStatus) HaveHeader() bool {


Isn't this just the inverse of HaveData?

Roasbeef · 2025-10-31T23:31:42Z

blockchain/process_test.go

+			MerkleRoot: merkle,
+			Bits:       chainParams.PowLimitBits,
+			Timestamp:  ts,
+			Nonce:      0, // To be solved.


Superfluous comment.

Roasbeef · 2025-10-31T23:32:30Z

blockchain/process_test.go

+	lastHeader := headers[len(headers)-1]
+	lastHeaderHash := lastHeader.BlockHash()
+	tipNode := chain.bestHeader.Tip()
+	if !tipNode.hash.IsEqual(&lastHeaderHash) {


You can use the require package for all these assertions.

Roasbeef · 2025-10-31T23:32:50Z

blockchain/process_test.go

+	}
+
+	// Create invalid header at the checkpoint.
+	//


Style nit: blank line, same below.

Roasbeef · 2025-10-31T23:33:39Z

blockchain/process_test.go

+	lastSidechainHeader := sidechainExtendingHeaders[lastSideChainHeaderIdx]
+	lastSidechainHeaderHash := lastSidechainHeader.BlockHash()
+
+	// Check that the tip is now different.


Roasbeef · 2025-10-31T23:46:10Z

netsync/manager.go

-				"properly connect to the chain from peer %s "+
-				"-- disconnecting", peer.Addr())
+		_, err := sm.chain.ProcessBlockHeader(
+			blockHeader, blockchain.BFNone, false)


Style nit: close paren should go on the line below:

_, err := sm.chain.ProcessBlockHeader( blockHeader, blockchain.BFNone, false, )

Roasbeef · 2025-10-31T23:47:08Z

netsync/manager.go

-		// that is already in the database and is only used to ensure
-		// the next header links properly, it must be removed before
-		// fetching the blocks.
-		sm.headerList.Remove(sm.headerList.Front())


Can we remove the header list all together now? Or is it still needed for re-orgs?

Roasbeef · 2025-10-31T23:50:41Z

netsync/manager.go

+	// If we're in the initial block download mode, check if we have
+	// peers that we can download headers from.
+	_, bestHeaderHeight := sm.chain.BestHeader()
+	higherHeaderPeers := sm.fetchHigherPeers(bestHeaderHeight)


Even though many years have passed at this point, we should retain the logic that we removed to filter out non-segwit peers once segwit is active.

Roasbeef · 2025-10-31T23:52:10Z

netsync/manager.go

+	// If we're on a checkpointed block, check if we still have checkpoints
+	// to let the user know if we're switching to normal mode.
+	if isCheckpointBlock {
+		log.Infof("on checkpoint block %v(%v)",


Log suggestion:

Continuing IBD, on check point block %v(%v)

Roasbeef · 2025-10-31T23:53:29Z

netsync/manager.go

 	headersFirstMode bool
-	headerList       *list.List
-	startHeader      *list.Element
-	nextCheckpoint   *chaincfg.Checkpoint


kcalvinalvin added 2 commits September 23, 2025 00:55

blockchain: add bestHeader to BlockChain

b66e2f8

We add a chainview of bestHeaders so that we'll be able to keep track of headers separately from the bestChain. This is needed as we're getting headers for new block annoucements instead of invs.

blockchain: add statusHeaderStored for blockNode status

05ac50c

Since we may now have blockNodes with just the block header stored without the data, we add a new status to account for this.

saubyk assigned kcalvinalvin Sep 23, 2025

kcalvinalvin force-pushed the 2025-09-16-new-parallel-block-downloads branch 3 times, most recently from 2308a1e to 3519720 Compare September 23, 2025 09:43

kcalvinalvin added 4 commits September 24, 2025 14:41

blockchain: add maybeAcceptHeader

11661ac

maybeAcceptHeader performs checks to accept block headers into the header chain. This function allows for a true headers-first download where we only accpet block headers for downloading new blocks.

blockchain: add ProcessBlockHeader

31342f5

ProcessBlockHeader performs chain selection and context-free & contextual validation for the given block header. The function allows a header-first downloading of blocks even without checkpoints.

kcalvinalvin force-pushed the 2025-09-16-new-parallel-block-downloads branch from 3519720 to f48f8bf Compare September 24, 2025 05:41

blockchain: add exports methods based on block header tip

653e3ca

The added exported methods on BlockChain provide access to the block header tip like fetching block hashes and heights from the headers chain.

kcalvinalvin force-pushed the 2025-09-16-new-parallel-block-downloads branch 3 times, most recently from cf683a9 to 242fa3b Compare September 25, 2025 07:16

kcalvinalvin added 10 commits September 25, 2025 16:19

netsync: add checkHeadersList

00ffdac

checkHeadersList takes in a blockhash and returns if it's a checkpointed block and the correct behavior flags for the verification of the block.

netsync: add fetchHigherPeers

2c5ef64

fetchHigherPeers provides a convenient function to get peers that are sync candidates and are at a higher advertised height than the passed in height.

netsync: add isInIBDMode

3b210f0

isInIBDMode returns if the SyncManager needs to download blocks and sync to the latest chain tip. It determines if it's in ibd mode by checking if the blockchain thinks we're current and if we don't have peers that are at higher advertised blocks.

netsync: add fetchHeaders

11de96d

fetchHeaders picks a random peer at a higher advertised block and requests headers from them.

netsync: use ProcessBlockHeader in handleBlockHeader

53bc1dd

Instead of the old headerList based header processing, we make use of the new ProcessBlockHeader function.

netsync: change fetchHeaderBlocks to be based on the processed block

a1877f0

headers Since we now utilize ProcessBlockHeader, we change the fetchHeaderBlocks to be based off of the block index instead of the headerList in SyncManager.

netsync: change startSync to be based off of processed headers

f4dd857

ince we now utilize ProcessBlockHeaders, we change the startSync function to utilize the block index for downloading blocks/headers instead of using the headerList in SyncManager.

netsync: update handleBlockMsg

87e869f

handleBlockMsg is updated to not be based off of the headerList and sm.nextCheckpoint when doing operations related to checkpoints and headers.

netsync: remove unused headerList and checkpoint code

0af30ad

Since we no longer utilize the headerList for doing headers-first download and checkpoint tracking, we remove related code.

netsync: rename headersFirstMode to ibdMode

e6fadc9

ibdMode is a more fitting name than headersFirstMode since all blocks are downloaded headers-first during the initial block download.

kcalvinalvin force-pushed the 2025-09-16-new-parallel-block-downloads branch from 242fa3b to e6fadc9 Compare September 25, 2025 07:25

kcalvinalvin marked this pull request as ready for review September 25, 2025 13:17

Roasbeef reviewed Oct 31, 2025

View reviewed changes

blockchain, netsync: implement a complete headers-first download during ibd #2428

Are you sure you want to change the base?

blockchain, netsync: implement a complete headers-first download during ibd #2428

Uh oh!

Conversation

kcalvinalvin commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Description

Steps to Test

Pull Request Checklist

Testing

Code Style and Documentation

Uh oh!

coveralls commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 18000176133

Details

💛 - Coveralls

Uh oh!

saubyk commented Oct 21, 2025

Uh oh!

saubyk commented Oct 21, 2025

Uh oh!

Roasbeef left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kcalvinalvin commented Sep 22, 2025 •

edited

Loading

coveralls commented Sep 22, 2025 •

edited

Loading