BitcoinTalk

Faster initial block download (5x faster)

Faster initial block download (5x faster)

By making some adjustments to the database settings, I was able to make the initial block download about 5 times faster.  It downloads in about 30 minutes.

The database default had it writing each block to disk synchronously, which is not necessary.  I changed the settings to let it cache the changes in memory and write them out in a batch.  Blocks are still written transactionally, so either the complete change occurs or none of it does, in either case the data is left in a valid state.

I only enabled this change during the initial block download.  When you come within 2000 blocks of the latest block, these changes turn off and it slows down to the old way.

I built a test build if you'd like to start using it:

http://www.bitcoin.org/download/bitcoin-0.3.2.5-win32.zip
http://www.bitcoin.org/download/bitcoin-0.3.2.5-linux.tar.gz

These binaries also include Gavin Andresen's JSON-RPC HTTP authentication feature and the other important security improvements from 0.3.2.

I've been running a test over the last 24 hours that kills and restarts it randomly every 2-60 seconds (poor thing) while it's trying to do an initial block download and it's been fine.

There are no changes to the way it handles wallet.dat.  This change is only for blk*.dat and the non-critical addr.dat.  You can always delete blk*.dat if it gets screwed up and let it re-download.

Re: Faster initial block download

Are these for use with the test network you have setup or the current public network?

Re: Faster initial block download

Are these for use with the test network you have setup or the current public network?
This is for the regular network. The test network doesn't have enough blocks to test it effectively. From what I can tell, satoshi's builds are always for the live network unless he specifies otherwise.

Re: Faster initial block download

Awesome, got some extra slow machines to wipe/test with then. Will be curious to see what it does on faster machines as well.  Smiley

Re: Faster initial block download

So far, awesome increase for the Windows client. Doing about 500 blocks / second using a private node. After the timer is done, I'll try it on a wild node out on the Internet to see if the speed is just as good.

Re: Faster initial block download

By making some adjustments to the database settings, I was able to make the initial block download about 5 times faster.  It downloads in about 30 minutes.

The database default had it writing each block to disk synchronously, which is not necessary.  I changed the settings to let it cache the changes in memory and write them out in a batch.  Blocks are still written transactionally, so either the complete change occurs or none of it does, in either case the data is left in a valid state.

I only enabled this change during the initial block download.  When you come within 2000 blocks of the latest block, these changes turn off and it slows down to the old way.
The first part of the 70k downloaded in about 20 minutes, the rest of the remaining 2000 blocks in about 7 minutes, so only 27 minutes from 0 to 100%, very nice! Is there a safety reason to stop within the last 2000 blocks or can it be tweaked to stop at remaining 500 blocks for example?

In case anyone was curious, average download speed was around 3 Mbps for each burst of download.

Re: Faster initial block download

Blocks download just as fast in the wild (about 3 Mbps), so a big thumbs up, this will certainly get new users up and running much quicker.

Re: Faster initial block download

Is there a safety reason to stop within the last 2000 blocks or can it be tweaked to stop at remaining 500 blocks for example?
Not really.  I'll change it to 1000 next time.