Re: RFC: ship block chain 1-74000 with release tarballs?


2010-11-26 17:32:01 UTC - Original Post
I tested it on a slow 7 year old drive, where bandwidth and CPU were clearly not the bottleneck.  Initial download took 1 hour 20 minutes.

If it's taking a lot longer than that, certainly 24 hours, then it must be downloading from a very slow node, or your connection is much slower than around 15KB per sec (120kbps), or something else is wrong.  It would be nice to know what appears to be the bottleneck when that happens.

Every 10 minutes or so when the latest block is sent, it should have the chance to change to a faster node.  When the latest block is broadcast, it requests the next 500 blocks from other nodes, and continues the download from the one that sends it fastest.  At least, that's how it should work.

Maybe Berkeley DB has some tweaks we can make to enable or increase cache memory.
Which of the ACID properties do you need, while downloading?
It may only need more read caching.  It has to read randomly all over blk0001.dat and blkindex.dat to index.  It can't assume the file is smaller than memory, although it currently still is.  Caching would be effective, since most dependencies are recent.

Someone should experiment with different Berkeley DB settings and see if there's something that makes the download substantially faster.  If something substantial is discovered, then we can work out the particulars.

Adding BDB records is simply appending to a log file, until you issue a checkpoint.  The checkpoint then updates the main database file.
We checkpoint every 500 blocks.