It's not the downloading that takes the time, it's verifying and indexing it.
Bandwidthwise, it's more efficient than if you downloaded an archive. Bitcoin only downloads the data in blk0001.dat, which is currently 55MB, and builds blkindex.dat itself, which is 47MB. Building blkindex.dat is what causes all the disk activity.
During the block download, it only flushes the database to disk every 500 blocks. You may see the block count pause at ??499 and ??999. That's when it's flushing.
Doing your own verifying and indexing is the only way to be sure your index data is secure. If you copy blk0001.dat and blkindex.dat from an untrusted source, there's no way to know if you can trust all the contents in them.
Maybe Berkeley DB has some tweaks we can make to enable or increase cache memory.
Re: RFC: ship block chain 1-74000 with release tarballs?
I tested it on a slow 7 year old drive, where bandwidth and CPU were clearly not the bottleneck. Initial download took 1 hour 20 minutes.
If it's taking a lot longer than that, certainly 24 hours, then it must be downloading from a very slow node, or your connection is much slower than around 15KB per sec (120kbps), or something else is wrong. It would be nice to know what appears to be the bottleneck when that happens.
Every 10 minutes or so when the latest block is sent, it should have the chance to change to a faster node. When the latest block is broadcast, it requests the next 500 blocks from other nodes, and continues the download from the one that sends it fastest. At least, that's how it should work.
It may only need more read caching. It has to read randomly all over blk0001.dat and blkindex.dat to index. It can't assume the file is smaller than memory, although it currently still is. Caching would be effective, since most dependencies are recent.
Someone should experiment with different Berkeley DB settings and see if there's something that makes the download substantially faster. If something substantial is discovered, then we can work out the particulars.
If it's taking a lot longer than that, certainly 24 hours, then it must be downloading from a very slow node, or your connection is much slower than around 15KB per sec (120kbps), or something else is wrong. It would be nice to know what appears to be the bottleneck when that happens.
Every 10 minutes or so when the latest block is sent, it should have the chance to change to a faster node. When the latest block is broadcast, it requests the next 500 blocks from other nodes, and continues the download from the one that sends it fastest. At least, that's how it should work.
Maybe Berkeley DB has some tweaks we can make to enable or increase cache memory.
Which of the ACID properties do you need, while downloading?Someone should experiment with different Berkeley DB settings and see if there's something that makes the download substantially faster. If something substantial is discovered, then we can work out the particulars.
Quote
Adding BDB records is simply appending to a log file, until you issue a checkpoint. The checkpoint then updates the main database file.
We checkpoint every 500 blocks.Re: RFC: ship block chain 1-74000 with release tarballs?
Despite everything else said, the current next step is:
If there was a "verify it" step, that would take as long as the current normal initial download, in which it is the indexing, not the data download, that is the bottleneck.
The speed of initial download is not a reflection of the bulk data transfer rate of the protocol. The gating factor is the indexing while it downloads.
Quote
Someone should experiment with different Berkeley DB settings and see if there's something that makes the download substantially faster. If something substantial is discovered, then we can work out the particulars.
In particular, I suspect that more read caching might help a lot.Another new user on IRC, Linux this time, was downloading at a rate of 1 block every 4 seconds -- estimated total download time around 4 days.
Then something more specific was wrong. That's not due to normal initial download time. Without more details, it can't be diagnosed. If it was due to slow download, did it speed up after 10-20 minutes when the next block broadcast should have made it switch to a faster source? debug.log might have clues. How fast is their Internet connection? Was it steadily slow, or just slow down at one point?Quote
We have the hashes for genesis block through block 74000 hardcoded (compiled) into bitcoin, so there's no reason why we shouldn't be able to automatically download a compressed zipfile of the block database from anywhere, unpack it, verify it, and start running.
The 74000 checkpoint is not enough to protect you, and does nothing if the download is already past 74000. -checkblocks does more, but is still easily defeated. You still must trust the supplier of the zipfile.If there was a "verify it" step, that would take as long as the current normal initial download, in which it is the indexing, not the data download, that is the bottleneck.
Presumably at some point there will be a lightweight client that only downloads block headers, but there will still be hundreds of thousands of those...
80 bytes per header and no indexing work. Might take 1 minute.Quote
uncompressed data using a protocol (bitcoin P2P) that wasn't designed for bulk data transfer.
The data is mostly hashes and keys and signatures that are uncompressible.The speed of initial download is not a reflection of the bulk data transfer rate of the protocol. The gating factor is the indexing while it downloads.
Re: RFC: ship block chain 1-74000 with release tarballs?
It seems like you're inclined to assume everything is wrong more than is actually so.
Writing the block index is light work. Building the tx index is much more random access per block. I suspect reading all the prev txins is what's slow. Read caching would help that. It's best if the DB does that. Maybe it has a setting for how much cache memory to use.
if (strFile == "blkindex.dat" && IsInitialBlockDownload() && nBestHeight % 500 != 0)
nMinutes = 1;
dbenv.txn_checkpoint(0, nMinutes, 0);
Probably should add this:
if (!fReadOnly)
dbenv.txn_checkpoint(0, nMinutes, 0);
In the following cases, would case 1 flush once and case 2 flush twice?
case 1:
write
write
write
write
checkpoint
case 2:
begin transaction
write
write
commit transaction
begin transaction
write
write
commit transaction
checkpoint
Contorting our database usage will not be the right approach. It's going to be BDB settings and caching.
Writing the block index is light work. Building the tx index is much more random access per block. I suspect reading all the prev txins is what's slow. Read caching would help that. It's best if the DB does that. Maybe it has a setting for how much cache memory to use.
Quote
1) bitcoin should be opening databases, not just environment, at program startup, and closing database at program shutdown.
Already does that. See CDB. The lifetime of the (for instance) CTxDB object is only to support database transactions and to know if anything is still using the database at shutdown.Quote
And, additionally, bitcoin forces a database checkpoint, pushing all transactions from log into main database.
If it was doing that it would be much slower. It's supposed to be only once a minute or 500 blocks:if (strFile == "blkindex.dat" && IsInitialBlockDownload() && nBestHeight % 500 != 0)
nMinutes = 1;
dbenv.txn_checkpoint(0, nMinutes, 0);
Probably should add this:
if (!fReadOnly)
dbenv.txn_checkpoint(0, nMinutes, 0);
Quote
2) For the initial block download, txn commit should occur once every N records, not every record. I suggest N=1000.
Does transaction commit imply flush? That seems surprising to me. I assume a database op wrapped in a transaction would be logged like any other database op. Many database applications need to wrap almost every pair of ops in a transaction, such as moving money from one account to another. (debit a, credit b) I can't imagine they're required to batch all their stuff up themselves.In the following cases, would case 1 flush once and case 2 flush twice?
case 1:
write
write
write
write
checkpoint
case 2:
begin transaction
write
write
commit transaction
begin transaction
write
write
commit transaction
checkpoint
Contorting our database usage will not be the right approach. It's going to be BDB settings and caching.
Re: RFC: ship block chain 1-74000 with release tarballs?
That's a good optimisation. I'll add that next time I update SVN.
More generally, we could also consider this:
dbenv.set_lk_max_objects(10000);
dbenv.set_errfile(fopen(strErrorFile.c_str(), "a")); /// debug
dbenv.set_flags(DB_AUTO_COMMIT, 1);
+ dbenv.set_flags(DB_TXN_NOSYNC, 1);
ret = dbenv.open(strDataDir.c_str(),
DB_CREATE |
DB_INIT_LOCK |
DB_INIT_LOG |
We would then rely on dbenv.txn_checkpoint(0, 0, 0) in CDB::Close() to flush after wallet writes.
More generally, we could also consider this:
dbenv.set_lk_max_objects(10000);
dbenv.set_errfile(fopen(strErrorFile.c_str(), "a")); /// debug
dbenv.set_flags(DB_AUTO_COMMIT, 1);
+ dbenv.set_flags(DB_TXN_NOSYNC, 1);
ret = dbenv.open(strDataDir.c_str(),
DB_CREATE |
DB_INIT_LOCK |
DB_INIT_LOG |
We would then rely on dbenv.txn_checkpoint(0, 0, 0) in CDB::Close() to flush after wallet writes.