RFC: ship block chain 1-74000 with release tarballs?

It appears that blk0001.dat, where bitcoin stores block chain information, is compatible across Windows, Linux, 32-bit and 64-bit.

Therefore, why not save new users some time by shipping blocks 1-74000 with each release?

Presumably, indexing and verifying a local file would be faster, and use fewer network resources, than downloading all those blocks via P2P.

I have mixed feelings about this.  Part of the problem is that there is perceived to be a free good, a network hosting service in the form of Source Forge, which will certainly allow those performing software releases to include considerably more data than is currently the case for Bitcoins.  If somehow we were paying for this service as a community in terms of $$$ per MiB, I think it would be a no brainer that this should stay out of the distribution.  Unfortunately for this consideration, it is a free good from the perspective of most users.

The other issue is that the network bandwidth between nodes is also a free good.  I've suggested in this thread that perhaps the presumption of network bandwidth may also not be considered a free good either.  In fact, I believe that it shouldn't be the case, but that is a completely separate issue entirely.

The network bandwidth for downloading the blocks is to me a wash either way, although a new client coming "on-line" trying to get the full block chain does suck up a whole bunch of blocks through the Bitcoin network and that impacts anybody who happens to be connected to those nodes.  BTW, this is one of the reasons I think it would be incredibly useful to start "charging" for bandwidth as a means to discourage this behavior... and of course to earn a few extra Bitcoins on the side.  If you can obtain blocks "free" from another source, some people might get more creative on how to get that accomplished including downloading a second package on some free file hosting service (perhaps included with the main client distributions) or coming up with a scheme on how to bootstrap new clients that impacts the network in a less obtrusive fashion.

I guess what I'm saying is that while this is a simple solution to a complex problem, it doesn't solve all of the problems including perhaps clients which may store the block data in another format.  There also isn't any apparent reason to necessarily encourage other software client distros to include this kind of data or for that matter to put in more than the most minimum number of blocks.  Still, raising the issue is useful here and I hope it raises a discussion about the problem.


I agree it seems very odd that you would take something which is by its nature distributed through P2P channels and instead put it into a conventional client-server distribution model.  Part of why I'm saying that perhaps more thought ought to go into this is perhaps to encourage a bittorrent distribution connection of some sort for a large collection of blocks if somebody has had their client off for awhile or some other kind of experimentation on how to solve this same problem.  The problem is that new clients are demanding the whole block chain and really can't get into "mining" or confirming new transactions until they have that chain.  Let's solve that problem, which is a larger issue.

The other issue is that it seems like a waste of bandwidth to include these blocks in a client when all you are doing is updating the software.  I would be just as worried that the block chain might get wiped out by the installation software with this "older" version of the chain, forcing older clients to update to the current block all over again, although this is certainly an installation bug.  Just because it is a free good doesn't imply there are no other consequences to going this route.

It's not the downloading that takes the time, it's verifying and indexing it.

Bandwidthwise, it's more efficient than if you downloaded an archive.  Bitcoin only downloads the data in blk0001.dat, which is currently 55MB, and builds blkindex.dat itself, which is 47MB.  Building blkindex.dat is what causes all the disk activity.

During the block download, it only flushes the database to disk every 500 blocks.  You may see the block count pause at ??499 and ??999.  That's when it's flushing.

Doing your own verifying and indexing is the only way to be sure your index data is secure.  If you copy blk0001.dat and blkindex.dat from an untrusted source, there's no way to know if you can trust all the contents in them.

Maybe Berkeley DB has some tweaks we can make to enable or increase cache memory.

This is not true of many novice users, who say things like "well it took several hours to catch all the 90 000 blocks but finally it arrived" (quoted from one new user, on IRC, today).


Agreed.  Compressed in an archive, blk0001.dat is around 36MB.


It remains the download, not the verification, that has the highest variability of experience, where first time users see a delay of 30 minutes to several hours before the software is actually usable.  Some P2P nodes may be extremely slow (I see high variability in latency and throughput for old blocks, and blocks larger than 512 bytes).  End user bandwidth may be low, spotty or expensive.  Firewalls are often a problem.

I'm betting that the above complaint from a new user was due to a Microsoft firewall; but the point stands:  large variance of network configuration and capability implies the P2P download impact may be far, far greater than impact of on-disk verification of 90,000 blocks.


Who said untrusted?  The proposal is that you distribute blk0001.dat (and only blk0001.dat) in the bitcoin.org official client downloads.  And of course the client will spend some time verifying blk0001.dat upon first use.  This is unavoidable, and nobody has proposed changing or eliminating verification.

Just shipping blk0001.dat with official bitcoin would eliminate several headaches that new bitcoin users continue to experience.

I tested it on a slow 7 year old drive, where bandwidth and CPU were clearly not the bottleneck.  Initial download took 1 hour 20 minutes.

If it's taking a lot longer than that, certainly 24 hours, then it must be downloading from a very slow node, or your connection is much slower than around 15KB per sec (120kbps), or something else is wrong.  It would be nice to know what appears to be the bottleneck when that happens.

Every 10 minutes or so when the latest block is sent, it should have the chance to change to a faster node.  When the latest block is broadcast, it requests the next 500 blocks from other nodes, and continues the download from the one that sends it fastest.  At least, that's how it should work.

It may only need more read caching.  It has to read randomly all over blk0001.dat and blkindex.dat to index.  It can't assume the file is smaller than memory, although it currently still is.  Caching would be effective, since most dependencies are recent.

Someone should experiment with different Berkeley DB settings and see if there's something that makes the download substantially faster.  If something substantial is discovered, then we can work out the particulars.

We checkpoint every 500 blocks.

I'm not sure how it is for you, but when I upgrade Bitcoin I don't have to re-download any blocks. It just picks up right where it left off before the upgrade.

Another new user on IRC, Linux this time, was downloading at a rate of 1 block every 4 seconds -- estimated total download time around 4 days.

Other commenters in this thread are correct that upgrading users don't need a block database...  but something needs to be done to improve the initial block download experience for new users.  Improve the database all you want.. you'll still have peers giving you blocks slowly for any number of reasons.

We have the hashes for genesis block through block 74000 hardcoded (compiled) into bitcoin, so there's no reason why we shouldn't be able to automatically download a compressed zipfile of the block database from anywhere, unpack it, verify it, and start running.

Yes, correct.

Presumably at some point there will be a lightweight client that only downloads block headers, but there will still be hundreds of thousands of those...

Not quite true.  "-checkblocks" (CheckBlock()) performs quite a few checks on the contents of blk0001.dat / blkindex.dat.  AcceptBlock() does a bit more, adding context, but not much more.  But let's ignore that for the moment.

I think a more important point you're missing is that nobody is proposing that verification be skipped.  The bitcoin code is quite capable of verifying and indexing untrusted blk0001.dat data.  It would just need a few modifications to behave sensibly if blkindex.dat is missing.

The proposal is simply:  don't download massive amounts of uncompressed data using a protocol (bitcoin P2P) that wasn't designed for bulk data transfer.


The client is clearly capable of verifying the cryptographic integrity of blk0001.dat from an untrusted source, because it does that for blocks coming in over the network, and blk0001.dat contains... serialized blocks originally received from untrusted sources over the network.

It does not seem overly difficult to pass in blk0001.dat file position data to ProcessBlock(), and simply skip the WriteToDisk() storage call in downstream callee AcceptBlock().

My understanding was that the client already did a blockchain recheck upon startup if the index was missing.  I did this when I first started, and it sure seemed like it was marching through the chain.  Doesn't it require an index to function anyway?  Why would it assume that the blockchain was valid upon startup?  Anyone could have edited it.  The genesis block is encoded into the client, isn't it?  That and the blockchain checkpoints are the only parts that are assumed correct, or am I wrong? There is no good reason to prevent a blockchain download via other methods.  In a future with the bitcoin network running close to it's capacity, downloading the entire blockchain over the P2P network will be harmful. 

Even a chain that has already been pruned of it's merkle trees should be able to be verified from the start, otherwise what good is using a merkle tree at all?

Sorry, these users' disk and CPU were not at 100%.  It is clear the bottleneck is not the database or indexing, for many users.


bzip2 gives you 33% compression ratio, saving many megabytes off a download:


I wouldn't call 33% "uncompressible"

The following code in AddToBlockIndex(main.cpp) is horribly inefficient, and dramatically slows initial block download:


This makes it impossible to use a standard technique for loading large amounts of records into a database (db4 or SQL or otherwise):  wrap multiple record insertions into a single database transaction.  Ideally, bitcoin would only issue a TxnCommit() for each 1000 blocks or so, during initial block download.  If a crash occurs, the database remains in a consistent state.

Furthermore, database open + close for each new block is incredibly expensive.  For each database-open and database-close operation, db4

• diagnose health of database, to determine if recovery is needed.  this test may require data copying.
• re-init memory pools
• read database file metadata
• acquire file locks
• read and initialize b-tree or hash-specific metadata.  build hash table / b-tree roots.
• forces a sync, even if transactions called with DB_TXN_NOSYNC
• fsync memory pool

And, additionally, bitcoin forces a database checkpoint, pushing all transactions from log into main database.

That's right, that long list of operations is executed per-database (DB), not per-environment (DB_ENV), for a database close+open cycle.  To bitcoin, that means we do this for every new block.  Incredibly inefficient, and not how db4 was designed to be used.

Recommendations:

1) bitcoin should be opening databases, not just environment, at program startup, and closing database at program shutdown.  db4 is designed to handle crashes, if proper transactional use is maintained -- and bitcoin already uses db4 transactions properly.

2) For the initial block download, txn commit should occur once every N records, not every record.  I suggest N=1000.



EDIT:  Updated a couple minor details, and corrected some typos.

Yeah, I missed the database-open caching buried in all the C++ constructors.  Major red herring, sorry about that.

db4 cache control is http://download.oracle.com/docs/cd/E17076_01/html/api_reference/CXX/dbset_cachesize.html

I timed two runs with clean data directories (no contents), -noirc, -addnode=10.10.10.1, Linux 64-bit.  Hardware: SATA SSD

Mainline, no patches:
     32 minutes to download 94660 blocks.

Mainline + TxnBegin/TxnCommit in AddToBlockIndex():
     25 minutes to download 94660 blocks.