On IRC bootstrapping

To eliminate segmentation, every peer should have a list of every other peer. Tor also has this requirement, so we should copy them: have several trusted directory servers periodically create a signed list of all peers and publish it via HTTP. All BitCoin clients have the option to act as a directory mirror, which will be indicated in the dirServers' list. Generators need to ask to be added to the list (which could also include info like the maximum number of connections that peer will accept), but people just wishing to make transactions can just get the list from a dirMirror and connect to a few random peers.

If this is too centralized, we can do what I2P did and allow anyone to become a directory server. You need to be able to detect when a dirServer goes rogue in this case, though.

Thanks soultcer for talking with the Freenode staffer.  Good to know it's OK at the current size, and now they know who we are.  They're supportive of projects like TOR so I hope they would probably be friendly to us.  We don't want to overstay our welcome.  If we get too big, then by the same token, we're big enough that we don't need IRC anymore and we'll get off.

We only needed IRC because nobody had a static IP.  In the early days there were some steady supporters, but they all had pool-allocated IPs that change every few days.  IRC was only intended as a temporary solution.  Bitcoin's built-in addr system is the main solution.

Bitcoin can get the list of IPs from any bitcoin node.  In that sense, every node serves as a directory server.

When there are enough static IP nodes to have a good chance that at least one will still be running by the time the current version goes out of use, we can preprogram a seed list.

How do you think we should compile the seed list?  Would it be OK to create it from the currently connected IPs that have been static for a while?

BTW, if we want to supplement by deploying separate directory server software, may I suggest IRC?  IRC is a good directory server (I've heard it has other uses too), and there are mature IRC server implementations available that anyone can run.   Bitcoin's IRC client implementation is already thoroughly tested.

Hi all.

This may be a stupid idea, and if it's not stupid, may not be viable for the next few years, but I thought I'd toss it out there anywhere:

What about Multicasting?

IPv6 is supposed to have better multicasting support than IPv4, and if I did not misunderstand the Bitcoin protocol, most messages need to be broadcasted to the entire network. Theoretically a node could send such messages to a global multicast address, and everyone will receive it in a bandwith-efficient way.

That would make bootstrapping in the traditional sense obsolete, since the client would just have to subscribe to the multicast channel. The remaining "Give me block xyz" requests could be handled by optionally including a field in messages to the multicast channel that basically says "My address is 2001:db8::42, and I'm willing to answer direct queries for specific blocks". After listening to the channel for a while, such a packet should come around, because, at the very least, new blocks will be announced there every so often.

TCP doesn't work with multicasting. And I doubt it will ever be easy for a home user to join a multicast group.

I throughly agree with this. In the long run, IRC should be completely phased out and replaced with something like Gnutella's host caches or Tor's dictionary servers (as others have suggested).

At the very least, Bitcoin should disconnected from IRC as soon as it has a list of peers to connect to. It should also cache that list so it wouldn't need to reconnect upon the next startup.

Just my 0.02฿.

What's wrong with IRC?  It's just another method that's used to exchange the peer list.  You can just prevent it from connecting and use -addnode=1.2.3.4 to connect to a known node for bootstrapping if you want..

If nodes disconnect from IRC after they get their list then it makes it less useful for bootstrapping since it will be empty except for the node that's trying to bootstrap at the time.

IRC has been around forever and it's well documented (and easy to understand for newcomers) - why create something more complicated?

Bitcoin has its own distributed address directory using the "addr" message.  It's about time we coded in a list of the current long running static nodes to seed from.  I can add code so new nodes do not preferentially stay connected to the seed nodes, just connect and get the list, so it won't be a burden on them.

What do you think, should I go ahead with adding the seeds?

It'll still try IRC first.  The IRC has the advantage that it lists nodes that are currently online, since they have to stay connected to stay on the list, but the disadvantage that it's a single point of failure.  The "addr" system has no single point of failure, but can only tell you what nodes have recently been seen, so it takes a little longer to get connected since some of the nodes you try have gone offline.  The combination of the two gets us the best of both worlds and more total robustness.

Is there anyone who wants to volunteer to run an IRC server in case freenode gets tired of us?

This is a common problem in P2P, known as Original Introduction, although bootstrapping is also a good word for it. The problem with bootstrapping is that you can't decentralize it. Whether it's IRC or HTTP or DNS, the client needs to be hardcoded with an address or list of addresses which is sufficiently fresh that at least one of the listed addresses is still active. After the first node is reached, you are no longer in Original Introduction mode and can use the full range of techniques for decentralization, such as gossip. Unless, of course, you get disconnected from the network and all of your known peers go away, in which case you're back to bootstrapping.

There are two properties that are at odds when you chose a bootstrapping method: robustness (scalability/reliability) and freshness. Robustness is increased at the expense of freshness by caching on multiple servers, as is usually done with HTTP peer lists. Freshness is maximized (at least up to the TCP timeout) at the expense of robustness by having everyone connected, as with IRC. Of course, the key is finding the right mix of robustness and freshness because you need both for the bootstrap to be successful.

Here are some of my current favorite methods for bootstrapping:

Append list of fresh peers to executable or installer dynamically on download. People usually get the application from its official website, so the website is already a point of failure for new users. You're already hardcoding an address in the application, the address that the application will use to bootstrap. So instead just add fresh peers at the moment of download. You need some fancy code in the executable to read the list off the end, but I've implemented this in an NSIS installer and it's not that hard. Most software developers are upset by the idea of this method.

Connect via XMPP to Google App Engine application. This gives the freshness of IRC, but with more robust scaling. App Engine is mostly for writing web apps, but it provides email and XMPP handling as well. It would be simple to write one application that could handle peer lists via either XMPP or HTTP with the same handler code. I'm currently using this in an application and it works well and is very reliable. I only wish there was a second App Engine to use as a fallback because it does have occasional downtime.

An alternative to requiring all nodes to include the complexity of a protocol like IRC or XMPP is to have a few special sentinel nodes which sit on the network and collect addresses of connected nodes via the usual decentralized methods available to an active node. These sentinel nodes periodically upload fresh addresses, say via HTTP POST to a number of websites. A new node can then download a fresh address list from any of the websites which is currently functioning and reachable. If you have 5 sentinels each uploading every 5 minutes (staggered), then you'll have updates roughly once a minute. This is on par with IRC in terms of freshness and is robust as you care to make it by varying the number of HTTP mirrors and the number of sentinels.

The SVN version now uses IRC first and if that fails it falls back to a hardcoded list of seed nodes.  There are enough seed nodes now that many of them should still be up by the time of the next release.  It only briefly connects to a seed node to get the address list and then disconnects, so your connections drop back to zero for while.  At that point, be patient.  It's only slow to get connected the first time.

This means TOR users won't need to -addnode anymore, it'll get connected automatically.  

You may want to leave Freenode in as a fallback server -- if his server doesn't work, use Freenode's.

Everybody needs to connect to the same IRC server and channel so they can find each other.

It might not be good if we suddenly rushed freenode with a ton of users all at once.

The fallback is our own seed system.

irc.lfnet.org is pretty old and has impressive uptime.  I think it's going to be fine.

We could take IRC out at some point if we want, but I'd rather ease into it and just test our own seed system as a backup for now, and I really like the complementary redundant attributes of the two different systems.