Potential disaster scenario

The difficulty for generating bitcoins is periodically adjusted using a method
that has worked well this far.  However, I am afraid there are plausible
scenarios where the current method would misbehave quite spectacularly.

One scenario goes like this:

1) As bitcoins become more known, competition among minters continues to
    increase, with corresponding increases in difficulty.  The increased
    difficulty will eventually make bitcoin minting clearly unprofitable for
    those who do not have access to good energy prices and cheap access to an
    energy-efficient HW/SW combination.

2) Some bitcoin users may continue to mint bitcoins even though it is not
    profitable for them.  This could be due to ideology, the fun factor, or
    just ignorance.  But it is quite plausible that the vast majority of
    bitcoins will be minted by those who profit from it.  Let's say that 99% of
    all bitcoins are eventually minted by for-profit-minters.

3) The competition among for-profit-minters will drive profit margins down, to
    the point where it is profitable to continue minting, but barely so.  Let's
    say that the typical profit margin during one difficulty adjustment period
    (2016 blocks) is 10%.

4) Since bitcoin minting is a decentralized uncoordinated process, we can
    expect random fluctuations in bitcoin minting activity.  This does not
    affect the difficulty during a specific 2016-block period, so the minting
    activity can e.g. increase by 20% during a period without making minting
    unprofitable within that period.

Given the above assumptions, we now have a disaster at hand at the next
difficulty adjustment.  As bitcoin production was 20% more than target,
difficulty is adjusted upwards by 20%.  But the profit margin was only 10%, so
for-profit-minters would now lose money if they continued minting.  They will
therefore stop minting, and as they make up 99% of the minting capacity,
generating the next 2016 blocks will take 100 times longer than normal.
Everything that depends on block generation will slow to a crawl, and this
slowness will persist for a very long time, since the next 2016 blocks will
take 100 times longer to generate (almost 4 years rather than two weeks).

Now, if this was to happen, I guess a new client could be released that resets
the difficulty to some sensible number and started using a better algorithm
for difficulty adjustment.  But it would be much better to do it proactively
before it becomes a problem (perhaps with a predetermined "flag day"
activating the new algorithm at a certain time in the future, giving the new
client a chance to propagate).

A simple(?) modification of the algorithm would be to apply the adjustment
after a certain amount of time rather than at a certain block number.  The
switch could still be synced to take effect for the next block, so time
synchronization between clients would not need to be super exact to have the
vast majority of them agree on when the new difficulty is to be applied.

Also, the difficulty adjustment should probably take into account the
adjustments of the number of bitcoins minted per event (now 50, halved every 4
years).  Halving the number of bitcoins generated each time is equivalent to
doubling the difficulty as far as profitability is concerned, and such a
drastical drop in profitability is unnecessary if it can be avoided easily.
I'm not sure if the current adjustment algorithm already takes that into
account somehow, but I couldn't see any obvious adjustment for it in the
source code.

There is a limit to the difficulty adjustment, but for upward adjustments it is 300% (factor of 4 between new and old) rather than the 20% in my scenario, so it wouldn't help. 

I would argue that a 20% difficulty adjustment is in no sense extreme - the last five adjustments were 44%, 35%, 300%, 93%, and 21%.  This hasn't been a problem up to now, but obviously not because a 20% adjustment would be too extreme, but more likely because it hasn't been enough to make minting unprofitable (or otherwise uninteresting).  My scenario does not assume that a single entity is responsible for a jump in difficulty - it is just as problematic if e.g. publicity makes minting 20% more popular at a time when the profit margin is less than 20%.  The extreme effects come from the proportion of minters who are unwilling to continue minting at a loss.  If this proportion is large while the profit margin is low, the system becomes very unstable.

Also, if I am right about the 4-year halving of bitcoins generated (i.e. that it doesn't affect the difficulty adjustment) such events alone would be equivalent to a 100% upward difficulty adjustment.  And it seems very reasonable to assume that most minters will have a lower profit margin than 100% in the future.

The problem with your analysis is that you assume that all for-profit minters will have the same profit margin. They won't. Among other things, larger minters will have economies of scale in their favour, making them more profitable.

In addition, as Bitcoin grows, people will develop dedicated hardware that maximizes the khash/dollar spent. In addition, people will tune the software in more and more precise ways to squeeze slightly more khash/second out of the same hardware. The people who invest a large fixed cost to do that will receive a correspondingly lower variable cost per Bitcoin minted in return, so they'll be able to mint at price levels that would drive others out.

Finally, at the point that this becomes an issue, everyone will be including transaction fees with their transactions to incentivize minting.

You ignore all kinds of things that help solve this.

Transaction fees will increase if blocks come more slowly because it will be more desirable to get your transactions in the next block.

If an entity calculates that (fixed costs + variable costs < revenue) adjusted for risk the it is pretty unlikely that they will hit the (variable costs > revenue) easily at all and that is the shutdown point.

These minters will usually have and use bitcoins themselves. This gives two extra incentives to operate at a "loss". They want their transactions to get processed, and they want to maintain bitcoin as a working system.

What if I had a factory and hired you as a consultant? Would you tell me avoiding costs altogether was the best thing? Many, many costs are worth it.

Having a secure, open, P2P, pseudonymous, limited quantity money, is worth a ton. People who agree will pay the .001BTC or whatever to make a transaction if that's about what it ends up costing. 

Really? Source?

Sorry, I was wrong about that.  I though I had read something on the forum to that effect, but now I can't find it.  Looking at old sources, I see that the 4x cap was there as early as bitcoin-0.1.0.rar, so this is definitely not a new modification.

It's in my original post:

Could you elaborate on what kind of scenarios you see where the proposed algorithm would be more vulnerable than the current one? Note that the current algorithm also uses wallclock time in the difficulty adjustment calculation and is synchronized using newly generated blocks.

Transactions are verified by generating blocks, so although I don't think it will happen, blocks being too difficult to generate would cause the economy to grind to a halt.

Blocks are used to verify transactions. For example, Bitcoin Market requires two confirmations. If a block takes two hours to make, then it would take four hours to put Bitcoins into Bitcoin Market. This would not be good.

When people start adding transaction fees, the value of the current block will steadily rise until it is solved. If this gets very high, everyone will attempt to "win the jackpot". The market will solve the problem.

Botnets will probably be supporting Bitcoin in the future, so it doesn't really matter anyway.

Some places where generation will gravitate to:
1) places where it's cheapest or free
2) people who want to help for idealogical reasons
3) people who want to get some coins without the inconvenience of doing a transaction to buy them

There are legitimate places where it's free.  Generation is basically free anywhere that has electric heat, since your computer's heat is offsetting your baseboard electric heating.  Many small flats have electric heat out of convenience.

How expensive is heating oil?  With the price of oil so high, if it's actually more expensive than electric, then generating would have negative cost.

There's also kids putting it on their parent's power bill, employees their employer, botnets, etc.

Case 3 comes into play for small amounts.  The overhead of doing an exchange doesn't make sense if you just need a small bit of pocket change for incidental micropayments.  I think this is a nice advantage vs fiat currency, instead of all the seigniorage going to one big entity, let it go in convenience amounts to people who need to scrape up a small amount of change.