BitcoinTalk

How to overthrow the GPU Oligarchs

BitcoinTalk
#1
From:
chickenado
Subject:
How to overthrow the GPU Oligarchs
Date:
The strength of this project was meant to be its distributedness.

But now, it seems that an increasingly small and exclusive elite has taken charge of coin/block generation. It's dominated by specialists who have access to wholesale means of production and secret, proprietary GPU code.

The average user no longer has a fighting chance and has given up generating blocks altogether.

What does this remind me of? That's right, bitcoin is becoming like the physical money economy, where a small number of central banks now guarantee the trustworthiness of paper money.

There are two problems: One, the possibility of market manipulation, and more importantly, if only a handful of people do most of the proof of work, what is the point of a decentralized currency in the first place? It weakens us, because a powerful attacker would only need to bribe a few people to gain more than 50% of khash/s.

To stop this, I suggest the following:

1) Add more hash functions that must be solved in addition to SHA-2. The more, the better. The function ecosystem should include functions with very different requirements, eg. some that require lots of memory, some that can't be paralellized, some that require lots of disk space,...
Also, function parameters that change randomly every time difficulty changes. This would take away some advantage from people who are highly specialized at solving just one hash function.

2) Pooled block generation. For instance, if 100 users join a pool and one block is solved, the 50 coins are distributed among those users. This would encourage amateur users to start generating again.
BitcoinTalk
#2
From:
nelisky
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
Pooled generation: +1

Add more hash functions... how would this balance the generation? People with better hw would still generate more, in all hash functions. Maybe not GPU / massive parallel, but any problem more/better hardware can't help resolve is probably out of general user's boundaries.

GPU and unfairness. Realize that the hardware needed to stand a chance of generating isn't all that cutting edge or expensive. Any mid-level gamer machine has the engine to put out 20 or 30 million hashes/sec, at least. The real issue is software, as this gives the owner of such software a huge advantage for a very small price, thus people will obviously take advantage of this as long as possible.

I feel stupid, really. I could put a little more money/effort into this, say nothing about it, and generate away. Instead here I am with my sense of community duty and such... damn, I'll never get rich this way Smiley

If you feel strongly as to the need for fairness, why don't you Help the little ol CUDA developer?
BitcoinTalk
#3
From:
hugolp
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
Someone should work on making the GPU code open. That is the best way to stop this.
BitcoinTalk
#4
From:
chickenado
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
If you feel strongly as to the need for fairness, why don't you Help the little ol CUDA developer?

My concern isn't really fairness, it's stability.

I buy about $20 worth of BTC each month, but I spend most of them straight away. What is keeping me from buying $1000 worth of BTC is my concern with the problems I mentioned. As a way of transferring money Bitcoin is great, as a way of saving money it is still too risky for my tastes. To succeed as a currency it must be good at both.
BitcoinTalk
#5
From:
nelisky
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
Someone should work on making the GPU code open. That is the best way to stop this.
That's exactly what nelisky is doing. Feel free to follow his link and help him.

Have done, actually. There is working (and generating) code already available here. It is known to work on OSX 10.6 (my dev environment) and linux (myself on ubuntu 10.04 and the Russian folks I'm not sure what OS).

The mentioned thread is a request for help in pursuing this, which I will with or without said help so you can safely ignore and still have a go at the outcome. The difference will be the amount of effort I put in this and thus the time to completion.

I'd also be happy to have someone else pick up the code too, but I fear all that have done so already are generating on their own, not giving anything back. The complete lack of developer feedback seems to point that way, at least. But, hey, we're only human, and we were taught to take advantage of the situation everytime we can, I understand. I'm just trying to teach my kids otherwise, by showing them being generous can work too Smiley
BitcoinTalk
#6
From:
Anonymous
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
Someone should work on making the GPU code open. That is the best way to stop this.
That's exactly what nelisky is doing. Feel free to follow his link and help him.

Have done, actually. There is working (and generating) code already available here. It is known to work on OSX 10.6 (my dev environment) and linux (myself on ubuntu 10.04 and the Russian folks I'm not sure what OS).

The mentioned thread is a request for help in pursuing this, which I will with or without said help so you can safely ignore and still have a go at the outcome. The difference will be the amount of effort I put in this and thus the time to completion.

I'd also be happy to have someone else pick up the code too, but I fear all that have done so already are generating on their own, not giving anything back. The complete lack of developer feedback seems to point that way, at least. But, hey, we're only human, and we were taught to take advantage of the situation everytime we can, I understand. I'm just trying to teach my kids otherwise, by showing them being generous can work too Smiley

Thanks for the work you are doing.
BitcoinTalk
#7
From:
nelisky
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
Someone should work on making the GPU code open. That is the best way to stop this.
That's exactly what nelisky is doing. Feel free to follow his link and help him.

Have done, actually. There is working (and generating) code already available here. It is known to work on OSX 10.6 (my dev environment) and linux (myself on ubuntu 10.04 and the Russian folks I'm not sure what OS).

The mentioned thread is a request for help in pursuing this, which I will with or without said help so you can safely ignore and still have a go at the outcome. The difference will be the amount of effort I put in this and thus the time to completion.

I'd also be happy to have someone else pick up the code too, but I fear all that have done so already are generating on their own, not giving anything back. The complete lack of developer feedback seems to point that way, at least. But, hey, we're only human, and we were taught to take advantage of the situation everytime we can, I understand. I'm just trying to teach my kids otherwise, by showing them being generous can work too Smiley

Thanks for the work you are doing.

My pleasure, really. But in all fairness, it is not true I don't get both appreciation and donations. What happens is the folks that do support me do so in a very altruistic way, as in "that sounds great for the community, keep it up" kind of way. These helping members do not have, to the best of my knowledge, any particular interest in the work I do, just think it helps everyone out.

The ones that have the hardware to take advantage of this, if they do exist, keep very, very silent Smiley
BitcoinTalk
#8
From:
theymos
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
But now, it seems that an increasingly small and exclusive elite has taken charge of coin/block generation. It's dominated by specialists who have access to wholesale means of production and secret, proprietary GPU code.

The average user no longer has a fighting chance and has given up generating blocks altogether.

It's always been the plan for block generation to not be profitable for most people. It's supposed to be done mainly by dedicated "backbone" entities. What's stronger: a thousand people producing 1 Gh/s with hardly any individual economic interest in preserving the network's integrity, or five businesses producing the same Gh/s that will fail if they or someone else destabilizes the network?

I'm not saying that GPU code shouldn't be made, but it's not fair to say that GPU generators are "taking without giving back". People dedicated to generating deserve a head-start on new technologies.
BitcoinTalk
#9
From:
kiba
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
This is a non-issue. Nothing to see here, keep moving.
BitcoinTalk
#10
From:
nelisky
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
But now, it seems that an increasingly small and exclusive elite has taken charge of coin/block generation. It's dominated by specialists who have access to wholesale means of production and secret, proprietary GPU code.

The average user no longer has a fighting chance and has given up generating blocks altogether.

It's always been the plan for block generation to not be profitable for most people. It's supposed to be done mainly by dedicated "backbone" entities. What's stronger: a thousand people producing 1 Gh/s with hardly any individual economic interest in preserving the network's integrity, or five businesses producing the same Gh/s that will fail if they or someone else destabilizes the network?

I'm not saying that GPU code shouldn't be made, but it's not fair to say that GPU generators are "taking without giving back". People dedicated to generating deserve a head-start on new technologies.

I was under the impression that coin generation was a way to make sure everyone had a fair chance of putting effort against coins, thus avoiding any central committee of deciding the future of bitcoins. Decentralized is the word, I believe...

But I get it, if you put the time and money into getting better miners, you should be able to keep it to yourself, so the 4way patch should never had been merged into mainline, as its creator would have 2x+ performance over the same machine running common clients.

I have to disagree here. Yes, competition is good and valuable, and people should not get into bitcoins for the mining alone, but if you let a small subset of people hold most of the future generated coins you are putting the whole project in danger of abuse and destruction... but I may be overreacting.

I'll be glad to stop posting code, buy some serious hw and just do the generation myself. As difficulty goes up and people stop generating, this gets more and more statistically interesting... you say I should, right?
BitcoinTalk
#11
From:
theymos
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
I'll be glad to stop posting code, buy some serious hw and just do the generation myself. As difficulty goes up and people stop generating, this gets more and more statistically interesting... you say I should, right?

The network will eventually be run by "oligarchs". Once software is optimized as far as it can be, it will come down to hardware, bandwidth, and, in the long-term, electricity generation. Most people won't be able to keep up.

Posting GPU code now will just prolong the period when generation is feasible for normal people. This will attract a few users, and it might increase the network's total power on the short-term, but on the long-term it'll have little value. If I were you, I'd keep the code private. Publishing it wouldn't be bad for the network, though.
BitcoinTalk
#12
From:
TTBit
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
Start a pool yourself. If you are generating 4000k/sec 10 others are generating 20,000k/sec combined, you get 4,000/24,000 of each block generated. Seems this code could be introduced. I don't think the GPU Oligarchs are making a mint, in fact, I bet they are losing money to my benefit.

I bet you could pay nelisky for proprietary code yourself. But, the economics won't work out.

BitcoinTalk
#13
From:
theymos
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
Pools won't eliminate the "problem" because pools are not more profitable than normal generation; they just pay out more often. They can't beat companies that have invested in specialized hardware. They also delegate all of the important network decisions to the pool maintainer, so there's no security benefit.
BitcoinTalk
#14
From:
chickenado
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
Quote
The network will eventually be run by "oligarchs". Once software is optimized as far as it can be, it will come down to hardware, bandwidth, and, in the long-term, electricity generation. Most people won't be able to keep up.

It will if proof-of-work comes down to something as specialized as solving a single, well-defined, simple, and predictable hash function such as SHA-2.

We need to make proof-of-work more choatic and complex if we want to avoid this. 

SHA-2 was actually designed to be computationally EASY because it's original purpose was digital signing, not proof-of-work. 

Otherwise, it will end just like you say.

Think of this vision of the year 2020: 2/3 of the world's Ghash/s is controlled by just three companies. They manufacture their own specialilsed HPUs (hash processing units) and feed them with custom-built power plants. The Chinese goverment then persuades the three companies to double spend 10% of all transactions into the Chinese government's bitcoin address, in return the companies get an exclusive deal on the rare earths they badly need for the HPU manufacture. Some people leave Bitcoin for alternative currencies without the 10% "tax", but most tolerate it because by then Bitcoin is accepted almost universally.



BitcoinTalk
#15
From:
nelisky
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
Quote
The network will eventually be run by "oligarchs". Once software is optimized as far as it can be, it will come down to hardware, bandwidth, and, in the long-term, electricity generation. Most people won't be able to keep up.

It will if proof-of-work comes down to something as specialized as solving a single, well-defined, simple, and predictable hash function such as SHA-2.

We need to make proof-of-work more choatic and complex if we want to avoid this. 

SHA-2 was actually designed to be computationally EASY because it's original purpose was digital signing, not proof-of-work. 

Otherwise, it will end just like you say.

Think of this vision of the year 2020: 2/3 of the world's Ghash/s is controlled by just three companies. They manufacture their own specialilsed HPUs (hash processing units) and feed them with custom-built power plants. The Chinese goverment then persuades the three companies to double spend 10% of all transactions into the Chinese government's bitcoin address, in return the companies get an exclusive deal on the rare earths they badly need for the HPU manufacture. Some people leave Bitcoin for alternative currencies without the 10% "tax", but most tolerate it because by then Bitcoin is accepted almost universally.





Generation will be less and less interesting in that way, as the coins per block will divide by 2 until there's no coins generated at all, and the system will need to be run by "volunteers", which aren't really volunteers because if no block is generated no coins can be transfered, thus removing all value from all coins...

I think it's an arms wrestle to try to avoid having a few people holding most of the coins, that's all. I was trying to leverage that by making everyone have a fair chance at getting coins while they are still being generated, but I fear that it is not practical.. When it comes to currencies (and I don't want to get into a semantics discussion, humour me with 'currency') people are just wired in a very egotistic way, and even when offered the chance to balance the scoreboard, they will just try and use that to their own advantage, not everyone's... it saddens me, really, but hey I'll keep on doing what I do because I strongly believe in it, just as others believe in taking the largest slice.
BitcoinTalk
#16
From:
theymos
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
Generation will be less and less interesting in that way, as the coins per block will divide by 2 until there's no coins generated at all, and the system will need to be run by "volunteers", which aren't really volunteers because if no block is generated no coins can be transfered, thus removing all value from all coins...

The companies can raise fees if generations aren't enough to profit.

Every calculation can be made more efficient in hardware. Trying to prevent it is pointless. It'd be a lot like making effective DRM.
BitcoinTalk
#17
From:
MoonShadow
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
The strength of this project was meant to be its distributedness.

But now, it seems that an increasingly small and exclusive elite has taken charge of coin/block generation. It's dominated by specialists who have access to wholesale means of production and secret, proprietary GPU code.

The average user no longer has a fighting chance and has given up generating blocks altogether.

What does this remind me of? That's right, bitcoin is becoming like the physical money economy, where a small number of central banks now guarantee the trustworthiness of paper money.

You seem to have missed the point.  The distributed nature of Bitcoin is not compromised by the consentration of block generation.  Because, unlike fiat currencies and central banking, no one group or person has monopoly control of the currency, and no one can manipulate it as such.  Not even the GPU oligarchs.  Also, they are competing among themselves.  If one of them starts thinking that he has a majority of the block generation locked up, and decides he is going to try and fork-steal, the rest are going to notice the changes and they all have a strong incentive to overtake the market leader.
BitcoinTalk
#18
From:
martin
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
SHA-2 was actually designed to be computationally EASY because it's original purpose was digital signing, not proof-of-work.

SHA is easy to calculate, but it's difficult to calculate a piece of data which will generate a given hash (which is [almost] what bitcoindoes). The proof of work must be difficult to calculate but easy to prove, which SHA is
BitcoinTalk
#19
From:
hugolp
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
If you feel strongly as to the need for fairness, why don't you Help the little ol CUDA developer?

My concern isn't really fairness, it's stability.

I buy about $20 worth of BTC each month, but I spend most of them straight away. What is keeping me from buying $1000 worth of BTC is my concern with the problems I mentioned. As a way of transferring money Bitcoin is great, as a way of saving money it is still too risky for my tastes. To succeed as a currency it must be good at both.

Mmm... Are you sure your dollars dont depreciate quicker than bitcoins? Dollars are worthless to create.

Also, thinking that holding dollars is less risky than bitcoins might be a bit naive.
BitcoinTalk
#20
From:
Anonymous
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
How many Ghash's do you think Google would do if it started to generate?

BitcoinTalk
#21
From:
LZ
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
BitcoinTalk
#22
From:
MoonShadow
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
How many Ghash's do you think Google would do if it started to generate?


Probably all of them, but difference would it make?  Even if they are willing to commit the full brunt of their computing resources to overtake the generation of new coins, they can't make them faster than the difficulty system would permit.  At least not for more than two weeks.  Nor can they do anything to manipulate the value of the coins already in existance, beyond the effects of the regular generation awards.  That's the beauty of the design, even Google couldn't do more than disrupt the system temporarily.  Even at this early stage, the total proof-of-work represents a supercomputing class cluster.  For what gain would anyone do such a thing?  For a max return of 50K coins over two weeks of supercomputing time?  Even Google doesn't have the computing power to rebuild the blockchain from the beginning.  Even if they did, to what end?  To steal a quarter million dollars in wealth?  Which would likely collapse the system if it could be done at all, so that value would drop to zero.  As we have seen, that difficulty level increases as the user base increases, regardless of an analysis as to the economics of each person generating.

I can accept that, however unlikely, a takeover of the system is possible; but if such a thing were to happen, it certainly wouldn't go unnoticed by the Bitcoin community.  From where I stand, the odds that the Federal Reserve will cease to exist and the FRN collapse is more likely to occur first.  For that matter, I would consider a worldwide extinction-level-event to be about as likely; but I'm not going to let the risks of a meteor strike stop me from walking out on the surface of the Earth.
BitcoinTalk
#23
From:
asdf
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
Pools won't eliminate the "problem" because pools are not more profitable than normal generation; they just pay out more often. They can't beat companies that have invested in specialized hardware. They also delegate all of the important network decisions to the pool maintainer, so there's no security benefit.

Pools offer the advantage that nodes can co-ordinate their hashing so that they aren't generating the same hashes as each other. It's not about "total hash/s", it's about "total unique hash/s". If everyone in the pool is assigned a subset of all hashes to work on (sizes based on each nodes average hash/s), then we'll guarantee that no hashes will be repeated. The node that generates the coin can just keep the full amount. The group will be better off together than alone. The bigger the pool, the better. In fact if these pools get much bigger than the companies (in terms of hash/s), then THEY will be severely marginalized and be forced to join the main pool or get nothing.

If the pool gets too big, then coins may be generated almost instantly at some point. This could be bad for the stability of the currency. Can anyone do the math on how many parallel coperating cpus it would take to find a coin in say, one minute?

I think this is a very important issue for the success of Bitcoin.
BitcoinTalk
#24
From:
theymos
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
Pools offer the advantage that nodes can co-ordinate their hashing so that they aren't generating the same hashes as each other. It's not about "total hash/s", it's about "total unique hash/s". If everyone in the pool is assigned a subset of all hashes to work on (sizes based on each nodes average hash/s), then we'll guarantee that no hashes will be repeated.

This is already guaranteed because everyone has a unique public key in their block. You reminded me of another way that pools are bad, though: since everyone uses the same public key, they have to do weird things with extraNonce, which increases the size of the block header and makes generating more difficult for them.
BitcoinTalk
#25
From:
LZ
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
Can you tell more about this:
"they have to do weird things with extraNonce, which increases the size of the block header".
BitcoinTalk
#26
From:
FreeMoney
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
Can you tell more about this:
"they have to do weird things with extraNonce, which increases the size of the block header".

Usually you can start the nonce at 0 because they block you are working on is made unique by the inclusion of your unique address in the generate transaction. If a group of people all send generates to the same address you need another number included to make sure that the members are not repeating work.

I have no idea how important that fact is, it doesn't seem like too big of a deal to me, but theymos can elaborate.
BitcoinTalk
#27
From:
theymos
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
Can you tell more about it:
"they have to do weird things with extraNonce, which increases the size of the block header".

When you generate, you calculate hashes of the block header. Hashing more data is slower than hashing less data, so the block header is critically of a fixed size for everyone, with one exception. After every hash attempt, you increment the Nonce header field, but since this field is only 32 bytes long, it overflows a lot. Whenever it overflows, you increment the variable-size extraNonce field. The larger extraNonce gets, the slower generating will get. It doesn't get significantly large with normal incrementing, though.

If you have a lot of computers and they're all working on the same block with the same public key, then they're all very likely to be hashing the same block at the same time, which is pointless. To fix this, each computer is given a unique extraNonce modifier value. This might be very large to prevent collisions, and it therefore slows down hashing.

Undoubtedly you could design a pooling system without this flaw, but it'd be more difficult.

I see that m0mchil's getwork is doing something with extraNonce. I don't know how bad that implementation is, but it theoretically must be slower than a client without it (all things being equal; clearly adding GPU support will improve performance).
BitcoinTalk
#28
From:
chickenado
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
Quote
Even Google doesn't have the computing power to rebuild the blockchain from the beginning.  Even if they did, to what end?  To steal a quarter million dollars in wealth?  Which would likely collapse the system if it could be done at all, so that value would drop to zero. 

They wouldn't need to rebuild the blockchain from the beginning. They can double spend any transaction that happens after the 50% Ghash/s takeover. Of course, if they get too greedy, the system would collapse as users become fed up with their coins being stolen all the time. But what if they decide to double spend a certain "optimal income" percentage of transactions, just short of what will piss off most users?
BitcoinTalk
#29
From:
grondilu
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
Non issue, imo.

Those guys are just working for us.  I'll just buy their bitcoins.  It will probably be cheaper than buying expensive hardware and running it 24/24.

By selling bitcoins those might think they screw us.  I think they're just wrong.
BitcoinTalk
#30
From:
gebler
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
The larger extraNonce gets, the slower generating will get.

But that effect would be totally insignificant.  Even the standard client is optimized so that it doesn't need to hash the entire block header if only the last part of it has changed.  That last part includes the nonce but not the extraNonce, so a larger extraNonce wouldn't cost anything for the vast majority of all hashes, namely those where only the normal nonce has been incremented.  With today's transaction volumes, the cost of hashing the extraNonce should be less than a millionth of a percent for pools with thousands of members. 
BitcoinTalk
#31
From:
gebler
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
If you have a lot of computers and they're all working on the same block with the same public key, then they're all very likely to be hashing the same block at the same time, which is pointless. To fix this, each computer is given a unique extraNonce modifier value. This might be very large to prevent collisions, and it therefore slows down hashing.

In a centralized system, the server could simply keep a list of extraNonces in active use and give out the lowest free one.  Then you would avoid collisions without requiring more than one unique extraNonce per client.  A 3-byte extraNonce would be sufficient for millions of clients.  Note that clients of realistic performance do not really need the extraNonce to handle nonce overflow today, since such overflows happen less frequently than the periodic nTime updates (but this is not considered by the standard client today, which updates the extraNonce more liberally).
BitcoinTalk
#32
From:
chrisdbc
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
I'm just getting into this whole bitcoin thing. It seems to me already, though, that getting
great results from bitcoin mining would require a server farm, super computer, etc; worrying
about individuals taking advantage of their GPUs or crypto accelerators is probably pointless.
What can 30000 khash/s really get you? Maybe around the equivalent of $1.50 per day at
current difficulty levels? Not really a very efficient means of recouping the cost of an expensive
gaming rig.

I'd be much more concerned about what somebody with a botnet might be capable of.

BitcoinTalk
#33
From:
kiba
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
I'd be much more concerned about what somebody with a botnet might be capable of.

So what if a botnet generate all these bitcoins? He'll be able to move markets for his money. Big deal.
BitcoinTalk
#34
From:
satoshi
Subject:
Re: How to overthrow the GPU Oligarchs
Date:
Can you tell more about it:
"they have to do weird things with extraNonce, which increases the size of the block header".
When you generate, you calculate hashes of the block header. Hashing more data is slower than hashing less data, so the block header is critically of a fixed size for everyone, with one exception.
This is the point of confusion.  extraNonce is not part of the block header, it is part of the first transaction.  It does not slow down your hashing.  It does not change the size of the header.

We need to be vigilant and nip in the bud any misconception that the contents of your block slows down your hash speed.  It doesn't.

extraNonce never needs to be very big.  We could reset it every second whenever the time changes if we wanted.  Worst case, if you didn't want to keep track of incrementing it, extraNonce could be 4 random bytes and the chance of wasting time from collision would be negligible.

Separate machines are automatically collision proof because they have different generated public keys in the first transaction.  That also goes for each thread too.