BitcoinTalk

JSON-RPC method idea: list transactions newer than a given txid

BitcoinTalk
#1
From:
davux
Subject:
JSON-RPC method idea: list transactions newer than a given txid
Date:
It would be great to have a JSON-RPC method for listing new transactions that are newer than a particular transaction id. This would enable developpers to watch new transactions easily, by just keeping track of the latest known txid and polling that method at the rate of their choice.

A possible way to do it would be to extend listttransactions so that it accepts an optional txid argument:
Code:
listtransactions <account> [count=10] [txid]

If txid is given, don't show it, nor any transaction older than it.

(As a sidenote, maybe listtransactions, with or without txid, should accept filtering by transaction category: generate, send, receive, move.)

I'm sorry I can't propose any patch here, as I can't do any decent C++.
BitcoinTalk
#2
From:
nelisky
Subject:
Re: JSON-RPC method idea: list transactions newer than a given txid
Date:
+1

I haven't got my hands dirty there yet, and would rather spend what time I find to do import/export of keys, but that is something that would help some of my scripts a lot.
BitcoinTalk
#3
From:
jgarzik
Subject:
Re: JSON-RPC method idea: list transactions newer than a given txid
Date:
It would be great to have a JSON-RPC method for listing new transactions that are newer than a particular transaction id. This would enable developpers to watch new transactions easily, by just keeping track of the latest known txid and polling that method at the rate of their choice.

If we're adding code to watch for new transactions, then we should be including gavin's monitor patches:
https://github.com/gavinandresen/bitcoin-git/tree/monitorreceived
BitcoinTalk
#4
From:
genjix
Subject:
Re: JSON-RPC method idea: list transactions newer than a given txid
Date:
Surely since the WalletTx are added sequentially to the wallet, we can assume that older txid's appear earlier in the wallet?

Then it's simply trivial to do a linear lookup and then continue on spitting out the txid's after that one.

Can anyone confirm this?
BitcoinTalk
#5
From:
satoshi
Subject:
Re: JSON-RPC method idea: list transactions newer than a given txid
Date:
It's not safe to use listtransactions this way.

I know I've been criticized for being reluctant about listtransactions.  Let me explain my reluctance.

Transactions are dynamic.  Past transactions can become unconfirmed, go away and come back, become invalid and disappear, or be replaced by a different double-spend.  Their date can change, their order can change.

Programmers are naturally inclined to want to use listtransactions like this: feed me the new transactions since I last asked, and I'll keep my own tally or static record of them.  This will seem to work in all regular use, but if you use the amounts for anything, it is highly exploitable:
1) How do you know if a past transaction becomes invalid and disappears?
2) When there's a block-chain reorg, it would be easy to double-count transactions when they get confirmed again.
3) A transaction can be replaced by a double-spend with a different txid.  You would count both spends.

The model where you assume you only need to see new transactions because you've already seen previous transactions is not true.  Old transactions can change at any time.

Any time you take an action based on payment amounts received, you always need to go back to bitcoin and ask for a current balance total (or use move or sendfrom), and be ready for the possibility that it can go down.

Now that we have the Accounts feature making it easier to do it the right way, we're better prepared to have listtransactions.
BitcoinTalk
#6
From:
jgarzik
Subject:
Re: JSON-RPC method idea: list transactions newer than a given txid
Date:
Surely since the WalletTx are added sequentially to the wallet, we can assume that older txid's appear earlier in the wallet?

Then it's simply trivial to do a linear lookup and then continue on spitting out the txid's after that one.

The wallet is a key/value db4 database (and a key/value map in RAM).

Neither data structure is ordered by time.
BitcoinTalk
#7
From:
mikegogulski
Subject:
Re: JSON-RPC method idea: list transactions newer than a given txid
Date:
I can see the motivation for davux's request. In developing the e-commerce plugins I've released, I've wondered about the viability long term of using a "dump" method against the wallet, since over time the data returned is just going to grow and grow.

What I'd prefer is being able to attach a JSON-RPC callback URL to an account or an address, maybe to be invoked each time a confirmation or any other status change to that account or address comes in, until the callback is cleared. That would eliminate the need for polling the wallet entirely.

I threw a "feature request" post in about that someplace.
BitcoinTalk
#8
From:
jgarzik
Subject:
Re: JSON-RPC method idea: list transactions newer than a given txid
Date:
I know I've been criticized for being reluctant about listtransactions.  Let me explain my reluctance.

Transactions are dynamic.  Past transactions can become unconfirmed, go away and come back, become invalid and disappear, or be replaced by a different double-spend.  Their date can change, their order can change.

Programmers are naturally inclined to want to use listtransactions like this: feed me the new transactions since I last asked, and I'll keep my own tally or static record of them.  This will seem to work in all regular use, but if you use the amounts for anything, it is highly exploitable:
1) How do you know if a past transaction becomes invalid and disappears?
2) When there's a block-chain reorg, it would be easy to double-count transactions when they get confirmed again.
3) A transaction can be replaced by a double-spend with a different txid.  You would count both spends.

At some point, a website or person accepting a transaction must take this risk.  It is unavoidable, whether you use listreceivedbyaddress or listtransactions.  This is why listtransactions reluctance seems so unusual.

Almost every exchange or website accepting bitcoins achieves a binary decisionpoint, where the transaction is accepted, and goods are shipped or money is exchanged.  After that binary decisionpoint, even if the block chain is reorg'd or transactions disappear, there is nothing the website can do but take a loss (or pursue a refund outside of bitcoin).

From the website's point of view, there is zero effective difference between 'listtransactions 6' and 'listreceivedbyaddress 6', because the end result to the website operator is the same:  the goods have been shipped / order accepted / money exchanged.
BitcoinTalk
#9
From:
satoshi
Subject:
Re: JSON-RPC method idea: list transactions newer than a given txid
Date:
Then how do you cope with the issues I listed in the message you quoted?
BitcoinTalk
#10
From:
jgarzik
Subject:
Re: JSON-RPC method idea: list transactions newer than a given txid
Date:
I think the important point is that these things are unavoidable unsafe after a certain point, even for listreceivedbyaddress.  So I think it's helpful to compare current website behavior under mainline bitcoin, with listtransactions.   Let us call this the confirmation point.  Addressing your queries...

1) How do you know if a past transaction becomes invalid and disappears?

bitcoinmarket and mtgox and other sites seem to consider 6 confirmations their "confirmation point", the moment at which a transaction may be considered "safe." If a past transaction becomes invalid and disappears, the website cannot avoid potential loss, because the user has already received their PayPal-USD or LR-USD or Pecunix GAU and disappeared.

Same for a web store or brick-and-mortar store.  There is a confirmation point at which the customer receives goods.  If a TX becomes invalid after that, the store takes an unavoidable loss, because the customer is already gone with the purchased goods.

2) When there's a block-chain reorg, it would be easy to double-count transactions when they get confirmed again.

Let us assume that confirmation count for txid 0x1234 is bouncing wildly between 0 to 10 to 0 and back and forth, due to a large number of block reorgs.

Whether it is listreceivedbyaddress or listtransactions, you still have a binary confirmation point, a moment in time, at which the transaction crosses the "approved by store" level of confidence.  At that confirmation point, the customer leaves with purchased goods, and store takes a loss regardless of further block chain or TX behavior.

I do agree a programmer may make a mistake, and assume that number of confirmations will always increase.  But that human mistake, too, will cause danger when used with listreceivedbyaddress.

3) A transaction can be replaced by a double-spend with a different txid.  You would count both spends.

On this point, I agree that listtransactions presents some additional danger here, due to changing txid, if and only if the new double-spend matches destination bitcoin address being sought by the JSON-RPC user.

Nevertheless, in this case too, the user's security rests entirely on their level of confidence:  if the TX is replaced before 6 confirmations, software will likely not notice anything.  If the TX is replaced after 6 confirmations, the customer has already left with purchased goods, and the bitcoin user takes a loss.


This is simply inherent to bitcoin itself.  A block chain reorg might happen after 50 blocks.  But websites do not want to make their users wait 50 blocks before receiving goods.  It is the well-known snack machine problem.  listtransactions does not add anything to this problem, beyond that which is already vulnerable through listreceivedbyaddress.

Transactions can and will be replaced after the binary "confirmation point."  All users of bitcoin must figure this into their business plans, just like they account for credit card chargeback risk or shoplifting risk.
BitcoinTalk
#11
From:
davux
Subject:
Re: JSON-RPC method idea: list transactions newer than a given txid
Date:
Any time you take an action based on payment amounts received, you always need to go back to bitcoin and ask for a current balance total (or use move or sendfrom), and be ready for the possibility that it can go down.

So it would be a design problem in the developper's app, not a problem inherent to listtransactions. What you're saying is: if listtransactions is used without care, it can be dangerous. OK, but many programmatic tools have traps too. For example, programming with threads is full of design traps. That doesn't mean that that tool shouldn't be made available to developpers, but that the documentation must warn strongly about those traps.

My point: let's make listtransactions available, but document it precisely, like any API, and explain the design mistakes that shouldn't be made when using it.
BitcoinTalk
#12
From:
satoshi
Subject:
Re: JSON-RPC method idea: list transactions newer than a given txid
Date:
I'm not talking about the normal risk for a given minconf level, I'm talking about additional pitfalls from listtransactions when used this way.

2) When there's a block-chain reorg, it would be easy to double-count transactions when they get confirmed again.
The OP's example of listtransactions <account> [count=10] [txid] seems to imply and it would be very easy for programmers to assume that if they pass in the last txid of the previous call to listtransactions, they will never see the same transaction more than once, which is not the case.  It would be very easy to double-count payments if you don't maintain your own persistent map or dictionary to track which txid's you've already accepted.

It doesn't seem right to have a function that seems tailor made to be used a certain obvious way, and that way is a non-obvious trap.

3) A transaction can be replaced by a double-spend with a different txid.  You would count both spends.
listtransactions does not add anything to this problem, beyond that which is already vulnerable through listreceivedbyaddress.
Suppose both spends are to the same address.  getreceivedbyaddress would always count only one or the other spend at any given time, never both.

Using listtransactions, it would be very easy to count both.  You see the first spend, you count it.  You see the second spend, you count it.  Total is double counted.
BitcoinTalk
#13
From:
Gavin Andresen
Subject:
Re: JSON-RPC method idea: list transactions newer than a given txid
Date:
I'll and add another reason not to have a "list transactions that happened after <txid>" :

move "transactions" don't have a transaction id, but they do affect account balances (and are listed in listtransactions).

Your code is going to get really messy if you expect to call listtransactions and then squirrel away the txid of the last item returned.  If it was "category":"move",  there WILL be no txid...

RE: eliminating polling:  at some point fairly soon, I plan on cleaning up my "monitorreceived" patch, to POST to a URL when transactions come in or blocks are accepted... but I need to do some Deep Thinking to redesign based on lessons learned from 'accounts'.  It might turn into a very minimal API, where the notification is "Hey, txid <123ae4221...> just got to N confirmations, you might want to call gettransaction and getbalance to get up-to-date."
BitcoinTalk
#14
From:
jgarzik
Subject:
Re: JSON-RPC method idea: list transactions newer than a given txid
Date:
I'll and add another reason not to have a "list transactions that happened after <txid>" :

I agree with you and satoshi about "txs after <txid>".  My listtransactions (now xlisttransactions) patch pointedly does not have that feature, and never has.
BitcoinTalk
#15
From:
ribuck
Subject:
Re: JSON-RPC method idea: list transactions newer than a given txid
Date:
If a past transaction becomes invalid and disappears, the website cannot avoid potential loss, because the user has already received their PayPal-USD or LR-USD or Pecunix GAU and disappeared.

That's not always the case. Sometimes the customer will have a balance at the website. The operator of the website will certainly want to know if a past transactions becomes invalid and disappears.
BitcoinTalk
#16
From:
jgarzik
Subject:
Re: JSON-RPC method idea: list transactions newer than a given txid
Date:
If a past transaction becomes invalid and disappears, the website cannot avoid potential loss, because the user has already received their PayPal-USD or LR-USD or Pecunix GAU and disappeared.

That's not always the case. Sometimes the customer will have a balance at the website. The operator of the website will certainly want to know if a past transactions becomes invalid and disappears.

Sure, and that's easy enough to track with transactions.
BitcoinTalk
#17
From:
satoshi
Subject:
Re: JSON-RPC method idea: list transactions newer than a given txid
Date:
I agree with you and satoshi about "txs after <txid>".  My listtransactions (now xlisttransactions) patch pointedly does not have that feature, and never has.
As long as the interface is designed for things like showing the user the last N transactions history, it's fine, now that we have the Accounts feature making it easier to do payment detection the right way.

Gavin, could listtransactions have an option to list transactions for all accounts?

I'm not sure what the interface could be, maybe:
listtransactions <JSON null type> [count]

It would be hard to do that from the command line though.

I can't think of a good solution for the interface, that's the problem.  Maybe "*" special case like "" is.  Everyone would have to make sure no user can create account name "*".

Sure, and that's easy enough to track with transactions.
I don't get how that's "easy" to track with transactions.