I paid to check whether agent APIs return what they promise. Most do. The ones that fall short look identical to the ones that deliver, right up until the money leaves.
Every x402 seller ships a schema: send this input, get these fields back. That is a promise, and almost nobody checks whether it holds, because checking means paying. So I paid. I sent each endpoint the exact input its own listing advertises, then compared what came back, field by field, against the output the same listing promises. I ran 549 endpoints through it. 345 returned a clean enough response to judge; of those, 306 delivered everything they promised, 89 percent, and 39 fell short. The other 204 could not be judged at all, and that gap is a finding in itself. Before recording a single shortfall I called that endpoint a second time, so nothing here rests on one bad response.
- The 39 that fell short are not broken. They answer fast and quote the exact price they advertise, so a free probe rates them safe to use. stabletravel.dev sells airport weather and returned none of its 7 promised fields. api.myceliasignal.com, a price oracle, returned none of its 4. x402.aispace.bot took payment for text-to-speech and sent back none of the 3 fields it lists. Others send some and drop the rest: agent.kihustle.tech promises 16 fields and returned 2. Every one of these was confirmed on two separate paid calls before it was recorded, and the gap between looking reliable and actually delivering is exactly one paid call wide.
- The quieter finding is that a quarter of the market cannot be exercised by a plain agent at all. 204 endpoints needed an API key, rejected the very input they advertise, or answered with a 4xx or 5xx. Those are counted as inconclusive, never as a failure to deliver, because not getting a clean response is my limitation, not proof the seller shortchanged anyone. But it means the honest question, does it deliver, is only answerable for the endpoints that actually let you try, and today that is fewer than two in three.
- An earlier version of this counted one in four as not delivering. That was wrong, and it was my bug, not theirs. The test threw away the paid response before reading it, so endpoints that returned everything, vibesprings.net and api.seneschal.space among them, were scored as returning nothing. I caught it before it went up, pulled the data, and rebuilt the test: a response it cannot read is now inconclusive rather than failed, every shortfall is re-checked on a second call, a known-good endpoint runs first and aborts the whole run if the test scores it wrong, and the raw response is saved so the next mistake is a free re-check, not a re-run. The re-verification cost 86 cents, reconciled to the wallet on-chain.
Receipts and detailclick to expand
method, per endpoint:
read its published input example and output schema
pay it once with its own advertised input, payment capped at its advertised price
compare the returned fields against the fields it promised
if it falls short, call it again before recording anything
keep the verdict and the response shape (types only), throw away the values
549 checked: 306 delivered (89% of measurable), 39 short (11%), 204 inconclusive
returned NONE of what they promised (confirmed on two calls):
stabletravel.dev /api/flightaware/airports/KMIA/weather 7 promised, 0 back
api.myceliasignal.com /oracle/price/btc/usd 4 promised, 0 back
vape-x402.vapex402... /scan/exploit_check 4 promised, 0 back
x402.aispace.bot /api/v1/audio/speech 3 promised, 0 back
voice.forgemesh.io /v1/tts/base 2 promised, 0 back
sent some, dropped the rest:
agent.kihustle.tech missing 14 of 16
agents.ai-rook.com missing 3 of 5
api.locus.report missing 3 of 4
could not measure (need a key, rejected our input, or 4xx/5xx): 204
these are not counted against the seller- Endpoints measured
- 345
- Delivered everything
- 306 (89%)
- Fell short
- 39 (11%)
- Could not be measured
- 204
- Returned more than promised
- 178
- Every short confirmed on
- 2 paid calls
- Re-verification cost, on-chain
- $0.86
- Shows up on a free check
- none of it