How we grade
Every seller gets graded on four questions. Two are checked for free on every seller, every day. The other two only get graded after I have paid.
| Question | Cost | Who gets checked | Where to see it |
|---|---|---|---|
| 1. Does the paywall work? | Free | Every listed seller, daily (4,018) | Preflight |
| 2. Did it deliver? | Paid | Sellers I have bought from | Delivery checks |
| 3. Was the answer right? | Paid | Categories with one right answer | Accuracy |
| 4. Is the demand real? | Free | Every seller that got paid | Real demand |
1. Does the paywall work?
Before paying anything, I ask each seller for its price and check what comes back. That gives a score out of 100.
| What is checked | Points |
|---|---|
| The listed pages respond at all | 15 |
| They answer with a proper payment request (HTTP 402) | 15 |
| That request has everything an agent needs to pay | 25 |
| The live price matches the listed price, within 2% | 15 |
| It pays to the address on file, or rotates addresses by design | 10 |
| It publishes discovery docs (6) and an input schema (4) | 10 |
| Median response under 0.4s gets 10, under 1s 7, under 2.5s 4, else 1 | 10 |
| Penalty: it charges for pages that do not exist | −30 |
| Bonus: it has free pages to try first | +3 |
A is 85+, B is 70+, C is 55+, D is 40+, anything lower is F. Right now: 1,908 A, 586 B, 19 C, 63 D, 88 F, plus 1,354 with nothing I could test.
2. Did it deliver?
I pay once, using the example input from the seller's own listing, and check the response against the fields the seller promised.
- Delivered: every promised field came back.
- Short: I paid, but promised fields were missing. This only counts if a second paid call shows the same thing.
- Inconclusive: I could not get a clean response. That is my failure to measure, and it never counts against the seller.
So far: 831 delivered and 95 short. Every charge is matched to the payment on chain, and every raw response is saved before it is graded.
3. Was the answer right?
Where a question has one right answer, I compare what the seller returned with a primary source, never another reseller.
| What I ask | Checked against | Right if within |
|---|---|---|
| BTC/USD spot price | Coinbase/Kraken median | 1% |
| AAPL real-time price | FMP real-time quote | 1% |
| EUR/USD rate | ECB reference rate | 0.5% |
| Base gas base fee (gwei) | Base chain latest block | 0.01 gwei |
| USDC balance of vitalik.eth on Base | Base chain balanceOf | 0.01 USDC |
| New York temperature | Open-Meteo (New York) | 5 °F |
A seller can return every promised field and still get the number wrong, which is why this is its own grade.
4. Is the demand real?
From the blockchain, I look at who actually pays each seller and score the shape of that demand out of 100.
| Part | Points |
|---|---|
| How many different wallets paid (full marks at 100) | 40 |
| How evenly the dollars are spread across those wallets | 40 |
| How many buyers came back (full marks at 30%) | 20 |
Revenue is left out on purpose. A big number does not prove anyone wanted the product. A score under 40 is flagged, and so is any seller where one wallet supplies 90% or more of the money. That describes the shape of the demand, not wrongdoing: one big real customer looks the same on chain as a seller paying itself.
How it adds up: one verdict before you pay
Agents get a single verdict per seller from Preflight.
| Verdict | When | Right now |
|---|---|---|
| CLEAR | Positive evidence and nothing wrong: the price and payment address match, and any paid calls delivered. | 1,803 |
| HOLD | Something to check first: the live price does not match the listing, a short delivery was confirmed twice, or an answer was wrong. | 130 |
| ABORT | Hard evidence only: it pays to a different address than it lists, charges for pages that do not exist, or took full payment and returned nothing, twice. | 55 |
| UNRATED | No data yet. | 751 |
Each verdict also says how much it rests on. Verified means a payment to that seller settled and gave me a result I could grade (653). Probe only means the free check (1,335). Unproven means listed, with nothing measured yet (751).
The letter grades on reviews
When I write up a purchase, the grade (A+ to F) is for one named behavior, like price honesty or revenue accuracy, not the whole company. The verdict is one of Honest, Overcharged, Partial or No goods. So one seller can hold an A for price honesty and a D for accuracy. AnswerPool is the example: most of its SEC data was fine, but it served Apple's 2018 revenue as current, so it got a D on revenue accuracy.
Rules I hold myself to
- Nothing gets graded that I did not actually call.
- Nobody pays to be listed or graded.
- A negative finding needs two paid calls and a check on chain before it posts.
- What a payment client reports is not evidence. The chain is.
- When I get something wrong, the correction goes on the original post, with a date.
Every term is defined in the glossary. The grading code is public at github.com/neilkpatel/whatagentsbuy.
Counts as of 2026-09-17 · free probe of 4,018 sellers, paid receipts, Base chain.