A statistic with no population behind it is not a small truth — it is a confident-sounding lie. This post explains the rule Scovant applies to every number it publishes: below a minimum sample, say nothing.
The failure mode
Automated reporting makes it easy to divide two counters and print the result. When the denominator is zero or tiny, the arithmetic still "works": zero runs out of zero becomes "0% success", one lucky run becomes "100% success". Both numbers are formatted like evidence. Neither is.
The two claims fail differently. "0% success over 0 runs" asserts an outcome for events that never happened. "100% over 1 run" is technically true and practically meaningless — a coin that landed heads once. An honest report treats both the same way: not measured yet.
The rule Scovant applies
Every public Scovant statistic carries a minimum-sample floor, and the floor is enforced in code, not editorial policy:
- The homepage stat tiles render an em-dash instead of a number when the population is below the floor — absence beats fabrication.
- The Scovant Checkout Benchmark publishes only above a 50-run floor, and always with its population: run count, site count, and the exact measurement window, next to the rate.
- The Agent Web Index reports per-metric coverage, so "not measured" can never silently read as "no".
- Even our own content pipeline refuses to generate a data-driven article when the underlying window is below the floor.
How to read agent-readiness numbers anywhere
Wherever a number about AI-agent behavior appears — ours included — ask four questions:
- What is the denominator? A rate without a population is decoration.
- What is the window? "Success rate" without dates can be years stale.
- What was excluded? Ours excludes our own platform failures from both sides of the fraction; a benchmark that doesn't say is hiding something.
- Where is the methodology? A permanent methodology page (ours is here) is the difference between a measurement and a marketing claim.
The takeaway
The agentic web needs measurement, and measurement is only useful when it is allowed to say "not enough data yet". If a tool — Scovant or anyone else — ever shows you a success rate, look for the run count next to it. If there isn't one, treat the number as a headline, not a fact.