Pantheon / Hive — dogfood experiment
Four tiers of AI planning, measured on one real task
I run an open-source agent framework called Hive that plans and executes work through a structured, multi-document process: research, design discussion, an adversarial “grill” pass, horizontal/vertical slicing, and a long structured outline. That costs real tokens and real time, so I tested whether it earns its cost — on a real planning task from this site’s own backlog, run four different ways, same day.
The four tiers
Tier 0 — no process
Told the requirement the way a user would actually type it. No framework, no template, no second look.
Tier 1 — base pass
One structured planning document. Real research, written once. No adversarial review.
Tier 2 — +grill
Same as Tier 1, then one real adversarial self-review pass, revised against the findings.
Tier 3 — full ceremony
Everything: research, design discussion, grill, horizontal plan, vertical plan, a full structured outline with a risk registry.
Real cost, three of the four tiers
Each ran as a fresh, isolated agent with no shared context — real harness numbers, not estimates. (Tier 2 is left out here on purpose: it ran inside a session that had already done hours of related research, so its isolated cost isn’t knowable. Its document volume — 874 lines — sat between Tier 1 and Tier 3.)
What each tier actually found
Tier 0 wasn’t reckless — it read 11 real files and flagged its own unverified claims. But it missed 2 of 4 real pricing conflicts, and recommended building a second, separate calculator rather than extending the one that already had the right logic — a meaningfully more expensive path than every other tier chose. It also found something none of the other three did: a 3-tier booking setup that was designed but never actually wired into the live page.
Tier 1 — the cheapest structured pass, with zero adversarial review — made the single biggest correction of the whole experiment: a “never-shipped” $4,500 product rung turned out to be fully built and live, just not linked from any pricing page. Both Tier 2 and Tier 3 got this wrong, because neither happened to open that one file.
Tier 2’s grill pass caught 6 real internal-consistency problems in its own draft — a risk with no owner, an overclaimed certainty, two self-contradicting sections.
Tier 3’s grill pass caught 7, including its own first draft flatly asserting “no test framework exists” in a repo with real, documented test suites — a false claim that could have caused a redundant test to get written later.
Tier 3’s 657-line structured outline — the single largest document across all four tiers — found exactly 2 genuinely new things in its entire length. Both real. Both would’ve fit in one line of a shorter document’s open questions.
So which combination actually wins?
I went back and ran grill directly against Tier 1’s own draft, to check: does adversarial review on top of good research lose anything, or just add? It added — 6 more real findings (a cron-poll pattern that’d leave a paying customer waiting up to a day for their report to start; a paid upsell wired into a booking flow the same document already calls unverified), the same yield grill produced everywhere else. It did not touch, dilute, or need to re-find the $4,500 catch, because that’s a research problem, not a self-consistency problem. The two mechanisms are additive.
That makes structured research + grill, skipping the horizontal/vertical/ structured-outline layer, the best combination actually measured here: a fraction of full ceremony’s tokens, with everything full ceremony found and one thing it missed.
1. Running several cheap, independent passes beats running one expensive pass harder — Tier 1 cost a fifth of Tier 3 and still found more.
2. Grill is additive, not a substitute for research — it caught real errors every time (6, 6, 7 across three separate drafts) without ever diluting a strong finding underneath it.
3. Structure beat no process on the things that mattered most here, but not uniformly — the zero-process pass found a real wiring gap nothing else did. The strongest takeaway isn’t “process always wins,” it’s that independent passes keep finding different real things, which is itself the case for running more than one.