Ponytail for Claude Code, tested: less code, weaker tests, one security hole

We tested Ponytail for Claude Code on real backend work before letting it near a client project. It did what it promises: about a quarter fewer output tokens and a quarter lower cost than plain Claude Code, and every run passed its tests.

It also wrote weaker tests, and on a security fix it missed a hole that plain Claude Code closed. Adding one rule, write the tests first and watch them fail, closed the test gap. It did not close the security gap.

So we now use the tests-first version for everyday build work, keep anything that decides who can see what on plain Claude Code, and keep code review plain. Below are the numbers, the prompt we use, and a recipe for running the same test on your own codebase in an afternoon.

Key takeaways

• Ponytail saves real money. Against plain Claude Code on Opus 5.5 it cut output tokens by 26%, cost by 25% and time by 19%, with every run passing.
• Its tests were the weak spot. On a privacy feature, its tests let 2 of 12 planted bugs through, including a sensitive ID number written into the audit log.
• Tests-first fixes the tests, not the savings. Ponytail plus a tests-first rule caught 12 of 12 planted bugs in all three runs, but saved only 12% of output tokens and ran 8% slower than plain.
• Lean prompts miss what the brief does not name. On a security fix, 4 of 6 candidate solutions leaked whether an account was blocked. Plain Claude Code and the version we shipped did not.
• Asking for “less code” in one line does nothing. A one-sentence brevity instruction saved nothing overall and, on one task, wrote 61% more code than plain.

Ponytail for Claude Code promises 53% less code

Ponytail is an open-source rule file for coding agents, released under the MIT licence, that tells the agent to act like “a lazy senior developer”. Before writing anything, the agent works down a ladder: does this need to exist, is it already in the codebase, does the standard library do it, does an installed dependency do it, can it be one line. Only then does it write the minimum code that works.

The author’s benchmark is striking. The README reports “-53% code · -41% time · -26% cost · -45% tokens” across 39 tasks on Opus 5.5, five runs each, and says “98% of risky logic ships with a test. Without Ponytail: 68%.”

Those are the author’s numbers, on the author’s tasks. We wanted our own, on code we know well, before changing how our engineers work. We did not install the plugin. We pinned one version of its rule file and added it to Claude Code’s system prompt, which is what the plugin’s start-up hook does.

Benchmark method: replaying changes we had already shipped

The fairest test is work you have already finished, because you know what a good answer looks like. We took three backend changes from a production multi-tenant SaaS our team builds, on Postgres with row-level security:

1. A bug fix. A scheduled cleanup job sometimes cleared more rows than its batch limit allowed.
2. A security fix. When the database refused a write, the error message told an outsider whether another account was blocked.
3. A privacy feature. Users had to be able to withdraw their own sensitive document details, even when their account area was locked for billing.

Every run started from a copy of the code as it was before the change, with no git history, so the agent could not find the answer we shipped. Each got the same written brief, the same project rules and the same list of allowed commands, and ran unattended on Claude Opus 5.5.

We graded every run the same way:

• Hidden tests: the tests that shipped with the real change, which the agent never saw.
• The full test suite, a database lint and safety checks on the database catalogue (permissions, function settings).
• Planted bugs: 8 to 12 per task, applied to the shipped fix. Each run’s own tests had to catch them.
• Two blind judges: Claude Opus 5.5 and Claude Fable 5.1 each scored every solution on six dimensions and ranked them all, without knowing which setup wrote which.

We compared four setups:

That came to 23 agent runs across two rounds, plus the 3 shipped fixes graded the same way as a reference. All 23 runs passed the hidden tests, the full suite and the safety checks. The differences are in what that green tick does not show.

Ponytail results: a quarter cheaper, same pass rate

Averaged across the three tasks, with each task weighted equally, Ponytail against plain Claude Code looked like this:

The savings held on every task, not just on average. On the cleanup-job fix, where we ran each setup three times, the ranges did not overlap: Ponytail used 12,000 to 16,000 output tokens per run, plain used 18,700 to 24,600.

In the first round, our first judge also scored Ponytail’s code the simplest and most readable of the three setups. If the story stopped at the green tick, this would be an easy yes.

Weaker tests were the catch

The privacy feature is where it went wrong. We planted 12 bugs in the shipped version and ran each setup’s own tests against them. Ponytail’s tests let 2 through:

• A sensitive ID number written into the audit log. The feature exists to remove that data, and the bug copied it into the one place it must never go.
• A blocked account allowed to act. The function was meant to refuse blocked accounts, and the bug quietly stopped it.

Plain Claude Code’s tests caught 12 of 12 in two runs and 11 of 12 in the third. On this task Ponytail wrote 29% fewer test lines than plain, and the missing lines were the ones that mattered.

In this benchmark the hidden tests caught everything anyway. In real work there are no hidden tests. The agent’s own tests and a code review are the whole safety net, so a weaker test suite ships its gaps.

A one-line “write less” instruction does nothing

Critics of Ponytail argue that its savings come from simply asking for less code. Our control setup tested that with one sentence.

It saved nothing consistently. On the cleanup job it used 5% more output tokens than plain. On the privacy feature it used 16% fewer. On the security fix it used 16% more and wrote 254 lines of production code against plain’s 158, rewriting two existing security functions from an old copy, which both judges flagged as a regression risk.

The structure in Ponytail is doing the work, not the request for brevity. A ladder of questions changes how the agent plans. A single “be brief” line does not.

If an agent already writes code for your product, the useful question is what checks its work before it reaches your users. In a free website audit we look at your site with you and point out where the risks sit.

Tests-first closed the gap, at a price

Ponytail’s own rule asks for “one small test” for non-trivial logic. We replaced that with a tests-first section: read the code, write a plan, write a test for every requirement, run the tests and watch each one fail for the expected reason, and only then write the smallest code that makes them pass. Claude Code’s own best practices point the same way: “write a failing test that reproduces it, then fix it”, and “Give Claude something it can run that comes back pass or fail.”

On the privacy feature, Ponytail plus tests-first caught 12 of 12 planted bugs in all three runs. One run went further on the cleanup job without being asked: it added a check that the test really reproduced the bug, and reverted half its own fix to prove the test caught that too.

The savings shrank, as you would expect when the agent writes and runs tests before any code:

Task by task, tests-first used 8 to 18% fewer output tokens than plain and wrote 15 to 37% less production code. It is the leanest code of any setup, with tests close to plain’s. It is not faster.

Full numbers by task

Averages per run. Runs per setup varied by task, so the single-run rows show a direction, not a spread.

On the cleanup job, the existing test suite already catches 7 of the 8 planted bugs. The one it misses is the real bug, and every setup’s tests caught that one, so there is no meaningful test gap on that task.

The security hole every lean prompt missed

The security fix’s brief listed five ways an outsider could learn something about another account from an error message. Every setup fixed all five, and every hidden test passed.

Then we probed one path the brief did not name: a signed-in member writing to a table through their own account area’s ID, while naming somebody else’s account in the same row. If the answer differs depending on whether that other account is blocked, the database is leaking account status.

Four of six candidate fixes leaked. Ponytail, control and both tests-first runs answered “read-only” for a blocked account and “not allowed” for an active one. Plain Claude Code and the fix we shipped gave the same answer either way.

The pattern is worth knowing even if you never use an AI agent. A common way to fix this kind of leak is to let your own members through early and refuse everyone else:

-- Members pass straight through; everyone else gets one fixed refusal.
if new.tenant_id in (select my_tenant_ids()) then
  return new;   -- the next check now answers about whoever the member named
end if;
raise exception 'not allowed' using errcode = '42501';

That early exit is the leak. Once a member is through, the next check answers a question about a third party, and the answer differs. Where no user should ever write a table directly, the safer fix refuses every direct write, member or not.

Why did the lean setups miss it? They did exactly what the brief asked, which is what Ponytail tells them to do: “Skip features, options and flexibility nobody asked for.” Plain Claude Code went looking for adjacent holes the brief did not mention.

AI judges disagree more than you would expect

The more uncomfortable finding was about review. When each judge reviewed a tests-first solution on its own, neither flagged the leak. When the same judges ranked all the solutions side by side, both caught it. Comparison surfaced what a single-diff review did not.

The judges also disagreed with each other often. The fix we actually shipped, after our own review, ranked 3rd of 11 with one judge and 9th of 11 with the other on the cleanup job. An AI review score is an opinion. Planted bugs and probes are evidence. We now weigh them that way.

Lean AI coding prompts: how we use them now

1. Tests-first for everyday build work. Ponytail plus the tests-first section is our default build mode. It writes the leanest code we measured, and its tests held up.
2. Plain sessions for access control. Row-level security policies, authentication, permission checks and anything that answers “who may see this” run without lean rules. That is where doing exactly the brief costs the most.
3. Never in shared instruction files. We run the lean rules as a command an engineer types, not as text in a project-wide instruction file. Claude Code’s subagent documentation says subagents “receive only this system prompt plus basic environment details”, but load the project’s CLAUDE.md files “unless you set omitClaudeMd”. Put lean rules in CLAUDE.md and your review agents inherit them too.
4. Review stays plain. Our code-review agents run without lean rules. A reviewer told to skip what nobody asked for is the wrong reviewer.
5. Probe the edges on security fixes. For any fix to a refusal or a permission check, we test the caller’s own ID path, the update path and third-party fields, not only the cases in the ticket.
6. Budget in tokens, not dollars. Claude Code’s headless documentation says its cost figures are “client-side estimates and can differ from your actual bill”. Token counts and time are the numbers to trust.

The tests-first prompt we use

This is the section we add after Ponytail’s rules. It replaces Ponytail’s “one small test” line:

## Tests first (this overrides "one small test")
Before you change production code:
1. Read the code the change touches. Say in 3-5 lines how it works now,
   then give the plan: files, order, and what could break.
2. Write the tests first. Write at least one assertion for each requirement
   in the task, including every refusal and every "must not change". Run
   them, and confirm each one fails on today's code for the reason you
   expect. A test that passes before the fix proves nothing, so fix the test.
3. Then write the smallest code that makes them pass, and run the related
   suite again.
If something is unclear, ask instead of guessing. If you cannot ask, take
the safer reading and list it as an assumption.
Lean code, complete tests: never cut a test to make the diff shorter.

Ponytail itself is released under the MIT licence, so you can pin a version, read it and combine it the same way.

Run your own AI coding benchmark in an afternoon

You do not need our codebase to repeat this. You need a few changes you have already shipped.

1. Replay finished work. Pick three changes you shipped and reviewed. You already know what good looks like, and you already have their tests.
2. Start from a copy with no history. Export the code as it was before each change. With the git history present, the agent can find the answer in the log.
3. Strip your own rule files. Remove any lesson you wrote down because of that change. Ours gave away the answer to two of the three tasks before we caught it.
4. Keep the shipped tests hidden. Use them only to grade. Score any assertion that names the shipped design separately, so a different correct answer is not marked wrong.
5. Plant bugs in the shipped fix. Write 8 to 12 small, realistic mistakes and check each one breaks the hidden tests. Then run every setup’s own tests against them.
6. Use two blind judges from different models. Score each solution alone, then rank them side by side. Disagreement between judges is a finding in itself.
7. Probe what the brief did not say. For security work, write one test for the path nobody listed. That is where our setups differed most.

Budget for the judging. Our runs cost about $94 at list price, the two judges about $110, and a pilot $6: about $210 in all. Reviewing every solution twice, with two models, cost more than writing them.

Should you install Ponytail?

For build work, yes, with tests-first added and the rules kept out of your shared instruction files. On our numbers it writes the leanest code with tests that hold up, at a modest saving.

Ponytail alone is cheaper still, and fine for small changes your existing tests already cover. We would not use either for security or permission work, and we would not let lean rules anywhere near the agent that reviews the code.

What this benchmark cannot tell you

• The sample is small. 23 runs over three tasks, with one to three runs per setup per task. It shows direction, not statistical certainty.
• One stack. All three tasks were database work on Postgres. We did not test front-end or TypeScript tasks.
• Unattended runs only. “Ask instead of guessing” and plan-first workflows behave differently when a person is there to answer.
• Same-family judges. Both judges are Claude models, so they may share blind spots with the agents they judged.
• Our rules are already strict. The codebase’s own instructions push hard for tests and for changing existing code rather than adding beside it. Ponytail may save more on a codebase with looser rules.

How Lunova builds software

We use AI coding agents every day, and we measure them before we trust them. Every engineer on client work is part of our in-house team, and the people you meet are the people who build your product. If you want to see how our web design team approaches builds, or how we think about agents touching a live site, start with our seven questions to ask before an AI agent touches your website and the work we have shipped.

If you are planning a web app or SaaS product and want it built by people who check the machine’s work, tell us about it.