Most of my development now is writing specs, not code. So I started paying attention to how the spec actually reaches the agent.

The default is to pile everything into markdown files. I wanted to see if there was a better shape. I found a project called acai.sh and implemented it on my codebase.

acai.sh formalizes how requirements get to the agent. Each requirement gets an ID. The ID links to tests, so you can answer a real question: is this requirement actually tested? And it uses YAML instead of Markdown, which gives it more structure. Intuitively, I thought that would be of benefit.

Then I couldn’t tell if it was.

It clearly added some load. The agent had to spend time maintaining requirements. I added a CI test to keep the requirements honest, which is another thing to maintain. I liked the clarity and the organization. But I started to wonder if the clarity was mostly for me — the human reading the files — and not for the agent at all. And if the extra context was quietly hurting the agent, the whole trade-off changes.

The papers don’t decide it for you

There’s research on this now, and the headline looks bad for context files.

The most careful study, from ETH Zurich and LogicStar, found that repository context files did not improve task success while raising cost by over 20%. Worth being precise: the success drop was small and not statistically significant. The reliable harm was cost. Extra context made the agent do more — more exploration, more steps — without a matching payoff.

But read a little wider and it stops being one-sided. Another paper measured AGENTS.md files making agents faster and cheaper at comparable completion — the opposite direction. A third ablated the individual signals you can hand an agent and found a precise reproduction test is worth a huge jump in success, while a repo overview is worth roughly nothing. Not all context is equal, and the value of any piece of it shifts with the setup.

The mechanism behind the cost is well documented. Long context rots — models get less reliable as you stuff more in, even on easy tasks. True-but-irrelevant detail distracts them. So a fat context file is a plausible tax, not a free gift.

Every one of these papers, read straight, says the same thing: don’t trust someone else’s aggregate. The result depends on the model, the harness, and the repo. Measure your own setup. And nobody ships a cheap way to do that.

The cheap version of an experiment

Designing a real experiment means writing tasks and answer keys, and, not long ago, that would have been too expensive.

Having agents do the work changes that. The implementation labor is close to free now.

My original plan was a clean experiment: take a shipped feature, delete it, then have the agent rebuild it under each context configuration. I still want to run that one. But I didn’t wait for it. I used features I was already building. For each one, I ran the same task under different context configurations, then took the best result forward and actually shipped it.

It felt like a variation on agent swarms. Instead of firing the same task at N agents and keeping the best, each arm got a different context configuration, and the difference between them was the thing I was trying to measure. Multishot with a control group. The work was going to happen anyway — I just did it three ways and kept the comparison.

The answer key was already there too. My own tests are the oracle. A feature that passes the tests it shipped with is a feature rebuilt correctly. No rubric to write from scratch.

I always ran three arms:

  • the skills and agents as they are
  • everything deleted — strip the skills entirely
  • the specific variation I wanted to test, like removing the acai layer but keeping the rest

If you’re not sure your skills are any good, delete them and see what happens.

How To Measure

A few rules keep it honest:

  • Same task for every arm. The difference has to come from the configuration, not from one arm drawing an easier job.
  • Clean baseline, no leaks. When you delete a layer, scrub every trace. Half-deleted references tell the agent what used to be there. I had to build a real procedure to strip all the acai traces.
  • Wipe the agent’s memory between arms. Otherwise you’re measuring what it remembers, not what the context did.
  • Write the prediction down first. In git, before the first run. It’s the only thing that stops you nodding along to whatever result you get.
  • Judge blind. Strip the labels, shuffle the order, then score.

Every arm goes through the same scoring. I split it into separate questions, because “it works” hides too much:

  • Does it build and pass? Build, typecheck, lint, and the tests that were already there. The cheap objective gate — plus how many fix cycles it took to get green.
  • Does it do the job? The oracle. For a rebuilt feature, the tests it shipped with, re-run against the new code. For new work, a frozen acceptance checklist scored pass / partial / fail. Same checklist for every arm.
  • Is it actually tested, or just green? For each item on the checklist, is there a real test behind it, or did it pass on a read-through? An arm that writes thin tests scores lower here even when everything is green.
  • Does it look like the rest of the codebase? Scored against the real peer modules — folder layout, the shared layers, the patterns already in the repo. This is the one that caught the delete-everything arm: working code, wrong shape.
  • What did it cost? Tokens and dollars where I can capture them, proxies where I can’t — fix cycles, turns, tool calls, files touched, wall time.
  • A human pass. Someone exercises the feature for defects the tests miss, and throws out any “miss” that’s really my spec being wrong rather than the agent. This is the step that costs real time, and the one I keep skipping.

A blind judge does most of this — labels stripped, arms shuffled — and I adjudicate the spec-error calls, since the spec is mine to own.

What it told me, which is: it depends

Two real runs. They disagreed.

On one feature, the stripped arms produced the strongest result — better tests, cleaner evidence — and the full context layer scored lowest. On another, the full layer was the safest base to ship from, though it took more back-and-forth to get there.

That’s n=1 each. Wide error bars. I’m not going to pretend it’s a verdict on acai, because it isn’t. But the disagreement is the actual finding.

I don’t think there’s a universal truth for the best skills configuration. Perhaps if your codebase is already written the way you want, the agent copies the patterns it sees and the explicit skill adds little. In a messier codebase, the skill earns its keep by enforcing what the code doesn’t show. The right answer is tied to your repo.

And it’s tied to your model. When something like Fable lands, the scaffolding you wrote to prop up an older model’s blind spots can turn into dead weight — or start mattering more. A result from this experiment has a shelf life. You don’t measure once; you recalibrate when the model moves.

So the real decision — which configuration to keep — is still in limbo for me. I need more data. The delete-a-feature-and-rebuild-it run is next.

Where it’s weak

I won’t oversell it.

You need decent tests. They’re the primary oracle. A thin suite means a passing rebuild proves almost nothing, and the whole thing gets weak.

Measuring tokens is its own mess. I wanted cost per arm and mostly didn’t get it cleanly. Claude Code hands you cost and a full token breakdown in JSON — one line. Cursor, which I used for the careful run, gives you no per-run number at all: not inline, not in the dashboard, not in the API. So I leaned on proxies — fix cycles, wall time, files touched. Confounded, but cheap, and honest about being proxies.

I also skipped the manual testing pass the procedure asks for, because I was busy shipping. Which is the real catch: agents make running the arms nearly free, but the evaluation still costs human time. To get real value out of the experiment, you have to dedicate the time.

And to be clear about what this measures: the context tooling, not the model. Every result is bound to one model, one harness, one repo.

The skill

I packaged the method as a skill called context-experiment. Install it with npx skills add wandir-tech/context-experiment --skill context-experiment.

It’s an installer, really. It bootstraps the process for trying different context configurations against a task. You bring the configuration variations and the task. The easiest alternate configuration to start with is deleting everything. Yup. Delete those skills you’ve been grooming and see what happens.

This is the kind of thing that wouldn’t have made sense a year ago. The implications of the cost of code trending toward zero keep surprising me. Experiments are now a Tuesday afternoon. The agents don’t just write the code — they can also measure the way you’re asking.