As models get stronger,
what should the harness control
— and what should it let go?
Twenty minutes · five beats
We build an open-source coding agent — and we build it with it. These are lessons, not a product tour.
Which parts of
your harness just
became dead weight?
A harness decides four things.
Each has a part to let go — and a part to keep.
Judgment — anything the model can now decide better than your if-statements: which tool, what plan, how to split the work, what to summarize.
Invariants — anything the model can't see, can't undo, or can't be trusted to remember: the cache prefix, the clock, process lifetimes, state that must survive.
Ask four
questions per feature
Every harness feature is a bet against the model. When the model changes, re-check the bet.
if hidden(x) or irreversible(x):
harness.enforce(x)
else:
expose(x); model.decide(x)
# then: eval before vs. after
If you can't name the failure, it's a candidate for deletion. If you can, you know what it's for.
Cache lifetimes, wall-clock time, file mtimes, other sessions, a flaky network — invisible to the model, so the harness owns them.
Irreversible means enforce it in code. A rule in the prompt is a hint, not a gate.
Measure first, change one thing at a time, keep what survives testing. Look at requests and tokens, not only the score.
Inject them every turn, or find them at call time?
Connectors are third-party MCP servers in the cloud, signed in as the user: connect once, use from any device — so the set of connected apps can change at any time.
- Wrap every app tool as
connector__<app>__<tool>, fetch the connected list before every turn - Short timeout, to protect time-to-first-token
- Network blip → empty app list: the model can't call anything
- Fetch fails, or the user connects / disconnects between turns → tools[] changed, cached prefix gone
- One small CLI the model calls; the system prompt and a Skill teach it how
- A tool call can afford a longer timeout
- Apps never enter the tools array
- Connect / disconnect no longer touches the cached prefix
Cut → defer → program
Measure at every step
Merge tools around user intent, make descriptions distinguishable, group by domain. Past 30–50 tool definitions, models get worse at picking the right one.
You can't delete the MCP servers users install themselves.
Core tools stay resident. Rare built-ins and large MCP sets move behind search automatically past a size threshold.
If it can't be found, it doesn't exist. Catalogs fetched over the network every turn slowed first token — we moved them back to resident.
A run_code tool: JavaScript calls tools.* through the same permissions and hooks. Failures return results, not exceptions.
run_code next to every native tool: barely used, and tokens went up 5%. It worked only once most tools became code-only.
Should the agent just wait for npm run dev?
Foreground for 6 minutes and the prior tokens likely fall out of cache. Bigger bill — and the model guesses durations wrong.
Every command starts as a background task; the foreground just waits alongside it
Far below a 5-minute cache lifetime, with room for the model's own generation
No kill-and-rerun; the model is told not to start it again
Foreground time counts; backgrounding or reading output never extends the deadline
More subagents
isn't better.
Subagents are for isolating context, not for carving up territory — one agent per team or per problem type is Conway's law, agent edition.
Splitting by problem type: lossy handoffs, a game of telephone. Moving subagent dispatch inside PTC programs: it got used noticeably less.
The window is a workbench, not a warehouse
Harness engineering is an experimental science
Requests, tokens, cache behavior — PTC looked flat on score and was a clear win on requests
Change several at once and you can't tell which one did it
All tests pass? Check who changed the tests. Did it re-read files, repeat work, drop a constraint?
Run each task more than once; look at which tasks flipped, not only the average
Rules written for older models
Hard-coded workflows and modes
Which tool, which order, which program
Orchestration across many subagents
The cache prefix: keep it byte-stable
Permissions: one path, even inside programs
Time: timeouts, cleanup, notifications
Truth outside the context window
Every harness feature is a bet against the model.
Re-check the bet every time the model changes — then ask what stays scarce.
When experiments are free,
what stays scarce?
Let go of judgment; keep the invariants. What's left for people: ask the right question, push the idea all the way through, and verify with controls — stacked; pull one layer and the rest idles.
Agents answer questions; they don't care which ones matter. The person still asking why / what if is steering.
Friction drops; quitting halfway gets easier. Write a checkable Done; demand evidence, not a report; finish the last 20%.
Cheap experiments amplify bad method. Hypotheses you can falsify; change one variable; design the control.
You own the question
and the control.
After the harness lets go of judgment, cheap experiments amplify method — not volume. You're designing the loop that runs the experiments.
"Maybe useful" doesn't count.
A clean failure is fine; a half-finished "success" is not.
A hundred uncontrolled runs = a hundred anecdotes.