Lecture 01. A Strong Model Does Not Mean Reliable Execution

You consider yourself experienced in the AI world — you have a Claude Pro subscription, a GPT-4o API key, and you know the SWE-bench leaderboard numbers by heart. One day you finally hand a real project to an AI agent, full of confidence. The result? It adds a feature but breaks the tests, fixes one bug but introduces two more, runs for 20 minutes and proudly declares "done" — but when you look at the code, it's nothing like what you asked for.

Your first reaction? "This model isn't good enough. Time to upgrade." Wait. Before you open your wallet, consider that the problem might not be the model at all.

Let's look at some numbers. As of late 2025, the strongest coding agents on SWE-bench Verified reach around 50-60%. And that's on carefully curated tasks with clear problem descriptions and existing tests. Move to your everyday development environment — vague requirements, no tests, hidden business rules scattered everywhere — and that number drops further.

But behind these numbers lies a counterintuitive truth.

Same Horse, Different Fate

Anthropic ran a controlled experiment. Same prompt ("build a retro 2D game generator"), same model (Opus 4.5). First run: bare, no support — 20 minutes, $9, the game's core features completely non-functional. Second run: full harness (a three-agent architecture: planner + generator + evaluator) — 6 hours, $200, a playable game.

They didn't change the model. Opus 4.5 was still Opus 4.5. What changed was the harness.

OpenAI's 2025 harness engineering post says it plainly: Codex in a well-harnessed repository goes from "unreliable" to "reliable." Notice their choice of words — not "a bit better," but a qualitative shift. Like a thoroughbred horse: you can ride it without the right tack, but you won't get far, you won't go fast, and it's no surprise when you fall off. The harness is that whole tack — everything in the technical infrastructure outside the model weights.

Where Agents Actually Get Stuck

So what exactly goes wrong?

Most common: you never clearly define the task. You say "add a search feature," and the agent's understanding is completely different from yours — search what? Full-text or structured? Pagination? Highlighting? You don't specify, so the agent guesses. Guessing right is luck; guessing wrong costs more to fix than being specific from the start. It's like walking into a restaurant and telling the chef "I want fish" — whether you get braised fish, steamed fish, or fish hotpot is entirely up to chance.

Even when you have specified things, the project has hidden architectural conventions the agent doesn't know about. Your team standardized on SQLAlchemy 2.0 syntax, but the agent defaults to writing 1.x-style code. All API endpoints must use OAuth 2.0 authentication, but that rule only exists in your head and in a Slack message from three months ago. The agent can't see these things — not because it refuses to comply, but because it genuinely doesn't know these rules exist.

The environment is also a trap. An incomplete dev environment, missing dependencies, wrong tool versions. The agent burns precious context window on pip install errors and Node version conflicts instead of solving your actual task. It's like hiring a skilled carpenter but forgetting to provide a hammer, nails, or a flat workbench — no matter how talented, they can't get the job done.

Even more common: there's simply no way to verify. No tests, no lint, or verification commands that were never communicated to the agent. The agent writes code, looks at it, decides it's fine, says "done." It's like asking a student to turn in homework with no answer key — they think they got it right, but when you grade it there are piles of errors. Anthropic also observed an interesting phenomenon: when agents sense the context running low, they rush to finish, skip verification, and pick a simpler solution over the optimal one. They call this "context anxiety" — the same thing that happens when you realize time is almost up on a test and start randomly guessing the remaining multiple-choice questions.

Long tasks spanning multiple sessions are even worse — all the findings from the previous session are lost, and every new session has to rediscover the project structure and re-understand how the code is organized. Agents without persistent state see failure rates spike on tasks exceeding 30 minutes.

Key Terms Explained

Given these situations, the following concepts stop being mere jargon:

Capability Gap: The large gap between a model's benchmark performance and its performance on real tasks. A 50-60% pass rate on SWE-bench Verified means nearly half of real-world issues can't be solved.
Harness: Everything outside the model — instructions, tools, environment, state management, verification feedback. If it's not the model weights, it's the harness. This is what we call the "tack."
Harness-Induced Failure: The model is capable enough, but the execution environment has a structural flaw. Anthropic's controlled experiment demonstrated this.
Verification Gap: The gap between an agent's confidence in its own output and actual correctness. The agent says "I'm done" when it isn't — this is the most common failure mode.
Diagnostic Loop: Execute, observe the failure, attribute it to a specific harness class, fix that class, execute again. This is the core methodology of harness engineering.
Definition of Done: A set of machine-verifiable conditions — tests pass, lint is clean, type check passes. Without a clear definition of done, the agent will invent its own.

When Something Breaks, Fix the Harness First

Core principle: When something breaks, don't switch models first — check the harness. If the same model succeeds on similar, well-structured tasks, assume it's a harness problem. It's like a car that breaks down — you don't immediately suspect the engine. You check the gas first.

Concrete steps:

Attribute every failure to a specific class. Don't just say "the model isn't good." Ask: was the task ambiguous? Was context insufficient? Was there no verification method? Map each failure to one of five defense layers: task specification, context provisioning, execution environment, verification feedback, state management. These are practical diagnostic classes, not the six glossary terms above. Build this habit, and you'll see "the model isn't good enough" show up less and less in your logs.

Write a clear Definition of Done for every task. Don't say "add a search feature." Say:

Completion criteria:
- New GET /api/search?q=xxx endpoint
- Supports pagination, default 20 items
- Results include highlighted excerpts
- All new code passes pytest
- Type checking passes (mypy --strict)

Create an AGENTS.md file. Put it at the repo root to tell the agent the project's tech stack, architectural conventions, and verification commands. This is the first step in harness engineering and the highest-ROI move you can make. A single AGENTS.md file can be more effective than upgrading to a more expensive model — I'm not exaggerating.

Build a diagnostic loop. Don't treat failures as "the model being dumb again." Treat them as signals that your harness has a flaw. For every failure, identify the class, fix it, never fail that way again. After a few rounds, your harness gets stronger and agent performance becomes more stable. It's like patching a road — every pothole you fill makes the next stretch smoother.

Quantify improvement. Keep a simple log: whether each task succeeded or failed, and which class caused the failure. After a few rounds, you'll see which class is the bottleneck — focus your energy there.

The Million-Line Codebase Experiment

OpenAI ran a bold experiment in 2025: use Codex to build a complete internal product from an empty git repository. Five months later, the repo had around a million lines of code — application logic, infrastructure, tooling, documentation, internal dev tools — all generated by the agent. Three engineers drove Codex, opening and merging about 1,500 PRs. An average of 3.5 PRs per person per day.

The key constraint: humans never wrote code directly. This wasn't a gimmick — it was designed to force the team to figure out what changes when an engineer's core job is no longer writing code, but designing the environment, articulating intent, and building feedback loops.

Initial progress was slower than expected. Not because Codex lacked capability, but because the environment wasn't complete enough yet — the agent lacked the tools, abstractions, and internal structure needed to advance toward high-level goals. The engineers' job became: break large goals down into small building blocks (design, coding, evaluation, testing), let the agent assemble them, then use those blocks to unlock more complex tasks. When something went wrong, the answer was almost never "try harder" — it was "what capability is the agent missing, and how do we make it something it can understand and execute?"

This experiment directly demonstrates the core thesis of this lecture: the same model produces fundamentally different results in a bare environment versus a fully-harnessed one. The model didn't change. The environment did.

Source: OpenAI: Harness engineering: leveraging Codex in an agent-first world

A More Down-to-Earth Example

A team used Claude Sonnet to add a new API endpoint to a medium-sized Python web app (FastAPI + PostgreSQL + Redis, ~15,000 lines of code).

Initially they gave just a single sentence: "add user preference endpoints under /api/v2/users." The result? The agent spent 40% of its context window exploring the repo structure, produced code that looked plausible but didn't follow the project's error-handling patterns, used outdated SQLAlchemy syntax, and declared completion while the endpoint had runtime errors. The next session had to redo all the exploration work from scratch.

They then added AGENTS.md (describing the project architecture and tech stack versions), clear verification commands (pytest tests/api/v2/ && python -m mypy src/), and architectural decision records. The same model succeeded across all three independent runs, with roughly 60% better context efficiency.

They didn't change the model. They changed the harness.

Key Takeaways

Model capability and execution reliability are two different things. A thoroughbred horse still needs good tack.
When something breaks, check the harness first, then the model. Switching models is the most expensive option — and often not even the actual problem.
Every failure is a signal: your harness has a structural flaw. Find it, fix it.
Five defense layers: task specification, context provisioning, execution environment, verification feedback, state management. Check them systematically, like a doctor ruling out the most common causes first.
A single AGENTS.md file can be more effective than upgrading to a more expensive model. It really is.

Exercises

Comparative experiment: Pick a codebase you know well and a non-trivial modification task. First, run the agent with no harness support and log the failures. Then add `AGENTS.md` with clear verification commands and run again with the same agent. Compare the results, attributing each failure to one of the five defense layers.

Measure the verification gap: Pick 5 programming tasks. After each one, record whether the agent declared completion, then verify actual correctness with independent tests. Calculate the rate at which the agent declared "done" when it wasn't — that's your verification gap. Then think: which verification commands would reduce that rate?

Practice the diagnostic loop: Find a task your agent keeps failing in your project. Run it once, log the failure. Attribute it to one of the five classes. Fix that class. Run it again. Repeat three to five rounds, recording the improvement each time.