Lecture 02. What a Harness Actually Means

The word "harness" gets thrown around a lot in the AI coding agent world, but honestly, most people mean "a prompt file" when they say harness. That's not a harness. It's like opening a restaurant with only ingredients — no kitchen, no knives, no recipes, no plating process. That's not a restaurant. That's a refrigerator.

This lecture gives you a precise, actionable definition of harness. Not an academic abstraction, but a framework you can use today: a harness consists of five subsystems, each with clear responsibilities and evaluation criteria.

Starting With an Analogy

Imagine you're a newly hired engineer, dropped into a project with no documentation. No README, no comments in the code, no one tells you how to run the tests, the CI config is buried somewhere. Can you write good code? Maybe — if you're smart and patient enough. But you'll spend a lot of time "figuring out what this project is about" instead of "solving the problem."

An AI agent faces exactly the same situation. And it's even worse — at least you can ask a colleague. The agent can only see the files placed in front of it and the commands it can execute. It can't tap someone on the shoulder and ask "hey, what ORM version does this project use?"

OpenAI frames the core principle as "the repo IS the spec" — all necessary context must live in the repository, conveyed through structured instruction files, clear verification commands, and clear directory organization. Anthropic's documentation on long-running agents emphasizes state continuity, clear recovery paths, and structured progress tracking. The two companies focus on different aspects, but they're saying the same thing: everything in the technical infrastructure outside the model determines how much of the model's capability actually gets realized.

Let's look at some tools you already know:

Claude Code embodies harness thinking. It reads CLAUDE.md from your repo (the recipe shelf), can run shell commands (the knife rack), executes in your local environment (the kitchen), maintains session history (the prep table), and can run tests and see results (the quality-check window). But if you don't tell it how to run tests, the quality-check window is broken — nobody knows if the dish is cooked.

Cursor follows the same logic. Its .cursorrules file is the recipe shelf, the terminal is the knife rack, it reads project structure and lint config for the kitchen. But Cursor's state management is relatively weak — close the IDE and reopen it, and the prior context is gone.

Codex (OpenAI's coding agent) uses git worktrees to isolate each task's runtime environment, combined with a local observability stack (logs, metrics, traces), so every change is verified in an isolated environment. In repos with AGENTS.md and clear verification commands, it performs far better than in "bare" repos.

AutoGPT is a cautionary tale — lacking structured state management leads to context accumulation on long tasks, and lacking precise feedback mechanisms causes the agent to loop. Many say AutoGPT "doesn't work," but really it's AutoGPT's harness that doesn't work — give a chef a broken stove and even the finest ingredients won't produce a meal.

Core Concepts

What a harness is: Everything in the technical infrastructure outside the model weights. OpenAI distills the core work of an engineer down to three things: designing the environment, articulating intent, and building feedback loops. Anthropic calls their Claude Agent SDK a "general-purpose agent harness."
The repo as the single source of truth: Anything the agent can't see, in every practical sense, doesn't exist. OpenAI treats the repo as the "system of record" — all necessary context must live there, through structured files and clear directory organization.
Give a map, not a manual: OpenAI's experience — AGENTS.md should be a directory page, not an encyclopedia. About 100 lines is enough. If it doesn't fit, split it into a docs/ folder and let the agent read on demand.
Constrain, don't micromanage: A good harness uses enforceable rules to constrain the agent, rather than listing instructions one by one. OpenAI says "enforce invariants, don't micromanage implementation"; Anthropic found that agents confidently praise their own work, and the solution is to separate the "doer" from the "checker."
Remove components one at a time: To quantify the marginal contribution of each harness component, remove them one at a time and see which removal causes the biggest performance drop. Anthropic used this method and found that as models get more capable, some components matter less — but new ones always emerge.

The Five-Subsystem Harness Model

Back to the kitchen analogy. A complete kitchen has five functional areas, and a harness has five subsystems:

Instruction Subsystem (the recipe shelf): Create an AGENTS.md (or CLAUDE.md) containing a project overview and purpose (one sentence), tech stack and versions (Python 3.11, FastAPI 0.100+, PostgreSQL 15), the first command to run (make setup, make test), non-negotiable hard constraints ("All APIs must use OAuth 2.0"), and links to more detailed documentation.

Tooling Subsystem (the knife rack): Make sure the agent has adequate tool access. Don't disable the shell "for security" — if the agent can't even run pip install, how is it supposed to work? But don't open everything either — follow the principle of least privilege.

Environment Subsystem (the kitchen): Make environment state self-describing. Use pyproject.toml or package.json to lock dependencies, .nvmrc or .python-version for runtime versions, Docker or devcontainers for reproducibility.

State Subsystem (the prep table): Long tasks need progress tracking. Use a simple PROGRESS.md file recording: what's done, what's in progress, what's blocked. Update it before each session ends, read it when the next session starts.

Feedback Subsystem (the quality-check window): This is the highest-ROI subsystem. Clearly list verification commands in AGENTS.md:

Verification commands:
- Test: pytest tests/ -x
- Type check: mypy src/ --strict
- Lint: ruff check src/
- Full verification: make check (includes all of the above)

Missing any subsystem is like missing a functional area in the kitchen — you can still cook, but it's always awkward.

Quantifying harness component value: Use "isobaric model control." Keep the model fixed, remove one subsystem at a time and measure which removal causes the biggest performance drop. The biggest drop indicates the highest marginal-contribution component for that task, not automatically the bottleneck. To identify the real bottleneck, combine the experiment with failure logs and root-cause attribution: ambiguous task, insufficient context, non-reproducible environment, missing verification feedback, or broken state management.

A Real Team's Story

A team used GPT-4o on a TypeScript + React frontend app (~20,000 lines of code). They went through four phases — essentially adding kitchen equipment one piece at a time:

Phase 1 — Empty kitchen: Only a basic project description in the README. 1 out of 5 runs succeeded (20%). Main failures: wrong package manager chosen (npm vs yarn), not following component naming conventions, unable to run tests.

Phase 2 — Recipe shelf installed: Added AGENTS.md with tech stack versions, naming conventions, key architectural decisions. Success rate rose to 60%. Remaining failures were mostly environment issues and lack of verification.

Phase 3 — Quality-check window opened: Listed verification commands in AGENTS.md: yarn test && yarn lint && yarn build. Success rate rose to 80%.

Phase 4 — Prep table ready: Introduced progress file templates where agents record completed and pending work after each run. Success rate stabilized at 80-100%.

Four iterations, the model never changed, success rate went from 20% to nearly 100%. That's the power of harness engineering. You're not buying more expensive ingredients — you're just organizing the kitchen properly.

Key Takeaways

Harness = Instructions + Tools + Environment + State + Feedback. Five subsystems, like five functional areas of a kitchen — all essential.
If it's not the model weights, it's the harness. Your harness determines how much of the model's capability actually gets realized.
Among the five subsystems, the feedback subsystem usually has the lowest investment and highest return. Get verification commands right first — the quality-check window is the most worthwhile upgrade.
Use "isobaric model control" to quantify each subsystem's marginal contribution; use failure logs and root-cause attribution to identify the real bottleneck.
Harnesses rot like source code. Check them regularly, pay down harness debt the way you pay down technical debt.

Exercises

Five-tuple harness audit: Take a project where you use an AI agent and perform a full audit using the five-tuple framework. Score each subsystem from 1-5. Find the lowest-scoring subsystem, spend 30 minutes improving it, then observe the change in agent performance.

Isobaric model control experiment: Pick a model and a challenging task. Remove instructions (delete AGENTS.md), remove feedback (don't provide verification commands), remove state (no progress file) — one at a time — and measure the performance drop. Based on the results, rank the subsystems' importance for your project.

Affordance analysis: Find a situation where an agent in your project "wants to do something but can't" (e.g., knows it should use parameterized queries but doesn't know your project's ORM patterns). Analyze whether this is a Gulf of Execution (doesn't know how) or a Gulf of Evaluation (doesn't know if it's correct), then design a harness improvement to close that gap.