Lecture 11. Making the Agent's Runtime Observable

What Problem Does This Lecture Solve?

You ask an agent to implement a feature. It runs for 20 minutes, modifies a batch of files, then tells you "done, but two tests are failing." You ask why they're failing — "not sure, maybe a timing issue." You ask which critical paths it changed — "let me look at the code..."

This isn't about the agent lacking capability. It's about your harness not providing enough observability. Without observability, the agent makes decisions under uncertainty, evaluation becomes subjective judgment, and retries become blind wandering. Both OpenAI and Anthropic define reliability as an evidence problem — the harness must expose runtime behavior and evaluation signals in a form that can guide the next decision.

Core Concepts

Runtime observability: System-level signals — logs, traces, process events, health checks. Answers "what did the system do."
Process observability: Visibility into harness decision artifacts — plans, scoring rubrics, acceptance criteria. Answers "why this change should be accepted."
Task trace: A complete record of the decision path from task start to completion, similar to request tracing in distributed systems. Every step the agent takes, with context, is recorded.
Sprint contract: A short-term agreement negotiated before coding begins — specifying task scope, verification standards, and exceptions. The core tool for process observability.
Evaluator rubric: Turns quality evaluation from subjective judgment into structured, evidence-based scoring. Makes different evaluators produce similar results for the same output.
Layered observability: System-layer observability and process-layer observability are designed simultaneously and reinforce each other. Runtime signals explain behavior; process artifacts explain intent.

Layered Observability

Why This Happens

The Real Cost of Missing Observability

When a harness lacks observability, four categories of problems appear systematically:

Can't distinguish "correct" from "looks correct": A function looks perfect in code review — correct syntax, sound logic. But at runtime, an edge-case handling bug produces incorrect results for specific inputs. Only a runtime trace can reveal that the actual execution path deviates from expectations.

Evaluation becomes mystical: Without scoring rubrics and acceptance criteria, evaluators (human or agent) rely on hidden assumptions. The same output can receive completely different evaluations from different reviewers. Quality evaluation becomes irreproducible.

Retries become blind guessing: When the agent doesn't know why something failed, the direction of a retry is random. It might try repeatedly in the wrong direction — fixing unrelated code paths while ignoring the actual cause of failure. Every blind retry costs tokens and time.

Session handoff information cliff: When unfinished work is handed off to the next session, lack of observability means the new session must diagnose system state from scratch. Anthropic's observations of long-running agents show this redundant diagnosis can consume 30-50% of total session time.

A Realistic Claude Code Scenario

Imagine a harness using a three-role "planner-generator-evaluator" workflow, executing the task "add dark mode to the app."

Without observability: The planner gives a vague description. The generator implements dark mode based on that ambiguity, but it doesn't match the planner's hidden expectations. The evaluator rejects it based on its own hidden standards but can't say specifically what's wrong. The generator retries blindly based on the vague rejection reason. The cycle repeats 3-4 times, taking about 45 minutes, producing a barely acceptable result.

With full observability: The planner produces a sprint contract — listing the components to modify, the verification standard for each, and exceptions (print styles not handled). The generator implements according to the contract. Runtime observability records the style-loading and application process for each component. The evaluator uses a scoring rubric to assess each dimension, with specific evidence citations. One iteration produces a high-quality result, in about 15 minutes.

A 3x efficiency difference. The only change was observability.

Why the Agent Can't Solve This on Its Own

You might be thinking: "Can't the agent just print its own logs?" The problems are:

The agent doesn't know what it doesn't know — it won't proactively log signals it doesn't realize are necessary.
Log formats are inconsistent — different sessions use different log formats, making systematic analysis impossible.
Process observability can't be solved with logs — sprint contracts and scoring rubrics are structured artifacts that need harness-level support.

How to Do It Right

1. Build Runtime Signal Collection Into the Harness

Don't rely on the agent to print its own logs. The harness should automatically collect these signals:

Application lifecycle: Startup, ready, running, shutdown phase states
Feature path execution: Records of critical-path execution, including entry points, checkpoints, and exit points
Data flow: Records of data flowing between components
Resource usage: Unusual resource-usage patterns (e.g., continuously increasing memory)
Errors and exceptions: Full error context, not just error messages

2. Implement a Sprint Contract

Before each task begins, the generator and evaluator (which may be different calls of the same agent) negotiate a contract:

# Sprint Contract: Dark Mode Support

## Scope
- Modify the theme-switching component
- Update global CSS variables
- Add dark mode tests

## Verification Standards
- Visual regression tests pass for each component
- End-to-end test for the main flow passes
- No flash of unstyled content (FOUC)

## Exceptions
- Print styles not handled
- Dark mode for third-party components not handled

3. Establish an Evaluator Rubric

Turn "good or not" into quantifiable scoring:

# Scoring Rubric

| Dimension | A | B | C | D |
|-------|---|---|---|---|
| Code correctness | All tests pass | Main flow passes | Partially passes | Build fails |
| Architecture compliance | Fully compliant | Minor deviation | Clear deviation | Serious violation |
| Test coverage | Main flow + edge cases | Main flow only | Skeleton only | No tests |

4. Standardize With OpenTelemetry

Create one trace per harness session, one span per task, and sub-spans per verification step. Use standard attributes to annotate key information. This way, observability data integrates with standard tools (Jaeger, Zipkin).

Real-World Case

A harness using a planner-generator-evaluator workflow, executing "add dark mode support":

Non-observable version: 3-4 rounds of blind retries, 45 minutes, barely acceptable result. The evaluator says "it doesn't feel right" but can't say specifically what. The generator wastes a lot of time in the wrong direction.

Fully observable version:

Sprint contract clarifies scope, standards, and exceptions
Runtime trace records the style-loading process for each component
Scoring rubric provides structured, per-dimension evaluation
One iteration produces a high-quality result, 15 minutes

3x efficiency improvement, more stable quality, reproducible evaluation.

Key Takeaways

Observability is a harness architectural property — not a feature bolted on later, but a core capability that must be considered during design.
Both observability layers are necessary — runtime signals explain "what happened," process artifacts explain "why it was done that way."
Sprint contracts front-load alignment — preventing "the generator building something the evaluator immediately rejects for predictable reasons."
Scoring rubrics make evaluation reproducible — different evaluators produce similar scores for the same output.
Missing observability wastes 30-50% of session time on redundant diagnosis.

Exercises

Observability Gap Analysis: Audit your current harness for system-layer and process-layer observability. Find system states that can't be distinguished from existing signals, and propose additions.

Sprint Contract Practice: Write a sprint contract for a real task. Have the agent execute according to the contract, and compare efficiency and quality with and without the contract.

Build a Task Trace: Record every step of an agent's operations during a complete coding task. Annotate with OpenTelemetry semantic conventions. Analyze the information bottlenecks in the trace — which steps lack sufficient signal support for decisions.