Sample code for this lecture: code/ Practice: Project 05. Letting the agent verify its own work

Lecture 10. Only End-to-End Testing Is Real Verification

You ask an agent to add a file-export feature to an Electron app. It writes the render-process component, the preload script, and the service-layer logic. Unit tests for each component pass perfectly. The agent says, "Done." When you actually click the export button — the file path format is wrong, the progress bar doesn't update, and exporting large files leaks memory. Five component-boundary bugs, and not one was caught by a unit test.

It's like a choir rehearsal — every section sounds perfect singing alone, but when they sing together, the soprano section is half a beat ahead of the bass, and the accompaniment is a half-step off from the main melody. Each part is "correct" on its own, yet the whole is out of tune.

Google's testing pyramid tells us: a large number of unit tests is the foundation, but if you stop there, you'll systematically miss component-interaction problems. For AI coding agents this problem is even more severe — agents tend to run only the fastest tests and then declare completion. Only end-to-end testing can prove that system-level bugs don't exist.

The Blind Spot of Unit Testing

The design philosophy of unit testing is isolation — mock the dependencies and focus only on the unit under test. This philosophy makes unit testing fast and precise, but it also creates systematic blind spots. It's like each section rehearsing with headphones in a choir practice — sounds fine to them, but problems only surface when they sing together:

Interface mismatch: The file path passed by the render process to the preload script is relative, but the preload script expects an absolute path. Their respective unit tests both use mocks and pass. The problem only surfaces when the end-to-end flow is executed — like two sections rehearsing independently and feeling fine, only to realize in the full ensemble that one section is singing in 4/4 and the other in 3/4.

State propagation error: A database migration changes a table schema, but the ORM caching layer still holds cache entries for the old schema. Unit tests provide a completely fresh mock environment each time, which will never expose this cross-layer state inconsistency. It's like changing the lyrics, but someone is still singing the old version.

Resource lifecycle issue: Acquiring and releasing file handles, database connections, and network sockets spans multiple components. Unit tests create and destroy resources independently for each test, so they can't expose resource contention or leaks. It's like each section taking turns with the microphone during rehearsal, but when everyone gets on stage together, there aren't enough mics to go around.

Environment dependency: Code works correctly in the test environment (where everything is mocked) but fails in the real environment due to configuration differences, network latency, or unavailable services. It's like singing perfectly in the rehearsal room, but hitting audio feedback and wind interference at an outdoor festival.

End-to-End Testing Doesn't Just Change Results, It Changes Behavior

Here's something many people don't realize: when an agent knows its work will go through end-to-end testing, its coding behavior changes.

• Considers component interactions: While writing code, it thinks about "how does this interface connect upstream," instead of focusing on a single function in isolation. It's like knowing you'll eventually sing together, so you pay attention to the other sections while rehearsing.
• Respects architectural boundaries: In systems with architectural constraints, end-to-end testing forces the agent to follow boundary rules. It's like a score marked "raise the volume here" — you have to follow it.
• Handles error paths: End-to-end tests often include failure scenarios, forcing the agent to consider exception handling. It's like rehearsing "what happens if the mic suddenly cuts out," so you know what to do.

The Testing Pyramid and Promoting Review Feedback Upward

In OpenAI's Codex engineering practices, they emphasize: error messages written for agents must include repair instructions. Don't just write "Direct filesystem access in renderer"; write "Direct filesystem access in renderer. All file operations must go through the preload bridge. Move this call to preload/file-ops.ts and invoke it via window.api." This turns architectural rules into a self-correcting loop. It's like a choir conductor who doesn't just say "you sang that wrong," but says "you're half a beat ahead here, listen to the alto section's tempo, and come in on beat 32."

Core Concepts

• Component boundary bug: Component A and B each pass their own unit tests, but their interaction produces incorrect behavior. This is the kind of problem end-to-end testing is best at catching — like choir sections that are each correct alone but out of sync together.
• Testing Adequacy Gradient: bugs caught by unit tests <= bugs caught by integration tests <= bugs caught by end-to-end tests. Each higher layer increases detection capability.
• Architectural Boundary Enforcement Rule: Turns rules from architecture docs (like "the render process can't access the filesystem directly") into automated, enforceable checks. From "written on paper" to "runs in CI."
• Review Feedback Promotion: Converts recurring code-review comments into automated tests. Every time a recurring issue is found, add a rule, and the harness automatically gets stronger. It's like a conductor turning common rehearsal mistakes into warm-up exercises — next time the same mistake happens, the exercise itself exposes it without the conductor having to say anything.
• Agent-Directed Error Message: An error message shouldn't just state "what went wrong," it should tell the agent exactly how to fix it. This turns test failures into a self-correcting feedback loop.

How to Do It

0. Define Architectural Boundaries First, Then Write E2E Tests

A prerequisite for end-to-end testing is a clear system boundary. If the architecture is a plate of spaghetti, end-to-end testing only proves "this plate of spaghetti runs" — it won't tell you where design intent is being violated. It's like a choir that hasn't even been split into sections yet — no amount of rehearsal will make it sound good.

OpenAI's experience: for agent-generated codebases, architectural constraints must be an initial prerequisite established from day one, not something to consider once the team grows. The reason is simple — agents will copy existing patterns in the repository, even if those patterns are inconsistent or suboptimal. Without architectural constraints, the agent will introduce more drift with every session.

OpenAI adopted a "Layered Domain Architecture" — each business domain is split into fixed layers: Types → Config → Repo → Service → Runtime → UI. Dependencies flow strictly forward, and cross-domain concerns enter through explicit Provider interfaces. Any other dependency is forbidden and mechanically enforced via custom linting.

Key principle: enforce invariants, don't micromanage implementation. For example, require "data is parsed at the boundary," but don't specify which library to use. Error messages must include repair instructions — not just say "violation," but tell the agent exactly how to change it.

Source: OpenAI: Harness engineering: leveraging Codex in an agent-first world

1. The Harness Must Include an End-to-End Layer

Make it explicit in your verification flow: for tasks involving cross-component changes, passing end-to-end tests is a prerequisite for completion:

## Verification Hierarchy
- Level 1: Unit tests (must pass)
- Level 2: Integration tests (must pass)
- Level 3: End-to-end tests (must pass when there are cross-component changes)
- Skipping any required level = Not Done

2. Turn Architectural Rules Into Enforceable Checks

Every architectural constraint should have a corresponding test or lint rule:

# Check whether the render process calls Node.js APIs directly
grep -r "require('fs')" src/renderer/ && exit 1 || echo "OK: no direct fs access in renderer"

3. Design Agent-Directed Error Messages

Error messages should contain three elements: what went wrong, why, and how to fix it:

ERROR: Found direct 'fs' import in src/renderer/App.tsx:12
WHY: The render process has no access to Node.js APIs for security reasons
FIX: Move the file operations to src/preload/file-ops.ts and call them via window.api.readFile()

4. Establish a Review Feedback Promotion Process

Every time a new category of agent mistake is found in code review, turn it into an automated check. A month later, your harness will be significantly stronger than it was at the start of the month. It's like keeping rehearsal notes for a choir — recording issues found in each rehearsal so they can be checked before the next one. Over time, common mistakes decrease, and the music becomes more harmonious.

Real-World Case

Task: Implement a file-export feature in an Electron app. Involves the render-process UI, the preload script's filesystem proxy, and service-layer data conversion.

Singing each section alone (unit tests pass): Render component test (passes, file operations mocked), preload script test (passes, filesystem mocked), service layer test (passes, data source mocked). The agent declares completion.

Singing together (bugs revealed by end-to-end testing):

| Bug | Description | Unit Test | E2E | |---|---|---|---| | Interface mismatch | Inconsistent file path formatting | Missed | Caught | | State propagation | Export progress not sent back to UI via IPC | Missed | Caught | | Resource leak | File handle for large export not released | Missed | Caught | | Permission issue | Different permissions in packaged environment | Missed | Caught | | Error propagation | Service-layer exception doesn't reach the UI layer | Missed | Caught |

All 5 bugs were caught by end-to-end testing, while unit tests caught none of them. The cost was an increase in test time from 2 seconds to 15 seconds — entirely acceptable in an agent workflow. However well each section sings, it's no match for a full ensemble rehearsal.

Key Takeaways

• Unit tests are systematically blind to component-boundary bugs — their isolated design is precisely what prevents them from detecting interaction problems. Everyone singing correctly doesn't mean the choir isn't out of tune.
• End-to-end testing doesn't just catch bugs, it changes the agent's coding behavior — making it pay more attention to integration and boundaries.
• Architectural rules must be enforceable — not written in docs to be read, but automatically checked on every commit.
• Error messages must be designed for the agent — including concrete "how to fix it" steps to form a self-correcting loop.
• Promoting review feedback makes the harness automatically stronger — every caught bug category becomes a permanent line of defense.

Further Reading

• How Google Tests Software - Whittaker et al. — The classic source for the Testing Pyramid model
• Harness Engineering - OpenAI — Engineering practices for automatically enforcing architectural constraints
• Chaos Engineering - Netflix (Basiri et al.) — Proactively injecting failures to verify system resilience
• QuickCheck - Claessen & Hughes — Property-testing methodology, sitting between example testing and formal verification

Exercises

Detect Cross-Component Bugs: Choose a modification task involving at least three components. First, run only unit tests and record the results, then run end-to-end tests. Analyze which cross-layer interaction category each newly detected bug belongs to.

Automate an Architectural Rule: Pick an architectural constraint from your project and turn it into an enforceable check (with an agent-directed error message). Integrate it into the harness and verify its effectiveness with a baseline task.

Promote Review Feedback: Find a recurring type of comment from your code review history and convert it into an automated check using the five-step process. Compare the frequency of the issue before and after promotion.