Evaluation
You judged the output, noticed failure, and named it.
// 00 Loop engineering · the masterclass, made interactive
Most people use AI one prompt at a time. Loop engineering builds the jobs you do in your head—evaluation, memory, guardrails, and stopping conditions—into a system that can tell whether its own work is improving.
A strong prompt can impress once. Real work has a quality bar, happens at volume, and must stay consistent long after attention starts to drift.
You judged the output, noticed failure, and named it.
You carried what failed into the next attempt.
You enforced boundaries: no invented dates and no broken tone.
You decided when an answer was ready. The model never could.
→ Without naming it, you were running a loop manually, inside your head.
Drafts one reply. Nothing happens if it is weak unless a person notices.
Runs a designed sequence: retrieve, check, draft, route.
Receives an objective and tools; inspects sources and asks when they conflict.
Evaluates work against a defined standard; failures inform another attempt, passing checks stop, and conflict escalates.
Treat a loop like a new employee with infinite patience and no earned judgment.It evaluates the fiftieth item with the same energy as the first, but cannot decide which failure matters.
The model handles generation and reasoning. The system around it supplies the controls. Select any part to see it resolved and the matching architecture lines light up.
01 · A measurable state, not an activity
Reviewing a pull request is an activity. A reliable goal defines completion: inspect every changed file, run the required tests, link failures to evidence, flag security-sensitive changes, then recommend approve, request changes, or escalate.
If the goal is an activity, the loop has no definition of done.
while target_not_reached:result = take_action(context, memory)evaluation = evaluate(result, rubric)log(result, evaluation)if evaluation.passes:stop_with_success(result)if human_decision_required(evaluation):escalate(result, evaluation)if budget_exhausted() or progress_stalled():stop_without_success()memory = update_memory(result, evaluation)
The active prompt is the champion. A challenger takes the title only by beating it on a holdout it never studied.
The same building blocks support several distinct patterns. Choose the architecture that matches the task and the evidence you can collect.
The active version keeps its title until a challenger beats it on a holdout it never studied.
Process interviews, tickets, or surveys in batches; attach original quotes and sources to every evolving cluster.
One model builds a document. Another attacks assumptions, edge cases, metrics, costs, and irreversible decisions without rewriting it.
A fresh browser session attempts a real product task and scores the full journey instead of one component.
Expand each failure. Every one is a specification problem, not a model problem.
A score can rise while useful work rots. Restate the real objective, use a multidimensional rubric, and inspect behavior direction as well as a number.
Stage 01
The loop runs on real work but cannot affect anything. Compare what it would have done with the human decision and turn each mismatch into a rule.
Boundary at this stageRead-only. No sends, writes, or merges; output is a logged shadow decision.
Clarify what good work looks like before automating it.
Automate what is checkable. Escalate what is consequential.
An unattended system needs an operating policy before it runs.
If a specification field is blank, that is design work—not an invitation to hide uncertainty inside a longer prompt.
One change preserves causality; bundled changes make it impossible to know what helped.
An improvement-set win is only a rumor until it survives unfamiliar work.
A tie goes to the active version. Every challenger needs recorded evidence.
Customers, money, legal commitments, private data, and irreversible changes need a human gate.
Shadow, suggest, approve, bounded autonomy—grant it on observable evidence and keep sampling.
Choose one narrow job you have performed at least three times. Fill this in before building—the preview updates live.