The Karpathy Loop, Three Months Later

A reflection on what survived from the original design, what had to change, and what it took to make the system's improvement loop trustworthy.

---

In early versions of this system, I was trying to solve a very specific problem. A user would describe a business application in natural language, and the system had to infer the structure behind that description: what the core records were, how they related to each other, what behaviors mattered, and what views or workflows the final application should expose. The quality of that inference depended heavily on one prompt. That made the prompt an obvious candidate for optimization.

The appeal of the Karpathy-loop pattern was that it made this optimization feel engineerable. If the model could propose prompt changes, and a frozen evaluator could score those changes honestly, then prompt iteration no longer had to be an informal exercise in intuition. It could become a ratcheting process: propose, measure, keep only genuine improvement, repeat.

That part was real. The part that was harder than it first appeared was the word "honestly."

The original system did work, in a limited sense. It optimized a large generation prompt against a fixed set of reference applications and scored candidates with a composite made up of compile success, structural quality checks, and gate pass rate. Candidate prompts were evaluated in-process through a temporary override rather than by mutating files or relying on git-backed revert logic. That decision was good then and is still good now. It kept evaluation bounded, auditable, and compatible with a database-versioned runtime.

But the original loop had two weaknesses that mattered more than I appreciated at the time. The first was that the benchmark sat too close to the final compiled artifact. A generated result could look structurally respectable while still reflecting weak understanding upstream. The second was more serious: I did not have a satisfactory hold-out story. I had a benchmark, but not a convincing answer to the question that matters most in any optimization loop: is the system learning the task, or is it learning the benchmark?

That question ended up driving almost all of the redesign.

The biggest architectural change was to move the optimization target one layer earlier in the pipeline. The old loop effectively asked whether the final generated structure compiled and passed downstream checks. The newer loop asks a more direct question: did the system correctly understand the user's request in the first place?

That shift sounds narrow, but it changes what the benchmark means. Instead of judging quality indirectly through the final artifact, the current evaluator works against a curated corpus of intent examples. Each example contains a natural-language goal prompt and an explicit gold expectation: the expected application type, the expected entities, the expected relations, and the expected projections. The benchmark no longer has to reconstruct correctness from downstream output. It states, directly, what a correct interpretation should contain.

The scoring function is correspondingly simpler and more honest:

composite = 0.25 × application-type accuracy
          + 0.35 × entity recall
          + 0.20 × role accuracy
          + 0.20 × behavior coverage

This is a better optimization surface for a straightforward reason. Users do not care whether an internal graph compiled. They care whether the system understood the business problem they described. Moving the metric closer to that promise made the whole loop more meaningful.

At the same time, several things from the original design stayed exactly because they had already proven to be structurally right. The editable surface remained narrow. Candidate prompts are still evaluated in-process through a context-local override, which in Python allows a temporary change inside one run without mutating global state for every other request. The generator and verifier are still separated. The ratchet is still strict. Calibration is still a prerequisite for autonomous operation. Those choices are less glamorous than the scoring formula, but they are what keep the loop from dissolving into self-congratulation.

The context-local override still looks essentially like this:

_prompt_override: ContextVar[str | None] = ContextVar("prompt_override", default=None)

@contextmanager
def prompt_override_context(candidate: str):
    token = _prompt_override.set(candidate)
    try:
        yield
    finally:
        _prompt_override.reset(token)

def get_active_prompt() -> str:
    override = _prompt_override.get()
    if override is not None:
        return override
    return _load_from_db()

That small mechanism is more important than it looks. It preserves the clean separation between evaluation and activation. A candidate prompt can be tested in the real harness without being written into shared runtime state, and rejection leaves no residue behind.

What changed more dramatically was the surrounding system. The original write-up described one loop because, at the time, that was the useful mental model. The current system is better understood as a small family of loops. Loop A is the slower end-to-end quality loop. Loop B is the faster prompt-optimization loop for intent extraction. Loop C looks for missing recurring domain patterns in the benchmark set. Loop D closes the production-feedback path by accumulating and analyzing operational signals from real usage. This split matters because different learning problems need different evaluators, different cadences, and different acceptance rules. Some changes are safe to ratchet automatically. Others should only produce candidates for human review. Still others should feed the benchmark rather than directly modify the system.

In practice, the center of gravity moved from the older end-to-end loop toward the faster and more interpretable intent-extraction loop. That made it possible to see regressions more clearly. Instead of only observing that a final output was somehow worse, I could now ask which entity disappeared, which role was misclassified, or which expected behavior was missing. That level of diagnostic clarity matters because it turns the loop from a blunt optimizer into something closer to an instrument.

The benchmark also became better in another important way: it stopped treating average improvement as sufficient. The corpus now includes a fixed set of anchor examples representing core application shapes. Those anchors function as invariants. They are not allowed to regress just because the global mean inches upward. This sounds like a small policy choice, but it closes a real failure mode. A loop can improve its average by sacrificing a category that matters disproportionately in practice. Anchor cases prevent the system from buying cheap gains at the cost of core coverage.

That insight led directly into the hardest unresolved issue from the first version: the hold-out problem.

At first I thought the missing hold-out set was mainly a data problem. I did not have enough clean spare examples to reserve a separate verification corpus. Over time it became clear that this was the wrong framing. The real issue was not the absence of extra data. It was the absence of a corpus policy that preserved the distinction between optimization and verification as the system evolved.

The solution was to make hold-out structure part of the data lifecycle itself.

The current design has three layers. First, there is an optimization corpus: a curated labeled set that supplies the main ratchet signal. Second, there is an anchor subset: a fixed representative slice that protects invariants and prevents silent collapse on important categories. Third, there is a shadow partition: new examples do not enter the optimization corpus immediately. They first spend time outside the ratchet as unseen verification cases.

This turned out to be the real resolution. Instead of waiting for a permanently separate test set to materialize, the system manufactures hold-out structure through corpus growth. Newly added examples are first quality-gated. If they pass, they do not immediately become optimization targets. They enter a shadow phase. During that phase, they test whether prompt changes generalize beyond the active optimization corpus. If a revision improves the main corpus but fails on the shadow slice, that is not improvement. It is specialization. Only after serving as unseen verification cases do those examples graduate into the optimization corpus, while another slice rotates out.

At the policy level, the mechanism is simple enough to describe directly:

def admit_example(example):
    if not passes_quality_gate(example):
        return "reject"
    if not completes_shadow_phase(example):
        return "shadow_only"
    return "promote_to_optimization_corpus"

What matters is not the literal implementation but the sequencing. Admission to the corpus is no longer the same thing as admission to optimization. That separation is what creates a moving verification boundary.

This approach is better than a permanently frozen test set for a long-running learning loop. A static test set eventually becomes familiar, even if only indirectly. People learn its shape. Prompts drift toward it. A rotating shadow partition preserves novelty, while the anchor cases preserve continuity. Together, the two solve different parts of the same problem.

So the hold-out problem was not resolved by finding more data. It was resolved by introducing corpus admission gates, shadow-first admission, rotating hold-out partitions, and fixed anchor invariants. That combination finally gave the loop a credible answer to the generalization question.

The statistical side of the loop also had to become more disciplined. The original article spent a lot of time on calibration, and that was not accidental. A ratchet is only as honest as its noise estimate. In practice the key lesson survived unchanged: estimate noise and signal separately. Noise estimation needs enough repeats to stabilize the standard deviation; signal detection only needs enough contrast to show that the landscape is not flat. Conflating those two jobs into one repeat count either wastes compute or produces a threshold that is too noisy to trust.

The thresholding logic also became more conservative. Mean-based rolling estimates understated the required acceptance margin in high-variance sessions. A rolling maximum behaved better because it explicitly guarded the tail rather than averaging it away. In effect, the loop stopped asking "what is typical noise?" and started asking "what level of noise must this ratchet survive without fooling itself?" That is the statistically correct question for an unattended acceptance gate.

The other major change since the earlier design is that production feedback is no longer treated as a future ambition. It now exists as a real operational loop. Loop D is finished in the sense that matters architecturally: production assemblies emit structured signals, those signals are persisted, aggregated over a rolling window, clustered by low-confidence or decomposed outcomes, and reviewed for recurring gaps, deprecation candidates, and candidate new patterns.

A simplified version of that analysis looks like this:

def cluster_unmatched_assemblies(signals):
    low_confidence = [
        s for s in signals
        if s["assembly_method"] in ("decomposed", "partial")
        or s["confidence"] < 0.70
    ]

    clusters = group_by_shape(
        low_confidence,
        keys=("app_class", "has_commitments", "entity_count_bucket"),
    )

    return [
        summarize_cluster(cluster)
        for cluster in clusters
        if cluster.size >= MIN_CLUSTER_SIZE
    ]

This is a good example of what "finished" means in this context. Loop D does not need to auto-edit prompts to be real. It needs to convert production outcomes into structured evidence that can guide subsequent prompt, pattern, or coverage work.

What Loop D does not do is directly rewrite prompts, update model weights online, or accept structural changes from raw production data. That would collapse the boundary between evidence and acceptance, which is exactly the boundary the rest of the system works to protect. In this context, a production-feedback loop is complete when it turns operational outcomes into governed evidence. It does not need to become live self-modification in order to count as finished.

That distinction is worth stating plainly because it generalizes beyond this system. The hard part of a Karpathy-style loop is not building a mechanism that keeps improving a score. The hard part is building an evaluator and a data lifecycle that resist the loop's pressure to exploit them.

Three conclusions feel solid after a year of working on this.

First, the metric needs to sit as close as possible to the user-facing promise. If the user needs correct intent extraction, then the loop should be judged on correct intent extraction rather than on a downstream proxy that is easier to measure but less faithful.

Second, a scalar objective is necessary but not sufficient. You also need protected cases that preserve coverage shape and stop the loop from trading away important categories for marginal average gains.

Third, hold-out evaluation cannot be treated as an optional external luxury. In a living system, it has to be designed into the corpus lifecycle itself.

The older article leaned more heavily on named statistical tools and algorithms, and that emphasis was justified. The details matter here. Cohen's d is useful because it tells you whether the evaluator can discriminate between meaningfully different prompts. Welch's t-test is useful because it does not assume equal variance between groups. Bigram Jaccard distance turned out to be a better diversity metric than truncated Levenshtein because it measured proposal deltas without collapsing under shared boilerplate. None of those techniques are exotic. Their value is that they force the loop to confront the shape of its own uncertainty instead of pretending that a scalar score is self-authenticating.

Taken together, those changes made the system less elegant on paper and much more trustworthy in practice. The original loop proved that prompt optimization could be mechanized. The revised system is what emerged after asking a more uncomfortable question: not whether the loop could improve, but whether it could improve without teaching itself the wrong lesson.

My current answer is more straightforward: the system improves more reliably when the metric sits close to the user promise, the editable surface stays narrow, anchor cases are protected as invariants, evidence is kept separate from acceptance, and hold-out evaluation is designed into the corpus from the start rather than added later.

That is the version of the Karpathy loop worth keeping.