Someone on the team fixed a typo in the support-ticket classifier’s prompt directly through an admin panel field on a Thursday afternoon, the kind of one-line change that feels too small to bother anyone about. By Monday, ticket misrouting had crept up noticeably, nobody had deployed anything, and it took most of a day to even locate the cause, because there was no diff to look at — the prompt lived in a database column, the old version was gone the moment the new one was saved, and the only record that anything had changed at all was a vague memory of “I think I touched that Thursday.”
We’d applied every discipline we had to the code around our AI features — code review, CI, a real eval harness — and left the prompts themselves, arguably the single highest-leverage piece of text in the whole system, living outside all of it. This is why that gap opens up so easily, what we changed, and where we deliberately left a faster, less formal path open for the cases that genuinely need one.
Prompts drift out of version control for a reasonable-sounding reason
Nobody decided prompts shouldn’t be reviewed. It happens by accretion: a prompt starts as a string literal in code, someone adds an admin panel so non-engineers can tweak the tone without waiting on a deploy, that panel writes straight to a database field for convenience, and within a few months the “real” prompt — the one actually running in production — is whatever’s in that field, disconnected from the repository entirely. Every step along that path was individually reasonable. The end state is a piece of application logic with no code review, no diff history, and no CI gate, sitting in the one place nobody thinks to look for a regression because it doesn’t feel like a deploy.
Treating a prompt like the code it actually is
The fix is mechanically simple and, in hindsight, obvious: prompts live as versioned files in the same repository as the code that calls them, get reviewed the same way, and only reach production through the same deploy pipeline as everything else.
prompts/
ticket-classifier/
v-2026-06-15.txt
v-2026-09-01.txt ← current, referenced by CURRENT symlink or a config constant
autonomy-reply/
v-2026-08-10.txt
// loaded explicitly, never edited in place at runtime
const CLASSIFIER_PROMPT = await readPromptFile('ticket-classifier', 'v-2026-09-01');
This alone gets you the thing that was actually missing during the incident above: git log -p prompts/ticket-classifier/ shows exactly what changed, when, and who changed it, the same as any other file. The Thursday-afternoon fix that took a day to find would have taken two minutes to spot with a working diff in hand.
A prompt review checks different things than a code review
Reviewing a prompt change isn’t the same exercise as reviewing code, and a reviewer who doesn’t know that will rubber-stamp changes that read as “just wording” but actually shift behavior. We run three checks on every prompt pull request, only one of which is a human reading the diff.
# .github/workflows/prompt-change-check.yml
on:
pull_request:
paths:
- 'prompts/**'
jobs:
eval-gate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run eval suite against the changed prompt
run: |
node scripts/run-evals.js \
--prompt-dir=prompts/ \
--suite=all \
--fail-below=0.92
- name: Diff pass/fail against main
run: node scripts/diff-eval-results.js main-results.json pr-results.json
The eval gate is the same harness we already built for catching model-version drift, pointed at the prompt diff instead — reused wholesale, because a prompt change and a model-version change are both just “the thing generating output changed” from the eval suite’s point of view. The gate catches measurable regressions. It does not catch scope creep — a prompt that quietly grows a new instruction nobody asked for, or drifts in tone — which is why a human still reads every prompt diff before merge, the same as they would for a code change with no test coverage on the specific thing that matters.
A prompt is the part of the system most likely to be edited by someone who doesn’t think of it as code. That’s exactly why it needs the same review discipline as code — the informal edit is precisely the one with no safety net underneath it.
Rollback needs a name, not just a memory of “the old one”
The other thing a database field can’t give you is a clean rollback. With versioned files, reverting a bad prompt change is the same one-line operation as reverting a bad code change — point the config constant back at the previous file, or revert the commit — instead of someone trying to reconstruct from memory or a Slack message what the wording used to be.
# Rolling back is just changing which file the config points to
git revert <bad-prompt-commit-sha>
# or, for a same-day rollback without a full revert:
export CLASSIFIER_PROMPT_VERSION=v-2026-06-15
Keeping a fast path for the people who genuinely need one
The honest trade-off is that this is slower than editing a database field, and there’s a real case for a non-engineer needing to adjust user-facing copy — a support team lead fixing an awkward phrasing in an auto-reply template — without waiting on a full PR cycle. We didn’t try to eliminate that need; we scoped it instead. Wording inside clearly marked template slots (a greeting, a sign-off, a tone adjective) stays editable through the admin panel, versioned automatically on save with a visible history and one-click revert built into that same panel. The parts of the prompt that define the model’s actual task, constraints, and output schema are not exposed there at all — those only change through the reviewed, eval-gated path.
Key takeaways
- Prompts drift out of version control gradually and reasonably — an admin panel added for convenience becomes the de facto production prompt with no diff history, code review, or CI gate behind it.
- Store prompts as versioned files in the same repository as the code that calls them, loaded explicitly rather than edited in place at runtime — this alone restores diff history and a working rollback.
- Reuse your eval harness as a CI gate on prompt pull requests — a prompt change and a model-version change are both “the output-generating thing changed” from the harness’s point of view.
- An eval gate catches measurable regressions but not scope creep or tone drift — keep a human reviewer reading every prompt diff, the same as you would for a code change with weak test coverage.
- Rollback should be a one-line operation (revert a commit, or point a version constant back) — not someone reconstructing the previous wording from memory or a chat message.
- Don’t eliminate a legitimate need for fast, non-engineer copy edits — scope it narrowly to clearly marked template slots that don’t touch the model’s actual task definition or output schema, and version those edits automatically too.
Frequently asked questions
Doesn’t putting prompts behind a PR process slow down legitimate quick fixes?
For changes to the model’s task definition, yes, and that’s deliberate — the whole point is that those changes deserve the same friction as a code change with production impact. For copy-only tweaks, we solved the speed problem separately with a scoped, versioned template-slot editor rather than removing review from the parts that actually shift behavior.
How do you handle prompts that need to differ per customer or per environment?
Per-environment prompt files (staging vs. production) go through the same versioning and review — they’re just different files in the same directory structure. Per-customer variation is scoped the same way as the copy-editing case: a base reviewed prompt with clearly bounded customer-specific slots, not a fully separate unreviewed prompt per customer.
What if the eval suite doesn’t have coverage for the specific thing a prompt change affects?
Then the eval gate will pass a change it shouldn’t, which is exactly the same failure mode an undertested code change has — the fix is adding eval cases for the gap once you find it, not distrusting the whole gate. This is also a reason the human review step never goes away, whatever the eval suite’s coverage looks like.
Is a full git-based workflow overkill for a small project with one or two prompts?
The core habit — prompts as files, not database fields, so you get a diff — is worth it at any scale and costs almost nothing to set up. The CI eval gate is worth adding once you actually have an eval suite to run; until then, file-based versioning alone still would have turned our day-long incident into a two-minute diff read.
How do you decide what counts as a “template slot” versus part of the reviewed prompt?
Anything that changes what the model is asked to do, what constraints it operates under, or what shape its output must take stays in the reviewed path. Anything that only changes the words a human reads — greetings, sign-offs, a specific adjective in a fixed sentence structure — is a reasonable candidate for a scoped, versioned slot. When in doubt, we default to the reviewed path; the cost of an extra review is much lower than the cost of another Thursday-afternoon incident.
