AI Automations/Quality
A minimal evaluation harness for content automation
Fifty fixtures, three assertions each, and a CI gate — enough to catch a prompt regression before it publishes.
advanced40 min
// read first
Prompt changes are code changes with none of the safety. Without fixtures you find out a prompt regressed when someone reads the output — which, in a publishing pipeline, is after it is live.
This harness is deliberately small. Fifty fixtures is enough signal to catch the regressions that matter and few enough that the suite runs in under a minute.
Structure
evals/
├── fixtures/
│ ├── 001-simple-doc.md
│ ├── 002-tabbed-code.md
│ ├── 003-steps-and-callouts.md
│ └── ...
├── expected/
│ └── 001-simple-doc.json # golden Portable Text
├── assertions.ts
└── run.tsThe three assertions
- Structural — the output parses as valid Portable Text and every block has a
_key. - Semantic — heading levels are monotonic, every image has alt text, every internal reference resolves.
- Golden — for a fixed input, the output matches the committed snapshot, ignoring generated keys.
The third catches drift. The first two catch the failures that would corrupt a document even when the shape is stable.
# whole suite
npx tsx evals/run.ts
# one fixture, verbose
npx tsx evals/run.ts --only 002 --verbose
# accept new snapshots after an intentional change
npx tsx evals/run.ts --updateexport function assertStructure(blocks: PortableTextValue) {
const seen = new Set<string>();
for (const block of blocks) {
if (!block._key) throw new Error(`block missing _key: ${block._type}`);
if (seen.has(block._key)) throw new Error(`duplicate _key ${block._key}`);
seen.add(block._key);
}
}
export function assertHeadingOrder(blocks: PortableTextValue) {
let previous = 1;
for (const block of blocks) {
if (block._type !== "block") continue;
const level = Number(String(block.style).replace("h", ""));
if (!Number.isFinite(level)) continue;
if (level > previous + 1) {
throw new Error(`heading jumped h${previous} → h${level}`);
}
previous = level;
}
}name: evals
on: [pull_request]
jobs:
run:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
cache: npm
- run: npm ci
- run: npx tsx evals/run.ts