Evals before prompts: measure or do not improve
A golden set of 30 to 50 cases, a scoring harness, and a CI gate. That is how you change a prompt without praying.
The prompt that “feels better”
Changing a prompt and sensing a better answer is the most expensive way to fool yourself in LLM engineering. It fixes one case and breaks three you never checked. The way out is treating prompts as tested code: a fixed evaluation set that scores every change before deployment.
The golden set
Collect 30 to 50 real cases from your domain (not invented ones): hard, ambiguous and typical inputs, each with the expected answer or written acceptance criteria. Grade them by hand once to fix the baseline. This set is your executable specification; everything else is decoration.
The minimal harness
One script that, on any prompt, model or temperature change, runs the full set and reports: pass rate, total cost and p95 latency. Store every run with date and commit to compare over time. You do not need an eval platform to start; you need the habit of running the script before every deploy.
The CI gate
Once the harness exists, the rule is mechanical: no prompt change reaches production if it drops the golden-set pass rate or pushes cost past the lesson 1 budget. Regressions stop being Monday scares and become red builds nobody merges.
Version your prompts
Every prompt carries a version, a date and its last run score. When a provider updates the model underneath you (it happens every few weeks), you rerun the set and know within minutes whether anything moved, instead of learning it from user complaints.
Exercise
Write 10 cases from your domain today, with expected answers. Tomorrow, 10 more. In a week you own a set worth more than any prompting framework.
That closes module 1: budgets that pick models, secrets that never touch your server, and measured changes instead of hunches. Modules 2 and 3 (RAG and production) open with the waitlist.