AI Engineering Practices

AI Engineering Practices

Evals — How to Actually Test If Your AI Feature Works

Traditional software testing assumes deterministic output — same input, same output, assert equality. LLM output is non-deterministic and graded on quality, not correctness in the boolean sense, so “testing” an AI feature means building an evaluation framework, not a unit test suite in the old sense — though the discipline maps closely if you think of it that way.

The components of a real eval setup:

  1. A golden dataset — a curated, representative set of inputs with known-good expected outputs (or at minimum, known-good criteria for a good output, since exact-match isn’t always meaningful for generative text). This is your regression baseline. Without it, you’re eyeballing outputs and calling it QA.
  2. Scoring methods, layered by cost and precision:
    • Exact match / rule-based checks — for structured outputs (does the JSON parse, does it contain required fields, does a classification label match exactly). Cheap, deterministic, use wherever the task allows it.
    • LLM-as-judge — use a separate model call to score the output against a rubric (e.g., “rate this summary’s faithfulness to the source on 1–5”). This is the most common technique for grading free-text quality at scale, but it has its own reliability limits — judge models have their own biases and inconsistencies, so the rubric needs to be specific and you should periodically spot-check the judge against human ratings.
    • Human review — the ground truth check, used to validate that automated scoring (rule-based or LLM-judge) actually correlates with real quality, and as the final check for high-stakes outputs before automated scoring is trusted to run unsupervised.
  3. Task-specific metrics for RAG specifically — beyond generic quality, you need retrieval-specific scores: faithfulness/groundedness (does the answer actually follow from the retrieved context, or did the model add unsupported claims), context relevance (did retrieval actually surface the right documents), and answer relevance (does the response actually address the query). Frameworks like RAGAS formalize these.
  4. Regression testing on every change — re-run the golden dataset against the eval suite whenever you change a prompt, swap a model, adjust chunking, or the underlying provider silently updates the model. This is the direct analogue of running your CI test suite before merging — except here a “passing” eval might mean “average faithfulness score stayed above threshold,” not a binary pass/fail.

The practical starting point for most teams: before any of the above tooling, write down 20–50 real example inputs your feature needs to handle well, along with what “good” looks like for each, and run them by hand against any prompt/model change. That’s the minimum viable eval — most teams skip this entirely and ship on vibes, which is exactly how silent quality regressions creep in after a “small prompt tweak.”

Guardrails — Input/Output Validation, Content Filtering

Guardrails are the validation/sanitization layer around the model call — the same instinct as input validation and output encoding around any user-facing service, applied to a probabilistic component instead of deterministic code.

Input-side guardrails:

  • Prompt injection defense — detecting attempts to override system instructions embedded in user input or in retrieved/tool-sourced content (e.g., a malicious instruction hidden inside a document your RAG pipeline retrieves). This is a genuinely hard, unsolved problem — defense is layered (input scanning, strict system/user role separation, least-privilege tool permissions) rather than fully solvable with one technique.
  • PII/sensitive data detection — scanning input before it’s sent to a model (especially a third-party API) to catch and redact data that shouldn’t leave your boundary, relevant directly to your healthcare/financial client work.
  • Topic/scope filtering — rejecting or redirecting requests clearly outside the feature’s intended scope before they consume a model call at all.

Output-side guardrails:

  • Content filtering — checking generated output against safety categories (harmful content, toxicity) before it reaches the user. Bedrock Guardrails, Azure Content Safety, and similar managed offerings handle this without custom infra.
  • Schema/format validation — for structured output (JSON, specific fields), validate against a schema before passing downstream; reject and retry (or fall back) if validation fails, rather than letting malformed output propagate into a system expecting a strict contract.
  • Factuality/groundedness checks — for RAG-grounded answers, a guardrail can check whether claims in the output are actually supported by the retrieved context before the response goes out, catching hallucination at the output boundary rather than relying on the model to self-police.
  • Business-rule enforcement — guardrails that aren’t about AI safety at all but about your domain (e.g., never let the model autonomously confirm a financial transaction above a threshold without a human-approval step). This is where AI output boundaries intersect with your existing authorization/approval workflow logic — the guardrail isn’t an AI concept here, it’s just your existing business rule applied at a new integration point.

The architectural principle: guardrails belong in the application/gateway layer, not as a hope that the model “behaves.” Treat every model output the way you’d treat any external, semi-trusted input — validate before it’s trusted, especially before it triggers a side effect (a database write, an email send, a transaction).

Cost Optimization — Token Economics, Model Selection by Task

The token economics, concretely: providers price input and output tokens separately, with output tokens typically costing more than input (generation is more expensive than processing). This means prompt design has direct cost implications — a bloated system prompt or unnecessarily verbose RAG context gets paid on every single call, not once.

Model selection by task — the core lever. The single biggest cost optimization most teams skip: not every task needs your most capable (and most expensive) model.

  • Task complexity triage — classify your use cases by how much reasoning they actually require. Simple classification, extraction, or formatting tasks often perform nearly as well on a small/fast/cheap model as on a frontier model, while complex multi-step reasoning, ambiguous judgment calls, or high-stakes outputs justify the expensive model. Routing tasks to the cheapest model that meets your quality bar — rather than defaulting everything to the flagship model — is usually the highest-leverage cost lever available, bigger than any caching or prompt-trimming optimization.
  • Cascade/escalation pattern — try the cheap/fast model first; only escalate to the expensive model if the cheap model’s output fails a confidence check or validation rule. This gets you the cost profile of the cheap model on the majority of “easy” cases while still catching the harder ones.
  • Prompt compression — trim unnecessary verbosity from system prompts and few-shot examples; use prompt caching (covered earlier) for repeated static context so you’re not paying full input-token cost for the same boilerplate on every call.
  • Output length control — explicitly bounding expected output length (via instructions or max_tokens) where verbose responses aren’t adding value, since output tokens are typically the more expensive side of the ledger.
  • Batching for non-real-time workloads — most providers offer a batch API at a meaningful discount (often ~50%) for workloads that don’t need synchronous response, trading latency for cost on anything that can tolerate async processing (bulk classification, overnight summarization jobs).

The framing that’ll resonate given your AWS background: this is the same reserved-vs-on-demand-vs-spot triage you already do for compute, just applied to model tier selection — match the resource tier to the actual workload requirement instead of over-provisioning the most capable (most expensive) option by default.

Versioning Prompts Like Code

Prompts are production logic, not configuration trivia — they directly determine application behavior — so they need the same discipline you’d apply to any code change, and treating them as “just text in a string” is how silent regressions creep into production.

What this looks like in practice:

  • Version control, literally — prompts live in your repo (not hardcoded inline scattered across services, not edited via some separate UI with no history), so every change is a diff, has a commit history, and can be reverted.
  • Decoupling prompt content from application code — store prompts as separate versioned artifacts (templates, files, or a dedicated prompt-management service) rather than embedded as string literals deep inside business logic. This lets you change/test a prompt without a full code deploy, and lets non-engineers (if relevant) iterate on prompt wording without touching application code.
  • Semantic versioning for prompts — tag prompt versions, and pin which version a given deployment/environment uses, the same way you’d pin a dependency version. This matters specifically because prompt changes can silently shift output quality in ways a code reviewer reading the diff might not catch just by reading the words.
  • Eval-gated changes — every prompt version change runs against your eval suite (above) before being promoted, the same way a code change runs against CI before merge. A prompt diff that “looks like an improvement” can regress quality on edge cases the golden dataset would catch and a manual read-through wouldn’t.
  • A/B and gradual rollout — for changes where eval scores are close or ambiguous, route a percentage of real traffic to the new prompt version and compare real-world outcome metrics (user satisfaction signals, task completion rate, escalation rate) before full rollout — same canary deployment pattern you’d use for any risky code change, applied to prompt changes instead of binary deploys.
  • Tooling: dedicated prompt-management platforms (LangSmith, PromptLayer, Langfuse) exist specifically to give prompts this version-history-plus-eval-plus-rollout workflow without building it from scratch; for simpler setups, a versioned prompt-template directory in your existing repo with eval scripts in CI gets you most of the value.

The mindset shift that matters most: a prompt change is a behavior change to a production system, full stop — it deserves the same review, testing, and rollback discipline as a code change, not the “just edit the string and redeploy” treatment that’s tempting because prompts look like plain text rather than logic.