Module 10 — Evaluation Observability & Hallucination Guardrails

Module 10 — Evaluation Observability & Hallucination Guardrails

1. AI Model Evaluation Utilities — Protecting Against Hallucinated Responses

The core utilities Spring AI provides at the core-framework level (distinct from Module 9’s Agent Judge project, which operates at the agentic-goal level — this section is about evaluating a single model response, not a multi-step agent run):

@Autowired
private ChatModel chatModel;

EvaluationRequest evaluationRequest = new EvaluationRequest(
    userQuestion,
    retrievedContext,      // the documents actually retrieved/used as grounding (RAG context from Module 7)
    generatedResponse
);

RelevancyEvaluator relevancyEvaluator = new RelevancyEvaluator(chatClient);
EvaluationResponse result = relevancyEvaluator.evaluate(evaluationRequest);

if (!result.isPass()) {
    log.warn("Response failed relevancy check: {}", result.getFeedback());
}

FactCheckingEvaluator — specifically targets hallucination by checking whether a generated response is actually supported by the provided context, rather than checking general response quality:

FactCheckingEvaluator factChecker = new FactCheckingEvaluator(chatClient);
EvaluationResponse factCheck = factChecker.evaluate(
    new EvaluationRequest(userQuestion, retrievedContext, generatedResponse)
);

The architecturally important point, directly extending Module 7’s RAG-grounding discussion: these evaluators are themselves model calls — FactCheckingEvaluator and RelevancyEvaluator use a ChatClient under the hood to judge another ChatClient‘s output. This is the LLM-as-judge pattern from Module 9, applied here at the single-response level rather than the agentic-goal level, and it inherits the exact same caveat already flagged in Module 9: a model judging a model shares structural blind spots with the thing it’s judging, so this catches some hallucinations (especially clear contradictions between context and response) but isn’t a complete safety net.

Talking point: “I treat FactCheckingEvaluator as a useful, cheap first-pass filter — it’s genuinely good at catching the case where a generated response directly contradicts the retrieved context, which is a common and detectable hallucination pattern in RAG systems specifically. But it’s not a general hallucination solution, because it’s still a probabilistic model checking another probabilistic model’s work. For anything high-stakes, I’d pair this with deterministic checks where they’re available — exactly the Judge-framework hierarchy from Module 9: deterministic first, command-based second, LLM-based evaluation as the layer that catches what the cheaper checks can’t, not as the sole line of defense.”

Where this fits relative to Module 7’s RAG discussion specifically: Module 7 noted that a poorly-grounded RAG system can produce more confidently wrong answers than an ungrounded one, because injected context lends false authority. FactCheckingEvaluator is the direct mitigation for that exact failure mode — explicitly checking “does the generated answer actually follow from what was retrieved,” rather than trusting that grounding alone guarantees faithfulness.

2. Observability — Micrometer-Based Tracing, Span Hierarchy

The foundation: Spring AI’s observability integrates with Micrometer, the same metrics/tracing abstraction underlying observability across the broader Spring ecosystem (Spring Boot Actuator, Spring Cloud Sleuth’s successor) — meaning AI-specific spans compose into the same distributed tracing picture as the rest of a Spring application’s existing observability stack, rather than requiring a separate, AI-specific tracing tool.

// Auto-instrumented when the observability starter is present — 
// every ChatClient call produces spans without manual instrumentation
@Bean
ObservationRegistry observationRegistry() {
    return ObservationRegistry.create();
}

A single ChatClient call naturally produces a span hierarchy, not a flat trace — the outer span for the overall request, child spans for each advisor in the chain (Module 5), and further child spans for the actual model call, any tool invocations (Module 4), and any vector store queries (Module 7). This hierarchy is what lets you see, in a single trace, exactly where time and cost were spent across a complex chain — was it the RAG retrieval, the tool call, or the model generation itself.

Why this area has had real, recent bug fixes worth knowing about specifically — and worth treating as a pattern, not a list of trivia: the hardest part of getting span hierarchy right is maintaining correct parent-child relationships across asynchronous and streaming boundaries. A synchronous call naturally preserves call-stack-based context propagation; a streaming response, where chunks arrive over time via a reactive pipeline, has to carry tracing context forward explicitly, and several real fixes in the release history addressed exactly this:

  • Span hierarchy bugs in streaming paths across chat models — parent spans losing their relationship to child spans specifically when responses were being streamed rather than returned synchronously.
  • Parent span being brought into ThreadLocals to maintain Observation/span parity — a fix addressing the fact that reactive/streaming execution doesn’t run on a single thread the way synchronous code does, so naive ThreadLocal-based context propagation (which works fine for synchronous call stacks) breaks across the thread-hopping that’s normal in reactive streaming pipelines.
  • Custom observation missing on advisors executed after the tool-calling advisor — a gap where an advisor positioned after tool-calling in the chain wasn’t getting its own observation/span recorded correctly, meaning that portion of the chain was effectively invisible in traces even though it was executing.

Talking point that ties this directly to your Java/Spring background and to the recurring theme of this whole module: “This is genuinely the same class of problem as ThreadLocal-based context propagation breaking across @Async boundaries — a recurring theme flagged repeatedly in the Java/Spring track’s coverage of the AOP self-invocation pitfall, just showing up here in the tracing infrastructure itself rather than in business logic. Reactive, streaming execution doesn’t preserve the implicit thread-bound context that synchronous code can lean on, and every layer that assumes it does — security context, transaction context, or here, tracing span hierarchy — has to be deliberately re-engineered to propagate explicitly instead. I’d treat ‘does this observability setup correctly show parent-child span relationships specifically for streaming and tool-calling paths’ as something to verify directly with a real trace inspection before trusting a production dashboard’s latency breakdown, rather than assuming span hierarchy ‘just works’ the same way it does for a simple synchronous call.”

3. Usage Metrics — Unified Cache Usage Tracking Across Providers

The Usage interface — a portable abstraction over token consumption and cost-relevant metadata, returned as part of ChatResponseMetadata on every response, regardless of provider:

ChatResponse response = chatClient.prompt()
    .user("Summarize this document.")
    .call()
    .chatResponse();

Usage usage = response.getMetadata().getUsage();
System.out.println("Prompt tokens: " + usage.getPromptTokens());
System.out.println("Completion tokens: " + usage.getCompletionTokens());
System.out.println("Total tokens: " + usage.getTotalTokens());

The recently added unified cache usage metrics specifically: several providers (Anthropic prompt caching, OpenAI’s cached input tokens, and similar mechanisms elsewhere) let you reuse previously-processed prompt content at a steep discount — caching the system prompt, a large RAG context block, or repeated few-shot examples across calls. The framework now surfaces cache hit/miss token counts through the same unified Usage interface, rather than requiring provider-specific code to inspect cache behavior.

Usage usage = response.getMetadata().getUsage();
// Unified access regardless of provider — no need to cast to a provider-specific 
// usage type to see cache-related token counts
Integer cachedTokens = usage.getNativeUsage() instanceof CacheAwareUsage cacheUsage 
    ? cacheUsage.getCacheReadTokens() 
    : null;

Why this matters as a cost-engineering tool, directly connecting to Module 6’s memory-windowing cost discussion: caching is one of the most effective levers for controlling the compounding cost growth problem flagged in Module 6 — if a long system prompt or a large RAG context block is being resent on every turn of a conversation, prompt caching can make repeated portions of that payload dramatically cheaper on subsequent calls, if the application is structured to take advantage of it (stable content placed where the provider’s caching mechanism expects it, e.g., earlier and unchanged across calls).

Talking point: “Having unified cache usage metrics matters for the exact same reason unified Usage mattered before caching was added — without it, optimizing for cache hits would require provider-specific instrumentation, which discourages teams from even checking whether their caching strategy is working. With it, I can build one piece of cost-monitoring logic — flag when cache read tokens are unexpectedly low for a conversation that should be hitting cache — and have it apply across whichever provider the application happens to be using, the same portability principle that’s run through every layer of this framework.”

4. Designing Evaluation Harnesses for Non-Deterministic Output

This closes the loop on a tension that’s been implicit since Module 1 and explicit since Module 9: traditional software testing assumes determinism — given the same input, you assert the same output. An LLM call, by design, doesn’t guarantee that. A harness for non-deterministic systems has to be built around a fundamentally different testing philosophy.

The layered approach worth presenting as the actual answer to this problem, synthesizing everything from Modules 4, 9, and this module:

Layer What it tests Mechanism
Structural/schema tests Did the output have the right shape? Deterministic — does it deserialize into the expected POJO (Module 3), does a tool call have valid arguments matching its schema
Golden-set regression tests For a fixed set of representative inputs, does output quality stay above a threshold over time? A mix of deterministic checks (keyword/fact presence) and evaluator-based scoring (Module 9’s Judge hierarchy), run on every model/prompt/dependency change
Evaluator-based assertions Is the content acceptable, even though exact wording varies run to run? RelevancyEvaluator/FactCheckingEvaluator (this module) or custom LLM-as-judge checks (Module 9), asserting a pass/fail or score threshold rather than exact string equality
Statistical/sampling-based evaluation Across N repeated runs of the same input, what’s the distribution of outcomes — not just one sample? Direct application of the Statistics module’s distribution thinking: a single passing run tells you little about reliability; running the same prompt N times and measuring the pass rate is what actually characterizes non-deterministic behavior
Production monitoring as an extension of testing Is real, live traffic still behaving acceptably? Sampling live responses through the same evaluators used in pre-release testing — testing doesn’t stop at deployment for a system this variable

Talking point that explicitly connects this back to the A/B Testing and Statistics modules from your Data Analyst track, which is genuinely the strongest possible framing for this point: “Building an evaluation harness for non-deterministic output is, almost exactly, applying the experimentation-design discipline from the Data Analyst track to software testing instead of product metrics. A single test run passing tells you as little about a non-deterministic system’s real reliability as a single user’s experience tells you about an A/B test’s true effect — you need a sample size, you need to think about the pass-rate as a distribution rather than a binary, and you need to be honest about statistical power: if you only run a golden-set test once per release, you genuinely don’t have the resolution to detect anything but a very large quality regression, the exact same MDE-and-sample-size reasoning from the experimentation module, just applied to test suites instead of product experiments.”

A concrete pattern that operationalizes this — running the same evaluation prompt multiple times and tracking the pass rate rather than a single pass/fail:

int passCount = 0;
int totalRuns = 10;
for (int i = 0; i < totalRuns; i++) {
    ChatResponse response = chatClient.prompt().user(testPrompt).call().chatResponse();
    EvaluationResponse eval = factChecker.evaluate(
        new EvaluationRequest(testPrompt, context, response.getResult().getOutput().getText())
    );
    if (eval.isPass()) passCount++;
}
double passRate = (double) passCount / totalRuns;
assertThat(passRate).isGreaterThanOrEqualTo(0.9);  // tolerate some variance, not zero

Closing talking point that ties the entire module together: “Evaluation, observability, and usage metrics are three different lenses on the same underlying problem — how do you trust and control a system that’s fundamentally probabilistic. Evaluation utilities and the Judge framework answer ‘is the output actually good.’ Observability and span hierarchy answer ‘where did the time and the failure actually happen inside a complex chain.’ Usage metrics answer ‘what is this actually costing, and is the cost behaving the way I expect.’ None of these are optional add-ons to an AI feature — they’re the direct analogue of monitoring, logging, and testing for any production system, just adapted for a component that doesn’t behave deterministically, which is exactly why treating ‘it ran without throwing an exception’ as sufficient validation is so much more dangerous here than in traditional software.”