Module 3 — Structured Outputs & Multimodality

 

Module 3 — Structured Outputs & Multimodality

1. Mapping Model Output to POJOs (Structured Outputs)

The core problem this solves: LLMs natively produce unstructured text. Any real application almost always wants a typed object back — not a string to regex-parse. Spring AI’s Structured Outputs pipeline handles the full round trip: it instructs the model (via prompt engineering and, where supported, native schema constraints) to produce JSON matching a target shape, then deserializes that JSON into your POJO.

record ArchitectureRecommendation(
    String pattern,
    String justification,
    List<String> tradeoffs,
    int confidenceScore
) {}

ArchitectureRecommendation rec = chatClient.prompt()
    .user("""
        Given a system handling 50k orders/day with bursty traffic,
        recommend an architecture pattern.
        """)
    .call()
    .entity(ArchitectureRecommendation.class);

System.out.println(rec.pattern());        // typed access, no manual parsing
System.out.println(rec.tradeoffs());      // a real List<String>, not a string to split

What’s actually happening under the hood — worth knowing because it explains both the convenience and the failure modes:

  1. Spring AI generates a JSON Schema from your target type (via reflection over the record/class).
  2. It injects formatting instructions (and, on providers that support it, a native structured-output/JSON-mode constraint) into the prompt so the model is steered toward emitting conforming JSON.
  3. The response is parsed (typically via Jackson — relevant given the Jackson 2→3 migration flagged in Module 1’s versioning section) into your target type.

Native structured output vs. prompt-based structured output — a distinction worth naming explicitly:

Prompt-based (older/fallback approach) Native structured output
Mechanism Instructions injected into the prompt asking the model to “respond only in JSON matching this schema” Provider-level API constraint (e.g., OpenAI’s JSON mode/structured outputs, similar mechanisms across providers) that forces the model’s token generation to conform
Reliability The model usually complies but can occasionally drift — add preamble text, miss a field, produce slightly malformed JSON Strongly enforced at the generation level — much higher reliability, since malformed output is largely prevented at the source rather than caught after the fact
Provider support Universal — works with any model that can follow instructions Provider/model-dependent — not every provider or every model version supports native constrained generation

Talking point: “Whenever the provider supports native structured outputs, I prefer it over prompt-based JSON instructions, because prompt-based formatting is still fundamentally probabilistic — the model is just very likely to comply, not guaranteed to. Native structured outputs push the guarantee down to the token-generation level itself, which matters a lot if a malformed response would break a downstream pipeline rather than just degrade a chat UI gracefully. For anything feeding into automated processing rather than direct human reading, I treat native structured-output support as close to a requirement, not a nice-to-have.”

A genuinely important practical caution worth raising: even with native structured outputs, deserialization can still fail — a model can produce technically valid JSON that doesn’t satisfy business-level constraints your POJO’s schema doesn’t capture (e.g., a numeric field within a valid JSON number range but outside a meaningful business range). Structured output correctness at the JSON-shape level is not the same as correctness at the business-logic level — this connects directly back to the AI Model Evaluation utilities from the table of contents’ Module 10; structured output parsing succeeding doesn’t mean the content is right, just that the shape is right.

2. Custom StructuredOutputConverter Implementations

The built-in .entity(Class) path covers the common case, but StructuredOutputConverter is the extension point when you need custom parsing logic — non-JSON output formats, legacy system integration, or post-processing beyond what reflection-based deserialization can express.

public interface StructuredOutputConverter<T> extends Converter<String, T>, FormatProvider {
    // Converter<String, T>.convert(String) — parses the raw model output into T
    // FormatProvider.getFormat() — returns formatting instructions injected into the prompt
}

A custom implementation — e.g., parsing a pipe-delimited legacy format some downstream system requires, rather than JSON:

public class PipeDelimitedConverter implements StructuredOutputConverter<List<String>> {

    @Override
    public String getFormat() {
        return """
            Respond with values separated by the pipe character, e.g.:
            value1|value2|value3
            Do not include any other text or explanation.
            """;
    }

    @Override
    public List<String> convert(String source) {
        if (source == null || source.isBlank()) {
            throw new IllegalArgumentException("Empty model response — cannot parse pipe-delimited output");
        }
        return Arrays.stream(source.trim().split("\\|"))
                      .map(String::trim)
                      .toList();
    }
}

List<String> result = chatClient.prompt()
    .user("List the top 3 risks of this migration plan.")
    .call()
    .responseEntity(new PipeDelimitedConverter())
    .getEntity();

Why this extension point matters architecturally: it’s the same “interface as extension seam” pattern that runs through Spring generally — StructuredOutputConverter lets custom output handling plug into the same ChatClient fluent chain as the built-in JSON path, rather than requiring a bypass of the framework’s abstractions whenever a non-standard format shows up.

A robust custom converter should handle the failure modes the built-in JSON path handles implicitly:

  • Malformed/partial output — the model truncated mid-response (common with token limits) — fail clearly rather than silently returning a partially-parsed result.
  • Defensive validation, not just parsing — converting successfully doesn’t mean the content is trustworthy; a converter is a good place to enforce structural invariants (e.g., “exactly 3 items expected”) before handing control back to application code.

Talking point: “I treat a custom StructuredOutputConverter the same way I’d treat a custom Jackson deserializer — it needs to fail loudly and specifically on malformed input, not swallow an error and return a default or null that silently corrupts something three layers downstream. Given that the input here is model output rather than a well-formed system-to-system payload, I assume malformed output will happen eventually and design the converter’s error handling for that reality rather than the happy path alone.”

3. Multimodal Input/Output — Image, Audio, Document Inputs

Image input — passing visual content alongside text, for providers/models supporting vision:

var imageResource = new ClassPathResource("architecture-diagram.png");

String analysis = chatClient.prompt()
    .user(u -> u.text("What architectural pattern does this diagram show? Identify any obvious issues.")
                .media(MimeTypeUtils.IMAGE_PNG, imageResource))
    .call()
    .content();

Image output (generation) — via the ImageModel/ImageClient abstraction, parallel in structure to ChatModel/ChatClient:

ImageResponse imageResponse = imageModel.call(
    new ImagePrompt("A minimalist navy and teal logo for a software architecture brand",
        OpenAiImageOptions.builder().withModel("dall-e-3").withHeight(1024).withWidth(1024).build())
);
String imageUrl = imageResponse.getResult().getOutput().getUrl();

Audio — transcription (speech-to-text):

AudioTranscriptionResponse transcription = transcriptionModel.call(
    new AudioTranscriptionPrompt(new ClassPathResource("sermon-recording.mp3"))
);
String transcript = transcription.getResult().getOutput();

This is directly relevant to your Tech & Faith channel’s bilingual Tamil-English content workflow — transcribing recorded sermon/teaching audio programmatically rather than manually, as a building block for generating searchable transcripts or repurposed written content from spoken material.

Audio — text-to-speech (TTS), now available through multiple provider integrations including the recently added ElevenLabs and OpenAI’s own Audio models:

SpeechResponse speech = speechModel.call(
    new SpeechPrompt("Welcome back to Tech and Faith.",
        OpenAiAudioSpeechOptions.builder().voice("alloy").responseFormat(Mp3).build())
);
byte[] audioBytes = speech.getResult().getOutput();

Document inputs — feeding PDFs/documents directly as model input (where the provider supports native document understanding), distinct from the ETL/RAG ingestion pipeline (Module 7) which is about indexing documents for retrieval rather than passing one directly into a single prompt:

chatClient.prompt()
    .user(u -> u.text("Summarize the key risks in this contract.")
                .media(MimeTypeUtils.APPLICATION_PDF, new ClassPathResource("contract.pdf")))
    .call()
    .content();

Talking point connecting multimodality back to the rest of the curriculum: “The same ChatClient fluent chain handles text, image, and document input — .media() is just another input attached to the user message, not a separate API to learn. That consistency is the actual point of the abstraction: a developer who’s learned the text-prompt pattern from Module 2 doesn’t need a fundamentally different mental model to add multimodal input, just an additional parameter on the same builder.”

4. Handling Provider-Specific Model Deprecations — Designing for Model Churn

The concrete example from the release history: Mistral’s Pixtral 12B model was removed, and Pixtral Large was deprecated in favor of newer Mistral vision models — a real, recent instance of the churn every team building on any of these providers will face repeatedly, not a hypothetical.

Why this is an architectural problem, not just a config update: if model names are hardcoded as string literals scattered through application code ("pixtral-12b-2409" typed directly into a dozen call sites), a provider deprecation becomes a multi-file search-and-replace under time pressure. If model selection is centralized and externalized, it becomes a one-line config change.

The pattern worth establishing from day one:

# Externalize model identifiers — never hardcode in application code
spring.ai.mistralai.chat.options.model=${MISTRAL_MODEL:mistral-large-latest}

// Or, centralized in a single configuration class rather than scattered string literals
@ConfigurationProperties(prefix = "app.ai.models")
public record ModelConfig(String visionModel, String chatModel, String embeddingModel) {}

A layered defensive strategy worth presenting as a checklist:

  1. Externalize all model identifiers to configuration, never inline string literals in business logic.
  2. Subscribe to/regularly check provider release notes and Spring AI’s own release notes — deprecation announcements are usually given a runway, but only if someone’s actually watching for them.
  3. Abstract model selection behind your own service layer, not just the model call — e.g., a VisionModelSelector that knows “use the current recommended vision model,” so a deprecation only requires updating that one piece of central logic, regardless of how many features used vision capability.
  4. Treat “latest” aliases carefully — providers sometimes offer a -latest style alias (as in the Mistral example above) that auto-rolls to the newest recommended model; convenient for staying current automatically, but it trades determinism for currency — a prompt that worked well against one model version might behave subtly differently after a silent -latest rollover. Pin to an explicit version for anything where reproducibility matters (e.g., regression-tested prompts, compliance-sensitive output), and use -latest-style aliases only where staying current matters more than exact reproducibility.
  5. Build an evaluation suite (Module 10 territory) that runs against model changes, not just code changes — the most dangerous version of this churn is a model swap that doesn’t break compilation but silently degrades output quality, which only a behavioral test, not a unit test, would catch.

Talking point that ties this back to your architecture background directly: “This is the same discipline as designing against any external dependency that you don’t control the lifecycle of — exactly the posture I’d take with a third-party API or a cloud-managed service version. AWS deprecates instance types and SDK versions on its own schedule too; the mitigation is identical in shape: externalize the dependency identifier, centralize the selection logic in one place, and have an automated way to detect behavioral regressions before a forced migration becomes an emergency instead of a planned upgrade.”

A closing synthesis point for the module: “Structured outputs, custom converters, and model-churn resilience are really the same underlying lesson from three angles — don’t let probabilistic model behavior leak unguarded into the rest of the system. Structured outputs constrain the shape of output; converters add a validation seam before untrusted parsed data reaches business logic; and externalized model selection means a provider’s own lifecycle decisions don’t directly destabilize application code. Each is a different layer of the same boundary-drawing discipline.”