Module 2 — ChatModel & ChatClient Fundamentals

 

Module 2 — ChatModel & ChatClient Fundamentals

1. ChatModel Interface — The Low-Level, Portable Abstraction

ChatModel is the contract every provider integration implements — deliberately minimal, so that portability doesn’t get compromised by trying to expose every provider’s unique surface area at this layer.

public interface ChatModel extends Model<Prompt, ChatResponse> {
    ChatResponse call(Prompt prompt);
    default String call(String message) { ... }   // convenience overload
}

// Streaming variant
public interface StreamingChatModel extends StreamingModel<Prompt, ChatResponse> {
    Flux<ChatResponse> stream(Prompt prompt);
}

Provider implementations — OpenAiChatModel, AnthropicChatModel, BedrockProxyChatModel, OllamaChatModel, and so on — each wraps that provider’s actual HTTP client/SDK underneath, but all expose the same call(Prompt) → ChatResponse contract.

@Autowired
private ChatModel chatModel;   // injected concretely as whatever provider is on the classpath/configured

Prompt prompt = new Prompt("Explain CAP theorem in two sentences.");
ChatResponse response = chatModel.call(prompt);

String text = response.getResult().getOutput().getText();
ChatResponseMetadata metadata = response.getMetadata();   // usage, model name, rate-limit info, etc.

Why this layer exists at all, given that ChatClient is the recommended app-facing API: ChatModel is the extension point — anyone building a new provider integration, a custom Advisor, or low-level infrastructure (e.g., a custom load-balancer across multiple model deployments) works against this interface directly. Application code calling business logic should almost always prefer ChatClient; ChatModel is the substrate ChatClient is built on top of.

Talking point: “ChatModel is the layer I’d reach for if I were writing infrastructure — a custom advisor, a provider failover mechanism, a testing mock — rather than feature code. It’s the same relationship as JdbcTemplate to EntityManager in the Java/Spring world: one is the low-level, portable primitive; the other is the ergonomic, fluent API most application code should actually be written against. Knowing both layers exist, and which one to reach for, is the signal of someone who understands the framework’s architecture rather than just copy-pasting from a tutorial.”

2. ChatClient — The Fluent, App-Facing API

ChatClient is deliberately modeled on WebClient/RestClient‘s builder-chain ergonomics — anyone fluent in modern Spring HTTP clients should find this immediately familiar.

@Autowired
private ChatClient chatClient;   // auto-configured from whatever ChatModel is on the classpath

// Simple call
String answer = chatClient.prompt()
    .user("What's the difference between optimistic and pessimistic locking?")
    .call()
    .content();

// Full builder usage — system prompt, parameters, response shape
ChatResponse response = chatClient.prompt()
    .system("You are a senior Java architect giving concise, technically precise answers.")
    .user(u -> u.text("Explain {concept} with a short code example.")
                .param("concept", "the AOP proxy self-invocation pitfall"))
    .call()
    .chatResponse();   // full response object, vs. .content() for just the text

Building a custom ChatClient bean explicitly (rather than relying purely on the auto-configured default) — useful when you want a pre-configured client with default system prompts or default advisors baked in:

@Bean
ChatClient supportChatClient(ChatModel chatModel) {
    return ChatClient.builder(chatModel)
        .defaultSystem("You are a customer support assistant. Be concise and accurate.")
        .defaultAdvisors(new SimpleLoggerAdvisor())
        .build();
}

Mapping the response directly to a POJO (Structured Outputs, previewed here, full depth in Module 3):

record Sentiment(String label, double confidence) {}

Sentiment result = chatClient.prompt()
    .user("Classify the sentiment of: 'This product completely changed how I work.'")
    .call()
    .entity(Sentiment.class);

Talking point on why this matters architecturally, not just stylistically: “The fluent builder isn’t just nicer syntax — it’s what makes Advisors composable inline at the call site, the same way .filter() on a WebClient lets you attach cross-cutting behavior per-request rather than only globally. ChatClient is the layer where Advisors, prompts, and structured output mapping all come together into one readable chain, which is exactly why almost all real Spring AI application code should be written against ChatClient, with ChatModel reserved for infrastructure-level code.”

3. Provider Starters

Each provider gets its own starter, all sitting under the same spring-ai-bom-managed dependency umbrella from Module 1 — the point of this section is knowing the practical differences worth caring about when choosing one.

Provider Starter artifact Notable practical considerations
OpenAI spring-ai-starter-model-openai Now wraps the official openai-java SDK underneath (recent architectural change) — chat, embedding, image, audio, and moderation models all unified through it
Anthropic spring-ai-starter-model-anthropic Recently gained access to rate-limit info from Anthropic’s response headers via ChatResponseMetadata — useful for production throttling logic
Azure OpenAI spring-ai-starter-model-azure-openai Distinct from the plain OpenAI starter — different auth model (Azure AD/API key + endpoint), relevant for enterprises already standardized on Azure
Amazon Bedrock spring-ai-starter-model-bedrock-* (per model family, e.g., Anthropic-on-Bedrock, Titan) Routes through AWS IAM auth rather than a vendor API key — directly relevant to your AWS background; lets you keep model calls inside existing AWS account/network boundaries
Google (GenAI) spring-ai-starter-model-google-genai Had a recent dependency declaration bug (incorrectly pulling in the embedding module) — worth checking current release notes before assuming a clean transitive dependency tree
Ollama spring-ai-starter-model-ollama The local/self-hosted option — runs models on your own infrastructure, no external API call at all; valuable for prototyping without API costs or for fully offline/air-gapped requirements
Mistral AI spring-ai-starter-model-mistral-ai Recently moved to immutable, builder-based options classes (mutable setters removed) — a pattern now spreading across other provider option classes too
DeepSeek spring-ai-starter-model-deepseek Newer addition to the supported provider list, reflecting how fast the provider landscape itself is moving, not just the framework

The architecturally important point underneath this table: because every provider implements the same ChatModel/ChatClient contract, switching providers is primarily a dependency + configuration change, not a code rewrite — assuming you’ve stuck to the portable API surface (the next section’s core distinction).

# Switching from OpenAI to Anthropic is mostly this:
spring.ai.anthropic.api-key=${ANTHROPIC_API_KEY}
spring.ai.anthropic.chat.options.model=claude-sonnet-4-6

Talking point: “Having Bedrock as a first-class provider option matters a lot in an AWS-centric architecture — it means model calls can stay inside existing IAM auth and VPC boundaries instead of introducing a new external vendor API key into the security model, which is often the actual deciding factor in an enterprise architecture review, more than which model performs marginally better on a benchmark.” (Good explicit tie-in to your AWS Solution Architect background.)

4. Synchronous vs. Streaming Responses; Portable API vs. Provider-Specific Options

Synchronous (.call()) — blocks until the full response is ready, returns a complete ChatResponse or extracted content. Right choice for: short responses, batch/background processing, anywhere a complete answer is needed before proceeding (e.g., feeding the output into a downstream structured-output mapping or a tool-calling decision).

Streaming (.stream()) — returns a Flux<String> (or Flux<ChatResponse> for the full metadata-bearing variant), emitting tokens/chunks as the model generates them. Right choice for: chat UIs where perceived latency matters (showing partial output immediately), long-form generation where waiting for the full response would feel sluggish.

// Synchronous
String fullAnswer = chatClient.prompt()
    .user("Summarize the CAP theorem.")
    .call()
    .content();

// Streaming — typically wired directly into a reactive web endpoint
Flux<String> tokens = chatClient.prompt()
    .user("Write a detailed explanation of event sourcing.")
    .stream()
    .content();

// e.g., exposed directly from a Spring WebFlux controller
@GetMapping(value = "/chat/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> streamChat(@RequestParam String question) {
    return chatClient.prompt().user(question).stream().content();
}

A genuinely important production nuance worth flagging (and a real bug from the release history): streaming + tool calling together is a known tricky combination — there have been real bugs around streaming responses dropping chunks specifically when buffering tool calls (switchMap/concatMap reactive operator interactions). Talking point: “Streaming a plain text response and streaming a response that might also trigger tool calls are architecturally different problems — the model has to decide whether it’s about to call a tool partway through token generation, which means the reactive pipeline has to buffer correctly without dropping chunks. This is exactly the kind of subtlety where I’d write an integration test against streaming + tool-calling together specifically, rather than assuming streaming behavior that works for plain chat automatically extends safely to the tool-calling case.” This is a strong, currently-accurate engineering-judgment point.

Portable API vs. provider-specific options — the practical mechanics of “abstraction with an escape hatch”:

// Portable — works identically regardless of provider
ChatOptions options = ChatOptions.builder()
    .temperature(0.7)
    .maxTokens(500)
    .build();

// Provider-specific — accessing options unique to one provider
AnthropicChatOptions anthropicOptions = AnthropicChatOptions.builder()
    .temperature(0.7)
    .topK(40)                          // Anthropic-specific parameter
    .thinking(AnthropicApi.ThinkingType.ENABLED, 2000)   // extended thinking — not portable
    .build();

ChatResponse response = chatClient.prompt()
    .user("Solve this step by step: ...")
    .options(anthropicOptions)
    .call()
    .chatResponse();

Talking point on this tradeoff explicitly: “This is the leaky-abstraction tradeoff every portability layer eventually faces — ChatOptions covers the lowest common denominator (temperature, max tokens, etc.) that every provider supports, while reaching for AnthropicChatOptions or OpenAiChatOptions directly gets you provider-specific capability at the cost of losing the ability to swap providers without a code change in that one call site. My architectural rule of thumb: keep the portable API as the default everywhere, and isolate provider-specific option usage behind a clearly named, single-purpose method or class — so the ‘this part is locked to Anthropic’ decision is visible and contained, not silently smeared across the codebase.”

5. Prompt Construction — PromptTemplate, System/User Messages, Multi-Turn Structure

The Prompt object’s anatomy — a Prompt is fundamentally a list of Messages plus optional ChatOptions:

List<Message> messages = List.of(
    new SystemMessage("You are a precise, concise technical assistant."),
    new UserMessage("What's the difference between optimistic and pessimistic locking?")
);
Prompt prompt = new Prompt(messages);

PromptTemplate — parameterized prompt construction, avoiding raw string concatenation (the prompt-engineering equivalent of avoiding string-concatenated SQL — same injection-adjacent and maintainability concerns):

PromptTemplate template = new PromptTemplate("""
    You are a {role}. Explain {concept} to a {audience_level} audience.
    Keep the explanation under {max_sentences} sentences.
    """);

Prompt prompt = template.create(Map.of(
    "role", "senior Java architect",
    "concept", "the AOP proxy self-invocation pitfall",
    "audience_level", "intermediate",
    "max_sentences", "3"
));

Via ChatClient, this is usually expressed inline rather than via a standalone PromptTemplate object, but the underlying templating mechanism is the same:

chatClient.prompt()
    .user(u -> u.text("Explain {concept} to a {level} audience in under {n} sentences.")
                .param("concept", "CQRS")
                .param("level", "intermediate")
                .param("n", "3"))
    .call()
    .content();

System vs. user messages — the architectural distinction that matters, not just labeling:

  • System message — sets persistent behavior/role/constraints for the entire conversation; conceptually the “configuration” of the model’s behavior, not part of the actual exchange.
  • User message — the actual turn-by-turn human input.
  • Assistant message — the model’s own prior responses, needed explicitly when manually constructing multi-turn history (vs. relying on a Memory advisor to do this automatically — Module 6).

Multi-turn conversation — manual construction (useful to understand even though Memory advisors usually handle this automatically in real applications):

List<Message> conversation = new ArrayList<>();
conversation.add(new SystemMessage("You are a helpful Java architecture assistant."));
conversation.add(new UserMessage("What is the AOP proxy self-invocation problem?"));

ChatResponse firstResponse = chatModel.call(new Prompt(conversation));
conversation.add(new AssistantMessage(firstResponse.getResult().getOutput().getText()));

conversation.add(new UserMessage("How would I fix it without restructuring the whole class?"));
ChatResponse secondResponse = chatModel.call(new Prompt(conversation));
// the model now has the full conversation history as context for this follow-up

Talking point connecting this directly to your Java/Spring track: “Manually managing this message list is illuminating because it makes explicit something the AOP module covered conceptually — context has to be deliberately carried forward, it doesn’t persist automatically. A multi-turn conversation is really just an accumulating list of messages resent on every call; the model itself is stateless between calls, the same way an HTTP request is stateless and any ‘session’ is really just data the client or server chooses to carry forward and resend. Once that clicks, the Memory advisor from Module 6 stops looking like magic — it’s doing exactly this list-accumulation-and-resend pattern automatically, persisted to a backend like JDBC or Redis instead of held in a local List<Message>.”

Talking point tying the whole module together: “PromptTemplate avoiding string concatenation, ChatOptions giving you a portable subset with a clearly-isolated provider-specific escape hatch, and the message list making conversation state explicit — these are all the same underlying engineering instinct applied to a new domain: keep the common case clean and portable, make the special case visible and contained, and never let implicit state hide where it can silently break.”