Table of Contents

Module 6 — Conversation Memory

1. Chat Memory Abstraction and Pluggable Persistent Backends

The core abstraction: ChatMemory is the interface defining how conversation history is stored and retrieved, decoupled entirely from where it’s stored. This is the same “abstraction over a swappable backend” philosophy as ChatModel over model providers, just applied to persistence instead of inference.

public interface ChatMemory {
    void add(String conversationId, List<Message> messages);
    List<Message> get(String conversationId);
    void clear(String conversationId);
}

Why a conversationId is the central key, not a user ID: a single user might have many distinct conversation threads (support chat #1, support chat #2, a separate assistant feature entirely) — the memory abstraction is scoped per-conversation, and it’s the application’s job to decide what conversationId means in its domain (a chat session UUID, a support ticket ID, etc.), not the framework’s.

Pluggable backends — same interface, different storage characteristics:

Backend	Storage repository	Best fit when…
JDBC	`JdbcChatMemoryRepository`	You already run a relational database for the rest of the app — lowest operational overhead, no new infrastructure
Redis	`RedisChatMemoryRepository`	Low-latency access matters (high-traffic chat UI), and you’re comfortable with memory being the primary store or backed by Redis persistence
MongoDB	`MongoChatMemoryRepository`	Conversation history is naturally document-shaped, and you’re already on a Mongo-centric stack
Cassandra	`CassandraChatMemoryRepository`	Very high write throughput across many concurrent conversations, with a need for horizontal scale beyond what a single relational instance handles comfortably
Neo4j	`Neo4jChatMemoryRepository`	Conversation/entity relationships themselves are graph-shaped — less common, but relevant if the broader application is already graph-modeled

@Bean
ChatMemory chatMemory(JdbcChatMemoryRepository repository) {
    return MessageWindowChatMemory.builder()
        .chatMemoryRepository(repository)
        .maxMessages(20)
        .build();
}

Talking point: “The backend choice here is exactly the same decision tree as choosing a session store for a traditional web application — JDBC if you want one less moving part and your existing database can handle the write volume, Redis if latency or scale pushes you toward an in-memory store, and the more specialized backends (Cassandra, Neo4j) only when there’s a specific scale or data-shape reason to reach for them. I wouldn’t default to anything more exotic than JDBC or Redis without a concrete justification — chat memory storage doesn’t usually need a graph database’s relationship-traversal strengths, even though Neo4j support exists for the cases where the broader app already lives there.”

2. JdbcChatMemoryRepository — Schema, Timestamp Access, Programmatic Management

The underlying schema — a spring_ai_chat_memory table, conceptually similar to any audit-log or event-table design:

CREATE TABLE spring_ai_chat_memory (
    conversation_id VARCHAR(256) NOT NULL,
    content         TEXT NOT NULL,
    type            VARCHAR(20) NOT NULL,   -- USER / ASSISTANT / SYSTEM / TOOL
    timestamp       TIMESTAMP NOT NULL,
    -- additional indexing typically on (conversation_id, timestamp)
);

Spring AI provides a schema initializer that can auto-create this table on startup (similar to Spring Session’s JDBC schema bootstrapping, or Spring Batch’s metadata tables) — useful for getting started quickly, though for production you’d typically manage this via your own migration tooling (Flyway/Liquibase) for the same reasons you’d manage any other schema that way: auditability, rollback safety, and consistency across environments.

Programmatic access to the timestamp field — a specific, recently-added capability worth knowing: earlier versions of JdbcChatMemoryRepository reportedly didn’t expose the stored timestamp back to application code cleanly; a recent fix allows accessing the timestamp field programmatically. This matters in practice for anything beyond simple memory replay — e.g., showing “last active” times in a support dashboard, implementing time-based retention policies (covered in section 4), or auditing when a particular exchange occurred for compliance purposes.

@Autowired
private JdbcChatMemoryRepository repository;

List<Message> history = repository.findByConversationId(conversationId);
// with timestamp now accessible, you can build retention/audit logic 
// directly against actual message age, not just message count

Programmatic management — the operations beyond simple add/get:

// Clearing a specific conversation (e.g., user explicitly requests history deletion — 
// a real requirement under privacy regulations)
chatMemoryRepository.deleteByConversationId(conversationId);

// Listing all conversation IDs (e.g., for an admin/support tooling view)
List<String> activeConversations = chatMemoryRepository.findConversationIds();

Talking point connecting this to your data-governance background from the Data Analyst track: “Chat memory is personal data, often containing PII by nature of being a real conversation — which means the same governance instincts from the data quality and access-control modules apply directly here: who can query this table, how long is it retained, and is there a clean, explicit deletion path for a user’s ‘forget me’ request. A JDBC-backed chat memory table is just another table subject to the same RLS, retention, and auditing discipline as any other store of personal data in the system — it shouldn’t get a free pass just because it’s ‘AI conversation data’ rather than ‘traditional application data.'”

3. Memory Advisor Placement — Why Ordering Matters for Correctness

This is the third time this specific issue has come up across the track (Modules 1, 4, and 5) — which is itself worth naming explicitly as a sign of how architecturally central this single ordering decision is, rather than treating it as a minor footnote each time.

Recap of the concrete behavior change: memory advisors are now placed outside ToolCallingAdvisor in the default chain construction, rather than interleaved within the tool-calling iteration.

Walking through why, mechanically, with the actual message flow:

User: "What's the status of my last order?"

WITHOUT correct ordering (memory advisor inside the tool-calling loop):
  1. Tool-calling advisor begins resolving the request
  2. Model decides to call getOrderStatus(orderId="4471")
  3. ⚠️ Memory advisor (positioned inside) might persist this 
     intermediate tool-call exchange as if it were a real conversational turn
  4. Tool returns "Shipped on June 20"
  5. ⚠️ This tool-result message might ALSO get persisted as a turn
  6. Model produces final answer: "Your order shipped on June 20."
  7. Final answer persisted

  → Stored history now contains the tool-call plumbing AND the final answer,
    cluttering future context windows with internal mechanics the user 
    never saw and didn't say.

WITH correct ordering (memory advisor outside the tool-calling loop):
  1. Memory advisor loads prior history, passes request inward
  2. Tool-calling advisor resolves the FULL exchange internally — 
     calling getOrderStatus, getting the result, producing the final answer — 
     all as internal iteration invisible to memory
  3. Only the FINAL, complete exchange (user question + final answer) 
     returns to the memory advisor
  4. Memory advisor persists exactly that — one clean user turn, 
     one clean assistant turn

  → Stored history stays clean and matches what a human reading the 
    transcript would actually expect to see.

Why this is a correctness bug, not just a style preference: a polluted memory store doesn’t just look messy — it actively degrades future model behavior, because that polluted history gets resent as context on every subsequent turn. If intermediate tool-call noise accumulates in memory, the context window fills with irrelevant internal mechanics instead of useful conversational history, the context budget gets consumed faster (directly increasing cost — section 4’s concern), and in the worst case the model can become confused by seeing its own internal tool-call plumbing presented back as if it were prior conversational content.

Talking point that makes the stakes concrete: “This isn’t a cosmetic ordering preference — a chat memory store polluted with tool-call internals actively makes every future turn in that conversation worse, because that polluted history gets resent as context on every subsequent call. It’s the same blast radius as a logging interceptor accidentally capturing and replaying internal RPC retries as if they were real business events — the bug doesn’t crash anything, it just silently corrupts the data that everything downstream relies on being clean. That’s exactly why this needed an explicit default-ordering fix rather than being left as ‘something the developer should configure correctly themselves’ — the failure mode is too subtle and too costly to leave to chance.”

4. Designing Memory Retention/Windowing Strategies for Cost and Context-Length Control

The core constraint driving this whole topic: every model call has a finite context window, and providers bill by token volume — unbounded conversation history growth is both a hard technical ceiling (eventually you exceed the context window) and a direct, compounding cost problem (every turn resends the entire accumulated history, so cost per turn grows as the conversation lengthens, not just total cost across the conversation).

Windowing strategies, from simplest to most sophisticated:

Strategy	Mechanism	Tradeoff
Fixed message-count window (`MessageWindowChatMemory`, shown in section 1)	Keep only the last N messages	Simple, predictable cost ceiling — but can abruptly drop context the model still needed (e.g., a detail mentioned 25 messages ago that’s still relevant)
Token-budget window	Keep as many recent messages as fit under a token budget, dropping oldest first	More precise cost control than a flat message count, since message lengths vary wildly
Summarization-based compaction	Periodically replace older messages with a model-generated summary of “what’s been discussed so far”	Preserves long-range context cheaply, but introduces a new failure mode — summarization itself can lose or distort detail, and it’s an additional model call (additional cost and latency) to produce the summary
Selective/semantic retention	Keep messages that are semantically important (decisions made, facts stated) regardless of recency, discard small-talk/filler	Best fidelity-to-cost ratio, but requires building or relying on logic to judge “importance” — meaningfully more engineering effort, and itself imperfect

Talking point connecting this directly to the Statistics/A-B-Testing track’s “every choice is a tradeoff, state it explicitly” theme: “There’s no free option here — every windowing strategy is trading completeness of context against cost and context-window ceiling, the same shape of tradeoff as choosing a sampling rate in any observability system. I’d start with a simple fixed message-count or token-budget window for most applications, since it’s cheap, predictable, and easy to reason about, and only move to summarization-based compaction once there’s a concrete, demonstrated case where users are losing important context that a simple window is dropping — building summarization-based memory before you have evidence you need it is premature complexity, the same caution I’d apply to over-engineering any caching or retention layer.”

A practical hybrid worth naming, since it’s common in production: keep a token-budget sliding window for recent turns, plus a separately-stored, persistent set of “key facts” extracted once (e.g., “customer’s account tier,” “stated preference for email over SMS”) that gets re-injected into every prompt regardless of windowing — similar in spirit to a system prompt that’s always present, just populated dynamically per-conversation rather than statically per-application.

Closing talking point tying the whole module together: “Chat memory design is really three separate concerns wearing one trenchcoat: a storage abstraction (which backend), a correctness concern (advisor ordering, so what gets stored is actually what should be stored), and a cost/capacity concern (windowing, so what gets stored doesn’t grow without bound). Treating all three as ‘the memory feature’ without separating them is how teams end up with bugs that look like cost problems but are actually correctness problems — exactly the polluted-history scenario from section 3, which manifests first as ‘why is this conversation suddenly expensive’ before anyone realizes the actual root cause is advisor ordering, not genuine conversational growth.”