Module 7 — Vector Stores & RAG

Module 7 — Vector Stores & RAG

1. VectorStore Abstraction — Portable API Across Providers

The interface, deliberately minimal in the same spirit as ChatModel:

public interface VectorStore {
    void add(List<Document> documents);
    void delete(List<String> idList);
    List<Document> similaritySearch(SearchRequest request);
}

Every provider implementation — PgVectorStore, PineconeVectorStore, QdrantVectorStore, RedisVectorStore, MongoDBAtlasVectorStore, ElasticsearchVectorStore, ChromaVectorStore, MilvusVectorStore, Neo4jVectorStore, OpenSearchVectorStore, WeaviateVectorStore, and others — implements this same contract, so application code written against VectorStore doesn’t change when the underlying store changes, identical in spirit to ChatModel‘s provider portability from Module 2.

@Autowired
private VectorStore vectorStore;

vectorStore.add(List.of(
    new Document("Refunds are processed within 5-7 business days.", 
                  Map.of("category", "policy", "topic", "refunds"))
));

List<Document> results = vectorStore.similaritySearch(
    SearchRequest.builder()
        .query("How long do refunds take?")
        .topK(3)
        .similarityThreshold(0.75)
        .build()
);

Why portability matters more here than almost anywhere else in the framework: vector store choice is frequently driven by infrastructure that already exists (a team already running PostgreSQL reaches for PGVector; a team already on Elasticsearch for search reaches for its vector capability) rather than by the AI feature’s needs in isolation. The portable API means a RAG feature prototyped against PGVector can migrate to a dedicated vector database like Pinecone or Qdrant later, if scale or recall-quality demands it, without rewriting the retrieval logic itself.

Practical selection considerations worth naming, since “which vector store” is a genuine architecture decision, not a coin flip:

Consideration Favors
Already running PostgreSQL PGVector — zero new infrastructure, transactional consistency with the rest of your data
Already running Elasticsearch/OpenSearch for text search Elasticsearch/OpenSearch vector support — hybrid keyword + semantic search in one system
Need a purpose-built, horizontally-scalable vector index at large scale Pinecone, Qdrant, Milvus — built specifically for vector workloads, generally stronger at very large-scale approximate nearest-neighbor search
Already running MongoDB MongoDB Atlas Vector Search — avoids a second datastore for document-shaped data that’s already in Mongo
Already running Redis for caching Redis vector support — but note the real bug flagged in the release history: RedisVectorStore#doDelete only deleting the first 10 messages — a concrete reminder that even widely-used integrations can have incomplete edge-case coverage, worth testing deletion behavior explicitly before relying on it in production

Talking point: “My default heuristic is to start with whichever vector-capable store is already part of the existing infrastructure — PGVector if there’s already a Postgres instance, Elasticsearch if hybrid search is already a requirement — and only introduce a dedicated vector database once there’s a measured recall or scale problem the existing store can’t solve. That’s the same ‘don’t add infrastructure before you have evidence you need it’ instinct that applies to the chat-memory windowing decision from the last module.”

2. SQL-Like Metadata Filter API

The problem this solves: pure similarity search alone often isn’t precise enough — you frequently need to combine semantic similarity with structured filtering (only search within a specific document category, a specific tenant’s data, documents from after a certain date). Spring AI exposes a portable, SQL-like filter expression syntax that works consistently across all the otherwise very differently-shaped underlying provider filter APIs.

List<Document> results = vectorStore.similaritySearch(
    SearchRequest.builder()
        .query("refund timeline")
        .topK(5)
        .filterExpression("category == 'policy' && region in ['US', 'EU'] && year >= 2025")
        .build()
);

// Programmatic filter builder, avoiding raw string construction — 
// directly analogous to PreparedStatement parameterization avoiding string-concatenated SQL
Filter.Expression filter = new FilterExpressionBuilder()
    .and(
        new FilterExpressionBuilder().eq("category", "policy"),
        new FilterExpressionBuilder().gte("year", 2025)
    ).build();

vectorStore.similaritySearch(SearchRequest.builder()
    .query("refund timeline")
    .filterExpression(filter)
    .build());

Why this is architecturally significant, not just a convenience: every underlying vector store has its own native filter syntax (Pinecone’s metadata filter JSON, Elasticsearch’s query DSL, Qdrant’s filter objects) — without a portable abstraction here, switching vector stores wouldn’t just mean swapping a connection string, it would mean rewriting every filtered query by hand for the new provider’s filter dialect. The SQL-like expression syntax is what makes the whole VectorStore abstraction genuinely portable, not just the basic similarity-search call.

Talking point connecting directly back to the SQL module of your Data Analyst track: “This is deliberately SQL-shaped syntax specifically because it transfers existing knowledge — ==, &&, in [...] read almost exactly like a WHERE clause, which means anyone fluent in SQL filtering logic from the analyst track can apply that same mental model here immediately, rather than learning a fourth different filter DSL for a fourth different vector store provider.”

Multi-tenancy is the most common real-world use of this filter API — scoping every RAG query to tenant_id == :currentTenant as a non-negotiable filter, enforced in code rather than trusted to be correct by convention, is the vector-store equivalent of row-level security from the BI/DAX module — the same governance principle, just enforced at the retrieval-filter layer instead of the database layer.

3. Document ETL Framework — Ingestion, Transformation, Chunking

This is, structurally, the same Extract-Transform-Load pattern from the Data Architecture module of your Data Analyst track, applied specifically to preparing unstructured documents for vector retrieval — worth naming that parallel explicitly, since the underlying discipline transfers directly.

DocumentReader → DocumentTransformer(s) → VectorStore.add()
   (Extract)         (Transform)              (Load)

DocumentReader — extracts raw content from a source format:

DocumentReader reader = new PagePdfDocumentReader(new ClassPathResource("policy-handbook.pdf"));
List<Document> rawDocuments = reader.get();

Multiple reader implementations exist for different sources (PDF, JSON, plain text, Markdown, and others) — the “Extract” stage’s job is purely getting raw content plus basic structural metadata (page number, source filename) into the common Document shape.

DocumentTransformer — the “Transform” stage, most critically chunking (splitting large documents into smaller passages):

TextSplitter splitter = TokenTextSplitter.builder()
    .withChunkSize(800)
    .withMinChunkSizeChars(350)
    .withMinChunkLengthToEmbed(5)
    .build();

List<Document> chunks = splitter.apply(rawDocuments);

Why chunking strategy is a genuinely consequential design decision, not a mechanical step: chunk size directly trades off two failure modes —

  • Chunks too large → each retrieved chunk contains a lot of irrelevant surrounding text alongside the relevant part, diluting the model’s attention and wasting context-window budget on noise.
  • Chunks too small → a chunk loses necessary surrounding context to be meaningful on its own (a sentence fragment without the paragraph that explains what it’s referring to), and you also multiply the number of vectors, increasing storage and search cost.

Metadata enrichment during transformation — attaching structured metadata (source document, section, category, date) to each chunk is what later makes the SQL-like filtering from section 2 possible at all; metadata has to be deliberately carried through or attached during the ETL pipeline, it doesn’t appear automatically.

List<Document> enriched = chunks.stream()
    .map(doc -> {
        doc.getMetadata().put("category", "policy");
        doc.getMetadata().put("source", "policy-handbook.pdf");
        return doc;
    })
    .toList();

vectorStore.add(enriched);   // Load stage

Talking point connecting this directly to your Data Cleaning module: “Chunking and metadata enrichment are RAG’s version of the data-cleaning discipline from the analyst track — garbage in, garbage out applies identically here. A RAG system retrieving badly-chunked, unlabeled passages produces unreliable answers for exactly the same root-cause reason a dashboard built on dirty, untyped data produces unreliable numbers. I’d treat the document ETL pipeline with the same rigor as any other data pipeline — version it, test it, and review chunk quality deliberately, rather than treating it as a one-line splitter.apply() call and moving on.”

4. QuestionAnswerAdvisor and the Full RAG Request Flow

Building directly on Module 5’s introduction — here’s the complete, end-to-end mechanics of what happens on a single RAG-enabled call:

chatClient.prompt()
    .advisors(QuestionAnswerAdvisor.builder(vectorStore)
        .searchRequest(SearchRequest.builder()
            .topK(5)
            .similarityThreshold(0.75)
            .filterExpression("category == 'policy'")
            .build())
        .build())
    .user("What does our refund policy say about items damaged in transit?")
    .call()
    .content();

1. User question arrives at QuestionAnswerAdvisor (request side)
2. Advisor performs vectorStore.similaritySearch() using the user's question 
   as the query, applying the configured topK/threshold/filter
3. Retrieved Document chunks are formatted and injected into the prompt 
   as additional context (typically appended to or merged with the system message)
4. The augmented prompt — original question + injected context — is sent to the model
5. Model generates an answer GROUNDED in the injected context, 
   rather than relying solely on its parametric training knowledge
6. Response flows back up through the chain (response side) — 
   QuestionAnswerAdvisor itself typically doesn't transform the response content,
   though custom advisors could add citation formatting, etc.

A critical, frequently-overlooked detail: retrieval quality is the ceiling on RAG answer quality. If similaritySearch returns irrelevant or low-quality chunks (because of poor chunking, a mismatched embedding model, or an overly loose similarity threshold), the model is being asked to answer grounded in bad context — and a model grounded in bad context often produces a more confidently wrong answer than one operating ungrounded, because the injected context lends false authority to whatever it contains.

Talking point that connects directly to the correlation-vs-causation and statistical-honesty themes from the Data Analyst track: “RAG doesn’t eliminate hallucination — it changes what the model is hallucinating from. An ungrounded model hallucinates from its training data; a poorly-grounded RAG system can hallucinate from bad retrieved context with more apparent confidence, because the answer now looks evidence-backed even when the evidence itself was irrelevant. That’s exactly why retrieval quality testing and evaluation (Module 10’s territory) matters as much as prompt engineering — the failure mode shifts, it doesn’t disappear, and a RAG system that looks reliable in casual testing can still be confidently wrong if nobody’s specifically evaluated the retrieval step’s precision.”

5. Embedding Models — Provider Options, Dimension/Compatibility Considerations

The role of embedding models in this pipeline: every document chunk and every incoming query gets converted into a numeric vector (an embedding) via an EmbeddingModel, and similarity search operates entirely on those vectors — the actual text is retrieved by mapping back from a matched vector to its source document, but the matching itself happens purely in vector space.

@Autowired
private EmbeddingModel embeddingModel;

float[] vector = embeddingModel.embed("How long do refunds take?");

The dimension-compatibility trap — a genuinely important, easy-to-get-wrong operational detail: a vector store index is built for a fixed embedding dimension (e.g., 1536 dimensions for one OpenAI embedding model, a different dimension for another provider’s model). Every document in the store and every query against it must use the same embedding model — mixing embedding models, or changing embedding models after documents are already indexed, produces vectors that aren’t comparable, and similarity search results become meaningless (or the system errors outright on a dimension mismatch).

Provider embedding options — most chat-model providers also offer a corresponding embedding model (OpenAI’s embedding models, Google’s, and so on), and there’s no requirement that the embedding provider matches the chat-model provider — it’s entirely valid to embed with one provider and generate the final answer with a different one, since they’re independent EmbeddingModel and ChatModel abstractions composed together by the application, not coupled to each other by the framework.

Talking point on why this matters operationally, with a concrete failure scenario worth naming explicitly: “The dimension-compatibility issue is the kind of thing that doesn’t show up in development — you build and test against one embedding model, everything works — and then bites later, specifically if someone swaps the embedding model in configuration without realizing every previously-indexed document is now vector-incompatible with new queries. The safe operational practice is: changing the embedding model requires re-embedding and reloading the entire vector store, not just updating a config value — and that re-indexing step needs to be a deliberate, planned migration, not something that can happen silently as a side effect of a routine config change.”

6. Recently Removed Integrations — SAP HANA, Infinispan — Migration Awareness

What happened: the SAP HANA DB vector store module and the Infinispan vector store module were both removed from Spring AI. Users relying on either integration must migrate to an alternative vector store provider — there’s no in-place replacement; it’s an explicit removal, not a rename or deprecation-with-shim.

Why this is worth knowing as a standing practice, not just a one-time fact: this is the vector-store-specific instance of the model-churn discipline established in Module 3 (the Pixtral deprecation) — a fast-moving project pruning less-widely-used integrations as it matures is a recurring pattern, not a one-off event, and it applies just as much to infrastructure-layer integrations (vector stores) as to model-layer integrations (specific chat/vision models).

The same defensive posture from Module 3 applies directly here:

  • Don’t build deep, integration-specific code against a less mainstream vector store provider without accepting some migration risk — the more niche the provider, the higher the chance of it being deprioritized or removed in a future version.
  • Keep vector-store-specific code behind the VectorStore interface as strictly as possible (avoid casting down to provider-specific implementation classes in business logic) — exactly the portability discipline section 1 already establishes, which is precisely what makes a forced migration like this tractable rather than catastrophic.
  • Track release notes for infrastructure integrations with the same discipline as for model deprecations — it’s easy to only watch for chat-model changes since they’re more visible day-to-day, while a vector store removal can be just as disruptive and easier to miss.

Talking point that closes the module by tying it to the recurring churn theme across the whole Spring AI track: “SAP HANA and Infinispan being removed is the same lesson as the Pixtral deprecation from Module 3, just one layer down the stack — this is a young, fast-evolving framework, and both the model layer and the infrastructure-integration layer are subject to real churn, not just one or the other. The mitigation is identical in both cases: code against the portable interface, never the concrete provider class, and treat ‘which integrations does this framework still support’ as something to actively check before every major version upgrade, not something to assume is stable just because it worked last time.”