RAG & Knowledge Integration

RAG & Knowledge Integration

Retrieval-Augmented Generation (RAG) — Architecture and When to Use It

Core architecture, end to end:

  1. Ingestion pipeline — source documents (PDFs, wikis, tickets, code) get chunked, embedded (converted to vectors), and stored in a vector database, usually alongside metadata (source, timestamp, access permissions).
  2. Query time — the user’s query gets embedded using the same embedding model, and a similarity search runs against the vector store to retrieve the top-K most relevant chunks.
  3. Augmentation — those retrieved chunks get injected into the LLM’s context window alongside the original query, typically with an instruction like “answer using only the provided context.”
  4. Generation — the LLM produces a response grounded in the retrieved material, ideally with citations back to source chunks.

That’s the “naive RAG” pipeline. Production systems usually add: a re-ranking step (a cheaper model scores retrieved chunks for relevance before they hit the expensive LLM call), query rewriting/expansion (the user’s raw query is often a poor search query — rephrase it first), and hybrid search (combining vector similarity with traditional keyword/BM25 search, since pure embedding similarity misses exact-match cases like part numbers or error codes).

When to use RAG vs. not:

  • Use it when the answer depends on information that changes (policies, pricing, current docs) or is too large/proprietary to bake into the model (your internal knowledge base, codebase, ticket history).
  • Use it when you need traceability — the user/compliance needs to know which document the answer came from.
  • Don’t reach for it when the task is purely about style, format, or behavior (tone, structured output, role-play) rather than factual knowledge — that’s a prompting or fine-tuning problem, not a retrieval problem.
  • Don’t reach for it when your knowledge base is small enough to fit entirely in context — at that point, just put it in the prompt directly; the retrieval layer adds complexity and failure points (bad chunking, bad retrieval ranking) for no benefit.

The architectural framing that’ll land with your audience: RAG is to LLMs what a search index + cache layer is to a traditional app — it’s an external data access pattern bolted onto a stateless compute layer, not a property of the model itself.

Vector Databases — Embeddings, Similarity Search, and the Major Options

The mechanics. Each chunk of text gets converted into a high-dimensional vector (embedding) by an embedding model. “Similar meaning” maps to “vectors close together in that space” — measured typically by cosine similarity or dot product. A vector database’s job is to store millions/billions of these vectors and answer “find me the K closest vectors to this query vector” fast — which is non-trivial at scale, since brute-force comparison against every stored vector doesn’t scale past a small dataset.

This is solved with approximate nearest neighbor (ANN) algorithms — most commonly HNSW (Hierarchical Navigable Small World graphs) — which trade a small amount of recall accuracy for massive speed gains by building a navigable graph structure instead of scanning everything.

The options, mapped to where you’d actually deploy them:

  • pgvector — a Postgres extension that adds vector columns and ANN indexing directly into a relational database you likely already run. The pitch: no new infrastructure, transactional consistency with your existing relational data, and you can JOIN vector search results against normal SQL tables. Best fit when your retrieval corpus is moderate scale and you don’t want to operate a separate system. Given your AWS/RDS background, this is the lowest-friction entry point — Aurora PostgreSQL supports pgvector natively.
  • OpenSearch — adds a k-NN plugin on top of what’s already a mature, horizontally-scalable search engine. Strong fit when you need hybrid search (full-text/BM25 + vector) in one system, and you’re already in the AWS ecosystem (Amazon OpenSearch Service is managed). Good for log/document-heavy enterprise search use cases.
  • Pinecone — a fully managed, purpose-built vector database with no underlying infra to operate. Strong fit when vector search is the product and you want to move fast without owning the ops burden — at the cost of another external managed dependency and per-query pricing.

The architectural decision pattern is the same one you already apply elsewhere: do you bolt a capability onto infrastructure you already operate (pgvector on RDS/Aurora) or bring in a purpose-built managed service (Pinecone) for performance/scale headroom you don’t want to engineer yourself.

Chunking Strategies for Document Retrieval

Chunking is where most RAG quality problems actually originate — bad retrieval is very often a bad chunking problem wearing a “the model is dumb” disguise.

  • Fixed-size chunking — split every N tokens (e.g., 500), often with overlap (e.g., 50 tokens) between consecutive chunks so context isn’t severed mid-thought at a hard boundary. Simple, predictable, but blind to actual document structure — can split a table or a step list mid-way.
  • Semantic/structure-aware chunking — split along natural boundaries: paragraphs, sections, markdown headers, code function boundaries. Much better retrieval quality because each chunk is a coherent unit of meaning, not an arbitrary token-count slice. This is the right default for structured technical docs (which is most of what you’re producing).
  • Recursive chunking — try splitting on the largest structural unit first (section), and if it’s still too big for the target chunk size, recursively split on the next smaller unit (paragraph → sentence). This is what most production frameworks (LangChain’s RecursiveCharacterTextSplitter, etc.) default to.
  • Hierarchical / parent-child chunking — embed small chunks for precise retrieval matching, but when a chunk is retrieved, return its larger parent section as the actual context sent to the LLM. Solves the tension between “small chunks retrieve more precisely” and “small chunks lack enough surrounding context for the LLM to answer well.”

The chunk-size tradeoff is direct: smaller chunks → more precise retrieval matching but less context per chunk (model might miss surrounding nuance); larger chunks → more context but noisier matching (a chunk about three different things gets retrieved for all three, diluting relevance). Most production systems land in the 200–800 token range with deliberate overlap, then tune empirically against real queries.

RAG vs Fine-Tuning vs Prompt Engineering — Decision Framework

These solve fundamentally different problems, and the most common architecture mistake is reaching for the expensive one (fine-tuning) when a cheaper one would’ve worked.

Need Right tool Why
Model needs facts it doesn’t know, or facts that change RAG Knowledge problem — inject the facts at query time, don’t bake them into weights
Model needs to behave/respond in a specific format, tone, or role Prompt engineering Behavior problem — usually solvable with instructions + few-shot examples, no training needed
Model needs a specialized skill (domain-specific reasoning pattern, a style of output that’s hard to specify via prompt, consistent behavior across thousands of edge cases) Fine-tuning Skill/pattern problem — you’re adjusting the model’s actual behavior distribution, not just feeding it facts
Data is highly proprietary/sensitive and traceability to source is required RAG You get citations and access control over what’s retrieved; fine-tuned facts are opaque and unauditable
Latency/cost sensitive at very high volume, and the task is narrow Fine-tuning (often a smaller model) A fine-tuned small model can match a much larger general model on a narrow task at a fraction of inference cost

The practical decision order most architects should follow, cheapest-and-fastest first:

  1. Prompt engineering first — always. It’s free, instant to iterate, and solves a surprising majority of “the model isn’t doing X” problems.
  2. RAG next — when the gap is knowledge, not behavior. Still no model training required, and you get auditability for free.
  3. Fine-tuning last — when you’ve proven prompting and RAG can’t get you the consistency/format/specialized-skill you need, or when you need to shrink a large model down to a smaller, cheaper one for a narrow repeated task. Fine-tuning requires labeled training data, retraining infrastructure or a fine-tuning API, evaluation rigor, and ongoing maintenance as your needs evolve — real ongoing cost, not a one-time fix.

In practice, production systems combine all three: a fine-tuned (or just well-prompted) model for tone/format, RAG for grounding in current/proprietary facts, and careful prompt engineering wrapping both. Treating these as mutually exclusive is the most common architectural mistake in this space.