Fine-Tuning & Customization
Fine-Tuning vs RAG vs Prompting — When Each Makes Sense (Recap)
Prompting — fixes behavior problems (tone, format, role). Free, instant, always try first.
RAG — fixes knowledge problems (facts the model doesn’t have, or that change). No retraining, full auditability/citations.
Fine-tuning — fixes skill/pattern problems prompting can’t reach: consistent behavior across thousands of edge cases, a specialized domain “voice” that’s hard to specify in a prompt, or shrinking a large general model down to a small cheap one for a narrow repeated task. Highest cost and ongoing maintenance burden — reach for it last, and only after prompting/RAG have proven insufficient.
The one-line test: “Does the model not know this fact?” → RAG. “Is the model not behaving the way I want?” → prompting first, fine-tuning if prompting plateaus.
LoRA / PEFT — Efficient Fine-Tuning Explained Simply
The problem PEFT solves. Full fine-tuning means updating every single parameter in a model — for a model with tens of billions of parameters, that requires enormous GPU memory (you need to store gradients and optimizer states for every parameter, not just the parameters themselves) and produces a complete new copy of the model per fine-tuned task. For most enterprise use cases, this is wildly disproportionate to the actual customization needed.
PEFT (Parameter-Efficient Fine-Tuning) is the umbrella term for techniques that adapt a model’s behavior by training only a small fraction of parameters, leaving the original model weights frozen.
LoRA (Low-Rank Adaptation) is the dominant PEFT technique. The intuition, without the linear algebra: instead of updating the full weight matrices of the model directly, LoRA inserts small, trainable “adapter” matrices alongside the original frozen weights. These adapter matrices are deliberately tiny relative to the original (low-rank — meaning they capture the adaptation using far fewer numbers than the full weight matrix would need). During training, only these small adapters get updated; the base model’s billions of original parameters never change.
Why this matters practically:
- Training cost drops drastically — you’re optimizing a tiny fraction of parameters (often <1% of the full model), so GPU memory and compute requirements shrink to something feasible on a handful of GPUs rather than a large training cluster.
- Storage is cheap and modular — instead of storing a full duplicate model per fine-tuned task, you store the small LoRA adapter (often megabytes, not gigabytes) and load it on top of the shared base model at inference time. This means you can maintain many task-specific adapters off one base model, swapping which adapter loads per request — directly analogous to how you’d run multiple lightweight configuration profiles on top of one shared service deployment rather than maintaining separate full deployments per customer/task.
- QLoRA extends this further by also quantizing the frozen base model (reducing numeric precision to save memory) while training the LoRA adapters on top — pushing fine-tuning feasibility down to even more modest hardware.
The architectural takeaway for you: LoRA is why fine-tuning open-weight models (Llama, Mistral, etc.) on your own infrastructure became practically affordable for mid-size engineering teams rather than requiring a dedicated ML infra investment — it’s the difference between “needs a training cluster” and “can run on a single multi-GPU instance for a few hours.” It’s also exactly the mechanism behind Bedrock’s and SageMaker’s “fine-tuning” offerings for supported models — under the hood, most managed fine-tuning APIs you’d call are LoRA-based, not full fine-tuning.
Embeddings — Custom Embeddings for Domain-Specific Search
Why off-the-shelf embeddings sometimes aren’t enough. General-purpose embedding models (OpenAI’s text-embedding-3, Amazon Titan Embeddings, Cohere embed, open-weight options like BGE/E5) are trained on broad web-scale text and capture general semantic similarity well. They struggle when your domain has specialized vocabulary, jargon, or notions of “similarity” that differ from general usage — e.g., in a legal or healthcare corpus, two documents might use very different general-language phrasing but mean the same regulatory concept, or conversely use similar-sounding language but mean legally distinct things. A general embedding model has no way to know your domain’s specific similarity structure.
What “custom embeddings” actually means in practice — there’s a spectrum, not a single technique:
- Embedding model selection/benchmarking — before any custom training, evaluate several existing embedding models against your actual retrieval task (using a held-out set of query→correct-document pairs from your domain) rather than assuming the most popular general model is best. Often this alone closes most of the gap.
- Fine-tuning an embedding model — using contrastive learning: you train the model on pairs/triplets of (query, relevant document, irrelevant document) from your own domain, adjusting the embedding space so domain-relevant items pull closer together and irrelevant ones push apart. This is conceptually the same LoRA/PEFT-style efficient fine-tuning described above, just applied to an embedding model rather than a generative one — you don’t need full retraining, you need the embedding space reshaped around your domain’s actual notion of relevance.
- Building embeddings from scratch — rare, and almost never justified given how strong general + fine-tuned options already are; reserved for genuinely novel data modalities general embedding models weren’t trained on at all.
The practical signal that you need this, rather than just better chunking or a different off-the-shelf model: your RAG retrieval step is consistently returning topically-similar-but-wrong-for-the-task documents — e.g., it retrieves the right kind of document but not the one that’s actually relevant to the specific query’s domain meaning. That’s an embedding-space problem, not a chunking or prompt problem, and it’s the point where fine-tuning an embedding model on your own (query, correct-doc) pairs starts paying off over swapping models or re-chunking.
Evaluation matters more here than the technique choice. Whatever route you take, you need a retrieval benchmark — a set of representative queries with known-correct documents — to measure recall@K before and after any change. Without that, you’re tuning blind; this is the same discipline as having an integration test suite before refactoring a service, applied to a retrieval system instead of code.