Table of Contents

Generative AI / LLMs

Prompt Engineering Fundamentals — Zero-Shot, Few-Shot, Chain-of-Thought

Zero-shot. You give the model an instruction with no examples and trust it to generalize from its training. “Summarize this email in 3 bullet points.” Works well for tasks the model has clearly seen analogues of during training. Fails when your task has an unusual format or domain-specific convention the model wouldn’t infer on its own.

Few-shot. You include 1–5 examples of input→output pairs in the prompt before the actual task, so the model pattern-matches the format/style you want rather than guessing. This is in-context learning — no weights are updated, the model is just conditioning its next-token predictions on the examples you handed it in the prompt window. Useful when output format matters more than raw capability (e.g., “extract these fields into this exact JSON shape”).

Chain-of-thought (CoT). You prompt the model to reason step-by-step before giving a final answer (“think through this carefully, then answer”), rather than jumping straight to the output. This measurably improves accuracy on multi-step problems — math, logic, multi-hop reasoning — because the model’s own intermediate tokens become additional context it can condition on for the final answer. It’s effectively giving the model scratch space. Modern “reasoning models” (o1-style, extended thinking in Claude) bake this in as a built-in behavior rather than something you have to explicitly prompt for.

The architect-relevant takeaway: these aren’t competing techniques, they’re a ladder of context-engineering effort vs. reliability. Production systems often combine all three — a system prompt with role instructions (zero-shot framing), a few curated examples for format control (few-shot), and an instruction to reason before answering on the harder cases (CoT).

Context Windows — What They Are, Why They Matter, Their Limits

The context window is the total amount of text (measured in tokens) the model can “see” at once — system prompt + conversation history + any documents/tools output + the response it’s generating. Everything the model knows about the current task has to fit inside this window; nothing persists across calls unless you re-send it.

Why it matters architecturally:

No implicit memory. Unlike a database, an LLM call is stateless. If you want multi-turn conversation, RAG context, or tool outputs available, you re-inject them into the prompt every single call. This is exactly the same statelessness principle you already design around with REST APIs and Lambda — except here the “state” is the entire conversation, and re-sending it costs tokens (= money + latency) every turn.
Cost and latency scale with context size, not linearly in the friendly sense — attention computation is roughly quadratic in sequence length for vanilla transformer attention (mitigated in practice by techniques like sliding-window or sparse attention in some model families, but still a real constraint).
“Lost in the middle.” Even with large windows (100K–1M+ tokens in current frontier models), retrieval quality across the context isn’t uniform — models tend to attend better to information near the start and end of a long context than buried in the middle. This is an empirical finding, not a hard architectural rule, but it shapes how you should structure long prompts (put critical instructions/facts near the edges, not deep in the middle of a huge document dump).
Why RAG exists at all. If you had infinite, uniformly-attended context, you wouldn’t need retrieval-augmented generation — you’d just stuff everything in. RAG is a workaround for context window limits and cost: retrieve only the relevant chunks, inject those, instead of paying token cost (and accuracy degradation) for dumping an entire knowledge base in every call.

Hallucination — Why It Happens, Mitigation Strategies

Why it happens, structurally: an LLM’s only trained objective is to predict the statistically most plausible next token given context — it has no built-in mechanism for “I don’t actually know this, let me say so.” When a query touches a gap in training data, an ambiguous fact, or something simply outside its knowledge, the model still produces fluent, confident-sounding text, because fluency is what it was optimized for, not calibrated uncertainty. It’s not “lying” in the human sense — there’s no internal model of truth being deliberately violated, just a probability distribution generating the next most likely-looking tokens.

This gets worse for: long-tail facts (rare entities, niche statistics), anything post-training-cutoff, precise numerical/citation claims, and multi-hop reasoning chains where an early subtle error compounds.

Mitigation strategies, in order of how directly they attack the problem:

Grounding via RAG / tool use. Give the model retrieved source documents or live tool output (search, database query) instead of relying on parametric memory. This is the single highest-leverage fix — you’re shifting the task from “recall” to “read and summarize,” which models do far more reliably.
Citations and attribution requirements. Prompt/require the model to cite specific source spans for claims; this both improves accuracy (forces grounding) and gives you a verification surface for the user.
Lower temperature / reduce sampling randomness for factual tasks — doesn’t eliminate hallucination but reduces variance on tasks where you want the most probable answer, not a creative one.
Structured output + verification layers — for high-stakes pipelines, have a second pass (another model call, or deterministic code) check claims against ground truth before they reach the user.
Self-consistency / multiple sampling — generate several independent answers and check agreement; divergence is a signal of low confidence.
Honest abstention prompting — explicitly instruct the model that it’s acceptable, even preferred, to say “I don’t know” or “I’m not certain” rather than guessing. This alone measurably reduces confident wrong answers, though it’s not a complete fix.

None of these fully eliminate hallucination — it’s a property of the architecture, not a bug to be patched out. The architectural job is risk-managing it: grounding where facts matter, human review where stakes are high, and clear UX signaling of confidence/uncertainty.

Model Families Landscape — GPT, Claude, Gemini, Llama, Open vs Closed Weight

Closed-weight (proprietary, API-only):

GPT family (OpenAI) — GPT-4-class and successor reasoning models (o-series/o1-style). Broad developer ecosystem, first-mover on ChatGPT consumer adoption.
Claude family (Anthropic) — strong on long-context reasoning, coding, and agentic tool-use; emphasis on safety/alignment research baked into training. Available via API, Claude.ai, AWS Bedrock, Google Vertex.
Gemini family (Google DeepMind) — natively multimodal architecture, very large context windows, deep integration with Google Cloud/Workspace.

You never get the weights for these — you call an API, the model runs on the provider’s infrastructure, and you have no ability to inspect or self-host the parameters.

Open-weight (downloadable, self-hostable):

Llama family (Meta) — the dominant open-weight line; weights are downloadable and can be self-hosted, fine-tuned, and run on your own infrastructure.
Other notable open-weight families: Mistral, Qwen (Alibaba), DeepSeek, Gemma (Google’s open line, distinct from closed Gemini).

“Open weight” ≠ “open source” in the strict sense — you typically get the trained parameters and permission to use/fine-tune them under a license, but not necessarily the training data or full training code. Still, it’s the meaningful distinction for architecture decisions: open-weight models you can run inside your own VPC (data never leaves your boundary), fine-tune on proprietary data, and control the entire serving stack — at the cost of owning the GPU provisioning, scaling, and ops burden yourself.

The architectural decision this maps to, directly analogous to build-vs-buy in cloud infra:

Factor	Closed-weight API	Open-weight self-hosted
Data residency / compliance	Data leaves your boundary (unless using VPC-isolated offerings like Bedrock)	Stays inside your infra
Ops burden	None — fully managed	You own GPU provisioning, scaling, patching
Cost model	Pay-per-token, scales with usage	Capex/reserved infra, better at high sustained volume
Frontier capability	Generally ahead on hardest reasoning tasks	Closing gap, especially for narrow fine-tuned tasks
Customization	Prompting, RAG, limited fine-tuning APIs	Full fine-tuning, architecture-level control

For most enterprise contexts (your Delta/World Bank/healthcare client background), the real decision driver is usually data governance and compliance posture more than raw capability — which is why VPC-isolated managed offerings (Bedrock, Vertex private endpoints) exist as a middle ground between fully open self-hosting and fully external API calls.