AI in Enterprise Architecture

AI in Enterprise Architecture

Building AI-Powered Microservices — Where AI Fits in Your Existing Architecture

The mistake to avoid first: treating “the LLM” as the architecture, rather than as one more dependency a service calls — like a database, a queue, or a third-party API. Once you frame it that way, it slots into patterns you already use.

Where it actually fits:

  • As a synchronous request-time dependency — a service calls the LLM inline as part of handling a request (e.g., a support-ticket service calls an LLM to draft a response). Same latency/timeout/retry/circuit-breaker discipline you’d apply to any external API call — except LLM calls have meaningfully higher and more variable latency (seconds, not milliseconds), so synchronous calls in a request path need generous timeout budgets and a fallback path for when the call is slow or fails.
  • As an async/event-driven worker — an LLM-calling service subscribes to a queue (SQS, EventBridge) and processes tasks asynchronously (e.g., summarizing every new document that lands in S3). This is the better fit for anything that doesn’t need sub-second response — and it naturally absorbs the latency variance and lets you control concurrency/cost via queue depth and worker scaling, exactly like any other async worker pool.
  • As a sidecar/enrichment layer — an existing service’s data pipeline gets an LLM step inserted (e.g., embedding generation before writing to a vector store, or classification before routing). This keeps the LLM dependency isolated to one stage rather than threading it through your core service logic.
  • Where it should NOT go: directly embedded in your core transactional/business-logic path with no fallback, no timeout strategy, and no separation from deterministic logic. If the LLM call fails or hallucinates, deterministic business rules (validation, authorization, anything regulatory) must not depend on it.

The pattern that scales: treat the LLM-calling logic as its own microservice behind a clean interface — your other services call “the summarization service” or “the classification service,” not “the LLM” directly. This gives you a single place to swap models, add caching, add fallback, and instrument observability, instead of that logic being scattered across every service that happens to need AI.

AI Gateway Pattern — Routing, Cost Control, Model Fallback

An AI gateway sits between your services and the various model providers (Bedrock, OpenAI, Anthropic API, self-hosted), centralizing everything you’d otherwise have to duplicate per service. It’s the same architectural instinct as an API gateway in front of microservices — except the “backends” are LLM providers instead of internal services.

What it actually does:

  • Routing — directs each request to the right model based on task complexity, cost target, or latency requirement. A simple classification task gets routed to a small/cheap model; a complex multi-step reasoning task gets routed to a frontier model. This routing logic lives in one place instead of being hardcoded into every calling service.
  • Cost control — centralized rate limiting, per-team/per-application token budgets, and usage quotas. Without this layer, cost runs away because every team independently calls whatever model is convenient with no shared visibility or limits — the same governance problem you’d have with uncontrolled cloud spend across teams, just one layer up.
  • Model fallback — if the primary model/provider is down, rate-limited, or returns an error, the gateway automatically retries against a secondary model or provider. This is a direct analogue to multi-AZ/multi-region failover, except failing over to a different model rather than a different infrastructure zone — and you have to design for the fact that a fallback model may produce subtly different output quality, not just “the same service running elsewhere.”
  • Caching and normalization — a single point to apply prompt caching (below), normalize request/response formats across providers so calling services don’t need provider-specific code, and apply consistent guardrails (PII redaction, content filtering) before requests leave your boundary or before responses return to callers.

For AWS-centric architecture, Bedrock itself partially absorbs this pattern — it gives you a single API surface across multiple model providers (Anthropic, Meta, Amazon’s own models) with unified IAM, logging, and some routing/fallback capability built in. Whether you still need a custom gateway layer on top depends on whether you need cross-cloud routing (e.g., Bedrock + Azure OpenAI + self-hosted) or governance requirements Bedrock doesn’t natively expose.

Caching Strategies for LLM Calls (Cost/Latency)

LLM calls are expensive and slow relative to typical backend calls, so caching has outsized leverage here — more than for a typical microservice call.

  • Exact-match response caching — if the same prompt (or prompt + parameters) has been seen before, return the cached response instead of calling the model again. Straightforward, like any standard cache, but LLM prompts rarely repeat verbatim in conversational use, limiting hit rate for chat-style traffic. Much higher hit rate for structured/repeated tasks (e.g., “classify this support ticket category” on similar recurring inputs).
  • Prompt caching (provider-native) — most providers (Anthropic, OpenAI, Bedrock) now support caching the static/repeated portion of a prompt — system instructions, few-shot examples, a large document you reuse across many queries — so the model doesn’t pay full processing cost for that portion on every call, only for the variable part (the actual user query). This is the highest-leverage caching technique for RAG and agentic systems where a large stable context (tool definitions, system prompt, retrieved documents) gets re-sent on every turn of a conversation.
  • Semantic caching — instead of requiring an exact prompt match, embed the incoming query and check similarity against previously cached query embeddings; if similarity exceeds a threshold, return the cached response (or use it as a strong prior). Higher hit rate than exact-match for natural-language queries that vary in phrasing but ask the same thing, at the cost of a small risk of returning a stale/wrong answer for a query that’s similar-but-not-identical — so this needs a conservative similarity threshold and is best applied to FAQ-style or narrow-domain queries, not open-ended conversation.
  • Embedding cache (for RAG pipelines) — cache the embedding vectors for documents that don’t change, so re-indexing or repeated ingestion doesn’t redundantly recompute embeddings for unchanged content.

The architectural tradeoff is the standard caching one — staleness vs. cost/latency — except here “staleness” can mean returning an answer that’s not just out-of-date but factually wrong for the new query if your cache-matching logic is too loose. Cache invalidation strategy (TTL, content-hash-based invalidation when source documents change) needs explicit design, same as any cache layer.

Observability for AI Systems — Tracing Prompts, Token Usage, Drift Detection

Standard observability (logs, metrics, traces) still applies, but LLM systems need additional dimensions that traditional APM tools don’t capture by default:

  • Prompt/response tracing — full visibility into the exact prompt sent (including injected RAG context, system prompt, conversation history) and the exact response received, per request, with the ability to replay a specific failure. This is the AI-specific equivalent of distributed tracing (think X-Ray/OpenTelemetry spans) — except the “payload” you need visibility into is the actual natural-language content, not just latency/status-code metadata. Tools like LangSmith, Langfuse, or Bedrock’s own logging integrations are purpose-built for this.
  • Token usage and cost tracking — per-request, per-user, per-feature token counts (input + output separately, since most providers price them differently), rolled up into cost dashboards. This is your AWS Cost Explorer equivalent for AI spend, and without it, cost surprises are common because token usage isn’t intuitively visible the way compute-hours or storage-GB are.
  • Quality/drift detection — this is the genuinely new category with no direct traditional-ops analogue. You need to track things like: response relevance/groundedness scores (does the output actually reflect the retrieved context, for RAG systems), hallucination rate sampling (periodic human or model-graded review of a sample of outputs), and model drift — when the underlying model is silently updated by the provider (this happens with hosted APIs) and your downstream behavior shifts even though you changed nothing on your end. Unlike a traditional dependency version pin, you often don’t control or even get notified of provider-side model updates, so periodic evaluation against a fixed golden dataset is the practical mitigation — re-run a known test set regularly and alert on output-quality regression, the same instinct as a regression test suite, just scored qualitatively rather than pass/fail.
  • Guardrail/safety event logging — track when content filters, PII redaction, or other safety layers actually trigger, both for compliance audit trails and to catch false-positive rates that hurt UX.

The architectural framing: traditional observability answers “is the system up and fast.” AI observability has to additionally answer “is the system still correct,” and correctness isn’t a status code — it requires sampling, scoring (often via a second LLM call acting as a judge), and human review loops baked into the pipeline, not just dashboards.

AWS AI Stack — Bedrock, SageMaker, Q Business

Mapping these to where they actually sit in an AWS-native architecture:

Amazon Bedrock — the managed foundation-model layer. Single API for calling multiple foundation models (Anthropic Claude, Meta Llama, Amazon’s own Titan/Nova models, others) without managing any infrastructure. Handles model invocation, fine-tuning (for supported models), Knowledge Bases (Bedrock’s managed RAG — handles ingestion, chunking, embedding, and vector store integration for you), Agents (Bedrock’s managed agent/orchestration framework with tool-calling), and Guardrails (built-in content filtering/PII redaction). This is your default choice when you want GenAI capability with AWS-native IAM, VPC isolation, and billing — the equivalent of choosing a managed service over self-hosting whenever the managed offering meets your requirements.

SageMaker — the platform for when you need to train or fully customize your own models, not just call a foundation model. Covers the full ML lifecycle: data labeling, training jobs (including fine-tuning open-weight models with full control), hyperparameter tuning, model registry, and custom inference endpoint hosting. This is where you go when Bedrock’s fine-tuning options aren’t sufficient — e.g., you’re training a model from scratch, doing heavy customization of an open-weight model, or running classical ML (non-LLM) workloads alongside your GenAI work. Think of SageMaker as the lower-level, more control-but-more-ops-burden layer relative to Bedrock’s managed simplicity — same tradeoff axis as EC2 vs. Lambda.

Amazon Q Business — the highest-level, most opinionated offering: a managed, enterprise-ready conversational assistant that connects to your existing data sources (S3, SharePoint, Salesforce, Confluence, etc.) via pre-built connectors and handles RAG, access control (respects existing source-system permissions, not just a flat knowledge base), and the chat UI out of the box. This is the right reach when the requirement is genuinely “give our employees a chat assistant over our internal knowledge” and you don’t want to build and maintain the RAG pipeline, ingestion connectors, and UI yourself. It trades flexibility for speed-to-value — appropriate when the use case is a known pattern Q Business already covers, not when you need custom agentic behavior or non-standard data sources.

The decision ladder, cheapest-effort first: Q Business if your need matches its pre-built assistant pattern → Bedrock (Knowledge Bases + Agents) if you need custom application logic but don’t want to manage model infrastructure → SageMaker if you need to train/fine-tune at a level Bedrock doesn’t expose, or run non-foundation-model ML alongside it. This mirrors the same AWS pattern you already apply elsewhere (Q Business ≈ a managed SaaS-like service, Bedrock ≈ a managed platform service, SageMaker ≈ raw infrastructure-level control) — most AI architecture decisions on AWS are really “how much of the stack do I want to own” decisions, same as everywhere else in the platform.