Databases & Data Architecture

Databases & Data Architecture

1. Relational Database Service (RDS)

AWS RDS manages traditional relational databases, abstracting away hardware provisioning and patching while offering robust high-availability and scaling patterns.

  • Multi-AZ (High Availability): * How it works: Synchronously replicates data to a standby instance in a different Availability Zone (AZ).

    • Purpose: Disaster recovery and high availability. The standby instance cannot take read traffic.

  • Read Replicas (Scalability): * How it works: Asynchronously replicates data to one or more reader instances (can be cross-region).

    • Purpose: Scaling read-heavy workloads and offloading reporting queries.

  • Failover Mechanism: * If the primary Multi-AZ instance fails, AWS automatically flips the DNS endpoint to the standby instance. This typically completes in 60–120 seconds with minimal data loss due to the synchronous nature of the replication.

2. Amazon DynamoDB

DynamoDB is a fully managed, serverless, NoSQL database designed for single-digit millisecond latency at any scale.

  • Partitions: Data is divided into physical storage segments (partitions) backed by SSDs. DynamoDB uses the Partition Key (PK) hash to determine which partition stores the item. Partitions split automatically based on size (>$10\text{ GB}$) or throughput exhaustion ($1,000$ WCU / $3,000$ RCU).

  • Global Secondary Indexes (GSIs):

    • Can be created at any time.

    • Can have a completely different PK and Sort Key (SK) from the base table.

    • Queries on GSIs are eventually consistent only.

  • Local Secondary Indexes (LSIs):

    • Must be created at table creation time.

    • Must share the same PK as the base table, but have a different SK.

    • Supports strongly consistent reads.

  • DynamoDB Streams: Captures a time-ordered sequence of item-level modifications (insert, update, delete) in real-time. Excellent for triggering AWS Lambda functions for event-driven architectures.

  • Time to Live (TTL): Automatically deletes items after a specified timestamp expires, at no extra cost, helping manage storage footprint without manual script maintenance.

3. Amazon Aurora Architecture

Aurora is AWS’s cloud-native relational database engine (MySQL/PostgreSQL compatible) engineered to decouple compute and storage.

  • Decoupled Storage Layer: Storage is completely independent of compute instances. Data is stripped across a virtualized, SSD-backed storage volume spanning 3 AZs with 6-way replication (2 copies per AZ).

  • Compute Scalability: You can spin up to 15 Read Replicas. Because all replicas share the exact same underlying storage volume, replication lag is near-zero (typically milliseconds), and failovers take under 30 seconds.

  • Aurora Serverless (v2): Scales compute up and down instantly in fractions of Aurora Capacity Units (ACUs) based on application demand, avoiding the need to over-provision for peak loads.

4. Analytics & ETL: Redshift, EMR, Glue, Athena

Service Type Core Use Case Key Characteristic
Amazon Redshift Data Warehouse Complex analytical querying (OLAP) over enterprise-scale data. Columnar storage, massive parallel processing (MPP).
Amazon EMR Big Data Framework Running open-source frameworks like Spark, Hive, Hadoop, and Presto. Highly customizable; ideal for massive, unstructured data processing.
AWS Glue Serverless ETL Discovering, preparing, and combining data for analytics. Contains the Glue Data Catalog (metadata metadata repository) and serverless ETL jobs.
Amazon Athena Serverless Query Ad-hoc querying of data directly in Amazon S3 using standard SQL. Pay-per-TB-scanned; zero infrastructure to manage.

5. Caching Strategies

Moving data closer to application compute to eliminate database bottlenecks.

Amazon ElastiCache (Redis / Memcached)

  • Application-level cache: Requires rewriting code to check the cache before hitting the database (Lazy Loading or Write-Through).

  • Redis: Supports complex data structures, geospatial data, pub/sub, and replication/high availability.

  • Memcached: Pure, simple key-value store optimized for demand spikes; multi-threaded.

DynamoDB Accelerator (DAX)

  • Inline-level cache: Sit directly in front of DynamoDB.

  • No code rewrite: You use the exact same DynamoDB API. If a cache miss occurs, DAX fetches it from DynamoDB seamlessly.

  • Performance: Drops read response times from single-digit milliseconds to microseconds.

6. Data Lake Architecture

A modern architectural pattern that allows organizations to store all their data (structured, semi-structured, unstructured) in a centralized, secure repository at any scale.

  • AWS Lake Formation: Simplifies building a data lake. It sits on top of S3 and centralizes security management. You can define granular, column-level and row-level access permissions across various analytics engines (like Athena or Redshift Spectrum) from a single location.

  • Apache Iceberg: An open-source, high-performance table format for massive analytic datasets. When implemented on S3, Iceberg brings ACID transactions, time-travel queries (querying historical snapshots), and schema evolution to the data lake, making object storage behave more like a robust relational data warehouse.