AWS Storage Architecture

AWS Storage Architecture Deep Dive & Guide

Choosing the right storage architecture requires balancing performance, cost, compliance, and data lifecycle requirements. This guide covers Amazon S3, EBS, EFS, FSx, and overall data resilience strategies.

1. Amazon S3 Deep Dive

Amazon Simple Storage Service (S3) is an object storage service offering industry-leading scalability, data availability, security, and performance.

Storage Classes

S3 offers a variety of storage classes tailored for different use cases and access patterns:

  • S3 Standard: High throughput, low latency. Designed for active, frequently accessed data (e.g., websites, mobile apps, gaming).

  • S3 Intelligent-Tiering: Automatically moves data between frequent, infrequent, and archive tiers based on access patterns without operational overhead or retrieval fees.

  • S3 Standard-Infrequent Access (Standard-IA): For data accessed less frequently but requires rapid access when needed. Lower storage cost but charges a per-GB retrieval fee.

  • S3 One Zone-IA: Similar to Standard-IA but stores data in a single Availability Zone (AZ). Costs 20% less but lacks multi-AZ resilience.

  • S3 Glacier Instant Retrieval: Archive storage for data accessed a few times a year with millisecond retrieval.

  • S3 Glacier Flexible Retrieval: Archive storage with retrieval times from minutes (Expedited) to hours (Standard/Bulk).

  • S3 Glacier Deep Archive: Lowest-cost storage class. Designed for long-term retention (7–10 years) with retrieval times within 12 hours.

Lifecycle Management

S3 Lifecycle configurations manage objects efficiently throughout their lifespan using two types of actions:

  1. Transition Actions: Define when objects transition to another storage class (e.g., move to Standard-IA 30 days after creation, then to Glacier after 90 days).

  2. Expiration Actions: Define when objects are automatically deleted.

S3 Replication

Replication enables automatic, asynchronous copying of objects across buckets.

  • Cross-Region Replication (CRR): Copies objects across different AWS Regions for compliance, disaster recovery (DR), or geographical proximity.

  • Same-Region Replication (SRR): Copies objects within the same AWS Region for log aggregation, testing/production separation, or strict sovereignty laws.

⚠️ Note: S3 Versioning must be enabled on both the source and destination buckets for replication to function.

Encryption

S3 ensures data security both in transit (via TLS) and at rest using three primary methods:

  • SSE-S3 (Amazon Managed Keys): S3 encrypts the object using AES-256. AWS fully manages the keys.

  • SSE-KMS (AWS KMS Keys): Uses AWS Key Management Service. Provides audit trails of key usage and allows separate permissions for the key and the object.

  • SSE-C (Customer-Provided Keys): You manage the encryption keys; AWS manages the encryption and decryption processes but does not store your keys.

2. Block vs. File: EBS & EFS Architecture

Amazon EBS (Elastic Block Store)

EBS provides block-level storage volumes for use with EC2 instances. It acts like a raw, unformatted physical hard drive.

  • Architecture: EBS volumes are replicated within a single Availability Zone (AZ) to protect against component failure.

  • Volume Types:

    • SSD-backed (gp3, io2 Block Express): Optimized for transactional workloads, databases, and boot volumes demanding high IOPS.

    • HDD-backed (st1, sc1): Optimized for large, sequential workloads like big data, data warehousing, and log processing.

  • Scope: Traditional EBS volumes can only be attached to a single EC2 instance within the same AZ (unless using EBS Multi-Attach on specific io1/io2 provisioned IOPS volumes, which allows attaching to multiple instances in the same AZ).

Amazon EFS (Elastic File System)

EFS provides a serverless, fully managed, elastic file system utilizing the NFSv4 protocol.

  • Architecture: EFS is a regional service. It stores data across multiple AZs concurrently, offering superior durability and availability compared to EBS.

  • Access: It can be mounted by thousands of EC2 instances, on-premises servers, and containers (ECS/EKS) simultaneously across different AZs.

  • Performance Modes: Offers General Purpose (ideal for latencies-sensitive use cases like CMS or DevOps tools) and Max I/O (scales to higher throughput and IOPS for big data analysis).

3. Amazon FSx: Specialized File Systems

Amazon FSx makes it easy and cost-effective to launch, run, and scale feature-rich, high-performance file systems powered by popular commercial and open-source file systems.

FSx Flavor Target Workloads Core Features Protocols
FSx for Windows File Server Enterprise Windows applications, Active Directory integration, user shares. Fully managed Windows file shares, natively supports NTFS permissions, shadow copies, and SMB. SMB (Server Message Block)
FSx for Lustre High-Performance Computing (HPC), machine learning, financial modeling, video rendering. Sub-millisecond latencies, massive throughput (hundreds of GB/s), seamless integration with Amazon S3. Lustre client
FSx for NetApp ONTAP Multi-protocol enterprise workloads, lift-and-shift of on-premises ONTAP storage. Data deduplication, compression, instant cloning (FlexClone), snapshots, cross-region replication. NFS, SMB, iSCSI

4. Data Durability & Consistency Models

Understanding how cloud storage handles data retention and state updates is crucial for application design.

Durability Comparison

  • Amazon S3: Designed for $99.999999999\%$ (11 9s) of durability by automatically replicating objects across a minimum of three geographically distinct AZs.

  • Amazon EFS: Also offers 11 9s of durability by replicating data across multiple AZs.

  • Amazon EBS: Designed for $99.999\%$ (5 9s) availability/durability within its single assigned AZ. It requires regular snapshots to S3 to achieve cross-AZ disaster resilience.

S3 Consistency Model

S3 offers strong read-after-write consistency for PUT and DELETE operations of objects in all AWS regions.

What this means: As soon as a write, update, or deletion is successful, any subsequent read request will immediately reflect the changes. There is no partial update state, and you will never receive stale data during a concurrent overwrite.

5. Backup & Archival Strategy

A robust backup architecture relies on automation, isolation, and compliance.

AWS Backup

Instead of managing custom snapshot scripts, AWS Backup acts as a centralized, policy-driven service to automate and manage backups across multiple AWS services (EBS, RDS, EFS, FSx, DynamoDB, etc.).

  • Backup Plans: Define retention periods, backup windows, and lifecycle rules (e.g., transition backups to cold storage after 30 days).

  • Cross-Account/Cross-Region Backup: Protects against ransomware and regional disasters by copying backups securely to entirely different AWS accounts or regions.

Archival Best Practices

  • Immutable Backups (S3 Object Lock): Enforce Write-Once-Read-Many (WORM) policies. Use Governance Mode (allows authorized users to delete locks) or Compliance Mode (nobody can delete the data or change the retention period, not even the root account) to prevent ransomware or accidental deletion.

  • S3 Vault Lock: For Glacier vaults, enforce compliance policies via a vault lock policy that cannot be altered once locked.

  • Monitoring: Use S3 Storage Lens to gain organization-wide visibility into object storage usage, cost-optimization opportunities, and data protection metrics.