awesome-kubernetes/v2-docs/aws-storage.md

AWS Storage. S3 & EBS. AWS Storage Gateway

!!! info "Architectural Context" Detailed reference for AWS Storage. S3 & EBS. AWS Storage Gateway in the context of Cloud Providers (Hyperscalers).

Standard Reference

• Making Requests to Amazon S3 over IPv6 [COMMUNITY-TOOL]
• Getting Started with AWS Storage Gateway [COMMUNITY-TOOL]
• awstip.com: Uploading files to S3 through API Gateway [COMMUNITY-TOOL]

Cloud Infrastructure

AWS Storage

Amazon EBS

• How to Build Sparse EBS Volumes for Fun and Easy Snapshotting [ADVANCED LEVEL] [COMMUNITY-TOOL] — An advanced architectural post detailing how to create sparse files inside loop devices on EBS volumes to minimize storage footprints. Analyzes the financial and performance benefits of sparse block allocations during automated snapshots.
• devopscube.com: How to Automate EBS Snapshot Creation, Retention and Deletion [COMMUNITY-TOOL] — Explains various mechanisms for lifecycle management of AWS EBS snapshots. Covers automated configuration using Amazon DLM (Data Lifecycle Manager), custom Lambda functions, and AWS CLI scripts.
• percona.com: Performance of Various EBS Storage Types in AWS [ADVANCED LEVEL] [COMMUNITY-TOOL] — An in-depth performance analysis of EBS volume classes (gp2, gp3, io2, etc.) when hosting database engines. Contains IOPS benchmarking, throughput limits, and cost-efficiency matrices vital for database architects.
• dev.to: Adding an EBS volume to a running AWS EC2 Instance [COMMUNITY-TOOL] — A step-by-step tutorial on attaching, partition formatting (ext4/xfs), and mounting a new EBS volume onto an active Linux EC2 instance without triggering server downtime.

Amazon EFS

• EFS Elastic File System [COMMUNITY-TOOL] — The historic production-ready announcement of AWS Elastic File System (EFS). Establishes the core characteristics of shared NFS storage scaling dynamically for concurrent EC2/container instances.
• Amazon Elastic File System triples read throughput [COMMUNITY-TOOL] — Announcement detailing architectural performance enhancements to AWS EFS. Details how the platform tripled read limits, providing immediate benefits to read-heavy workloads like distributed machine learning or CMS host arrays.
• infoq.com: AWS Transfer Family Introduces Support for EFS [COMMUNITY-TOOL] — Reports on the strategic integration between AWS Transfer Family (SFTP/FTPS/FTP) and EFS. Enables developers to expose shared file systems securely directly to remote partners without managing SFTP instance clusters.

Amazon S3

• (2023) blog.awsfundamentals.com: AWS S3 Sync - An Extensive Guide 🌟🌟🌟🌟 [ENTERPRISE-STABLE] — A detailed, high-density guide on utilizing the aws s3 sync CLI command. Includes performance tuning parameters like concurrency configs, exclusion patterns, and handling large scales securely.
• (2021) acloudguru.com: S3 Glacier Instant Retrieval deep dive: Which S3 Storage Class is right for me? 🌟🌟🌟 [COMMUNITY-TOOL] — A deep dive into S3 Glacier Instant Retrieval, analyzing costs and access latency. Explains the strategic sweet spot of the tier for archival data that requires millisecond retrieval speeds without standard Glacier delays.
• S3 FAQ [DOCUMENTATION] [DE FACTO STANDARD] — The official AWS documentation addressing technical details of Simple Storage Service (S3). Provides absolute specifications for consistency models, replication capabilities, access control lists, and durability guarantees.
• cloudkatha.com: Is S3 Region Specific or Global? What do you think? [COMMUNITY-TOOL] — Clarifies the architectural paradox of Amazon S3's namespace being globally unique while data is stored physically in regional clusters. Details the implications on DNS, endpoint routing, and latency.
• cloudkatha.com: This is why S3 Bucket Names are unique Globally [COMMUNITY-TOOL] — Explores the technical constraints under the hood that necessitate globally unique S3 bucket names. Outlines DNS mapping to S3 regional endpoints and the routing restrictions of the HTTP REST API.
• cloudkatha.com: AWS S3 Storage Classes: Everything You Need to Know [COMMUNITY-TOOL] — A complete comparative guide to S3 Storage Classes, detailing Standard, Intelligent-Tiering, Standard-IA, One Zone-IA, and Glacier options. Analyzes pricing structures, retrieval latency, and redundancy across availability zones.
• A step-by-step guide to synchronize data between Amazon S3 buckets [COMMUNITY-TOOL] — Official AWS blog showing patterns for cross-bucket data synchronization. Compares the use of AWS CLI Sync, S3 Batch Operations, and S3 Same-Region / Cross-Region Replication (SRR/CRR) for diverse use cases.

Analytics

• Monitor Amazon S3 activity using S3 server access logs and Pandas in Python [COMMUNITY-TOOL] — Shows how to parse raw S3 access logs using Python and Pandas. Walks through identifying bad actors, evaluating storage patterns, and monitoring data transfer metrics for cost control.

Hybrid Cloud

• Connect Amazon S3 File Gateway using AWS PrivateLink for Amazon S3 [ADVANCED LEVEL] [COMMUNITY-TOOL] — Architectural post detailing how to establish a secure, private connection from on-premises Storage Gateways to S3 via AWS PrivateLink. Prevents traffic traversal over the public internet, satisfying rigorous enterprise compliance models.

High Availability

Multi-Region

• Building an active-active, latency-based application across multiple Regions 🌟 [ADVANCED LEVEL] [ENTERPRISE-STABLE] — An elite architectural blueprint for active-active multi-region AWS applications. Uses S3 Cross-Region Replication (CRR), Route 53 latency-based routing, and DynamoDB Global Tables to design highly resilient services.

Object Storage

MinIO

• (2022) blog.min.io: Certificate-based Authentication for S3 [ADVANCED LEVEL] 🌟🌟🌟🌟 [ENTERPRISE-STABLE] — Explains how to implement robust Mutual TLS (mTLS) and client certificate-based authentication with MinIO. Outlines secure key exchanges and architectural mapping of certificates to IAM policies.

Storage

Distributed Filesystems

• Ceph: A Distributed Object, Block, and File Storage Platform ⭐ 16621 [ADVANCED LEVEL] [DE FACTO STANDARD] — The industry-standard unified, distributed storage system designed to provide excellent performance, reliability, and scalability.

• Provides object, block, and file storage within a single cluster.
• Acts as a foundational storage engine for large-scale Kubernetes PV platforms (Rook-Ceph) and private clouds.

---

💡 Explore Related: AWS Security | AWS Databases | AWS Backup