awesome-kubernetes/docs/aws-data.md

AWS Big Data

1. Introduction
2. AWS Data Lake
3. AWS Data Pipeline (aka Big Data Pipelines or Data Streams)

Introduction

• aws.amazon.com/big-data
• blogs.aws.amazon.com/bigdata
• Querying Amazon Kinesis Streams Directly with SQL and Spark Streaming
• Using Spark SQL for ETL
• whizlabs.com: AWS Kinesis vs Kafka Apache

AWS Data Lake

• Building a Data Lake on AWS AWS provides a highly scalable, flexible, secure, and cost-effective solution for your organization to build a Data Lake – a data repository for both structured and unstructured data that is designed to be easily accessible for on-demand data analytics enabling you to answer questions as they arise.

AWS Data Pipeline (aka Big Data Pipelines or Data Streams)

• AWS Data Pipeline
• AWS Data Pipeline Documentation
• medium: No-Code Data Collect API on AWS A No-Code Data Collections mechanism for Big Data Pipelines on AWS.
• AWS Big Data Blog: Category - AWS Data Pipeline
  • (2026) AWS Glue 6.0 now available with 30% lower price and full Apache Iceberg v3 support 🌟 - AWS Glue 6.0 launches with a significant 30% cost reduction and native, deep integration with Apache Iceberg v3 for data lakes.

Amazon DynamoDB

• (2026) Amazon DynamoDB now supports real-time vector search at any scale 🌟 - Amazon DynamoDB introduces native real-time vector search capabilities for hyperscale AI applications.