mirror of https://github.com/nubenetes/awesome-kubernetes.git synced 2026-07-12 18:00:37 +00:00

Files

Nubenetes Bot 19b7b44e20 feat(v2): restore in-page search/tag filter & drop redundant Markdown TOC

Fix: the per-page resource search box with maturity tag-pills stopped
appearing after navigation.instant was enabled (DOMContentLoaded fires only
on first SPA load). Reworked v2_filter.js to init via Material's document$
observable with a DOMContentLoaded fallback, guard double-injection, bind the
clickable-tag delegation once, and add a no-results state. Cache-bust ?v=2.9.12.

Change: removed the duplicated Markdown "## Table of Contents" from all 156 V2
content pages and from v2_optimizer.py — it forced hundreds of links of scroll
on large pages. Replaced by Material's native sticky "On this page" TOC
(removed toc.integrate, added toc.follow). tags.md keeps its TOC (it is an index).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-06-19 18:49:15 +02:00

11 KiB

Raw Permalink Blame History

Big Data and Kubernetes Big Data

!!! tip "Nubenetes V2 Elite Portal" You are browsing the AI-Curated V2 Elite Edition. Looking for the exhaustive list of references? Check out the V1 Historical Archive.

!!! info "Architectural Context" Detailed reference for Big Data and Kubernetes Big Data in the context of The Container Stack.

Architectural Foundations

Kubernetes Tools

General Reference

tomlous.medium.com: CI/CD for Data Engineers. Reliably Deploying Scala Spark' containers for Kubernetes with Github Actions [COMMUNITY-TOOL] — A curated technical resource and architectural guide covering tomlous.medium.com: CI/CD for Data Engineers. Reliably Deploying Scala Spark' containers for Kubernetes with Github Actions in the Kubernetes Tools ecosystem.
dzone: Run and Scale an Apache Spark Application on Kubernetes [COMMUNITY-TOOL] — A curated technical resource and architectural guide covering dzone: Run and Scale an Apache Spark Application on Kubernetes in the Kubernetes Tools ecosystem.
dzone: Running Apache Spark on Kubernetes [COMMUNITY-TOOL] — A curated technical resource and architectural guide covering dzone: Running Apache Spark on Kubernetes in the Kubernetes Tools ecosystem.
medium: Running Apache Spark on Kubernetes [COMMUNITY-TOOL] — A curated technical resource and architectural guide covering medium: Running Apache Spark on Kubernetes in the Kubernetes Tools ecosystem.
levelup.gitconnected.com: Master SparkML: Practical Guide for Machine Learning [COMMUNITY-TOOL] — A curated technical resource and architectural guide covering levelup.gitconnected.com: Master SparkML: Practical Guide for Machine Learning in the Kubernetes Tools ecosystem.

Data and AI

Apache Spark

Cloud Migration

(2021) itnext.io: Migrating Apache Spark workloads from AWS EMR to Kubernetes [MARKDOWN CONTENT] [ADVANCED LEVEL] 🌟🌟🌟🌟 [ENTERPRISE-STABLE] — A highly granular migration case study describing the engineering process of moving Apache Spark analytical jobs from AWS EMR to Amazon EKS. Addresses key architectural trade-offs such as node grouping, instance types, spot instance lifecycle management, and driver/executor scheduling. Live Grounding underscores that moving Spark workloads to Kubernetes optimizes compute utilization and reduces vendor lock-in.

Cost Optimization

(2023) spot.io: Setting up, Managing & Monitoring Spark on Kubernetes [MARKDOWN CONTENT] 🌟🌟🌟 [COMMUNITY-TOOL] — Discusses the financial and operational mechanics of running Spark executor pods on heterogeneous Kubernetes clusters. Emphasizes automated cost optimization, spot instance utilization, and intelligent scaling practices. Live Grounding shows that dynamically shifting Spark executor tasks to spot instances under strict fallback rules is essential for scaling modern, cost-sensitive big data pipelines.

OpenShift

(2022) cloud.redhat.com: Getting Started running Spark workloads on OpenShift [MARKDOWN CONTENT] 🌟🌟🌟 [COMMUNITY-TOOL] [GUIDE] — Provides an enterprise guide to running containerized Apache Spark workloads within Red Hat OpenShift, highlighting security-first practices and security context constraints (SCCs). Illustrates leveraging OpenShift's cluster monitoring and dynamic storage classes for big data analytics. Live Grounding confirms OpenShift remains a preferred, secure platform for regulated enterprises executing large-scale analytical tasks.

Performance and Tuning

(2022) ==coderstan.com: Apache Spark on Kubernetes—Lessons Learned from Launching Millions of Spark Executors (Databricks Data+AI Summit 2022)== [MARKDOWN CONTENT] [ADVANCED LEVEL] 🌟🌟🌟🌟🌟 [DE FACTO STANDARD] — Shares high-scale operational insights and hard-earned engineering lessons from managing millions of Spark executors on Kubernetes clusters. Details scheduling bottlenecks, DNS limits, local SSD configurations, and executor startup latency optimizations. Live Grounding highlights these exact tuning principles as standard operating procedures for operating hyperscale, cost-efficient data platforms today.

Streaming and Scheduling

(2023) docs.databricks.com: Use scheduler pools for multiple streaming workloads [MARKDOWN CONTENT] [ADVANCED LEVEL] 🌟🌟🌟🌟 [ENTERPRISE-STABLE] — Deep dives into configuring Spark scheduler pools to enforce Fair Scheduling (FAIR) when running multiple concurrent Structured Streaming queries in a shared production workspace. Prevents heavy resource queries from starving lightweight streaming jobs. Live Grounding verifies that proper allocation of pool weights remains a mandatory configuration practice for robust multi-tenant streaming pipelines.

Batch Scheduling

Kueue

(2024) Red Hat Build of Kueue [MARKDOWN CONTENT] [ADVANCED LEVEL] 🌟🌟🌟🌟 [ENTERPRISE-STABLE] — Focuses on Red Hat's enterprise integration of Kueue, a Kubernetes-native job queueing system designed to manage resource quotas, tenant isolation, and fair-share scheduling for high-performance AI/ML and batch workloads. Live Grounding confirms Kueue is crucial in 2026 for orchestrating GPU and CPU cluster resource allocation dynamically across large-scale enterprise clusters.

Cloud Platforms

Databricks

(2022) aprenderbigdata.com: Databricks: Introducción a Spark en la nube [SPANISH CONTENT] 🌟🌟🌟 [COMMUNITY-TOOL] [GUIDE] — Offers a clear Spanish-language introduction to Databricks, highlighting its managed Apache Spark ecosystem across AWS and Azure. Demystifies collaborative notebooks, workspace management, and optimized runtimes. Live Grounding shows that Databricks continues to expand its lakehouse dominance, abstracting underlying infrastructure away from data engineers while offering native cloud integrations.

Data Pipelines

Apache Spark (1)

(2022) hevodata.com: Building Apache Spark Data Pipeline? Made Easy 101 🌟 [MARKDOWN CONTENT] 🌟🌟🌟 [COMMUNITY-TOOL] [GUIDE] — An introductory primer outlining the core concepts of building end-to-end data processing pipelines with Apache Spark. Focuses on data ingestion, transformations, and loading into modern target data warehouses. Live Grounding confirms that while Spark remains a foundational ETL standard, contemporary architectures increasingly wrap these pipelines inside orchestrated dbt-on-Kubernetes or SparkOperator setups.

Databricks (1)

Governance

(2024) github.com/databrickslabs/ucx: Databricks Labs UCX ⭐ 308 [PYTHON CONTENT] [ADVANCED LEVEL] 🌟🌟🌟🌟 [ENTERPRISE-STABLE] [LEGACY] — The Databricks Labs UCX project provides a specialized framework designed to upgrade legacy Databricks workspaces to Unity Catalog governance standards. Simplifies catalog and privilege migrations automatically. Live Grounding confirms UCX is standard for enterprise organizations establishing secure, centralized data governance, metadata isolation, and unified access controls.

Market Analysis

Adoption Trends

(2021) opensourceforu.com: Kubernetes Adoption Widespread for Big Data: Survey [MARKDOWN CONTENT] 🌟🌟🌟 [COMMUNITY-TOOL] — Details industry survey results illustrating the widespread migration of Big Data and stateful analytics workloads onto Kubernetes. Shows the transition from static, dedicated bare-metal clusters to dynamic, container-orchestrated platforms. Live Grounding confirms this historical trajectory has culminated in 2026, where cloud-native orchestration is the unquestioned standard for running Spark, Flink, and ML training pipelines.

💡 Explore Related: Kubernetes Storage | Kubernetes Alternatives | Kubernetes Client Libraries

11 KiB Raw Permalink Blame History