github/awesome-kubernetes

Fork 0

mirror of https://github.com/nubenetes/awesome-kubernetes.git synced 2026-03-03 18:20:25 +00:00

Files

Inaki Fernandez b962617136 markdown lint

2022-10-02 22:51:53 +02:00

69 KiB

Raw Blame History

Cloud Based Integration & Messaging. Data Processing & Streaming (aka Data Pipeline). Open Data Hub

Message Queue in Kubernetes. Event-driven Messaging. Real-Time Data Streaming
RPC vs Messaging
Message Brokers
Cloud Based Integration. Integration Platform-as-a-Service (iPaaS) solutions
1. Red Hat Fuse and Red Hat Fuse Online
2. Syndesis open source integration platform
Debezium open source distributed platform for Change Data Capture (CDC) software design pattern
Red Hat Integration service registry and Apicurio
Data Mesh
Data Processing (aka Streaming Data, Data Pipeline or Big Data Pipeline)
Workflow Engines
Zeebe
1. Apache Airflow
2. Couler
Red Hat AMQ (ActiveMQ Artemis broker and Apache Kafka)
Open Data Hub AI-as-a-Service (AIaaS) platform
KEDA. Kubernetes Event Driven Autoscaling
Integration Platform as a Solution (iPaaS). Platforms for collecting, storing and routing customer event data
1. IpaaS Vendors
eBooks
Related
Questions and Answers
Videos
Tweets

Message Queue in Kubernetes. Event-driven Messaging. Real-Time Data Streaming

Wikipedia: Message Broker
Wikipedia: Event-driven messaging
Wikipedia: Streaming Data
nginx.com: Event-Driven Data Management for Microservices 🌟
dzone: Event-Driven Architecture as a Strategy Event-driven architecture provides five key benefits to modern application architecture: scalability, resilience, agility, data sharing, and cloud enabling.
infoq.com: From Monolith to Event-Driven: Finding Seams in Your Future Architecture
wikipedia: Enterprise service bus
thenewstack.io: The Rise of the Event Streaming Database 🌟
cncf.io: The need for Kubernetes Native Messaging Platform in Hybrid Cloud Environment
wiprodigital.com: A Guide to Enterprise Event-Driven Architecture
medium: Introduction to Event-Driven Architecture 🌟 The essential concepts that every developer should know
ibm.com: Event-driven cloud-native applications (microservices) The event backbone is being part of the microservices mesh, providing the publish-and-subscribe communication between microservices and enabling the support of loosely coupled event-driven microservices.
stackoverflow.blog: How event-driven architecture solves modern web app problems 🌟 In this article, we’ll discuss some of the problems driving innovation in modern web development. Then we’ll dive into the basics of event-driven architecture (EDA), which tries to address these problems by thinking about back-end architecture in a novel way.
sebalopezz.medium.com: Monolith to Microservices + Event-Driven Architecture 🌟
confluent.io: Event-Driven Microservices Architecture (white paper) 🌟 Microservices are an architectural pattern that structures an application as a collection of small, loosely coupled services that operate together to achieve a common goal. Because they work independently, they can be added, removed, or upgraded without interfering with other applications. While there are numerous benefits to microservices architecture, like easier deployment and testing, improved productivity, flexibility, and scalability, they also pose a few disadvantages, as independently run microservices require a seamless method of communication to operate as one larger application. Event-driven microservices allow for real-time microservices communication, enabling data to be consumed in the form of events before they’re even requested. In this white paper, we’ll cover how event-driven microservices work, presenting a sample currency exchange platform to illustrate the design and architecture of an application composed of event-driven microservices using Apache Kafka® and Confluent Platform. We also discuss other aspects of microservices architectures, such as team structure, continuous delivery, deployment, and testing. Lastly, we discuss how Apache Kafka and Confluent Platform enable and extend core principles of microservices, including decoupling, separation of concerns, agility, and real-time streaming of event data.
redhat.com: Event-driven architecture: Understanding the essential benefits 🌟 Event-driven architectures bring significant benefits when managing many endpoints, but it also has its complexities to be aware of.
medium: Introduction to Message Queues 🌟
headspring.com: Is Kafka or RabbitMQ the right messaging tool for you?
dzone: Why Pub/Sub Isn’t Enough for Modern Apps Pub/Sub is the most appropriate way of architecting the delivery side of an event-driven architecture (especially for the web).
baeldung.com: Pub-Sub vs. Message Queues 🌟
engineering.atspotify.com: Spotify’s Event Delivery – The Road to the Cloud (Part I)
medium: Monolithic to Microservices Architecture with Patterns & Best Practices 🌟
infoq.com: Turning Microservices Inside-Out
towardsdatascience.com: Architecture for High-Throughput Low-Latency Big Data Pipeline on Cloud 🌟 Scalable and efficient data pipelines are as important for the success of analytics, data science, and machine learning as reliable supply lines are for winning a war.
dzone: RESTful Applications in An Event-Driven Architecture Hybrid architecture with both RESTful and event-driven services.
developers.redhat.com: Distributed transaction patterns for microservices compared
thenewstack.io: The Rise of Event-Driven Architecture
jinwookim928.medium.com: Why Not Event Driven Architecture?

thenewstack.io: Streaming Data and the Modern Real-Time Data Stack

	Modern Data Stack	Modern Real-Time Data Stack
Language	SQL	SQL
Deployment	Cloud-native	Cloud-native
Data Ops	Complex batch transformations every 15 minutes, hourly or daily	Simple incremental transformations every second
Insights	Monthly, Weekly or Daily	Instantly
Cost	Affordable at massive scale	Affordable at massive scale and speed

blog.direktiv.io: Event driven orchestration with Knative (part 1)
blog.direktiv.io: Redefining event-driven orchestration for automation & applications
pub.towardsai.net: Deep Dive into Event-Driven architecture | Gul Ershad
developer.com: An Introduction to Event Driven Microservices
dzone.com: What Are Microservices and The Event Aggregator Pattern? 🌟 Learn about the Event Aggregator pattern for organizing event communication in a microservices architecture with an example.
ibm.com: Event-driven cloud-native applications (microservices) The event backbone is being part of the microservices mesh, providing the publish-and-subscribe communication between microservices and enabling the support of loosely coupled event-driven microservices.
irfanyusanif.medium.com: Best practices to communicate between microservices
swapnil-chougule.medium.com: Rapid Feature Engineering through SQL
==blog.twitter.com: Processing billions of events in real time at Twitter==
codeopinion.com: Event Sourcing vs Event Driven Architecture
thenewstack.io: The Path to Getting the Full Data Stack on Kubernetes
medium.com/tinyclues-vision: 4 Design Principles for Robust Data Pipelines Design Principles for traditional Software Engineering quickly fail when working with large and diverse sets of data — a new way of thinking about this difference.
medium.com/fiverr-engineering: How to Share Data Between Microservices on High Scale Several approaches suitable for a scalable system considering tradeoffs between availability and consistency
medium.com/codex: Microservices Communication — Queues Topics and Streams In this article, I will speak about types of messages broker that are famous in microservices communication.
blog.bitsrc.io: Why Microservices Should use Event Sourcing 🌟 6 Reasons where you need to use Event Sourcing in Micro-services
==emirayhan.medium.com: What is the difference Message Queue and Message Bus?== 🌟
verraes.net: DDD and Messaging Architectures 🌟 An overview of my different series on patterns in distributed systems. A good collection of Messaging Patterns

RPC vs Messaging

particular.net: RPC vs. Messaging – which is faster?

Message Brokers

ActiveMQ message broker

ActiveMQ 5.x "classic"
ActiveMQ Artemis Apache ActiveMQ is a subproject of Apache ActiveMQ. It has been donated to the Apache Software Foundation in 2015. There were lots of changes in project names in the past. The Artemis project first started as JBoss Messaging and got renamed to HornetQ in August 2009.
Apache Artemis JMeter Running the ActiveMQ Artemis JMeter Performance Testing Examples.
developers.redhat.com: Implementing Apache ActiveMQ-style broker meshes with Apache Artemis

RabbitMQ message broker

K8s prevent queue worker Pod from being killed during deployment How to prevent a Kubernetes (like RabbitMQ) queue worker Pod from being killed during deployment while handling a message?
medium.com: RabbitMQ vs. Kafka An architect’s dilemma
blog.rabbitmq.com: First Application With RabbitMQ Streams
geshan.com.np: How to use RabbitMQ and Node.js with Docker and Docker-compose
salaboy.com: Event-Driven applications with CloudEvents on Kubernetes

Redis message broker

Apache Camel message broker

Apache Camel Camel is an Open Source integration framework that empowers you to quickly and easily integrate various systems consuming or producing data. In version 3 we use <5MB memory, including the JVM. Also reflection free, low GC, super modular, native compilation friendly.
Quora.com: What's the difference between Apache Camel and Kafka?
dzone: Hybrid multi-cloud event mesh architectural design Building the event mesh with Camel
developers.redhat.com: Integrating systems with Apache Camel and Quarkus on Red Hat OpenShift

Apache Camel K

Apache Camel K is a lightweight cloud-integration platform that runs natively on Kubernetes. Based on the famous Apache Camel, Camel K is designed and optimized for serverless and microservices architectures.
developers.redhat.com: Six reasons to love Camel K
developers.redhat.com: Extending Kafka connectivity with Apache Camel Kafka connectors
developers.redhat.com: Design event-driven integrations with Kamelets and Camel K
thenewstack.io: Camel K Brings Apache Camel to Kubernetes for Event-Driven Architectures

KubeMQ message broker

KubeMQ.io: Kubernetes Native Message Queue Broker
devops.com: Best of 2019: Implementing Message Queue in Kubernetes
kubemq.io: Kafka VS KubeMQ 🌟
github.com/kubemq-io/kubemq-community 🌟 KubeMQ community version is now available as an open-source project!
dzone: KubeMQ: A Modern Alternative to Kafka This article introduces a modern, Kubernetes-native message queue called KubeMQ, to show how organizations trying to implement Kafka on Kubernetes can benefit from it.

Google Cloud Platform Pub/Sub

Google Cloud Platform Pub/Sub
A generic framework of concurrent consumers for Google Cloud Platform Pub/Sub 🌟 An example shows how to publish messages to Pub/Sub and build a service to consume the messages concurrently using the Python multiprocessing module

JMS Message Queue vs. Apache Kafka

==kai-waehner.de: Comparison: JMS Message Queue vs. Apache Kafka==

Cloud Based Integration. Integration Platform-as-a-Service (iPaaS) solutions

Wikipedia: Cloud Based Integration (iPaaS)
Integration Platform as a Service (iPaaS) is a suite of cloud services enabling development, execution and governance of integration flows connecting any combination of on premises and cloud-based processes, services, applications and data within individual or across multiple organizations.
Integration platform as a service (iPaaS) is a set of automated tools for connecting software applications that are deployed in different environments. iPaaS is often used by large business-to-business (B2B) enterprises that need to integrate on-premises applications and data with cloud applications and data.
blog.axway.com: What is iPaaS?
ibm.com: iPaaS (Integration-Platform-as-a-Service): iPaaS is a cloud-based solution that simplifies application integration across on-premises and cloud environments, to help you accelerate innovation and lower your integration and operations costs.

Red Hat Fuse and Red Hat Fuse Online

Syndesis open source integration platform

Syndesis open source integration platform (OpenSource Project for Red Hat Fuse Online)
developers.redhat.com: Low-code microservices orchestration with Syndesis

Debezium open source distributed platform for Change Data Capture (CDC) software design pattern

Change Data Capture, or CDC, is a well-established software design pattern for a system that monitors and captures the changes in data so that other software can respond to those changes. CDC captures row-level changes to database tables and passes corresponding change events to a data streaming bus. Applications can read these change event streams and access these change events in the order in which they occurred.
Debezium: Stream changes from your database
developers.redhat.com: Decoupling microservices with Apache Camel and Debezium
A good explanation of how to avoid distributed transactions using outbox pattern: Transaction Log Tailing With Debezium
developers.redhat.com: Capture database changes with Debezium Apache Kafka connectors
developers.redhat.com: Change data capture for microservices without writing any code
debezium.io: Lessons Learned from Running Debezium with PostgreSQL on Amazon RDS
info.crunchydata.com: PostgreSQL Change Data Capture With Debezium
medium.com: Stream Your Database into Kafka with Debezium An Introduction and Experience Report. Insightful post by David Hettler of comsysto about their usage of Debezium, touching on many details like outbox pattern, Avro schemas, Postgres on RDS etc.
noti.st: Change Data Capture with Flink SQL and Debezium 🌟
vladmihalcea.com: A beginner’s guide to CDC (Change Data Capture)
shopify.engineering: Capturing Every Change From Shopify’s Sharded Monolith
developers.redhat.com: Db2 and Oracle connectors coming to Debezium 1.4 GA
medium: Change Data Capture — Using Debezium
daily.dev: Building a fault-tolerant event-driven architecture with Google Cloud, Pulumi and Debezium
pradeepdaniel.medium.com: Creating an ETL data pipeline to sync data to Snowflake using Kafka and Debezium Setting up a real-time data pipeline from scratch to sync data from transactional databases to Snowflake cloud warehouse.
medium: A Visual Introduction to Debezium 🌟 A story-based introduction to understanding what Debezium is, how it is made of, and how it works in a real-world scenario
debezium.io: Using Debezium to Create a Data Lake with Apache Iceberg
developers.redhat.com: Improve your Kafka Connect builds of Debezium. A Kubernetes native way of building Kafka Connect images in Debezium 1.6
conduktor.io: Capturing MySQL database changes using Debezium, Kafka, and Conduktor Using Kafka Connect in Conduktor and specifically how to use Debezium to monitor the changes in a MySQL database.

Red Hat Integration service registry and Apicurio

Red Hat Integration service registry
Apicurio Registry An API/Schema registry - stores APIs and Schemas.
Event streaming and data federation: A citizen integrator’s story
redhat.com: Using a schema registry to ensure data consistency between microservices Make interservice communication easier by using a schema registry.

Data Mesh

Data Processing (aka Streaming Data, Data Pipeline or Big Data Pipeline)

Awesome Streaming A curated list of awesome streaming (stream processing) frameworks, applications, readings and other resources.
cloudblog.withgoogle.com: Turn any Dataflow pipeline into a reusable template
thenewstack.io: Part 1: The Evolution of Data Pipeline Architecture
eng.uber.com: Uber’s Journey Toward Better Data Culture From First Principles
satishchandragupta.com: Scalable Efficient Big Data Pipeline Architecture
openshift.com: How to Orchestrate Data Pipelines with Applications Deployed on OpenShift

Apache Kafka

Apache Kafka
developers.redhat.com: Using secrets in Kafka Connect configuration
developers.redhat.com: Capture database changes with Debezium Apache Kafka connectors
Awesome Kafka
Single Message Transformations - The Swiss Army Knife of Kafka Connect
medium: Logs & Offsets: (Near) Real Time ELT with Apache Kafka + Snowflake Replacing Apache Airflow with Debezium.
medium: Apache Kafka Startup Guide: System Design Architectures: Notification System, Web Activity Tracker, ELT Pipeline, Storage System 🌟
medium: Getting Started With Kafka on OpenShift
containerjournal.com: Red Hat Platform Brings Kafka Closer to Kubernetes
lightbend.com: Monitor Kafka Consumer Group Latency with Kafka Lag Exporter
AKHQ (previously known as KafkaHQ) 🌟 Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more...
banzaicloud.com: Kafka Schema Registry on Kubernetes the declarative way
Build a simple cloud-native change data capture pipeline
banzaicloud.com: Bulletproof Kafka, and the tale of an Amazon outage=
confluent.fr: Infrastructure Modernization with Google Anthos and Apache Kafka
confluent.io: Apache Kafka DevOps with Kubernetes and GitOps
Build a data streaming pipeline using Kafka Streams and Quarkus
levelup.gitconnected.com: Kafka for Engineers 🌟 Here are things about Kafka that you need to understand as a software engineer.
confluent.io: How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka
banzaicloud.com: Kafka on Kubernetes - using etcd 🌟
softwareengineeringdaily.com: Kafka Applications with Tim Berglund (podcast) 🌟
medium: Logs & Offsets: (Near) Real Time ELT with Apache Kafka + Snowflake
infoq.com: Building a SQL Database Audit System using Kafka, MongoDB and Maxwell's Daemon
tecmint: How to Install Apache Kafka in CentOS/RHEL 7
medium: Processing guarantees in Kafka "Duplicates and lost messages are due not only to features of the messaging systems, but in the design of producer and consumer applications as well." One of the best posts on processing guarantees in kafka.
davidxiang.com: Kafka As A Database? Yes Or No
medium: How Pinterest runs Kafka at scale
medium: Google Pub/Sub Lite for Kafka Users
medium: 4 Microservices Caching Patterns at Wix
Confluent.io: Intro to Apache Kafka: How Kafka Works 🌟
levelup.gitconnected.com: Kafka for Engineers
medium: Microservices in Rust with Kafka
medium: Apache Kafka in a Nutshell 🌟 Architecture, Use Cases, and a Getting Started guide — rolled into one
confluent.io: Simplifying Apache Kafka Multi-Cluster Management Using Control Center and Cluster Registry
kai-waehner.de: App Modernization and Hybrid Cloud Architectures with Apache Kafka
kai-waehner.de: Apache Kafka and MQTT (Part 1 of 5) – Overview and Comparison
medium: Solutions to Communication Problems in Microservices using Apache Kafka and Kafka Lens
kafka-tutorials.confluent.io 🌟
- kafka-tutorials.confluent.io: How to join a stream and a lookup table 🌟 If I have events in a Kafka topic and a table of reference data (aka a lookup table), how can I join each event in the stream to a piece of data in the table based on a common key?
confluent.io: DevOps for Apache Kafka with Kubernetes and GitOps 🌟
dzone.com: Microservices, Event-Driven Architecture and Kafka 🌟
medium: Understanding Kafka Topic Partitions Everything in Kafka is modeled around partitions. They rule Kafka’s storage, scalability, replication, and message movement.
kafka-tutorials.confluent.io: How to count messages in a Kafka topic
confluent.io: Apache Kafka Made Simple: A First Glimpse of a Kafka Without ZooKeeper 🌟
piotrminkowski.com: Knative Eventing with Kafka and Quarkus
blog.cloudera.com: Scalability of Kafka Messaging using Consumer Groups
thenewstack.io: Beyond the Quickstart: Running Apache Kafka as a Service on Kubernetes
towardsdatascience.com: You Can Replace Kafka with a Database
Handling Retries in Kafka: If You’re Using Kafka With Your Microservices, You’re Probably Handling Retries Wrong
confluent.io: What’s New in Apache Kafka 2.8
devclass.com: Apache Kafka 2.8.0 previews life without ZooKeeper
instaclustr.com: Apache Kafka Architecture: A Complete Guide 🌟
youtube playlist: Kafka Connect Tutorials | Kafka Connect 101: REST API 🌟 KafkaConnect uses a REST API to expose its management capabilities. tlberglund demonstrates many of the key functions available using the REST API, including creating connectors, viewing their status, and accessing troubleshooting information.
developers.redhat.com: Event-driven APIs and schema governance for Apache Kafka: Get ready for Kafka Summit Europe 2021
developers.redhat.com: Building resilient event-driven architectures with Apache Kafka
tech.ebayinc.com: Resiliency and Disaster Recovery with Kafka
newrelic.com: Effective Strategies for Kafka Topic Partitioning 🌟
gentlydownthe.stream A children’s book about Apache Kafka.
confluent.io: Apache Kafka Made Simple: A First Glimpse of a Kafka Without ZooKeeper
dzone: Event-Driven APIs and Schema Governance for Apache Kafka As a developer, I'm always excited to attend so many great sessions addressing critical challenges in the Apache Kafka ecosystem like how changes to event-driven APIs are leading developers to focus on contract-first development for Kafka.
phoenixnap.com: How to Set Up and Run Kafka on Kubernetes 🌟
piotrminkowski.com: Knative Eventing with Quarkus, Kafka and Camel
itnext.io: Configuring Kafka Sources and Sinks declaratively in Kubernetes using Knative This solves the complexity in work flow of compiling JARs and uploading them to a Kafka connect cluster. Using Knative it can be possible to leverage the Kubernetes cluster and define Kafka sources and sinks with Kubernetes objects.
strimzi.io: Kafka upgrade improvements
==redhat.com: Introducing Red Hat OpenShift Streams for Apache Kafka==
developers.redhat.com: Getting started with Red Hat OpenShift Streams for Apache Kafka
developers.redhat.com: Managing the API life cycle in an event-driven architecture: A practical approach 🌟
baeldung.com: List Active Brokers in a Kafka Cluster Using Shell Commands 🌟
developers.redhat.com: How to secure Apache Kafka schemas with Red Hat Integration Service Registry 2.0
mercurytfs.blogspot.com: Colas Kafka
grafana.com: Get comprehensive monitoring for your Apache Kafka ecosystem instances quickly with Grafana Cloud
dzone: Next-Gen Data Pipes With Spark, Kafka and k8s 🌟 This article examines the architecture patterns and provides some sample code for the readers to implement in their own environment.
confluent.io: Making Apache Kafka Serverless: Lessons From Confluent Cloud
developer.confluent.io 🌟🌟 over ten hours of FREE video courses with hands-on exercises, 50+ event streaming patterns, deep-dive articles on Kafka's internals, and a ton more.
itnext.io: Sending Messages to Kafka in Kubernetes
cloudhut.dev: Running Apache Kafka on Kubernetes successfully A comparison for different installation methods for running Kafka in Kubernetes
developers.redhat.com: The outbox pattern with Apache Kafka and Debezium 🌟
towardsdatascience.com: Overview of UI Tools for Monitoring and Management of Apache Kafka Clusters
analyticsindiamag.com: How Uber is Leveraging Apache Kafka For More Than 300 Micro Services
itnext.io: Securely Decoupling Kubernetes-based Applications on Amazon EKS using Kafka with SASL/SCRAM Securely decoupling Go-based microservices on Amazon EKS using Amazon MSK with IRSA, SASL/SCRAM, and data encryption
medium: Running Kafka in Kubernetes, Part 1: Why we migrated our Kafka clusters to Kubernetes At Wise, we chose to migrate our Apache Kafka clusters, previously running on Amazon Web Services (AWS) EC2 instances, into a multi-cluster Kubernetes setup. This article is the first part of a two-part series aiming to outline the motivations behind this choice and the challenges we faced.
betterprogramming.pub: How to Handle Duplicate Messages and Message Ordering in Kafka Dealing with the challenges faced when using Apache Kafka
medium: Optimizing Kafka Streams Apps on Kubernetes by Splitting Topologies
blog.twitter.com: Processing billions of events in real time at Twitter
inder-devops.medium.com: Kafka- Best practices & Lessons Learned | By Inder
datadoghq.com: Monitoring Kafka performance metrics
blog.workwell.io: How to manage your Kafka consumers from the producer
slack.engineering: Building Self-driving Kafka clusters using open source components
adam-kotwasinski.medium.com: Kafka mesh filter in Envoy
==conduktor.io: What is Apache Kafka?== Learn about Apache Kafka and its ecosystem in 20 minutes.
medium.com/airwallex-engineering: Kafka Streams: Iterative Development and Blue-Green Deployment Blue-Green Deployment for Kafka Stream Applications.
==redhat.com: How we use Apache Kafka to improve event-driven architecture performance== When designing your event-driven architecture, consider these ways to configure Kafka to improve performance.
dev.to: Running Kafka on kubernetes for local development
medium.com/udemy-engineering: Introducing Hot and Cold Retries on Apache Kafka An overview of how to build a fault-tolerant event delivery system by using non-blocking retries of Apache Kafka® in Udemy Payments Team
==conduktor.io/kafka: Learn Apache Kafka like never before== Conduktor Kafkademy is the quickest, easiest and most effective way for you to learn Apache Kafka for free.
developers.redhat.com: Which is better: A single Kafka cluster to rule them all, or many?
medium.com/dna-technology: Why we dropped event sourcing with Kafka Streams when given a second chance
==kai-waehner.de: When NOT to use Apache Kafka?==
betterprogramming.pub: Everything You Need To Know About Kafka 🌟 A simple guide for beginners
==learnk8s.io/kafka-ha-kubernetes: Designing and testing a highly available Kafka cluster on Kubernetes== 🌟 Learn how to design a Kafka cluster to achieve high availability using standard kubernetes resources and test how it tolerates maintenance and total node failures
blog.developer.adobe.com: Exploring Kafka Producer’s Internals 🌟 This is the first part of a series where we explore Kafka client's internals. This post focuses on the Kafka Producer.
medium.com/altitudehq: Kafka retries and maintaining the order of retry events 🌟
medium.com/cloudnesil: Kafka Streams State Store at Scale Managing Interactive Queries on multiple application instances Alternative solutions and their comparisons.
towardsdev.com: Performance Testing Your Kubernetes Kafka Cluster In this article, you will learn how to benchmark Apache Kafka instances running on Kubernetes against parallel loads.

Kafka Tools

Kafdrop – Kafka Web UI 🌟
redpanda-data/kowl Kowl is a Web UI for Apache Kafka that allows exploring messages, consumers, configurations and more with a focus on a good UI & UX.
KLoadGen - Kafka + (Avro/Json Schema) Load Generator 🌟 KLoadGen is kafka load generator plugin for jmeter designed to work with AVRO and JSON schema. It allows sending kafka messages with a structure defined as an AVRO Schema or a Json Schema. It connects to the Scheme Registry Server, retrieve the subject to send and generate a random message every time.
dev.to: Learn how to use Kafkacat – the most versatile Kafka CLI client 🌟
github.com/lensesio/fast-data-dev (Lenses Box) Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, Landoop Tools, 20+ connectors. A apachekafka docker image that actually works without zookeeper. If you don't want do deal with docker-compose this one is for you.
dzone: Visualize your Apache Kafka Streams using the Quarkus Dev UI Visualize your Apache Kafka Streams using the Quarkus Dev UI
medium: Solutions to Communication Problems in Microservices using Apache Kafka and Kafka Lens

Strimzi kubernetes operator for apache kafka

strimzi.io
developers.redhat.com: how easy to deploy and configure a Kafka Connect on Kubernetes through strimziio operator and use secrets
developers.redhat.com: Introduction to Strimzi: Apache Kafka on Kubernetes (KubeCon Europe 2020) 🌟
strimzi.io: Optimizing Kafka producers
strimzi.io: Optimizing Kafka consumers 🌟
strimzi.io: Optimizing Kafka producers 🌟
pepy.tech/project/strimzi-kafka-cli 🌟 - pypi.org/project/strimzi-kafka-cli
strimzi/kafka-kubernetes-config-provider: Kubernetes Configuration Provider for Apache Kafka Apache Kafka supports pluggable configuration providers which can load configuration data from external sources. The configuration providers in this repo can be used to load data from Kubernetes Secrets and Config Maps. It can be used in all Kafka components and does not depend on the other Strimzi components. So you could, for example, use it with your producer or consumer applications even if you don't use the Strimzi operators to provide your Kafka cluster. One of the example use-cases is to load certificates or JAAS configuration from Kubernetes Secrets.
strimzi.io: Using Kubernetes Configuration Provider to load data from Secrets and Config Maps
strimzi.io: Using HTTP Bridge as a Kubernetes sidecar
strimzi.io: Using Open Policy Agent with Strimzi and Apache Kafka
strimzi/strimzi-canary This repository contains the Strimzi canary tool implementation. It acts as an indicator of whether Kafka clusters are operating correctly. This is achieved by creating a canary topic and periodically producing and consuming events on the topic and getting metrics out of these exchanges.
medium: Mastering Apache Kafka on Kubernetes — Strimzi K8s operator
medium.com/@ahmed.farhan: Kafka Setup in Kubernetes Using Strimzi K8s operator — Part 2

[![airflow vs kafka debezium](images/airflow_vs_debezium.jpg)](https://medium.com/convoy-tech/logs-offsets-near-real-time-elt-with-apache-kafka-snowflake-473da1e4d776)

Apache Kafka Desktop Clients

conduktor.io 🌟 Apache Kafka Desktop Client. We created Conduktor, the all-in-one friendly interface to work with the Kafka ecosystem. Develop and manage Apache Kafka with confidence.

AWS Kinesis

MQTT

mqtt.org MQTT: The Standard for IoT Messaging
developers.redhat.com: Deploying the Mosquitto MQTT message broker on Red Hat OpenShift, Part 1
- developers.redhat.com: Deploying the Mosquitto MQTT message broker on Red Hat OpenShift, Part 2

Banzai Cloud Supertubes (Cloud Native Kafka implementation)

Confluent Cloud (Apache Kafka Re-engineered for the Cloud)

confluent.io The Complete Event Streaming Platform for Apache Kafka.
Focus on building apps and not managing clusters with a scalable, resilient and secure event streaming platform. Event streaming with Kafka made simple on AWS, Azure and GCP clouds.
mongodb.com: DaaS with MongoDB and Confluent
confluent.io: Confluent and Microsoft Announce Strategic Alliance
confluent.io: Monitoring Your Event Streams: Integrating Confluent with Prometheus and Grafana

Redpanda (kafka alternative). A modern streaming platform for mission critical workloads

Redpanda 🌟 is a Kafka® compatible event streaming platform. No Zookeeper, no JVM, and no code changes required. Use all your favorite open source tooling - 10x faster.
hub.docker.com/r/vectorized/redpanda Easy Docker experience to use VectorizedIO Redpanda in a container. Streaming platform for mission critical workloads, Kafka compatible, no Zookeeper, no JVM, no code changes required - 10x faster.
Redpanda is now Free & Source Available
softwareengineeringdaily.com: Redpanda: Kafka Alternative with Alexander Gallego 🌟

KsqlDB

ksqlDB The event streaming database purpose-built for stream processing applications.
Kafka Streams and ksqlDB Compared – How to Choose

Apache Pulsar

Apache Pulsar is an open-source distributed pub-sub messaging system originally created at Yahoo and now part of the Apache Software Foundation
Pulsar vs Kafka – Comparison and Myths Explored

Apache Flink

Apache Flink Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale.
How to set up Apache Flink on Kubernetes for real time data processing
flink.apache.org: How to natively deploy Flink on Kubernetes with High-Availability (HA)

Hazelcast JET

Hazelcast JET Open-Source Distributed Stream Processing
devops.com: Hazelcast Simplifies Streaming for Extremely Fast Event Processing in IoT, Edge and Cloud Environments

Workflow Engines

wikipedia: Workflow Engine

Zeebe

Apache Airflow

towardsdatascience.com: A journey to Airflow on Kubernetes
dzone: Apache Airflow Architecture on OpenShift
redhat.com: Monitoring Apache Airflow using Prometheus
towardsdatascience.com: Apache Airflow for containerized data-pipelines Are you having problems running tasks with a different version of Python on Airflow? In this article, I explain how to solve this issue.
Apache Airflow official helm chart 🌟
youtube: Airflow Helm Chart : Quick Start For Beginners in 10mins
snowflake.com: Migrating Airflow from Amazon EC2 to Kubernetes
dev.to: Get started with Apache Airflow
betterprogramming.pub: Running Airflow Using Kubernetes Executor and Kubernetes Pod Operator with Istio There are many obstacles when you deploy Airflow with the Kubernetes executor, Pod Operator and Istio since the Istio proxy sidecar makes the worker pod hang and run forever. In this article, you will learn how to overcome that.
towardsdatascience.com: Apache Airflow Architecture 🌟 A deep dive into Apache Airflow architecture and how it orchestrates workflows

Couler

Couler Couler aims to provide a unified interface for constructing and managing workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.

Red Hat AMQ (ActiveMQ Artemis broker and Apache Kafka)

Red Hat AMQ overview
Red Hat AMQ = AMQ Broker (Apache ActiveMQ Artemis) + AMQ Streams (Apache Kafka)

Red Hat AMQ Broker (ActiveMQ Artemis)

Red Hat AMQ Streams

[![AMQ in a nutshell](images/AMQ.png)](https://developers.redhat.com/products/amq/overview)

Product	Also Known As	Components	URL
Red Hat AMQ 6	JBoss AMQ 6	Apache ActiveMQ	Ref
Red Hat AMQ 7	JBoss AMQ 7 (Broker) or Red Hat AMQ 7 Suite	AMQ Broker + AMQ Streams	Ref
Red Hat AMQ 7	JBoss AMQ 7 (Broker) or Red Hat AMQ 7 Suite	JBoss AMQ 7 (Broker) + Apache Kafka	Ref
Red Hat AMQ 7	JBoss AMQ 7 (Broker) or Red Hat AMQ 7 Suite	Apache ActiveMQ Artemis + Apache Kafka	Ref

Slides of Red Hat AMQ Streams

??? note "Click to expand!"

<center>
<script async class="speakerdeck-embed" data-id="54c1ce6ee6e44d68a0c311c31ddc8225" data-ratio="1.77777777777778" src="//speakerdeck.com/assets/embed.js"></script>
</center>

Open Data Hub AI-as-a-Service (AIaaS) platform

KEDA. Kubernetes Event Driven Autoscaling

KEDA Kubernetes Event-driven Autoscaling. Application autoscaling made simple. https://github.com/kedacore/keda
Dzone: Autoscaling Your Kubernetes Microservice with KEDA Introduction to KEDA—event-driven autoscaler for Kubernetes, Apache Camel, and ActiveMQ Artemis—and how to use it to scale a Java microservice on Kubernetes.
tomd.xyz: Event-driven integration on Kubernetes with Camel & KEDA 🌟 Can we develop apps in Kubernetes that autoscale based on events? Perhaps, with this example using KEDA, ActiveMQ and Apache Camel.
faun.pub: Scaling an app in Kubernetes with KEDA (no Prometheus is needed)
itnext.io: Event Driven Autoscaling KEDA expands the capabilities of Kubernetes by managing the integration with external sources allowing you to auto-scale your Kubernetes Deployments based on data from both internal and external metrics.
medium.com/@casperrubaek: Why KEDA is a game-changer for scaling in Kubernetes KEDA makes it possible to easily scale based on any metric imaginable from almost any metric provider and is running at a massive scale in production in the cloud at some of the largest corporations in the world.
levelup.gitconnected.com: Scale your Apps using KEDA in Kubernetes

Integration Platform as a Solution (iPaaS). Platforms for collecting, storing and routing customer event data

IpaaS Vendors

eBooks

O'Really: Streaming data

Service meshes to the rescue: Load balancing and scaling long-lived connections in Kubernetes 🌟 Kubernetes doesn't load balance long-lived connections, some Pods might receive more requests than others, In case you are using HTTP/2, gRPC, RSockets, AMQP. Any work around?

Questions and Answers

adambien.blog - 75th airhacks.tv Questions and Answers: Kafka, JAX-RS, MicroProfile, JSON-B, GSON, JWT, VSC, NetBeans, Java Fullstack "Kafka vs. JAX-RS / RPC, thoughts about APIs, JSON-B vs. GSON, Path.of over Paths.get, Java Records, MicroProfile JWT, beginners vs. expert content, best Java fullstack, code coverage, NetBeans in 2020, Visual Studio Setup for Java, screencast configuration, ReactJS / Angular over JSF?, JSON-P vs. JSON-B, security code scanning"

Videos

Click to expand!

Tweets