awesome-kubernetes/docs/message-queue.md

Cloud Based Integration & Messaging. Data Processing & Streaming (aka Data Pipeline). Open Data Hub

• Message Queue in Kubernetes. Event-driven Messaging. Real-Time Data Streaming
• Message Brokers
  • ActiveMQ message broker
  • RabbitMQ message broker
  • Redis message broker
  • Apache Camel message broker
    • Apache Camel K
  • KubeMQ message broker
  • Google Cloud Platform Pub/Sub
• Cloud Based Integration. Integration Platform-as-a-Service (iPaaS) solutions
  • Red Hat Fuse and Red Hat Fuse Online
  • Syndesis open source integration platform
• Debezium open source distributed platform for Change Data Capture (CDC) software design pattern
• Red Hat Integration service registry and Apicurio
• Data Processing (aka Streaming Data, Data Pipeline or Big Data Pipeline)
  • Apache Airflow
  • Apache Kafka
  • Banzai Cloud Supertubes (Cloud Native Kafka implementation)
  • Confluent Cloud (Apache Kafka Re-engineered for the Cloud)
    • KsqlDB
  • Apache Pulsar
  • Apache Flink
  • Hazelcast JET
• Red Hat AMQ (ActiveMQ Artemis broker and Apache Kafka)
  • Red Hat AMQ Broker (ActiveMQ Artemis)
  • Red Hat AMQ Streams
• Open Data Hub AI-as-a-Service (AIaaS) platform
• KEDA. Kubernetes Event Driven Autoscaling
• eBooks
• Related
• Questions and Answers

Message Queue in Kubernetes. Event-driven Messaging. Real-Time Data Streaming

• Wikipedia: Message Broker
• Wikipedia: Event-driven messaging
• Wikipedia: Streaming Data
• nginx.com: Event-Driven Data Management for Microservices 🌟
• dzone: Event-Driven Architecture as a Strategy Event-driven architecture provides five key benefits to modern application architecture: scalability, resilience, agility, data sharing, and cloud enabling.
• infoq.com: From Monolith to Event-Driven: Finding Seams in Your Future Architecture

Message Brokers

• Apache ActiveMQ
• Dzone: Introduction to Message Brokers. Part 1: Apache Kafka vs. RabbitMQ
• Dzone: Introduction to Message Brokers. Part 2: ActiveMQ vs. Redis Pub/Sub
• developers.redhat.com: Choosing the right asynchronous-messaging infrastructure for the job

ActiveMQ message broker

• ActiveMQ 5.x "classic"
• ActiveMQ Artemis Apache ActiveMQ is a subproject of Apache ActiveMQ. It has been donated to the Apache Software Foundation in 2015. There were lots of changes in project names in the past. The Artemis project first started as JBoss Messaging and got renamed to HornetQ in August 2009.
• Apache Artemis JMeter Running the ActiveMQ Artemis JMeter Performance Testing Examples.

RabbitMQ message broker

• K8s prevent queue worker Pod from being killed during deployment How to prevent a Kubernetes (like RabbitMQ) queue worker Pod from being killed during deployment while handling a message?
• medium.com: RabbitMQ vs. Kafka An architect’s dilemma

Redis message broker

• Redis
• Redis Pub/sub

Apache Camel message broker

• Apache Camel
• Quora.com: What's the difference between Apache Camel and Kafka?

Apache Camel K

• Apache Camel K is a lightweight cloud-integration platform that runs natively on Kubernetes. Based on the famous Apache Camel, Camel K is designed and optimized for serverless and microservices architectures.
• developers.redhat.com: Six reasons to love Camel K
• developers.redhat.com: Extending Kafka connectivity with Apache Camel Kafka connectors

KubeMQ message broker

• KubeMQ.io: Kubernetes Native Message Queue Broker
• devops.com: Best of 2019: Implementing Message Queue in Kubernetes
• kubemq.io: Kafka VS KubeMQ 🌟

Google Cloud Platform Pub/Sub

• Google Cloud Platform Pub/Sub
• A generic framework of concurrent consumers for Google Cloud Platform Pub/Sub 🌟 An example shows how to publish messages to Pub/Sub and build a service to consume the messages concurrently using the Python multiprocessing module

Cloud Based Integration. Integration Platform-as-a-Service (iPaaS) solutions

• Wikipedia: Cloud Based Integration (iPaaS)
• Integration Platform as a Service (iPaaS) is a suite of cloud services enabling development, execution and governance of integration flows connecting any combination of on premises and cloud-based processes, services, applications and data within individual or across multiple organizations.
• Integration platform as a service (iPaaS) is a set of automated tools for connecting software applications that are deployed in different environments. iPaaS is often used by large business-to-business (B2B) enterprises that need to integrate on-premises applications and data with cloud applications and data.
• blog.axway.com: What is iPaaS?
• ibm.com: iPaaS (Integration-Platform-as-a-Service): iPaaS is a cloud-based solution that simplifies application integration across on-premises and cloud environments, to help you accelerate innovation and lower your integration and operations costs.

Red Hat Fuse and Red Hat Fuse Online

• Red Hat Fuse
• Red Hat Fuse Online

Syndesis open source integration platform

• Syndesis open source integration platform (OpenSource Project for Red Hat Fuse Online)
• developers.redhat.com: Low-code microservices orchestration with Syndesis

Debezium open source distributed platform for Change Data Capture (CDC) software design pattern

• Change Data Capture, or CDC, is a well-established software design pattern for a system that monitors and captures the changes in data so that other software can respond to those changes. CDC captures row-level changes to database tables and passes corresponding change events to a data streaming bus. Applications can read these change event streams and access these change events in the order in which they occurred.
• Debezium: Stream changes from your database
• developers.redhat.com: Decoupling microservices with Apache Camel and Debezium
• A good explanation of how to avoid distributed transactions using outbox pattern: Transaction Log Tailing With Debezium
• developers.redhat.com: Capture database changes with Debezium Apache Kafka connectors
• developers.redhat.com: Change data capture for microservices without writing any code
• debezium.io: Lessons Learned from Running Debezium with PostgreSQL on Amazon RDS
• info.crunchydata.com: PostgreSQL Change Data Capture With Debezium
• medium.com: Stream Your Database into Kafka with Debezium An Introduction and Experience Report. Insightful post by David Hettler of comsysto about their usage of Debezium, touching on many details like outbox pattern, Avro schemas, Postgres on RDS etc.
• noti.st: Change Data Capture with Flink SQL and Debezium 🌟
• vladmihalcea.com: A beginner’s guide to CDC (Change Data Capture)

Red Hat Integration service registry and Apicurio

• Red Hat Integration service registry
• Apicurio Registry An API/Schema registry - stores APIs and Schemas.
• Event streaming and data federation: A citizen integrator’s story

Data Processing (aka Streaming Data, Data Pipeline or Big Data Pipeline)

• Awesome Streaming A curated list of awesome streaming (stream processing) frameworks, applications, readings and other resources.
• cloudblog.withgoogle.com: Turn any Dataflow pipeline into a reusable template

Apache Airflow

• towardsdatascience.com: A journey to Airflow on Kubernetes

Apache Kafka

• Apache Kafka
• developers.redhat.com: how easy to deploy and configure a Kafka Connect on Kubernetes through strimziio operator and use secrets
• developers.redhat.com: Using secrets in Kafka Connect configuration
• developers.redhat.com: Capture database changes with Debezium Apache Kafka connectors
• Awesome Kafka
• Single Message Transformations - The Swiss Army Knife of Kafka Connect
• medium: Logs & Offsets: (Near) Real Time ELT with Apache Kafka + Snowflake Replacing Apache Airflow with Debezium.
• medium: Apache Kafka Startup Guide: System Design Architectures: Notification System, Web Activity Tracker, ELT Pipeline, Storage System 🌟
• medium: Getting Started With Kafka on OpenShift
• containerjournal.com: Red Hat Platform Brings Kafka Closer to Kubernetes
• lightbend.com: Monitor Kafka Consumer Group Latency with Kafka Lag Exporter
• AKHQ (previously known as KafkaHQ) 🌟 Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more...
• banzaicloud.com: Kafka Schema Registry on Kubernetes the declarative way
• Build a simple cloud-native change data capture pipeline
• banzaicloud.com: Bulletproof Kafka, and the tale of an Amazon outage=
• confluent.fr: Infrastructure Modernization with Google Anthos and Apache Kafka
• confluent.io: Apache Kafka DevOps with Kubernetes and GitOps
• Build a data streaming pipeline using Kafka Streams and Quarkus

[](https://medium.com/convoy-tech/logs-offsets-near-real-time-elt-with-apache-kafka-snowflake-473da1e4d776)

Banzai Cloud Supertubes (Cloud Native Kafka implementation)

• Banzai Cloud
• Banzai Kafka Operator
• The benefits of integrating Apache Kafka with Istio

Confluent Cloud (Apache Kafka Re-engineered for the Cloud)

• confluent.io The Complete Event Streaming Platform for Apache Kafka.
• Focus on building apps and not managing clusters with a scalable, resilient and secure event streaming platform. Event streaming with Kafka made simple on AWS, Azure and GCP clouds.

KsqlDB

• ksqlDB The event streaming database purpose-built for stream processing applications.
• Kafka Streams and ksqlDB Compared – How to Choose

Apache Pulsar

• Apache Pulsar is an open-source distributed pub-sub messaging system originally created at Yahoo and now part of the Apache Software Foundation
• Pulsar vs Kafka – Comparison and Myths Explored

Apache Flink

• Apache Flink Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale.
• How to set up Apache Flink on Kubernetes for real time data processing

Hazelcast JET

• Hazelcast JET Open-Source Distributed Stream Processing
• devops.com: Hazelcast Simplifies Streaming for Extremely Fast Event Processing in IoT, Edge and Cloud Environments

Red Hat AMQ (ActiveMQ Artemis broker and Apache Kafka)

• Red Hat AMQ overview
• Red Hat AMQ = AMQ Broker (Apache ActiveMQ Artemis) + AMQ Streams (Apache Kafka)

Red Hat AMQ Broker (ActiveMQ Artemis)

• Apache ActiveMQ Artemis broker
• developers.redhat.com: JDBC Master-Slave Persistence setup with Activemq using Postgresql database
• developers.redhat.com: Connecting external clients to Red Hat AMQ Broker on Red Hat OpenShift

Red Hat AMQ Streams

• Understanding Red Hat AMQ Streams components for OpenShift and Kubernetes 🌟
• Red Hat AMQ streams (kafka): Simplify Apache Kafka on Red Hat OpenShift
• Set up Red Hat AMQ Streams custom certificates on OpenShift
• speakerdeck.com: Apache Kafka with Red Hat AMQ Streams 🌟
• HTTP-based Kafka messaging with Red Hat AMQ Streams
• blog.jromanmartin.io: How to upgrade Strimzi Operator using the CLI

[](https://developers.redhat.com/products/amq/overview)

Product	Also Known As	Components	URL
Red Hat AMQ 6	JBoss AMQ 6	Apache ActiveMQ	Ref
Red Hat AMQ 7	JBoss AMQ 7 (Broker) or Red Hat AMQ 7 Suite	AMQ Broker + AMQ Streams	Ref
Red Hat AMQ 7	JBoss AMQ 7 (Broker) or Red Hat AMQ 7 Suite	JBoss AMQ 7 (Broker) + Apache Kafka	Ref
Red Hat AMQ 7	JBoss AMQ 7 (Broker) or Red Hat AMQ 7 Suite	Apache ActiveMQ Artemis + Apache Kafka	Ref

<script async class="speakerdeck-embed" data-id="54c1ce6ee6e44d68a0c311c31ddc8225" data-ratio="1.77777777777778" src="//speakerdeck.com/assets/embed.js"></script>

Open Data Hub AI-as-a-Service (AIaaS) platform

• Open Data Hub
• Open Data Hub 0.6 brings component updates and Kubeflow architecture
• A development roadmap for Open Data Hub

KEDA. Kubernetes Event Driven Autoscaling

• KEDA Kubernetes Event-driven Autoscaling. Application autoscaling made simple.
• Dzone: Autoscaling Your Kubernetes Microservice with KEDA Introduction to KEDA—event-driven autoscaler for Kubernetes, Apache Camel, and ActiveMQ Artemis—and how to use it to scale a Java microservice on Kubernetes.

eBooks

• O'Really: Streaming data

Related

• Service meshes to the rescue: Load balancing and scaling long-lived connections in Kubernetes

Questions and Answers

• adambien.blog - 75th airhacks.tv Questions and Answers: Kafka, JAX-RS, MicroProfile, JSON-B, GSON, JWT, VSC, NetBeans, Java Fullstack "Kafka vs. JAX-RS / RPC, thoughts about APIs, JSON-B vs. GSON, Path.of over Paths.get, Java Records, MicroProfile JWT, beginners vs. expert content, best Java fullstack, code coverage, NetBeans in 2020, Visual Studio Setup for Java, screencast configuration, ReactJS / Angular over JSF?, JSON-P vs. JSON-B, security code scanning"

---

Pub-Sub ≠ Partitioning ≠ Multiplexing pic.twitter.com/0ZVaH9Mxvr

— Clemens Vasters 🇪🇺☁📨 (@clemensv) July 28, 2020

<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>