new images and entries

docs/matrix-table.md

@@ -1,4 +1,4 @@
-# Kubernetes Distributions or Installers Matrix Table
+# Kubernetes Distributions & Installers Matrix Table
 
 |  Kubernetes Installer or Distribution | Role | Ecosystem | Infra Provider | On-Premise | Licence | HA | Standalone | Runs in Docker | Ingress + Storage <br/>included | Automated <br/>Deployment | Details | 
 | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | 

docs/monitoring.md

@@ -35,10 +35,16 @@
     - [Validation of Artemis Broker Monitoring with JMeter](#validation-of-artemis-broker-monitoring-with-jmeter)
         - [JMeter Example Test Plans](#jmeter-example-test-plans)
 - [Kibana](#kibana)
-- [Interactive Learning](#interactive-learning)
+- [Prometheus and Grafana Interactive Learning](#prometheus-and-grafana-interactive-learning)
 - [Performance](#performance)
+- [List of Performance Analysis Tools](#list-of-performance-analysis-tools)
+    - [Thread Dumps. Debugging Java Applications](#thread-dumps-debugging-java-applications)
+- [Debugging Java Applications on OpenShift and Kubernetes](#debugging-java-applications-on-openshift-and-kubernetes)
 - [Distributed Tracing. OpenTelemetry and Jaeger](#distributed-tracing-opentelemetry-and-jaeger)
+    - [Microservice Observability with Distributed Tracing. OpenTelemetry.io](#microservice-observability-with-distributed-tracing-opentelemetryio)
+    - [Jaeger VS OpenTelemetry. How Jaeger works with OpenTelemetry](#jaeger-vs-opentelemetry-how-jaeger-works-with-opentelemetry)
 - [Application Performance Management (APM)](#application-performance-management-apm)
+    - [Elastic APM](#elastic-apm)
     - [Dynatrace APM](#dynatrace-apm)
 - [Message Queue Monitoring](#message-queue-monitoring)
     - [Red Hat AMQ 7 Broker Monitoring solutions based on Prometheus and Grafana](#red-hat-amq-7-broker-monitoring-solutions-based-on-prometheus-and-grafana)
@@ -657,20 +663,78 @@ JMeter|Artemis Grafana|Artemis Dashboard
 * [dzone: Getting Started With Kibana Advanced Searches](https://dzone.com/articles/getting-started-with-kibana-advanced-searches)
 * [dzone: Kibana Hacks: 5 Tips and Tricks](https://dzone.com/articles/kibana-hacks-5-tips-and-tricks)
 
-## Interactive Learning
+## Prometheus and Grafana Interactive Learning
 * [katacoda.com: Getting Started with Prometheus](https://www.katacoda.com/courses/prometheus/getting-started)
 * [katacoda.com: Creating Dashboards with Grafana](https://www.katacoda.com/courses/prometheus/creating-dashboards-with-grafana)
 
 ## Performance
 * [dzone.com: The Keys to Performance Tuning and Testing](https://dzone.com/articles/the-keys-to-performance-tuning-and-testing)
 * [dzone.com: How Performance Tuning and Testing are Changing](https://dzone.com/articles/how-performance-tuning-and-testing-are-changing)
-* Java:
-    * [developers.redhat.com: Troubleshooting java applications on openshift](https://developers.redhat.com/blog/2017/08/16/troubleshooting-java-applications-on-openshift/)
-    * [dzone.com: how to take thread dumps](https://dzone.com/articles/how-to-take-thread-dumps-7-options)
 * [Performance Patterns in Microservices-Based Integrations 🌟](https://dzone.com/articles/performance-patterns-in-microservices-based-integr-1) Almost all applications that perform anything useful for a given business need to be integrated with one or more applications. With microservices-based architecture, where a number of services are broken down based on the services or functionality offered, the number of integration points or touch points increases massively.
 
+## List of Performance Analysis Tools
+* Threadumps + heapdumps + GC analysis tools
+* [en.wikipedia.org/wiki/List_of_performance_analysis_tools](https://en.wikipedia.org/wiki/List_of_performance_analysis_tools)
+* [InspectIT](https://en.wikipedia.org/wiki/InspectIT)
+* [VisualVM 🌟](https://en.wikipedia.org/wiki/VisualVM)
+* [OverOps](https://en.wikipedia.org/wiki/OverOps)
+* [FusionReactor](https://en.wikipedia.org/wiki/FusionReactor)
+* [tier1app.com](https://tier1app.com/)
+* [fastthread.io 🌟](https://fastthread.io/)
+* [gceasy.io 🌟](https://gceasy.io/)
+* [heaphero.io](https://heaphero.io/)
+
+### Thread Dumps. Debugging Java Applications
+- [How to read a Thread Dump](https://dzone.com/articles/how-to-read-a-thread-dump)
+- [Performance Patterns in Microservices-Based Integrations 🌟](https://dzone.com/articles/performance-patterns-in-microservices-based-integr-1) **A must read!**
+- [Dzone: how to take thread dumps](https://dzone.com/articles/how-to-take-thread-dumps-7-options)
+- Thread Dump Analyzers: [fastThread](https://fastthread.io/), [Spotify TDA](https://spotify.github.io/threaddump-analyzer/), [IBM Thread and Monitor Dump Analyzer for Java](https://www.ibm.com/support/pages/ibm-thread-and-monitor-dump-analyzer-java-tmda), [TDA - Thread Dump Analyzer](https://github.com/irockel/tda)
+- [FastThread.io](https://fastthread.io/): Thread dumps can be uploaded via Web or API Call from within the POD (jstack must be available within the container):
+
+        ```bash
+        #!/bin/sh
+        # Generate N thread dumps of the process PID with an INTERVAL between each dump.
+        if [ $# -ne 3 ]; then
+           echo Generates Java thread dumps using the jstack command.
+           echo
+           echo usage: $0 process_id repetitions interval
+           exit 1
+        fi 
+        PID=$1
+        N=$2
+        INTERVAL=$3 
+        for ((i=1;i<=$N;i++))
+        do
+           d=$(date +%Y%m%d-%H%M%S)
+           dump="threaddump-$PID-$d.txt"
+           echo $i of $N: $dump
+           jstack -l $PID > $dump
+           curl -X POST --data-binary @./$dump https://fastthread.io/fastthread-api?apiKey=<APIKEY> --header "Content-Type:text"
+           sleep $INTERVAL
+        done
+        ```
+
+    - How to run this script from within the POD: ```./script_thread_dump.sh 1 15 3```, where:
+        - “1”: PID of java process (“1” in containers running a single process, check with “ps ux” command).
+        - “15”: 15 repetitions or thread dumps
+        - “3”: interval of 3 seconds between each thread dump.
+    - According to some references only 3 thread dumps captured in a timeframe of 10 seconds is necessary (when we want to troubleshoot a Java issue during a service degradation).
+    - Sample thread dump analysis reports generated by fastThread: 
+        - [Transitive blocks](https://fastthread.io/ft-thread-report.jsp?dumpId=1&s=t) 
+        - [Unresponsive JVM](https://fastthread.io/ft-thread-report.jsp?dumpId=1&s=t) 
+        - [Sudden CPU spike](https://fastthread.io/ft-thread-report.jsp?dumpId=1&s=t)
+        - [Thread Leaks](https://fastthread.io/ft-thread-report.jsp?dumpId=1&s=t)
+
+## Debugging Java Applications on OpenShift and Kubernetes
+- [developers.redhat.com: Troubleshooting java applications on openshift (Jolokia)](https://developers.redhat.com/blog/2017/08/16/troubleshooting-java-applications-on-openshift/)
+- [Debugging Java Applications On OpenShift and Kubernetes](https://www.openshift.com/blog/debugging-java-applications-on-openshift-kubernetes)
+- [Remote Debugging of Java Applications on OpenShift](https://servicesblog.redhat.com/2019/03/06/remote-debugging-of-java-applications-on-openshift/)
+    - https://dzone.com/articles/remote-debugging-of-java-applications-on-openshift (JMX + VisualVM)
+- [VisualVM: JVisualVM to an Openshift pod](https://fedidat.com/250-jvisualvm-openshift-pod/)  
+- [redhat.com: How do I analyze a Java heap dump?](https://access.redhat.com/solutions/18301)
+
 ## Distributed Tracing. OpenTelemetry and Jaeger
-- [**opentelemetry.io** 🌟](https://opentelemetry.io/) (**OpenTracing.io + OpenCensus.io = OpenTelemetry.io**)
+- [Microservice Observability with Distributed Tracing: **OpenTelemetry.io** 🌟](https://opentelemetry.io/) (**OpenTracing.io + OpenCensus.io = OpenTelemetry.io**)
 - [**Jaeger** 🌟](https://www.jaegertracing.io/)
      - [Jaeger Demo1](https://github.com/obitech/micro-obs)
      - [Jaeger Demo 2](https://github.com/open-telemetry/opentelemetry-collector/tree/master/examples/demo)
@@ -678,14 +742,50 @@ JMeter|Artemis Grafana|Artemis Dashboard
 - [**zipkin.io**](https://zipkin.io/)
 - [**OpenTracing.io**](https://opentracing.io/)
      - [lightstep.com: Understand Distributed Tracing](https://docs.lightstep.com/docs/understand-distributed-tracing)
-   
+
+### Microservice Observability with Distributed Tracing. OpenTelemetry.io
+- Used for monitoring and troubleshooting microservices-based distributed systems.
+- [OpenTelemetry.io](https://opentelemetry.io/): 
+    - **Unified standard** (open, vendor-neutral API), **merge of [OpenCensus.io](https://opencensus.io/) and [OpenTracing.io](https://opentracing.io/)**. 
+    - “A single set of system components and language-specific telemetry libraries” to standardize how the industry uses metrics, traces, and eventually logs to enable observability. 
+- A major component of the [OpenTelemetry specification](https://github.com/open-telemetry/opentelemetry-specification) is **distributed tracing**. 
+- **Tracing** is about analyzing, recording, and describing transactions.
+- **Distributed Tracing:** Troubleshooting requests between interconnected cloud-based microservices can’t always be done with logs and metrics alone. This is where distributed tracing comes into play: It provides developers with a  detailed view of individual requests as they “hop” through a system of microservices. With **distributed tracing** you can:
+    - Trace the path of a request as it travels across a complex system.
+    - Discover the latency of the components along that path.
+    - Know which component in the path is creating a bottleneck or failure.
+- **Performance:** Latency is a very important metric in microservices. Latency problems in one service will impact the overall request latency when chaining calls to different microservices. Every call to a microservice should record a trace, which is basically a record of how much time it took to respond. It's possible to add more details to the function level, including the action, the result, and the pass to the next service. The hard part is triaging all traces in a request from a client. Usually, a trace ID header has to be sent in every request. If there isn't one, the logging library creates it and it will represent the first trace in a request. **Adding traces with [OpenCensus](https://opencensus.io/) is simply a matter of including the libraries and registering an exporter.** 
+- **Monitoring in a Microservices/Kubernetes World:** In distributed system architectures like microservices, having visibility from different perspectives will be critical at troubleshooting time. Many things could happen in a request when there are many parts constantly interacting at the same time. The most common method is to write logs to the stdout and stderr streams.
+    - For example, a latency problem in the system could exist because a microservice is not responding correctly. Maybe Kubernetes is restarting the pod too frequently, or perhaps the cluster is out of capacity and can't schedule any more pods. But for this reason, tools like [Istio](https://istio.io/) exist; by injecting a container in every pod, you can get a pretty good baseline of telemetry. Additionally, when you add instrumentation with libraries like [OpenCensus](https://opencensus.io/), you can deeply understand what's happening with and within each service.
+    - All this information will need a storage location, and as a good practice, you might want to have it a centralized location to provide access to anyone in the team — not just for the operations team.
+- **Older Distributed Tracing Solutions:** 
+    - [Dapper](https://research.google/pubs/pub36356/) (Google)
+    - [Zipkin](https://zipkin.io/) (A.K.A. OpenZipkin, created by Twitter, inspired by Dapper)
+    - [Jaeger](https://www.jaegertracing.io/) (Uber Technologies, inspired by Dapper & Zipkin)
+    - etc.
+- [Medium: Distributed Tracing and Monitoring using OpenCensus](https://medium.com/@rghetia/distributed-tracing-and-monitoring-using-opencensus-fe5f6e9479fb)
+- [Dzone: Zipkin vs. Jaeger: Getting Started With Tracing](https://dzone.com/articles/zipkin-vs-jaeger-getting-started-with-tracing) Learn about Zipkin and Jaeger, how they work to add request tracing to your logging routine, and how to choose which one is the right fit for you.
+- [opensource.com: Distributed tracing in a microservices world](https://opensource.com/article/18/9/distributed-tracing-microservices-world) What is distributed tracing and why is it so important in a microservices environment?
+- [opensource.com: 3 open source distributed tracing tools](https://opensource.com/article/18/9/distributed-tracing-tools) Find performance issues quickly with these tools, which provide a graphical view of what's happening across complex software systems.
+- [newrelic.com: OpenTracing, OpenCensus, OpenTelemetry, and New Relic (Best overview of OpenTelemetry)](https://blog.newrelic.com/engineering/opentelemetry-opentracing-opencensus/)  
+- There’s no OpenTelemetry UI, instead Jaeger UI (or any APM like Dynatrace or New Relic) can be used as “Tracing backend + Visualization frontend + Data mining platform” of OpenTelemetry API/SDK.
+
+[Jaeger UI](https://www.jaegertracing.io/)|[Zipkin UI](https://zipkin.io/)
+:----:|:----:
+![Jaeger UI](images/jaeger_ui.png)|![Zipking UI](images/zipkin_ui.png)
+
+### Jaeger VS OpenTelemetry. How Jaeger works with OpenTelemetry
+- [Medium: Jaeger VS OpenTracing VS OpenTelemetry](https://medium.com/jaegertracing/jaeger-and-opentelemetry-1846f701d9f2)
+
+![Jaeger Vs OpenTelemetry](images/jaeger_vs_opentelemetry.png)
+
 ## Application Performance Management (APM)
-* [en.wikipedia.org/wiki/Application_performance_management](https://en.wikipedia.org/wiki/Application_performance_management)
+- [APM in wikipedia](https://en.wikipedia.org/wiki/Application_performance_management): The monitoring and management of performance and availability of software applications. APM strives to detect and diagnose complex application performance problems to maintain an expected level of service. APM is "the translation of IT metrics into business meaning.” 
+- Tip: [Download APM report from IT Central Station](https://www.itcentralstation.com/categories/application-performance-management-apm)
 * [Awesome APM 🌟](https://github.com/antonarhipov/awesome-apm)
 * [dzone.com: APM Tools Comparison](https://dzone.com/articles/apm-tools-comparison-which-one-should-you-choose)
 * [dzone.com: Java Performance Monitoring: 5 Open Source Tools You Should Know](https://dzone.com/articles/java-performance-monitoring-5-open-source-tools-you-should-know)
 * [dzone.com: 14 best performance testing tools and APM solutions](https://dzone.com/articles/14-best-performance-testing-tools-and-apm-solution)
-* [elastic.co: Using the Elastic APM Java Agent on Kubernetes](https://www.elastic.co/blog/using-elastic-apm-java-agent-on-kubernetes-k8s)
 * Exception Tracking:
     * [sentry.io](https://sentry.io/)
     * APMs like Dynatrace, etc.
@@ -693,22 +793,21 @@ JMeter|Artemis Grafana|Artemis Dashboard
     * [datadoghq.com](https://www.datadoghq.com/)
     * [honeycomb.io](https://www.honeycomb.io)
     * [lightstep.com](https://lightstep.com)
-    * [skywalking.apache.org](https://skywalking.apache.org/)
-    * [Elastic APM](https://www.elastic.co/products/apm)
+    * [skywalking.apache.org](https://skywalking.apache.org/)   
     * [AppDynamics 🌟](https://www.appdynamics.com/)
     * [New Relic 🌟](https://newrelic.com/)
     * [Dynatrace 🌟](https://www.dynatrace.com/)
-* List of Performance Analysis Tools:
-    * Threadumps + heapdumps + GC analysis tools
-    * [en.wikipedia.org/wiki/List_of_performance_analysis_tools](https://en.wikipedia.org/wiki/List_of_performance_analysis_tools)
-    * [InspectIT](https://en.wikipedia.org/wiki/InspectIT)
-    * [VisualVM 🌟](https://en.wikipedia.org/wiki/VisualVM)
-    * [OverOps](https://en.wikipedia.org/wiki/OverOps)
-    * [FusionReactor](https://en.wikipedia.org/wiki/FusionReactor)
-    * [tier1app.com](https://tier1app.com/)
-    * [fastthread.io 🌟](https://fastthread.io/)
-    * [gceasy.io 🌟](https://gceasy.io/)
-    * [heaphero.io](https://heaphero.io/)
+
+### Elastic APM
+- [Elastic APM](https://www.elastic.co/products/apm)
+- [Elastic APM Server](https://www.elastic.co/guide/en/apm/get-started/current/components.html): 
+- [Mininimum elasticsearch requirement is 6.2.x or higher](https://www.elastic.co/support/matrix#matrix_compatibility)
+- Built-in elasticsearch 5.6 in Openshift 3 & Openshift 4 cannot be integrated with Elastic APM Server.
+- Solutions: Deploy a higher version of [Elasticsearch + Kibana](https://hub.docker.com/_/elasticsearch) on a new Project dedicated to Elastic APM; or setup an Elastic Cloud account.
+- [Elastic APM Server Docker image](https://github.com/sls-dev1/openshift-elastic-apm-server) (“oss” & openshift compliant). 
+- [elastic.co: Using the Elastic APM Java Agent on Kubernetes](https://www.elastic.co/blog/using-elastic-apm-java-agent-on-kubernetes-k8s)
+
+![Elastic APM](images/elasticapm.png)
 
 ### Dynatrace APM
 * [adictosaltrabajo.com: Monitorización y análisis de rendimiento de aplicaciones con Dynatrace APM](https://www.adictosaltrabajo.com/tutoriales/monitorizacion-y-analisis-de-rendimiento-de-aplicaciones-con-dynatrace/)

docs/openshift.md

@@ -14,6 +14,30 @@
     - [Identity Management](#identity-management)
     - [Quota Management](#quota-management)
 - [OpenShift Container Platform 4 (OCP 4)](#openshift-container-platform-4-ocp-4)
+    - [OCP 4 Overview](#ocp-4-overview)
+        - [Three New Functionalities](#three-new-functionalities)
+        - [New Technical Components](#new-technical-components)
+        - [Installation & Cluster Autoscaler](#installation--cluster-autoscaler)
+            - [IPI & UPI](#ipi--upi)
+        - [Cluster Autoscaler Operator](#cluster-autoscaler-operator)
+        - [Operators](#operators)
+            - [Introduction](#introduction)
+            - [Catalog](#catalog)
+            - [Certified Opeators, OLM Operators and Red Hat Operators](#certified-opeators-olm-operators-and-red-hat-operators)
+            - [OpenShift Container Storage Operator (OCS)](#openshift-container-storage-operator-ocs)
+                - [OCS 3 (OpenShift 3)](#ocs-3-openshift-3)
+                - [OCS 4 (OpenShift 4)](#ocs-4-openshift-4)
+            - [Cluster Network Operator (CNO) & Routers](#cluster-network-operator-cno--routers)
+            - [ServiceMesh Operator](#servicemesh-operator)
+            - [Serverless Operator (Knative)](#serverless-operator-knative)
+            - [Monitoring & Observability](#monitoring--observability)
+                - [Grafana](#grafana)
+                - [Prometheus](#prometheus)
+                - [Alerts & Silences](#alerts--silences)
+                - [Cluster Logging (EFK)](#cluster-logging-efk)
+            - [Build Images. Next-Generation Container Image Building Tools](#build-images-next-generation-container-image-building-tools)
+            - [Registry & Quay](#registry--quay)
+            - [Local Development Environment](#local-development-environment)
     - [OpenShift Youtube](#openshift-youtube)
     - [OpenShift 4 Training](#openshift-4-training)
     - [OpenShift 4 roadmap](#openshift-4-roadmap)
@@ -188,6 +212,322 @@ Industry momentum has aligned behind Kubernetes as the orchestration platform fo
 
 [![OCP 4 Architecture](images/ocp4_arch.png)](https://www.openshift.com/blog/enterprise-kubernetes-with-openshift-part-one)
 
+### OCP 4 Overview
+- Result of RedHat’s (now IBM) acquisition of CoreOS -> [RHCOS](https://docs.openshift.com/container-platform/4.4/architecture/architecture-rhcos.html) (Red Hat Enterprise Linux CoreOS)
+- Merge of two leading Kubernetes distributions, Tectonic and OpenShift:
+    - **CoreOS Tectonic**:
+        - [Operator Framework](https://www.openshift.com/learn/topics/operators)
+        - [quay.io](https://quay.io/) container build and registry service
+        - Stable tiny Linux distribution with [ignition bootstrap](https://coreos.com/ignition/docs/latest/what-is-ignition.html) and transaction-based update engine.
+    - **OpenShift**:
+        - [Wide enterprise adoption](https://www.openshift.com/#success-stories-intro)
+        - [Security](https://docs.openshift.com/container-platform/4.4/authentication/managing-security-context-constraints.html)
+        - [Multi-tenancy features](https://www.slideshare.net/Smals_ICT/20171010-multitenancy-in-openshift) (self-service)
+- OpenShift 4 is built on top of Kubernetes 1.13+ 
+- [Roadmap](https://assets.openshift.com/hubfs/Commons-London-OpenShift-Container-Platform-4.3-Roadmap.pdf)
+- [Release Notes](https://docs.openshift.com/container-platform/4.4/release_notes/ocp-4-4-release-notes.html)
+
+![tenant](images/tenant.png)
+
+#### Three New Functionalities
+1. Self-Managing Platform
+2. Application Lifecycle Management ([OLM](https://docs.openshift.com/container-platform/4.4/operators/understanding_olm/olm-understanding-olm.html)):
+    - **OLM Operator**:
+        - Responsible for deploying applications defined by [ClusterServiceVersion (CSV) manifest](https://docs.openshift.com/container-platform/4.4/operators/understanding_olm/olm-understanding-olm.html#olm-csv_olm-understanding-olm).
+        - Not concerned with the creation of the required resources; users can choose to manually create these resources using the CLI, or  users can choose to create these resources using the Catalog Operator.
+    - **Catalog Operator**:
+        - Responsible for resolving and installing CSVs and the required resources they specify. It is also responsible for watching CatalogSources for updates to packages in channels and upgrading them (optionally automatically) to the latest available versions.
+        - A user that wishes to track a package in a channel creates a **Subscription resource** configuring the desired **package, channel, and the CatalogSource** from which to pull updates. When updates are found, an appropriate **InstallPlan** is written into the namespace on behalf of the user.
+3. Automated Infrastructure Management ([Over-The-Air Updates](https://access.redhat.com/documentation/en-us/openshift_container_platform/4.1/pdf/updating_clusters/OpenShift_Container_Platform-4.1-Updating_clusters-en-US.pdf))
+
+![ocp update1](images/ocp_update1.png)|![ocp update2](images/ocp_update2.png)|![ocp update3](images/ocp_update3.png)
+:---:|:---:|:---:
+
+#### New Technical Components
+- **[New Installer](https://cloud.redhat.com/openshift/install):**
+    - [try.openshift.com/](https://try.openshift.com/)
+    - [github.com/openshift/installer](https://github.com/openshift/installer)
+- **Storage:** Cloud integrated storage capability used by default via [OCS Operator](https://github.com/openshift/ocs-operator) (Red Hat)
+    - There are a number of persistent storage options available to you through the OperatorHub / Storage vendors that don’t involve Red Hat, NFS or Gluster.
+    - Kubernetes-native persistent storage technologies available (non-RedHat solutions):
+        - [Rook-Ceph](https://operatorhub.io/operator/rook-ceph): [Rook-Ceph storage Operator now on OperatorHub.io](https://www.redhat.com/en/blog/rook-ceph-storage-operator-now-operatorhubio)
+        - [Robin Storage Operator](https://operatorhub.io/operator/robin-operator): [get.robin.io/](https://get.robin.io/)
+- **Operators End-To-End!:** responsible for reconciling the system to the desired state
+    - Cluster configuration kept as API objects that ease its maintenance (“everything-as-code” approach):
+        - Every component is configured with [Custom Resources (CR)](https://docs.openshift.com/container-platform/4.4/operators/crds/crd-managing-resources-from-crds.html) that are processed by operators. 
+        - No more painful upgrades and synchronization among multiple nodes and no more configuration drift.
+    - List of operators that configure cluster components (API objects):
+        - API server
+        - Nodes via Machine API
+        - Ingress
+        - Internal DNS
+        - Logging (EFK) and Monitoring (Prometheus)
+        - Sample applications
+        - Networking
+        - Internal Registry
+        - Oauth (and authentication in general)
+        - etc
+- **At the Node Level:**
+    - [RHEL CoreOS](https://docs.openshift.com/container-platform/4.4/architecture/architecture-rhcos.html) is the result of merging CoreOS Container Linux and RedHat Atomic host functionality and is currently the only supported OS to host OpenShift 4.
+    - Node provisioning with [ignition](https://coreos.com/blog/introducing-ignition.html), which came with CoreOS Container Linux
+    - Atomic host updates with [rpm-ostree](https://github.com/coreos/rpm-ostree)
+    - [CRI-O](https://cri-o.io/) as a container runtime
+    - [SELinux](https://www.slideshare.net/openshift/openshift-18812162) enabled by default
+- [Machine API](https://github.com/openshift/machine-api-operator/tree/master): Provisioning of nodes. Abstraction mechanism added (API objects to declaratively manage the cluster):
+    - Based on [Kubernetes Cluster API project](https://github.com/kubernetes-sigs/cluster-api)
+    - Provides a new set of machine resources: 
+        - Machine
+        - Machine Deployment
+        - MachineSet: 
+            - distributes easily your nodes among different Availability Zones
+            - manages multiple node pools (e.g. pool for testing, pool for machine learning with GPU attached, etc)
+- **Everything “just another pod”**
+
+#### Installation & Cluster Autoscaler
+- New installer openshift-install tool, replacement for the old Ansible scripts.  
+- 40 min (AWS). Terraform.
+- 2 installation patterns:
+    1. **Installer Provisioned Infrastructure (IPI)**
+    2. **User Provisioned Infrastructure (UPI)**
+- The whole process can be done in one command and requires minimal infrastructure knowledge (IPI): ```openshift-install create cluster```
+
+![OCP IPI](images/ocp-ipi.png)
+![OCP IPI UPI](images/ocp_ipi_upi.png)
+
+##### IPI & UPI
+- 2 installation patterns:
+    1. **Installer Provisioned Infrastructure (IPI):** On supported platforms, the installer is capable of provisioning the underlying infrastructure for the cluster. The installer programmatically creates all portions of the networking, machines, and operating systems required to support the cluster. Think of it as best-practice reference architecture implemented in code.  It is recommended that most users make use of this functionality to avoid having to provision their own infrastructure.  The installer will create and destroy the infrastructure components it needs to be successful over the life of the cluster.
+    2. **User Provisioned Infrastructure (UPI):** For other platforms or in scenarios where installer provisioned infrastructure would be incompatible, the installer can stop short of creating the infrastructure, and allow the platform administrator to provision their own using the cluster assets generated by the install tool. Once the infrastructure has been created, OpenShift 4 is installed, maintaining its ability to support automated operations and over-the-air platform updates.
+
+![OCP IPI2](images/ocp_ipi2.png)|![OCP UPI](images/ocp_upi.png)
+:----:|:----:
+
+#### Cluster Autoscaler Operator
+- Adjusts the size of an OpenShift Container Platform cluster to meet its current deployment needs. It uses declarative, Kubernetes-style arguments
+- Increases the size of the cluster when there are pods that failed to schedule on any of the current nodes due to insufficient resources or when another node is necessary to meet deployment needs. The ClusterAutoscaler does not increase the cluster resources beyond the limits that you specify.
+- A huge improvement over the manual, error-prone process used in the previous version of OpenShift and RHEL nodes.
+
+![OCP Autoscaler1](images/ocp4_autoscaler1.png)|![OCP Autoscaler2](images/ocp4_autoscaler2.png)
+:----:|:----:
+
+#### Operators
+##### Introduction
+- Core of the platform
+- The hierarchy of operators, with clusterversion at the top, is the single door for configuration changes and is responsible for reconciling the system to the desired state.
+- For example, if you break a critical cluster resource directly, the system automatically recovers itself. 
+- Similarly to cluster maintenance, [operator framework](https://www.redhat.com/en/blog/introducing-operator-framework-building-apps-kubernetes) used for applications. As a user, you get SDK, [OLM](https://docs.openshift.com/container-platform/4.4/operators/understanding_olm/olm-understanding-olm.html) (Lifecycle Manager of all Operators and their associated services running across their clusters) and embedded [operator hub](https://www.redhat.com/en/blog/new-kubernetes-operatorhub-red-hat-openshift-enable-hybrid-cloud-flexibility-enterprises).
+- [OLM Arquitecture](https://github.com/operator-framework/operator-lifecycle-manager/blob/master/doc/design/architecture.md)
+- [Adding Operators to a Cluster](https://docs.openshift.com/container-platform/4.4/operators/olm-adding-operators-to-cluster.html) (They can be added via **CatalogSource**)
+- The supported method of using **Helm charts** with Openshift is via the [Helm Operator](https://www.openshift.com/blog/build-kubernetes-operators-from-helm-charts-in-5-steps)
+- [twitter.com/operatorhubio](https://twitter.com/operatorhubio)
+- View the list of Operators available to the cluster from the OperatorHub:
+    ```bash
+    $ oc get packagemanifests -n openshift-marketplace 
+    NAME AGE 
+    amq-streams 14h 
+    packageserver 15h 
+    couchbase-enterprise 14h 
+    mongodb-enterprise 14h 
+    etcd 14h myoperator 14h 
+    ...
+    ```
+
+![OCP Operators](images/ocp_operators.png)
+
+##### Catalog
+- Developer Catalog
+- Installed Operators
+- OperatorHub (OLM)
+- Operator Management:
+    - **Operator Catalogs** are groups of Operators you can make available on the cluster. They can be added via **CatalogSource** (i.e. “catalogsource.yaml”). Subscribe and grant a namespace access to use the installed Operators. 
+    - **Operator Subscriptions** keep your services up to date by tracking a channel in a package. The approval strategy determines either manual or automatic updates.
+
+![Operator Subscriptions](images/operator_subscriptions.png)
+
+##### Certified Opeators, OLM Operators and Red Hat Operators
+- **Certified Operators** packaged by Certified:
+    - Not provided by Red Hat 
+    - Supported by Red Hat
+    - Deployed via “Package Server” OLM Operator
+- **OLM Operators**:
+    - Packaged by Red Hat  
+    - **“Package Server”** OLM Operator includes a CatalogSource provided by Red Hat
+- **Red Hat Operators**:
+    - Packaged by Red Hat  
+    - Deployed via “Package Server” OLM Operator
+- **Community Edition Operators**:
+    - Deployed by any means
+    - **Not supported** by Red Hat
+
+![OCP Certified Operators](images/ocp_certified_operators.png)
+
+##### OpenShift Container Storage Operator (OCS)
+###### OCS 3 (OpenShift 3)
+- OpenShift Container Storage based on [GlusterFS](https://www.gluster.org/) technology.
+- Not OpenShift 4 compliant: Migration tooling will be available to facilitate the move to OCS 4.x (OpenShift Gluster APP Mitration Tool).
+
+###### OCS 4 (OpenShift 4)
+- **OCS Operator** based on Rook.io with Operator LifeCycle Manager (OLM).
+- Tech Stack:
+    - [Rook](https://rook.io) (don't confuse this with non-redhat ["Rook Ceph"](https://operatorhub.io/operator/rook-ceph) -> [RH ref](https://www.redhat.com/en/blog/rook-ceph-storage-operator-now-operatorhubio)).
+        - Replaces [Heketi](https://github.com/heketi/heketi)  (OpenShift 3)
+        - Uses **Red Hat Ceph Storage** and **Noobaa**.
+    - [Red Hat Ceph Storage](https://ceph.io)
+    - [Noobaa](https://www.noobaa.io):
+        - Red Hat Multi Cloud Gateway (AWS, Azure, GCP, etc)
+        - Asynchronous replication of data between my local ceph and my cloud provider
+        - Deduplication
+        - Compression
+        - Encryption
+- Backups available in OpenShift 4.2+ (Snapshots + Restore of Volumes)
+- OCS Dashboard in OCS Operator
+
+![OCS Dashboard](images/ocs_dashboard.png)
+
+##### Cluster Network Operator (CNO) & Routers
+- Cluster Network Operator (CNO): The cluster network is now configured and managed by an Operator. The Operator upgrades and monitors the cluster network.
+- [Router plug-ins in OCP3:](https://docs.openshift.com/container-platform/3.11/install_config/router/index.html)
+    - A « route » is the external entrypoint to a [Kubernetes Service](https://kubernetes.io/docs/concepts/services-networking/service/). This is one of the biggest differences between [Kubernetes](https://kubernetes.io/) and [OpenShift Enterprise (= OCP)](https://www.openshift.com/) and [origin](https://www.okd.io/). 
+    - OpenShift router has the endpoints as targets and therefore the pod of the application.
+    - [Shared/Stikcy sessions are enabled by default](https://dzone.com/articles/session-stickiness-in-openshift)
+    - [HAProxy template router](https://docs.openshift.com/container-platform/3.11/architecture/networking/assembly_available_router_plugins.html#architecture-haproxy-router) (default router): HTTP(s) & TLS-enabled traffic via SIN.
+        - [dzone.com/articles/updating-haproxy-configurations-openshift](https://dzone.com/articles/updating-haproxy-configurations-openshift)
+        - [dzone.com/articles/openshift-egress-options](https://dzone.com/articles/openshift-egress-options) 
+    - F5 BIG-IP Router plug-in integrates with an existing F5 BIG-IP system in your environment
+    - Since the 9th May 2018, [NGINX](https://www.openshift.com/blog/introducing-nginx-and-nginx-plus-routers-for-openshift) is also available as « router ».
+- Routers in OCP4: 
+    - [Ingress Controller](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/) is the most common way to allow external access to an OpenShift Container Platform cluster
+    - [Configuring Ingress Operator in OCP4](https://docs.openshift.com/container-platform/4.4/networking/ingress-operator.html)
+    - Limited to HTTP, HTTPS using SNI, and TLS using SNI (sufficient for web applications and services)
+    - Has two replicas by default, which means it should be running on two worker nodes.
+    - Can be scaled up to have more replicas on more nodes.
+    - The Ingress Operator implements the ingresscontroller API and is the component responsible for enabling external access to OpenShift Container Platform cluster services. 
+    - The operator makes this possible by deploying and managing one or more HAProxy-based [Ingress Controllers](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/) to handle routing.
+- [Network Security Zones in Openshift (DMZ)](https://blog.openshift.com/openshift-and-network-security-zones-coexistence-approaches/)
+
+```bash
+oc describe clusteroperators/ingress
+oc logs --namespace=openshift-ingress-operator deployments/ingress-operator
+```
+
+##### ServiceMesh Operator 
+- ServiceMesh: [Istio](https://istio.io/) + [kiali](https://kiali.io/) + [Jaeger](https://www.jaegertracing.io/)
+- ServiceMesh Community Edition: [github.com/maistra/istio](https://github.com/maistra/istio)
+    - Red Hat community installer compliant with OCP 4.1: [maistra.io/docs/getting_started/install](https://maistra.io/docs/getting_started/install)
+    - Outcome: publicly known errors in 2 or 3 components.
+- **Certified ServiceMesh Operator**
+    -  [ServiceMesh](https://access.redhat.com/documentation/en-us/openshift_container_platform/4.4/html-single/service_mesh/index) GA in September 2019 (available in OperatorHub):
+        - [blog.openshift.com/red-hat-openshift-service-mesh-is-now-available-what-you-should-know/](https://blog.openshift.com/red-hat-openshift-service-mesh-is-now-available-what-you-should-know/) 
+    - Certified & Packaged by Red Hat
+    - [“One-click” deployment](https://docs.openshift.com/container-platform/4.4/service_mesh/service_mesh_install/installing-ossm.html)
+    - [Preparing to install Red Hat OpenShift Service Mesh](https://docs.openshift.com/container-platform/4.4/service_mesh/service_mesh_install/preparing-ossm-installation.html). To install the Red Hat OpenShift Service Mesh Operator, you must first install these Operators:
+        - Elasticsearch
+        - Jaeger
+        - Kiali
+    - Do not install Community versions of the Operators. Community Operators are not supported.
+
+![OCS Servicemesh 1](images/ocp_servicemesh1.png)|![OCS Servicemesh 2](images/ocp_servicemesh2.png)|![OCS Servicemesh 3](images/ocp_servicemesh3.png)
+:----:|:----:|:----:
+
+![OCS Servicemesh 4](images/ocp_servicemesh4.png)
+
+##### Serverless Operator (Knative) 
+- Operator install on OperatorHub.io
+- Knative Eventing (Camel-K, Kafka, Cron, etc)
+- Integration with Openshift ServiceMesh, Logging, Monitoring.
+- [openshift.com/learn/topics/serverless](https://www.openshift.com/learn/topics/serverless)
+- [redhat-developer-demos.github.io/knative-tutorial](https://redhat-developer-demos.github.io/knative-tutorial)  
+
+##### Monitoring & Observability
+###### Grafana
+- Integrated Grafana v5.4.3 (deployed by default): 
+- Monitoring -> Dashboards
+- Project “openshift-monitoring”
+- https://grafana.com/docs/v5.4/ 
+
+###### Prometheus
+- Integrated Prometheus v2.7.2 (deployed by default): 
+- Monitoring -> metrics
+- Project “openshift-monitoring”
+- https://prometheus.io/docs/prometheus/2.7/getting_started/ 
+
+###### Alerts & Silences
+- Integrated Alertmanager 0.16.2 (deployed by default):
+    - Monitoring -> Alerts
+    - Monitoring -> Silences 
+    - Silences temporarily mute alerts based on a set of conditions that you define. Notifications are not sent for alerts that meet the given conditions.
+- Project “openshift-monitoring”
+- https://prometheus.io/docs/alerting/alertmanager/
+
+###### Cluster Logging (EFK)
+- EFK: Elasticsearch + Fluentd + Kibana
+- Cluster Logging EFK **not deployed by default**
+- As an OpenShift Container Platform cluster administrator, you can deploy cluster logging to aggregate logs for a range of OpenShift Container Platform services.
+- The OpenShift Container Platform cluster logging solution requires that you **install both the Cluster Logging Operator and Elasticsearch Operator**. There is no use case in OpenShift Container Platform for installing the operators individually. You must **install the Elasticsearch Operator using the CLI** following the directions below. You can **install the Cluster Logging Operator using the web console or CLI.**
+Deployment procedure based on CLI + web console: 
+    - [docs.openshift.com/container-platform/4.4/logging/cluster-logging-deploying.html](https://docs.openshift.com/container-platform/4.4/logging/cluster-logging-deploying.html) 
+    - **Elasticsearch Operator** must be installed in Project **“openshift-operators-redhat”**
+    - **Cluster Logging Operator** must be deployed in Project **“openshift-logging”**
+    - **CatalogSourceConfig** added to enable Elasticsearch Operator on the cluster
+    - etc.
+
+OCP Release|Elasticsearch|Fluentd|Kibana|EFK deployed by default
+:--|:--|:--|:--|:--
+OpenShift 3.11| 5.6.13.6|0.12.43|5.6.13|No
+OpenShift 4.1|5.6.16|?|5.6.16|No
+
+##### Build Images. Next-Generation Container Image Building Tools
+- Redesign of how images are built on the platform.
+- Instead of relying on a daemon on the host to manage containers, image creation, and image pushing, we are leveraging [Buildah](https://buildah.io/) running inside our build pods.
+- This aligns with the general OpenShift 4 theme of making everything “just another pod”
+- A simplified set of build workflows, not dependent on the node host having a specific container runtime available. 
+- Dockerfiles that built under OpenShift 3.x will continue to build under OpenShift 4.x and S2I builds will continue to function as well.
+- The actual BuildConfig API is unchanged, so a BuildConfig from a v3.x cluster can be imported into a v4.x cluster and work without modification.
+- [Podman & Buildah for docker users](https://developers.redhat.com/blog/2019/02/21/podman-and-buildah-for-docker-users/)
+- [Openshift ImageStreams](https://cloudowski.com/articles/why-managing-container-images-on-openshift-is-better-than-on-kubernetes/)
+- [Openshift 4 image builds](https://www.openshift.com/blog/openshift-4-image-builds)
+- [Custom image builds with Buildah](https://docs.openshift.com/container-platform/4.4/builds/custom-builds-buildah.html)
+- [Rootless podman and NFS](https://www.redhat.com/sysadmin/rootless-podman-nfs)
+
+![Buildah](images/Buildah.png)
+
+##### Registry & Quay
+- A Docker registry is a place to store and distribute Docker images.
+- It serves as a target for your docker push and docker pull commands.
+- [Openshift ImageStreams](https://cloudowski.com/articles/why-managing-container-images-on-openshift-is-better-than-on-kubernetes/)
+- The registry is now managed by an Operator instead of ```oc adm``` registry.
+- [Quay.io](https://quay.io/) is a hosted Docker registry from CoreOS:
+    - Main features:
+        - “Powerful build triggers” 
+        - “Advanced team permissions”
+        - “Secure storage”
+    - One of the more enterprise-friendly options out there, offering fine-grained permission controls.
+    - They support any git server and let you build advanced workflows by doing things like mapping git branches to Docker tags so that when you commit code it automatically builds a corresponding image.
+    - Quay offers unlimited free public repositories. Otherwise, you pay by the number of private repositories. There’s no extra charge for storage or bandwidth.
+- [Quay 3.0 released in May 2019](https://www.redhat.com/en/blog/introducing-red-hat-quay-3-registry-your-linux-and-windows-containers): support for multiple architectures, Windows containers, and a [Red Hat Enterprise Linux (RHEL)](https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux)-based image to this container image registry.
+- [Quay 3.1 released in September 2019](https://www.redhat.com/en/blog/red-hat-quay-31-now-even-better-across-distributed-environments): The newest Quay feature is repository mirroring, which complements our existing geographic replication features. Repository mirroring reflects content between distinct, different registries. With this, you can synchronize whitelisted repositories or a source registry subset into Quay. This makes it much easier to distribute images and related data through Quay. 
+- [Quay Community Edition operator](https://github.com/redhat-cop/quay-operator)
+- [Quay 3.1 Certified Operator is not available in Openshift and must be purchased](https://www.openshift.com/products/quay)
+- Open Source [ProjectQuay.io](https://www.projectquay.io/) Container Registry: 
+    - [Red Hat Introduces open source Project Quay container registry](https://www.redhat.com/en/blog/red-hat-introduces-open-source-project-quay-container-registry) 
+    - [github.com/quay](https://github.com/quay]
+
+##### Local Development Environment
+- For version 3 we have [Container Development Kit](https://developers.redhat.com/products/cdk/overview) (or its open source equivalent for OKD - [minishift](https://github.com/minishift/minishift/releases)) which launches a single node VM with Openshift and it does it in a few minutes. It’s perfect for testing also as a part of CI/CD pipeline.
+- Openshift 4 on your laptop: There is a working solution for single node OpenShift cluster. It is provided by a new project called [CodeReady Containers](https://github.com/code-ready/crc/).
+- [Procedure:](https://developers.redhat.com/blog/2019/09/05/red-hat-openshift-4-on-your-laptop-introducing-red-hat-codeready-containers/)
+
+    ```bash
+    untar
+    crc setup
+    crc start
+    environment variables
+    oc login
+    ```
+
+- [Red Hat OpenShift 4.2 on your laptop: Introducing Red Hat CodeReady Containers](https://developers.redhat.com/blog/2019/09/05/red-hat-openshift-4-on-your-laptop-introducing-red-hat-codeready-containers/)
+
+
 ### OpenShift Youtube
 * [OpenShift Youtube](https://www.youtube.com/user/rhopenshift/videos)
 * [youtube: Installing OpenShift 4 on AWS with operatorhub.io integration 🌟](https://www.youtube.com/watch?v=kQJxGtsqphk)
@@ -202,9 +542,9 @@ Industry momentum has aligned behind Kubernetes as the orchestration platform fo
 * [learn.crunchydata.com](https://learn.crunchydata.com/)
 
 ### OpenShift 4 roadmap
-* [blog.openshift.com: OpenShift 4 Roadmap (slides)](https://blog.openshift.com/wp-content/uploads/Red-Hat-OpenShift-4.0-Roadmap-Public-Feb-2019-Ali.pdf)
-* [blog.openshift.com: OpenShift Container Storage (OCS 3 & 4 slides)](https://blog.openshift.com/wp-content/uploads/OPENSHIFT-CONTAINER-STORAGE.pdf)
-* [blog.openshift.com: OpenShift 4 Roadmap Update (slides)](https://blog.openshift.com/wp-content/uploads/OpenShift-4-Roadmap-Update-William-Markito-and-Chris-Blum.pdf)
+* [blog.openshift.com: OpenShift 4 Roadmap (slides) - this link may change](https://assets.openshift.com/hubfs/Commons-London-OpenShift-Container-Platform-4.3-Roadmap.pdf)
+* [blog.openshift.com: OpenShift Container Storage (OCS 3 & 4 slides) - this link is now broken](https://blog.openshift.com/wp-content/uploads/OPENSHIFT-CONTAINER-STORAGE.pdf)
+* [blog.openshift.com: OpenShift 4 Roadmap Update (slides) - this link is now broken](https://blog.openshift.com/wp-content/uploads/OpenShift-4-Roadmap-Update-William-Markito-and-Chris-Blum.pdf)
 
 ### Kubevirt Virtual Machine Management on Kubernetes
 * [kubevirt.io 🌟](https://kubevirt.io/)