It only takes two commands to install x509-certificate-exporter, however you should read the instructions in the next section to take advantage of all the features!

Add our Charts repository :

$ helm repo add enix https://charts.enix.io

Install x509-certificate-exporter for TLS Secrets monitoring with prometheus-operator support :

$ helm install x509-certificate-exporter enix/x509-certificate-exporter

To remove built-in Prometheus alerts if you'd rather craft your own :

$ helm upgrade x509-certificate-exporter enix/x509-certificate-exporter --reuse-values --set prometheusRules.create=false

If you don't use the Prometheus operator at all, and don't have the CRD, disable resource creation and perhaps add Pod annotations for scrapping :

secretsExporter:
  podAnnotations:
    prometheus.io/port: "9793"
    prometheus.io/scrape: "true"
service:
  create: false
prometheusServiceMonitor:
  create: false
prometheusRules:
  create: false

📜 Using the Chart

This will guide you through writing the initial set of values.

Metrics for TLS Secrets

By default we only run a Deployment to provide metrics on TLS Secrets stored in the Kubernetes cluster. It helps detect expiring certificates whether you manage them on your own or rely on controllers such as cert-manager.

🙂 If you're only interested in this feature, you could probably install the Chart not specifying any value.

Disable this exporter when Secrets metrics are not wanted – if you're looking for hostPath DaemonSets only :

secretsExporter:
  enabled: false

Metrics for node certificates (hostPath)

Kubernetes components use many certificates to authenticate and secure communications between each others. This PKI is critical to the operation of a cluster and its health should be monitored carefully. Expiring Kubernetes certificates are a common source of outages, and depending on your distribution could happen a few months after installation if left unattended.

This Chart provides a facility to deploy DaemonSets so that each node of a cluster can run its own x509-certificate-exporter and export metrics for host files :

etcd server and client certificates
Kubernetes CA
kube-apiserver certificates
kubelet certificates
kubeconfig files with embedded certificates
etc. Obviously it also works with any other application deployed on cluster nodes as long as it uses PEM encoded certicates (deployment agents, security tools, etc.).

⚙️ You'll have to compile a list of files and directories of interest. There is no "one size fits all" configuration that we could recommend, or even a decent boilerplate. Examples below should give an idea of what to look after.

🏙️ While having a single DaemonSet sounds like a fair option, it is not uncommon for nodes to assume different roles, and as a result hold different sets of certificate files requiring targeted x509-certificate-exporter configurations. For example, with the help of node selectors and tolerations, we can have nodes of the control plane run their own exporter targeting API and etcd certificates, while regular nodes would have a simpler configuration for Kubelet alone.

Deployment of hostPath exporters is controlled under the hostPathsExporter key of Chart Values. All values are defaults that would apply to any number of DaemonSet you wish to run, unless overridden individually. Then you'll need to create at least one DaemonSet in hostPathsExporter.daemonSets.

This is the most basic configuration. It will create one DaemonSet named nodes with an empty configuration. Exporters won't export no certificate metric.

hostPathsExporter:
  daemonSets:
    nodes: {}

Moving on, we can add all flavors of "watch" settings :

watchDirectories : to monitor all PEM files found in a host directory (no recursion in subdirectories)
watchFiles : to target known file paths, this is highly recommended over the directory option when file paths are predictable
watchKubeconfFiles : look for base64 encoded embedded certificates in Kubeconfig files

This will create a DaemonSet able to monitor the same files on all nodes. It could fit a typical kubeadm cluster with no control plane dedicated nodes :

hostPathsExporter:
  daemonSets:
    nodes:
      watchFiles:
      - /var/lib/kubelet/pki/kubelet-client-current.pem
      - /etc/kubernetes/pki/apiserver.crt
      - /etc/kubernetes/pki/apiserver-etcd-client.crt
      - /etc/kubernetes/pki/apiserver-kubelet-client.crt
      - /etc/kubernetes/pki/ca.crt
      - /etc/kubernetes/pki/front-proxy-ca.crt
      - /etc/kubernetes/pki/front-proxy-client.crt
      - /etc/kubernetes/pki/etcd/ca.crt
      - /etc/kubernetes/pki/etcd/healthcheck-client.crt
      - /etc/kubernetes/pki/etcd/peer.crt
      - /etc/kubernetes/pki/etcd/server.crt
      watchKubeconfFiles:
      - /etc/kubernetes/admin.conf
      - /etc/kubernetes/controller-manager.conf
      - /etc/kubernetes/scheduler.conf

Dedicated nodes will require other DaemonSets. Based on our kubeadm example, it could be extended like this :

hostPathsExporter:
  podAnnotations:
    prometheus.io/port: "9793"
    prometheus.io/scrape: "true"

  daemonSets:
    cp:
      nodeSelector:
        node-role.kubernetes.io/master: ""
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
        operator: Exists
      watchFiles:
      - /var/lib/kubelet/pki/kubelet-client-current.pem
      - /etc/kubernetes/pki/apiserver.crt
      - /etc/kubernetes/pki/apiserver-etcd-client.crt
      - /etc/kubernetes/pki/apiserver-kubelet-client.crt
      - /etc/kubernetes/pki/ca.crt
      - /etc/kubernetes/pki/front-proxy-ca.crt
      - /etc/kubernetes/pki/front-proxy-client.crt
      - /etc/kubernetes/pki/etcd/ca.crt
      - /etc/kubernetes/pki/etcd/healthcheck-client.crt
      - /etc/kubernetes/pki/etcd/peer.crt
      - /etc/kubernetes/pki/etcd/server.crt
      watchKubeconfFiles:
      - /etc/kubernetes/admin.conf
      - /etc/kubernetes/controller-manager.conf
      - /etc/kubernetes/scheduler.conf

    nodes:
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/ingress
        operator: Exists
      watchFiles:
      - /var/lib/kubelet/pki/kubelet-client-current.pem
      - /etc/kubernetes/pki/ca.crt

With this last configuration we demonstrated :

using podAnnotations right under hostPathsExporter, it will apply to all hostPathsExporter.daemonSets because they don't override that setting
two DaemonSets, cp for control plane nodes, and nodes for regular ones
nodeSelector on the control plane DaemonSet to schedule Pods on "masters" only
tolerations for both. On cp it's required to be scheduled on those tainted nodes. Because this cluster would also have nodes dedicated to ingress controllers, DaemonSet nodes also get a toleration for this role.

Custom Resources for the Prometheus operator

Users of the prometheus-operator will immediately scrape exporters thanks to the creation of a ServiceMonitor resource, and get basic alerting rules from the new PrometheusRule. The operator is usually installed with kube-prometheus or by the Kubernetes distribution.

When it's missing and you don't have the CRD, helm will raise one of this error :

Error: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"]

Add the following values to disable creation of ServiceMonitors and PrometheusRules :

prometheusServiceMonitor:
  create: false
prometheusRules:
  create: false

Then perhaps you would need Pod annotations to work with the Kubernetes service discovery in Prometheus :

secretsExporter:
  podAnnotations:
    prometheus.io/port: "9793"
    prometheus.io/scrape: "true"

Also in such case the headless service may not serve any purpose and can be removed :

service:
  create: false

ℹ️ Chart Values provide a few knobs to control Prometheus rules, such as numbers of days before certificate expiration for warning and critical alerts are triggered.

⚠️ Special alert X509ExporterReadErrors is meant to report anomalies with the exporter, such as API authorization issues or unreadable files. If the Kubernetes API is unstable it could be disabled with prometheusRules.alertOnReadErrors.
When using hostPath exporters, and some nodes don't have all the files, it's better to add other DaemonSet profiles to target each situation and preserve this alert. Detecting configuration regressions is especially important when working with files that can change path over time and on cluster upgrades.

Installing the Chart

Create a file named x509-certificate-exporter.values.yaml with your values, as discussed previously and with the help of Chart Values.

Add our Charts repository :

$ helm repo add enix https://charts.enix.io

Install the x509-certificate-exporter with release name x509-certificate-exporter :

$ helm install x509-certificate-exporter enix/x509-certificate-exporter --values x509-certificate-exporter.values.yaml

The upgrade command is used to change configuration when values are modified :

$ helm upgrade x509-certificate-exporter enix/x509-certificate-exporter --values x509-certificate-exporter.values.yaml

Upgrading the Chart

Update Helm repositories :

$ helm repo update

Upgrade release names x509-certificate-exporter to the latest version :

$ helm upgrade x509-certificate-exporter enix/x509-certificate-exporter

📝 Notes

`watchFiles` and inode change

Because of limitations with mount binding that CRIs use to expose a single host file to a container, we cannot use subPath in volumeMounts. This feature would in fact result in the perfect implementation where the exporter can only access designated files. However if a certificate file is not replaced in-place and it's inode is altered, the exporter would keep seeing the old content and can't even tell if it still exists on the host. This situation is common with Kubernetes control-plane certificates.

Be aware that for every file path provided to watchFiles, the exporter container will be given read access to the parent directory. This is how we handle the problem of changing inodes. Metrics will of course be limited to the single targetted path, as the program is told to watch the real path from watchFiles.

This is to be taken into consideration if you're doing threat assessment. In such case it's recommended not to put secret keys in the same directory as certificate files.

Watching symbolic links

Starting with version 2.6.0, the exporter is now handling symlinks in a special way to better suit containerization and the limited access to host filesystem.

When a symlink is added to a watchFiles list, we are able to resolve and read it's target as long as it's in the same directory. This is dynamic for each metrics scrape and will track symlink changes.

A typical case of this is the Kubelet client certificate that now uses a symlink pointing to a new file each time the certificate gets rotated. For instance you may be using :

watchFiles:
- /var/lib/kubelet/pki/kubelet-client-current.pem

Because all client certificates reside in the pki directory, the exporter will be able to read kubelet-client-current.pem and it's target properly. Even though the Operating System cannot resolve the link itself in the container namespace.

Values

Key	Type	Default	Description
kubeVersion	string	`""`	Override Kubernetes version detection ; usefull with "helm template"
extraLabels	object	`{}`	Extra labels to add on chart resources
nameOverride	string	`""`	Partially override x509-certificate-exporter.fullname template (will prepend the release name)
fullnameOverride	string	`""`	Fully override x509-certificate-exporter.fullname template
imagePullSecrets	list	`[]`	Specify docker-registry secret names as an array
image.registry	string	`"docker.io"`	x509-certificate-exporter image registry
image.repository	string	`"enix/x509-certificate-exporter"`	x509-certificate-exporter image repository
image.tag	string	`nil`	x509-certificate-exporter image tag (defaults to Chart appVersion)
image.pullPolicy	string	`"IfNotPresent"`	x509-certificate-exporter image pull policy
psp.create	bool	`false`	Should Pod Security Policy resources be created
rbac.create	bool	`true`	Should RBAC resources be created
rbac.secretsExporter.serviceAccountName	string	`nil`	Name of the ServiceAccount for the Secrets exporter (required if `rbac.create=false`)
rbac.secretsExporter.serviceAccountAnnotations	object	`{}`	Annotations added to the ServiceAccount for the Secrets exporter
rbac.secretsExporter.clusterRoleAnnotations	object	`{}`	Annotations added to the ClusterRole for the Secrets exporter
rbac.secretsExporter.clusterRoleBindingAnnotations	object	`{}`	Annotations added to the ClusterRoleBinding for the Secrets exporter
rbac.hostPathsExporter.serviceAccountName	string	`nil`	Name of the ServiceAccount for hostPath exporters (required if `rbac.create=false`)
rbac.hostPathsExporter.serviceAccountAnnotations	object	`{}`	Annotations added to the ServiceAccount for the hostPath exporters
rbac.hostPathsExporter.clusterRoleAnnotations	object	`{}`	Annotations added to the ClusterRole for the hostPath exporters
rbac.hostPathsExporter.clusterRoleBindingAnnotations	object	`{}`	Annotations added to the ClusterRoleBinding for the hostPath exporters
podExtraLabels	object	`{}`	Extra labels added to all Pods
podAnnotations	object	`{}`	Annotations added to all Pods
exposePerCertificateErrorMetrics	bool	`false`	Enable additional metrics to report per-certificate errors ; helps with identifying read errors origin not having to look at exporter logs, at the expense of additional storage on Prometheus
exposeRelativeMetrics	bool	`false`	Enable additional metrics with relative durations instead of absolute timestamps ; not recommended with Prometheus
metricLabelsFilterList	list	`nil`	Restrict metric labels to this list if set. Warning : use with caution as reducing cardinality may yield metrics collisions and force the exporter to ignore certificates. This will also degrade the usability of the Grafana dashboard. This list should always include at least `filepath`, `secret_namespace` and `secret_name`. Also `subject_CN` is highly recommended for when a file contains multiple certificates.
secretsExporter.enabled	bool	`true`	Should the TLS Secrets exporter be running
secretsExporter.debugMode	bool	`false`	Should debug messages be produced by the TLS Secrets exporter
secretsExporter.replicas	int	`1`	Desired number of TLS Secrets exporter Pod
secretsExporter.restartPolicy	string	`"Always"`	restartPolicy for Pods of the TLS Secrets exporter
secretsExporter.strategy	object	`{}`	DeploymentStrategy for the TLS Secrets exporter
secretsExporter.resources	object	check `values.yaml`	ResourceRequirements for containers of the TLS Secrets exporter
secretsExporter.nodeSelector	object	`{}`	Node selector for Pods of the TLS Secrets exporter
secretsExporter.tolerations	list	`[]`	Toleration for Pods of the TLS Secrets exporter
secretsExporter.affinity	object	`{}`	Affinity for Pods of the TLS Secrets exporter
secretsExporter.podExtraLabels	object	`{}`	Extra labels added to Pods of the TLS Secrets exporter
secretsExporter.podAnnotations	object	`{}`	Annotations added to Pods of the TLS Secrets exporter
secretsExporter.podSecurityContext	object	`{}`	PodSecurityContext for Pods of the TLS Secrets exporter
secretsExporter.securityContext	object	check `values.yaml`	SecurityContext for containers of the TLS Secrets exporter
secretsExporter.secretTypes	list	check `values.yaml`	Which type of Secrets should be watched ; "key" is the map key in the secret data
secretsExporter.includeNamespaces	list	`[]`	Restrict the list of namespaces the TLS Secrets exporter should scan for certificates to watch (all namespaces if empty)
secretsExporter.excludeNamespaces	list	`[]`	Exclude namespaces from being scanned by the TLS Secrets exporter (evaluated after `includeNamespaces`)
secretsExporter.includeLabels	list	`[]`	Only watch TLS Secrets having these labels (all secrets if empty). Items can be keys such as `my-label` or also require a value with syntax `my-label=my-value`.
secretsExporter.excludeLabels	list	`[]`	Exclude TLS Secrets having these labels. Items can be keys such as `my-label` or also require a value with syntax `my-label=my-value`.
secretsExporter.cache.enabled	bool	`true`	Enable caching of Kubernetes resources to prevent scraping timeouts
secretsExporter.cache.maxDuration	int	`300`	Maximum time a resource can stay in cache unrefreshed (seconds) - it will be at least half of that
hostPathsExporter.debugMode	bool	`false`	Should debug messages be produced by hostPath exporters (default for all hostPathsExporter.daemonSets)
hostPathsExporter.restartPolicy	string	`"Always"`	restartPolicy for Pods of hostPath exporters (default for all hostPathsExporter.daemonSets)
hostPathsExporter.updateStrategy	object	`{}`	updateStrategy for DaemonSet of hostPath exporters (default for all hostPathsExporter.daemonSets)
hostPathsExporter.resources	object	check `values.yaml`	ResourceRequirements for containers of hostPath exporters (default for all hostPathsExporter.daemonSets)
hostPathsExporter.nodeSelector	object	`{}`	Node selector for Pods of hostPath exporters (default for all hostPathsExporter.daemonSets)
hostPathsExporter.tolerations	list	`[]`	Toleration for Pods of hostPath exporters (default for all hostPathsExporter.daemonSets)
hostPathsExporter.affinity	object	`{}`	Affinity for Pods of hostPath exporters (default for all hostPathsExporter.daemonSets)
hostPathsExporter.podExtraLabels	object	`{}`	Extra labels added to Pods of hostPath exporters (default for all hostPathsExporter.daemonSets)
hostPathsExporter.podAnnotations	object	`{}`	Annotations added to Pods of hostPath exporters (default for all hostPathsExporter.daemonSets)
hostPathsExporter.podSecurityContext	object	`{}`	PodSecurityContext for Pods of hostPath exporters (default for all hostPathsExporter.daemonSets)
hostPathsExporter.securityContext	object	check `values.yaml`	SecurityContext for containers of hostPath exporters (default for all hostPathsExporter.daemonSets)
hostPathsExporter.watchDirectories	list	`[]`	[SEE README] List of directory paths of the host to scan for PEM encoded certificate files to be watched and exported as metrics (one level deep)
hostPathsExporter.watchFiles	list	`[]`	[SEE README] List of file paths of the host for PEM encoded certificates to be watched and exported as metrics (one level deep)
hostPathsExporter.watchKubeconfFiles	list	`[]`	[SEE README] List of Kubeconf file paths of the host to scan for embedded certificates to export metrics about
hostPathsExporter.daemonSets	object	`{}`	[SEE README] Map to define one or many DaemonSets running hostPath exporters. Key is used as a name ; value is a map to override all default settings set by `hostPathsExporter.*`.
rbacProxy.enabled	bool	`false`	Should kube-rbac-proxy be used to expose exporters
rbacProxy.image.registry	string	`"quay.io"`	kube-rbac-proxy image registry
rbacProxy.image.repository	string	`"coreos/kube-rbac-proxy"`	kube-rbac-proxy image repository
rbacProxy.image.tag	string	`"v0.5.0"`	kube-rbac-proxy image version
rbacProxy.image.pullPolicy	string	`"IfNotPresent"`	kube-rbac-proxy image pull policy
rbacProxy.upstreamListenPort	int	`9091`	Listen port for the exporter running inside kube-rbac-proxy exposed Pods
rbacProxy.resources	object	check `values.yaml`	ResourceRequirements for all containers of kube-rbac-proxy
rbacProxy.securityContext	object	check `values.yaml`	SecurityContext for all containers of kube-rbac-proxy
podListenPort	int	`9793`	TCP port to expose Pods on (whether kube-rbac-proxy is enabled or not)
hostNetwork	bool	`false`	Enable hostNetwork mode. Useful when Prometheus is deployed outside of the Kubernetes cluster
service.create	bool	`true`	Should a headless Service be installed, targets all instances Deployment and DaemonSets (required for ServiceMonitor)
service.port	int	`9793`	TCP port to expose the Service on
service.annotations	object	`{}`	Annotations to add to the Service
service.extraLabels	object	`{}`	Extra labels to add to the Service
prometheusServiceMonitor.create	bool	`true`	Should a ServiceMonitor ressource be installed to scrape this exporter. For prometheus-operator (kube-prometheus) users.
prometheusServiceMonitor.scrapeInterval	string	`"60s"`	Target scrape interval set in the ServiceMonitor
prometheusServiceMonitor.scrapeTimeout	string	`"30s"`	Target scrape timeout set in the ServiceMonitor
prometheusServiceMonitor.extraLabels	object	`{}`	Extra labels to add on ServiceMonitor ressources
prometheusServiceMonitor.relabelings	object	`{}`	Relabel config for the ServiceMonitor, see: https://coreos.com/operators/prometheus/docs/latest/api.html#relabelconfig
prometheusPodMonitor.create	bool	`false`	Should a PodMonitor ressource be installed to scrape this exporter. For prometheus-operator (kube-prometheus) users.
prometheusPodMonitor.scrapeInterval	string	`"60s"`	Target scrape interval set in the PodMonitor
prometheusPodMonitor.scrapeTimeout	string	`"30s"`	Target scrape timeout set in the PodMonitor
prometheusPodMonitor.extraLabels	object	`{}`	Extra labels to add on PodMonitor ressources
prometheusPodMonitor.relabelings	object	`{}`	Relabel config for the PodMonitor, see: https://coreos.com/operators/prometheus/docs/latest/api.html#relabelconfig
prometheusRules.create	bool	`true`	Should a PrometheusRule ressource be installed to alert on certificate expiration. For prometheus-operator (kube-prometheus) users.
prometheusRules.alertOnReadErrors	bool	`true`	Should the X509ExporterReadErrors alerting rule be created to notify when the exporter can't read files or authenticate with the Kubernetes API. It aims at preventing undetected misconfigurations and monitoring regressions.
prometheusRules.readErrorsSeverity	string	`"warning"`	Severity for the X509ExporterReadErrors alerting rule
prometheusRules.alertOnCertificateErrors	bool	`true`	Should the CertificateError alerting rule be created to notify when the exporter can't decode or process a certificate. Depends on `exposePerCertificateErrorMetrics` to be enabled too.
prometheusRules.certificateErrorsSeverity	string	`"warning"`	Severity for the CertificateError alerting rule
prometheusRules.certificateRenewalsSeverity	string	`"warning"`	Severity for the CertificateRenewal alerting rule
prometheusRules.certificateExpirationsSeverity	string	`"critical"`	Severity for the CertificateExpiration alerting rule
prometheusRules.warningDaysLeft	int	`28`	Raise a warning alert when this little days are left before a certificate expiration (cert-manager would renew Let's Encrypt certs before day 29)
prometheusRules.criticalDaysLeft	int	`14`	Raise a critical alert when this little days are left before a certificate expiration (two weeks to deal with ACME rate limiting should this be an issue)
prometheusRules.extraLabels	object	`{}`	Extra labels to add on PrometheusRule ressources

⚖️ License

Copyright (c) 2020, 2021 ENIX

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

README.md Unescape Escape

🔏 X.509 Certificate Exporter

🏃 TL;DR