[stable/rabbitmq] Make metrics probes configurable (#18200)

* [stable/rabbitmq] Make metrics probes configurable

* The default timeouts are too low for clusters with a large
  queue count.
* Cleaned up parameter markdown table a bit in README.md

Signed-off-by: Sheldon Warkentin <sheldonw@forallsecure.com>

* [stable/rabbitmq] Bump chart version

Signed-off-by: Sheldon Warkentin <sheldonw@forallsecure.com>

* [stable/rabbitmq] Resolve review feedback

* A feature change requires minor version incremented
  to 6.10.0
* Make use of `metrics.*Probe.enabled` flags.
  * This will control the inclusion of the metric probes in
     the exporter sidecar.

Signed-off-by: Sheldon Warkentin <sheldonw@forallsecure.com>

* [stable/rabbitmq] Add probes to prod example yaml

Signed-off-by: Sheldon Warkentin <sheldonw@forallsecure.com>
This commit is contained in:
Sheldon Warkentin
2019-10-23 08:02:16 -07:00
committed by Kubernetes Prow Robot
parent 1e6196e9eb
commit bcfe46b9d4
5 changed files with 169 additions and 115 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
apiVersion: v1
name: rabbitmq
version: 6.9.0
version: 6.10.0
appVersion: 3.8.0
description: Open source message broker software that implements the Advanced Message Queuing Protocol (AMQP)
keywords:
+122 -110
View File
@@ -46,116 +46,128 @@ The command removes all the Kubernetes components associated with the chart and
The following table lists the configurable parameters of the RabbitMQ chart and their default values.
| Parameter | Description | Default |
| ------------------------------------ | ------------------------------------------------ | ------------------------------------------------------- |
| `global.imageRegistry` | Global Docker image registry | `nil` |
| `global.imagePullSecrets` | Global Docker registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) |
| `global.storageClass` | Global storage class for dynamic provisioning | `nil` |
| `image.registry` | Rabbitmq Image registry | `docker.io` |
| `image.repository` | Rabbitmq Image name | `bitnami/rabbitmq` |
| `image.tag` | Rabbitmq Image tag | `{TAG_NAME}` |
| `image.pullPolicy` | Image pull policy | `IfNotPresent` |
| `image.pullSecrets` | Specify docker-registry secret names as an array | `nil` |
| `image.debug` | Specify if debug values should be set | `false` |
| `nameOverride` | String to partially override rabbitmq.fullname template with a string (will prepend the release name) | `nil` |
| `fullnameOverride` | String to fully override rabbitmq.fullname template with a string | `nil` |
| `rbacEnabled` | Specify if rbac is enabled in your cluster | `true` |
| `podManagementPolicy` | Pod management policy | `OrderedReady` |
| `rabbitmq.username` | RabbitMQ application username | `user` |
| `rabbitmq.password` | RabbitMQ application password | _random 10 character long alphanumeric string_ |
| `rabbitmq.existingPasswordSecret` | Existing secret with RabbitMQ credentials | `nil` |
| `rabbitmq.erlangCookie` | Erlang cookie | _random 32 character long alphanumeric string_ |
| `rabbitmq.existingErlangSecret` | Existing secret with RabbitMQ Erlang cookie | `nil` |
| `rabbitmq.plugins` | List of plugins to enable | `rabbitmq_management rabbitmq_peer_discovery_k8s` |
| `rabbitmq.extraPlugins` | Extra plugings to enable | `nil` |
| `rabbitmq.clustering.address_type` | Switch clustering mode | `ip` or `hostname` |
| `rabbitmq.clustering.k8s_domain` | Customize internal k8s cluster domain | `cluster.local` |
| `rabbitmq.logs` | Value for the RABBITMQ_LOGS environment variable | `-` |
| `rabbitmq.setUlimitNofiles` | Specify if max file descriptor limit should be set | `true` |
| `rabbitmq.ulimitNofiles` | Max File Descriptor limit | `65536` |
| `rabbitmq.maxAvailableSchedulers` | RabbitMQ maximum available scheduler threads | `2` |
| `rabbitmq.onlineSchedulers` | RabbitMQ online scheduler threads | `1` |
| `rabbitmq.env` | RabbitMQ [environment variables](https://www.rabbitmq.com/configure.html#customise-environment) | `{}` |
| `rabbitmq.configuration` | Required cluster configuration | See values.yaml |
| `rabbitmq.extraConfiguration` | Extra configuration to add to rabbitmq.conf | See values.yaml |
| `rabbitmq.advancedConfiguration` | Extra configuration (in classic format) to add to advanced.config | See values.yaml |
| `rabbitmq.tls.enabled` | Enable TLS support to rabbitmq | | `false` |
| `rabbitmq.tls.failIfNoPeerCert` | When set to true, TLS connection will be rejected if client fails to provide a certificate | `true` |
| `rabbitmq.tls.sslOptionsVerify` | `verify_peer` | Should [peer verification](https://www.rabbitmq.com/ssl.html#peer-verification) be enabled? |
| `rabbitmq.tls.caCertificate` | Ca certificate | Certificate Authority (CA) bundle content |
| `rabbitmq.tls.serverCertificate` | Server certificate | Server certificate content |
| `rabbitmq.tls.serverKey` | Server Key | Server private key content |
| `rabbitmq.tls.existingSecret` | Existing secret with certificate content to rabbitmq credentials | `nil` |
| `service.type` | Kubernetes Service type | `ClusterIP` |
| `service.port` | Amqp port | `5672` |
| `service.tlsPort` | Amqp TLS port | `5671` |
| `service.distPort` | Erlang distribution server port | `25672` |
| `service.nodePort` | Node port override, if serviceType NodePort | _random available between 30000-32767_ |
| `service.nodeTlsPort` | Node port override, if serviceType NodePort | _random available between 30000-32767_ |
| `service.managerPort` | RabbitMQ Manager port | `15672` |
| `service.extraPorts` | Extra ports to expose in the service | `nil` |
| `service.extraContainerPorts` | Extra ports to be included in container spec, primarily informational | `nil` |
| `persistence.enabled` | Use a PVC to persist data | `true` |
| `service.annotations` | service annotations as an array | [] |
| `schedulerName` | Name of the k8s service (other than default) | `nil` |
| `persistence.storageClass` | Storage class of backing PVC | `nil` (uses alpha storage class annotation) |
| `persistence.existingClaim` | RabbitMQ data Persistent Volume existing claim name, evaluated as a template | "" |
| `persistence.accessMode` | Use volume as ReadOnly or ReadWrite | `ReadWriteOnce` |
| `persistence.size` | Size of data volume | `8Gi` |
| `persistence.path` | Mount path of the data volume | `/opt/bitnami/rabbitmq/var/lib/rabbitmq` |
| `securityContext.enabled` | Enable security context | `true` |
| `securityContext.fsGroup` | Group ID for the container | `1001` |
| `securityContext.runAsUser` | User ID for the container | `1001` |
| `resources` | resource needs and limits to apply to the pod | {} |
| `replicas` | Replica count | `1` |
| `priorityClassName` | Pod priority class name | `` |
| `nodeSelector` | Node labels for pod assignment | {} |
| `affinity` | Affinity settings for pod assignment | {} |
| `tolerations` | Toleration labels for pod assignment | [] |
| `updateStrategy` | Statefulset update strategy policy | `RollingUpdate` |
| `ingress.enabled` | Enable ingress resource for Management console | `false` |
| `ingress.hostName` | Hostname to your RabbitMQ installation | `nil` |
| `ingress.path` | Path within the url structure | `/` |
| `ingress.tls` | enable ingress with tls | `false` |
| `ingress.tlsSecret` | tls type secret to be used | `myTlsSecret` |
| `ingress.annotations` | ingress annotations as an array | [] |
| `livenessProbe.enabled` | would you like a livenessProbed to be enabled | `true` |
| `livenessProbe.initialDelaySeconds` | number of seconds | 120 |
| `livenessProbe.timeoutSeconds` | number of seconds | 20 |
| `livenessProbe.periodSeconds` | number of seconds | 30 |
| `livenessProbe.failureThreshold` | number of failures | 6 |
| `livenessProbe.successThreshold` | number of successes | 1 |
| `readinessProbe.enabled` | would you like a readinessProbe to be enabled | `true` |
| `readinessProbe.initialDelaySeconds` | number of seconds | 10 |
| `readinessProbe.timeoutSeconds` | number of seconds | 20 |
| `readinessProbe.periodSeconds` | number of seconds | 30 |
| `readinessProbe.failureThreshold` | number of failures | 3 |
| `readinessProbe.successThreshold` | number of successes | 1 |
| `metrics.enabled` | Start a side-car prometheus exporter | `false` |
| `metrics.image.registry` | Exporter image registry | `docker.io` |
| `metrics.image.repository` | Exporter image name | `bitnami/rabbitmq-exporter` |
| `metrics.image.tag` | Exporter image tag | `{TAG_NAME}` |
| `metrics.image.pullPolicy` | Exporter image pull policy | `IfNotPresent` |
| `metrics.serviceMonitor.enabled` | Create ServiceMonitor Resource for scraping metrics using PrometheusOperator | `false` |
| `metrics.serviceMonitor.namespace` | Namespace where servicemonitor resource should be created | `nil` |
| `metrics.serviceMonitor.interval` | Specify the interval at which metrics should be scraped | `30s` |
| `metrics.serviceMonitor.scrapeTimeout`| Specify the timeout after which the scrape is ended | `nil` |
| `metrics.serviceMonitor.relabellings`| Specify Metric Relabellings to add to the scrape endpoint | `nil` |
| `metrics.serviceMonitor.honorLabels` | honorLabels chooses the metric's labels on collisions with target labels. | `false` |
| `metrics.serviceMonitor.additionalLabels`| Used to pass Labels that are required by the Installed Prometheus Operator | `{}` |
| `metrics.port` | Prometheus metrics exporter port | `9419` |
| `metrics.env` | Exporter [configuration environment variables](https://github.com/kbudde/rabbitmq_exporter#configuration) | `{}` |
| `metrics.resources` | Exporter resource requests/limit | `nil` |
| `metrics.capabilities` | Exporter: Comma-separated list of extended [scraping capabilities supported by the target RabbitMQ server](https://github.com/kbudde/rabbitmq_exporter#extended-rabbitmq-capabilities) | `bert,no_sort` |
| `podLabels` | Additional labels for the statefulset pod(s). | {} |
| `volumePermissions.enabled` | Enable init container that changes volume permissions in the data directory (for cases where the default k8s `runAsUser` and `fsUser` values do not work) | `false` |
| `volumePermissions.image.registry` | Init container volume-permissions image registry | `docker.io` |
| `volumePermissions.image.repository` | Init container volume-permissions image name | `bitnami/minideb` |
| `volumePermissions.image.tag` | Init container volume-permissions image tag | `stretch` |
| `volumePermissions.image.pullPolicy` | Init container volume-permissions image pull policy | `Always` |
| `volumePermissions.resources` | Init container resource requests/limit | `nil` |
| `forceBoot.enabled` | Executes 'rabbitmqctl force_boot' to force boot cluster shut down unexpectedly in an unknown order. Use it only if you prefer availability over integrity. | `false` |
| `extraSecrets` | Optionally specify extra secrets to be created by the chart. | `{}` |
| Parameter | Description | Default |
| -------------------------------------------- | ------------------------------------------------ | ------------------------------------------------------- |
| `global.imageRegistry` | Global Docker image registry | `nil` |
| `global.imagePullSecrets` | Global Docker registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) |
| `global.storageClass` | Global storage class for dynamic provisioning | `nil` |
| `image.registry` | Rabbitmq Image registry | `docker.io` |
| `image.repository` | Rabbitmq Image name | `bitnami/rabbitmq` |
| `image.tag` | Rabbitmq Image tag | `{TAG_NAME}` |
| `image.pullPolicy` | Image pull policy | `IfNotPresent` |
| `image.pullSecrets` | Specify docker-registry secret names as an array | `nil` |
| `image.debug` | Specify if debug values should be set | `false` |
| `nameOverride` | String to partially override rabbitmq.fullname template with a string (will prepend the release name) | `nil` |
| `fullnameOverride` | String to fully override rabbitmq.fullname template with a string | `nil` |
| `rbacEnabled` | Specify if rbac is enabled in your cluster | `true` |
| `podManagementPolicy` | Pod management policy | `OrderedReady` |
| `rabbitmq.username` | RabbitMQ application username | `user` |
| `rabbitmq.password` | RabbitMQ application password | _random 10 character long alphanumeric string_ |
| `rabbitmq.existingPasswordSecret` | Existing secret with RabbitMQ credentials | `nil` |
| `rabbitmq.erlangCookie` | Erlang cookie | _random 32 character long alphanumeric string_ |
| `rabbitmq.existingErlangSecret` | Existing secret with RabbitMQ Erlang cookie | `nil` |
| `rabbitmq.plugins` | List of plugins to enable | `rabbitmq_management rabbitmq_peer_discovery_k8s` |
| `rabbitmq.extraPlugins` | Extra plugings to enable | `nil` |
| `rabbitmq.clustering.address_type` | Switch clustering mode | `ip` or `hostname` |
| `rabbitmq.clustering.k8s_domain` | Customize internal k8s cluster domain | `cluster.local` |
| `rabbitmq.logs` | Value for the RABBITMQ_LOGS environment variable | `-` |
| `rabbitmq.setUlimitNofiles` | Specify if max file descriptor limit should be set | `true` |
| `rabbitmq.ulimitNofiles` | Max File Descriptor limit | `65536` |
| `rabbitmq.maxAvailableSchedulers` | RabbitMQ maximum available scheduler threads | `2` |
| `rabbitmq.onlineSchedulers` | RabbitMQ online scheduler threads | `1` |
| `rabbitmq.env` | RabbitMQ [environment variables](https://www.rabbitmq.com/configure.html#customise-environment) | `{}` |
| `rabbitmq.configuration` | Required cluster configuration | See values.yaml |
| `rabbitmq.extraConfiguration` | Extra configuration to add to rabbitmq.conf | See values.yaml |
| `rabbitmq.advancedConfiguration` | Extra configuration (in classic format) to add to advanced.config | See values.yaml |
| `rabbitmq.tls.enabled` | Enable TLS support to rabbitmq | `false` |
| `rabbitmq.tls.failIfNoPeerCert` | When set to true, TLS connection will be rejected if client fails to provide a certificate | `true` |
| `rabbitmq.tls.sslOptionsVerify` | `verify_peer` | Should [peer verification](https://www.rabbitmq.com/ssl.html#peer-verification) be enabled? |
| `rabbitmq.tls.caCertificate` | Ca certificate | Certificate Authority (CA) bundle content |
| `rabbitmq.tls.serverCertificate` | Server certificate | Server certificate content |
| `rabbitmq.tls.serverKey` | Server Key | Server private key content |
| `rabbitmq.tls.existingSecret` | Existing secret with certificate content to rabbitmq credentials | `nil` |
| `service.type` | Kubernetes Service type | `ClusterIP` |
| `service.port` | Amqp port | `5672` |
| `service.tlsPort` | Amqp TLS port | `5671` |
| `service.distPort` | Erlang distribution server port | `25672` |
| `service.nodePort` | Node port override, if serviceType NodePort | _random available between 30000-32767_ |
| `service.nodeTlsPort` | Node port override, if serviceType NodePort | _random available between 30000-32767_ |
| `service.managerPort` | RabbitMQ Manager port | `15672` |
| `service.extraPorts` | Extra ports to expose in the service | `nil` |
| `service.extraContainerPorts` | Extra ports to be included in container spec, primarily informational | `nil` |
| `persistence.enabled` | Use a PVC to persist data | `true` |
| `service.annotations` | service annotations as an array | [] |
| `schedulerName` | Name of the k8s service (other than default) | `nil` |
| `persistence.storageClass` | Storage class of backing PVC | `nil` (uses alpha storage class annotation) |
| `persistence.existingClaim` | RabbitMQ data Persistent Volume existing claim name, evaluated as a template | "" |
| `persistence.accessMode` | Use volume as ReadOnly or ReadWrite | `ReadWriteOnce` |
| `persistence.size` | Size of data volume | `8Gi` |
| `persistence.path` | Mount path of the data volume | `/opt/bitnami/rabbitmq/var/lib/rabbitmq` |
| `securityContext.enabled` | Enable security context | `true` |
| `securityContext.fsGroup` | Group ID for the container | `1001` |
| `securityContext.runAsUser` | User ID for the container | `1001` |
| `resources` | resource needs and limits to apply to the pod | {} |
| `replicas` | Replica count | `1` |
| `priorityClassName` | Pod priority class name | `` |
| `nodeSelector` | Node labels for pod assignment | {} |
| `affinity` | Affinity settings for pod assignment | {} |
| `tolerations` | Toleration labels for pod assignment | [] |
| `updateStrategy` | Statefulset update strategy policy | `RollingUpdate` |
| `ingress.enabled` | Enable ingress resource for Management console | `false` |
| `ingress.hostName` | Hostname to your RabbitMQ installation | `nil` |
| `ingress.path` | Path within the url structure | `/` |
| `ingress.tls` | enable ingress with tls | `false` |
| `ingress.tlsSecret` | tls type secret to be used | `myTlsSecret` |
| `ingress.annotations` | ingress annotations as an array | [] |
| `livenessProbe.enabled` | would you like a livenessProbed to be enabled | `true` |
| `livenessProbe.initialDelaySeconds` | number of seconds | 120 |
| `livenessProbe.timeoutSeconds` | number of seconds | 20 |
| `livenessProbe.periodSeconds` | number of seconds | 30 |
| `livenessProbe.failureThreshold` | number of failures | 6 |
| `livenessProbe.successThreshold` | number of successes | 1 |
| `readinessProbe.enabled` | would you like a readinessProbe to be enabled | `true` |
| `readinessProbe.initialDelaySeconds` | number of seconds | 10 |
| `readinessProbe.timeoutSeconds` | number of seconds | 20 |
| `readinessProbe.periodSeconds` | number of seconds | 30 |
| `readinessProbe.failureThreshold` | number of failures | 3 |
| `readinessProbe.successThreshold` | number of successes | 1 |
| `metrics.enabled` | Start a side-car prometheus exporter | `false` |
| `metrics.image.registry` | Exporter image registry | `docker.io` |
| `metrics.image.repository` | Exporter image name | `bitnami/rabbitmq-exporter` |
| `metrics.image.tag` | Exporter image tag | `{TAG_NAME}` |
| `metrics.image.pullPolicy` | Exporter image pull policy | `IfNotPresent` |
| `metrics.livenessProbe.enabled` | would you like a livenessProbed to be enabled | `true` |
| `metrics.livenessProbe.initialDelaySeconds` | number of seconds | 15 |
| `metrics.livenessProbe.timeoutSeconds` | number of seconds | 5 |
| `metrics.livenessProbe.periodSeconds` | number of seconds | 30 |
| `metrics.livenessProbe.failureThreshold` | number of failures | 6 |
| `metrics.livenessProbe.successThreshold` | number of successes | 1 |
| `metrics.readinessProbe.enabled` | would you like a readinessProbe to be enabled | `true` |
| `metrics.readinessProbe.initialDelaySeconds` | number of seconds | 5 |
| `metrics.readinessProbe.timeoutSeconds` | number of seconds | 5 |
| `metrics.readinessProbe.periodSeconds` | number of seconds | 30 |
| `metrics.readinessProbe.failureThreshold` | number of failures | 3 |
| `metrics.readinessProbe.successThreshold` | number of successes | 1 |
| `metrics.serviceMonitor.enabled` | Create ServiceMonitor Resource for scraping metrics using PrometheusOperator | `false` |
| `metrics.serviceMonitor.namespace` | Namespace where servicemonitor resource should be created | `nil` |
| `metrics.serviceMonitor.interval` | Specify the interval at which metrics should be scraped | `30s` |
| `metrics.serviceMonitor.scrapeTimeout` | Specify the timeout after which the scrape is ended | `nil` |
| `metrics.serviceMonitor.relabellings` | Specify Metric Relabellings to add to the scrape endpoint | `nil` |
| `metrics.serviceMonitor.honorLabels` | honorLabels chooses the metric's labels on collisions with target labels. | `false` |
| `metrics.serviceMonitor.additionalLabels` | Used to pass Labels that are required by the Installed Prometheus Operator | `{}` |
| `metrics.port ` | Prometheus metrics exporter port | `9419` |
| `metrics.env` | Exporter [configuration environment variables](https://github.com/kbudde/rabbitmq_exporter#configuration) | `{}` |
| `metrics.resources` | Exporter resource requests/limit | `nil` |
| `metrics.capabilities` | Exporter: Comma-separated list of extended [scraping capabilities supported by the target RabbitMQ server](https://github.com/kbudde/rabbitmq_exporter#extended-rabbitmq-capabilities) | `bert,no_sort` |
| `podLabels` | Additional labels for the statefulset pod(s). | {} |
| `volumePermissions.enabled` | Enable init container that changes volume permissions in the data directory (for cases where the default k8s `runAsUser` and `fsUser` values do not work) | `false |
| `volumePermissions.image.registry` | Init container volume-permissions image registry | `docker.io` |
| `volumePermissions.image.repository` | Init container volume-permissions image name | `bitnami/minideb` |
| `volumePermissions.image.tag` | Init container volume-permissions image tag | `stretch` |
| `volumePermissions.image.pullPolicy` | Init container volume-permissions image pull policy | `Always` |
| `volumePermissions.resources` | Init container resource requests/limit | `nil` |
| `forceBoot.enabled` | Executes 'rabbitmqctl force_boot' to force boot cluster shut down unexpectedly in an unknown order. Use it only if you prefer availability over integrity. | `false` |
| `extraSecrets` | Optionally specify extra secrets to be created by the chart. | `{}` |
The above parameters map to the env variables defined in [bitnami/rabbitmq](http://github.com/bitnami/bitnami-docker-rabbitmq). For more information please refer to the [bitnami/rabbitmq](http://github.com/bitnami/bitnami-docker-rabbitmq) image documentation.
+14 -4
View File
@@ -246,18 +246,28 @@ spec:
ports:
- name: metrics
containerPort: 9090
{{- if .Values.metrics.livenessProbe.enabled }}
livenessProbe:
httpGet:
path: /metrics
port: metrics
initialDelaySeconds: 15
timeoutSeconds: 5
initialDelaySeconds: {{ .Values.metrics.livenessProbe.initialDelaySeconds }}
timeoutSeconds: {{ .Values.metrics.livenessProbe.timeoutSeconds }}
periodSeconds: {{ .Values.metrics.livenessProbe.periodSeconds }}
failureThreshold: {{ .Values.metrics.livenessProbe.failureThreshold }}
successThreshold: {{ .Values.metrics.livenessProbe.successThreshold }}
{{- end }}
{{- if .Values.metrics.readinessProbe.enabled }}
readinessProbe:
httpGet:
path: /metrics
port: metrics
initialDelaySeconds: 5
timeoutSeconds: 5
initialDelaySeconds: {{ .Values.metrics.readinessProbe.initialDelaySeconds }}
timeoutSeconds: {{ .Values.metrics.readinessProbe.timeoutSeconds }}
periodSeconds: {{ .Values.metrics.readinessProbe.periodSeconds }}
failureThreshold: {{ .Values.metrics.readinessProbe.failureThreshold }}
successThreshold: {{ .Values.metrics.readinessProbe.successThreshold }}
{{- end }}
resources:
{{ toYaml .Values.metrics.resources | indent 10 }}
{{- end }}
+16
View File
@@ -349,6 +349,22 @@ metrics:
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
livenessProbe:
enabled: true
initialDelaySeconds: 15
timeoutSeconds: 5
periodSeconds: 30
failureThreshold: 6
successThreshold: 1
readinessProbe:
enabled: true
initialDelaySeconds: 5
timeoutSeconds: 5
periodSeconds: 30
failureThreshold: 3
successThreshold: 1
## Prometheus Service Monitor
## ref: https://github.com/coreos/prometheus-operator
## https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#endpoint
+16
View File
@@ -349,6 +349,22 @@ metrics:
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
livenessProbe:
enabled: true
initialDelaySeconds: 15
timeoutSeconds: 5
periodSeconds: 30
failureThreshold: 6
successThreshold: 1
readinessProbe:
enabled: true
initialDelaySeconds: 5
timeoutSeconds: 5
periodSeconds: 30
failureThreshold: 3
successThreshold: 1
## Prometheus Service Monitor
## ref: https://github.com/coreos/prometheus-operator
## https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#endpoint