Prometheus supports scraping multiple instances of an application. Applications that run in orchestrated environments require to be discovered dynamically, since their IP addresses will change. Prometheus can be configured to use the Kubernetes API to discover changes in the list of running instances dynamically.
The following shows a minimalistic Prometheus example of instrumenting an application with multiple pod instances.
Prometheus configuration
We configure Prometheus to discover the pods of our config-example application.
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 | global: scrape_interval: 15s external_labels: monitor: 'example-monitor'scrape_configs:- job_name: 'example-metric' scrape_interval: 5s metrics_path: /metrics/ scheme: https basic_auth: username: admin password: adminadmin tls_config: insecure_skip_verify: true kubernetes_sd_configs: - role: endpoints namespaces: names: - default relabel_configs: - source_labels: [__meta_kubernetes_service_label_app] separator: ; regex: config-example replacement: $1 action: keep - source_labels: [__meta_kubernetes_endpoint_port_name] separator: ; regex: https replacement: $1 action: keep - source_labels: [__meta_kubernetes_namespace] separator: ; regex: (.*) target_label: namespace replacement: $1 action: replace - source_labels: [__meta_kubernetes_pod_name] separator: ; regex: (.*) target_label: pod replacement: $1 action: replace - source_labels: [__meta_kubernetes_service_name] separator: ; regex: (.*) target_label: service replacement: $1 action: replace - source_labels: [__meta_kubernetes_service_name] separator: ; regex: (.*) target_label: job replacement: ${1} action: replace - separator: ; regex: (.*) target_label: endpoint replacement: https action: replace |
We need to adjust the app label (here config-example) and the port name (https) under which the monitoring endpoint is available.
Instrumented application
This example uses the MicroProfile Metrics endpoint that runs on top of Open Liberty. The application exposes the metrics in the Prometheus format under port 9443 and path /metrics/.
Here you can see an example service and deployment:
01 02 03 04 05 06 07 08 09 10 11 12 13 14 | kind: ServiceapiVersion: v1metadata: name: config-example labels: app: config-examplespec: selector: app: config-example ports: - port: 9443 name: https - port: 9080 name: http |
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 | kind: DeploymentapiVersion: apps/v1beta1metadata: name: config-examplespec: replicas: 2 template: metadata: labels: app: config-example spec: containers: - name: config-example image: sdaschner/config-example:1 # ... |
You can find the full example including the running application on GitHub.
Prometheus setup & RBAC
In order to make the example work for a RBAC Kubernetes setup, the Prometheus service account needs to have the correct permissions. Therefore, we create a service account and corresponding cluster roles as included in the YAML definitions here. Similarly, we specify the service account when creating the Prometheus deployment.
We create a deployment and service for our Prometheus instance.
01 02 03 04 05 06 07 08 09 10 11 12 | kind: ServiceapiVersion: v1metadata: name: prometheus labels: app: prometheusspec: selector: app: prometheus ports: - port: 9090 name: http |
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | kind: DeploymentapiVersion: apps/v1beta1metadata: name: prometheusspec: replicas: 1 template: metadata: labels: app: prometheus version: v1 spec: serviceAccountName: prometheus containers: - name: prometheus image: prom/prometheus:v2.7.1 ports: - containerPort: 9090 volumeMounts: - name: prometheus-config-volume mountPath: /etc/prometheus/prometheus.yml subPath: prometheus.yml volumes: - name: prometheus-config-volume configMap: name: prometheus-config restartPolicy: Always |
The configuration shown earlier can be injected into the instance from a config map.
1 2 3 4 5 6 7 8 9 | kind: ConfigMapapiVersion: v1metadata: name: prometheus-configdata: prometheus.yml: | global: scrape_interval: 15s # content as shown earlier ... |
For a full explanation on the Prometheus configuration, have a look at the documentation.
Accessing the targets
The running Prometheus instance can now discover both running config example applications which run as Kubernetes pods:
1 | gt; kubectl get pods NAME READY STATUS RESTARTS AGE config-example-69974cbc96-dqd96 1/1 Running 0 4m config-example-69974cbc96-zstg7 1/1 Running 0 4m grafana-8694db9d4f-nvn5s 1/1 Running 0 3m prometheus-594dd9cdb8-95ftz 1/1 Running 0 3m |
We can see the actual targets including their IP address under the Prometheus configuration.
Have a look at the full example on GitHub. You might also see the following video how to implement business metrics in Java EE applications using MicroProfile.
For more complex microservice examples that define many more applications, this approach using the plain Prometheus configuration might be a bit cumbersome. To minimize boilerplate code, developers can use solutions that abstracts away the lower-level configuration, such as Prometheus Operator. In the next post we’ll see how Prometheus Operator facilitates the configuration of applications.
Happy monitoring!
Published on Java Code Geeks with permission by Sebastian Daschner, partner at our JCG program. See the original article here: Discover applications running on Kubernetes with Prometheus Opinions expressed by Java Code Geeks contributors are their own. |
