🇬🇧Creating a Monitoring Environment with Prometheus and Grafana

Posted on by

🇧🇷 – para ler este artigo em português clique aqui

Monitoring applications is crucial to ensure everything is working properly and to quickly detect potential issues. Moreover, it makes your applications much more professional by incorporating proper monitoring. In this article, I’ll show you how to set up a monitoring environment using Prometheus and Grafana.

What is Prometheus?

Prometheus is a monitoring tool that collects and stores metrics from services and applications. It is popular for its active data collection capability, meaning it gathers information directly from services exposed via HTTP without relying on external submissions.

Let’s build a compose.yaml:

services:

  web:

    image: kubedevio/rotten-potatoes:v1

    ports:

      - "8080:5000"

    networks:

      - live_prometheus

    depends_on:

      - mongodb

    environment:

      MONGODB: "admin"

      MONGODB_HOST: "mongodb"

      MONGODB_PORT: "27017"

      MONGODB_USERNAME: "mongouser"

      MONGODB_PASSWORD: "mongopwd"

  mongodb:

    image: mongo:7.0

    volumes:

      - db_mongo:/data/db

    networks:

      - live_prometheus

    environment:

      MONGO_INITDB_ROOT_USERNAME: mongouser

      MONGO_INITDB_ROOT_PASSWORD: mongopwd

  prometheus:

    image: prom/prometheus:v2.47.2

    ports:

      - "9090:9090"

    networks:

      - live_prometheus

    volumes:

      - ./prometheus.yaml:/etc/prometheus/prometheus.yml

    extra_hosts:

      host.docker.internal: 192.168.100.5

  grafana:

    image: grafana/grafana:10.2.2

    ports:

      - "3000:3000"

    networks:

      - live_prometheus

volumes:

  db_mongo:

networks:

  live_prometheus:

    driver: bridge

In this file, we are creating a Docker environment with the following services:

  • web: Running on port 8080 and connected to the live_prometheus network. This service hosts a sample web application that exposes metrics through the API at /metrics.

prometheus: Running on port 9090 and connected to the live_prometheus network. This service builds the Prometheus image and provides a web version of the application. Access Prometheus at localhost:9090, and click on Status > Targets in the top menu to see all endpoints being monitored by Prometheus.

  • mongodb: Builds the MongoDB image, used by the web application.
  • grafana: Running on port 3000 and connected to the live_prometheus network. It builds the Grafana image, providing the web application for visualization.

Adding web application monitoring in Prometheus:

We will add a prometheus.yaml file to the project, which will serve as Prometheus’ configuration file. Prometheus offers several configuration options, but we will use only two:

  • scrape_interval: 15s
  • scrape_timeout: 10s
global:

  scrape_interval: 15s

  scrape_timeout: 10s

scrape_configs:

  - job_name: rottenpotatoes

    scrape_interval: 15s

    scrape_timeout: 10s

    static_configs:

      - targets:

          - web:5000

  - job_name: prometheus

    static_configs:

      - targets:

          - localhost:9090

Once Prometheus is configured, it will automatically update the targets with our monitored web application.

Now, I can choose which metrics I want to collect

Using an example metric (localhost:8080/metrics) on the Prometheus homepage:

  • flask_http_request_duration_seconds_count:
  • You can filter specific parameters from this example.
flask_http_request_duration_seconds_count{status="404"}

This filters only the entries with status 404.

For a more professional view of the information, I use Grafana

Grafana is an open-source platform for data visualization, widely used for application and infrastructure monitoring. It allows the creation of interactive dashboards that display metrics, logs, and real-time alerts from various data sources like Prometheus, Elasticsearch, InfluxDB, and others.

Grafana is running on port 3000:

  • Username: admin
  • Password: admin

In Grafana:

  1. Access Menu > Prometheus > Connections > Datasource > Prometheus.
  2. Enter the following in Connections:
    http://prometheus:9090
  1. Save the settings.

This way, Grafana will now be aware of the web application and its metrics.

Creating a dashboard in Grafana:

For those unfamiliar with creating dashboards in Grafana, the tool offers several free templates:

Example template: https://grafana.com/grafana/dashboards/3662-prometheus-2-0-overview/
Click on Copy ID to clipboard, paste it into Grafana, and click Load

Configure the datasource for Prometheus and import.

Your dashboard is ready!

This was just an example of how to import dashboards in Grafana, but we won’t be using exactly this dashboard in our monitoring. So, save and return to the Grafana menu.

Now, how do we retrieve the application information and display it in Grafana?

Looking at the web application repository:

https://github.com/KubeDev/rotten-potatoes

in the file:

https://github.com/KubeDev/rotten-potatoes/blob/main/src/app.py

we are using:

from prometheus_flask_exporter import PrometheusMetrics

from the Prometheus library to expose the metrics and use them.

The import of PrometheusMetrics in the app.py file from the prometheus_flask_exporter library is essential to expose metrics from the Flask application and allow Prometheus to collect and monitor them.

This library acts as an integrator between the Flask application and Prometheus, providing specific metrics about the application’s behavior, such as request response times, error counts, and resource usage.

This is just one of the libraries, but you can check the official Grafana documentation and read more about each metrics library. The important thing is to prepare the application to expose its metrics.

In the prometheus_flask_exporter project: https://github.com/rycus86/prometheus_flask_exporter

there is a dashboards folder with a dashboard in JSON format:

https://github.com/rycus86/prometheus_flask_exporter/blob/master/dashboards/flask_webapp.json

In Grafana, create a new dashboard by importing a dashboard and place the project’s dashboard.

Your monitoring dashboard will be monitored within the application.

Are we going to monitor from a different source?

Monitoring the Docker metrics file:

Include the following:

{
  "metrics-addr": "0.0.0.0:9323"
}

in the Docker configuration file.

Let’s go:

sudo vim /etc/docker/daemon.json

Press I and insert:

{
  "metrics-addr": "0.0.0.0:9323"
}

Then:

:wq!

to save.

Run:

sudo systemctl restart docker

Go to:

http://localhost:9323/metrics

and you will see all Docker metrics exposed.

Include the Docker target in the prometheus.yaml file:

global:
  scrape_interval: 15s
  scrape_timeout: 10s

scrape_configs:
  - job_name: rottenpotatoes
    scrape_interval: 15s
    scrape_timeout: 10s
    static_configs:
      - targets:
          - web:5000

  - job_name: prometheus
    static_configs:
      - targets:
          - localhost:9090

  - job_name: docker  # <<<<<<<< modification
    static_configs:
      - targets:
          - host.docker.internal:9323

Restart the compose:

docker-compose up -d

The Docker target will be available again.

In Grafana, we will insert a new dashboard using a pre-made template (as taught earlier):

https://grafana.com/grafana/dashboards/9621-docker-registry

Notice that the dashboard is already monitoring our Docker at the target we defined (host.docker.internal:9223).

This host.docker.internal was defined in the compose file because Docker runs on the IP of our machine, not inside the Docker container like our web application.

You can navigate the same dashboard across different targets (like web:5000).

If you have any questions about how a visualization uses parameters to monitor metrics, click on the visualization options and then click on Edit.