HomeDockerHow to setup a Grafana and Prometheus stack

How to setup a Grafana and Prometheus stack

What we are going to achieve with this article is to realize a monitoring resolution, that monitors a number of services for you and shows this in a dashboard.

  • What each image does in a nutshell: Grafana visualizes data and shows it beautifully in a dashboard. Prometheus is the data source where all data ends up. Node_exporter and cadvisor monitors the system and Docker engine (i.e. all Docker services) and ultimately send the collected data to Prometheus.
  • What we also added: Adguard exporter and Speedtest. Briefly what Adguard does: a service that acts as a DNS server and blocks advertisements in your LAN community. You can show this data, for example how many ads have been blocked and which clients fire the most queries. And Speedtest makes a call every x minutes to monitor your download velocity, upload velocity and latency from your home community.

How the Adguard dashboard looks like this:

grafana-adguard

grafana-adguard

How the Speedtest dashboard like this:

grafana-speedtest

grafana-speedtest

Now let’s configure the whole thing!

Prerequisites

  • I assume that you have Docker operating and you already have basic knowledge of how to use Docker.
  • You already have Traefik operating as a reversed proxy.
  • It works with Docker swarm but you can easily convert this to Docker compose.
  • I can’t elaborate every detail in the yml file what it does, in that case please check the official page of that image for more information.
  • This has been examined on a QNAP, but it will probably work on another NAS system like Synology or OMV.
  • By default, the stack demands a lot of memory and CPU from the system where you are going to run it. The configuration is optimized to run without too many sources.

Before we proceed, we will make sure that the folder structure is in order. Create the following folders and files and copy paste the contents as described below.

.(your volume)
├── Docker-compose.yml
├── grafana/
├── prometheus/
    └── data/
    └── prometheus.yml

You can copy and paste this in prometheus.yml
Change the NASip to your internal NAS IP address.

global:
  scrape_interval: 30s
  scrape_timeout: 10s
  evaluation_interval: 30s

scrape_configs:
- job_name: prometheus
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - NASip:9090
- job_name: grafana
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - grafana:3000
- job_name: node_exporter
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - node_exporter:9100
- job_name: traefik
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - NASip:8090
- job_name: cadvisor
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - cadvisor:8080
- job_name: adguard
  static_configs:
  - targets:
    - adguard_exporter:9617
- job_name: 'speedtest'
  metrics_path: /probe
  params:
    script: [speedtest]
  static_configs:
  - targets:
    - NASip:9469
  scrape_interval: 30m
  scrape_timeout: 60s
- job_name: 'script_exporter'
  metrics_path: /metrics
  static_configs:
  - targets:
    - NASip:9469

You can copy paste this in the Docker-compose.yml file.

version: "3"
services:
  grafana:
    image: grafana/grafana:latest
    networks:
      - internal
      - traefik_public
    environment:
      - GF_SERVER_ROOT_URL=https://grafana.yourdomain.com
      - GF_METRICS_ENABLED=true
      - GF_INSTALL_PLUGINS=grafana-piechart-panel
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.grafana-rtr.entrypoints=https"
        - "traefik.http.routers.grafana-rtr.rule=Host(`grafana.yourdomain.com`)"
        - "[email protected]"
        - "traefik.http.routers.grafana-rtr.service=grafana-svc"
        - "traefik.http.services.grafana-svc.loadbalancer.server.port=3000"
    volumes:
      - /yourvolume/grafana:/var/lib/grafana
 
  prometheus:
    image: prom/prometheus:latest
    ports:
      - 9090:9090
    networks:
      - internal
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - --storage.tsdb.retention.time=7d
      - "--web.console.libraries=/usr/share/prometheus/console_libraries"
      - "--web.console.templates=/usr/share/prometheus/consoles"
    volumes:
      - /yourvolume/prometheus/data:/prometheus
      - /yourvolume/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - /yourvolume/prometheus/rules:/etc/prometheus/rules

  node_exporter:
    image: prom/node-exporter:latest
    networks:
     - internal
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro

  cadvisor:
    image: gcr.io/cadvisor/cadvisor
    ports:
    - 8084:8080
    command:
      - '--housekeeping_interval=30s'
      - '--max_housekeeping_interval=35s'
      - '--store_container_labels=untrue'
      - '--global_housekeeping_interval=30s'
      - '--Docker_only'
      - '--disable_root_cgroup_stats=untrue'
      - '--disable_metrics=percpu,process,sched,tcp,udp,diskIO,disk,community'      # enable only cpu, memory
      - '--allow_dynamic_housekeeping=true'
      - '--storage_duration=1m0s'
    networks:
      - internal
    volumes:
      - /var/run/Docker.sock:/var/run/Docker.sock:ro
      - /:/rootfs:ro
      - /var/run:/var/run
      - /sys:/sys:ro
      - /var/lib/Docker/:/var/lib/Docker:ro

  adguard_exporter:
    image: ebrianne/adguard-exporter:latest
    networks:
      - internal
    ports:
      - 9617:9617
    environment:
      - adguard_protocol=http
      - adguard_hostname=AdguardIP
      - adguard_username=username
      - "adguard_password=password
#      - adguard_port= #optional
      - interval=300s #5min
      - log_limit=1000

  speedtest:
    image: billimek/prometheus-speedtest-exporter:1.1.0
    networks:
      - internal
    ports:
      - 9469:9469

networks:
  traefik_public:
    external: true
  internal:
    driver: overlay
    ipam:
      config:
        - subnet: 172.16.84.0/24

Before you save this file, make sure to change ‘yourvolume’ to the correct location, so where you just created the folders and files.
At the Adguard part, change the hostname, username and password.
And at last change the domain name at the traefik labels.

Log in to your NAS using SSH and browse to the ‘volume’ / location where you just created the folders and files.
Now run the command: Docker-compose up
Then Docker will create all services and the folders will be stuffed with application data.

Now what?
It will not appear in Grafana by itself, we will have to create or download a dashboard for all the juicy graphics. For the sake of comfort I’m putting some dashboards ready here: (https://techsuited.com/dashboard.zip)

Start Grafana, in our case: https://grafana.yourdomain.com and press the + button at the left. Choose for Import and select the dashboard you want.

If you want to visualise Traefik data in a dashboard, you need to make a change in the Traefik static.yaml file. Add these lines:

metrics:
  prometheus:
    addServicesLabels: true

You will see something like this afterwards:

grafana-traefik

grafana-traefik

Source

Most Popular