# Playground with Prometheus, Grafana, Loki and k6

In this blog, we will set up a VM with a testing playground. This playground will have a microservice, with Prometheus, Grafana and Loki for runtime metrics and log observability and k6 for generating a constant load. It can be used for measuring the capabilities of your microservice, or for benchmarking your tools.

# Context

I’ve been following Doctor Droid for a bit and have been in touch with Founders to try their product. I recently started [Zurich Lab](https://zurichlab.io/), a tech consultancy, to explore different opportunities - so I hit them up and it turns out they were looking for someone for a project (win-win!).

# Setup

Doctor Droid is an observability tool to auto-diagnose production issues with [engineer-created playbooks](https://drdroid.io/). To test out their tool, they needed a few sandbox environments, with one or 10s of microservices, with syntactic points of failure and some chaotic traffic and ways to monitor the environment. We decided to start with a single microservice, which can be expanded to multiple.

![Setting up Prometheus, Grafana and Promtail on goRubu service](https://lh7-us.googleusercontent.com/CzG4XBcMUXp6tTfXqtthpEnw7LF7hfOzc7oujx74Zn-oP2cMKbAwaaR83PqGixoF8aNO7zgQJx4h1Xs8btgDiPvGTRJkIwRuOZu_vPqNpbNh2M3kAJovDNuHc6LvrLKmi-LOVTM7Pd19X9TgXjbB8pU align="left")

Here’s how we went about doing it

* Step 1: Provision VM.
    
* Step 2: Setup the observability environment.
    
    * [Prometheus](https://prometheus.io/)
        
    * [Grafana](https://grafana.com/)
        
    * [Loki](https://grafana.com/oss/loki/)
        
* Step 3: Setup the microservice
    
* Step 4: Setup traffic simulation with [k6](https://k6.io/).
    
* Conclusion: Explore metrics in Grafana.
    

Below are the details for all the steps

## Step 1: Provision VM.

We have the provision of an EC2 instance, with 4GB RAM and any basic CPU available, in our case, it’s  Intel Xeon E5-2686 v4 (2) @ 2.299GHz, with Ubuntu 22. You can either use Multipass to have a local VM or provision a VM on by any cloud provider. Not recommended, but you can also use your own Ubuntu machine to do the same setup, which will make permanent changes to your system, which can be undone, but we have not covered that in this blog. :)

## Step 2: Setup the observability environment

First, we will set up each part individually.

### Prometheus

* Get the latest release of Prometheus, and unzip it.
    
    ```bash
    wget https://github.com/prometheus/prometheus/releases/download/v2.37.6/prometheus-2.37.6.linux-amd64.tar.gz
    
    tar xvfz prometheus-*.tar.gz
    
    rm prometheus-*.tar.gz
    ```
    
* Move the Prometheus binary to local/bin, and make etc and lib folder for Prometheus.
    
    ```bash
    cd prometheus-2.37.6.linux-amd64
    sudo mv prometheus promtool /usr/local/bin/
    
    sudo mkdir /etc/prometheus /var/lib/prometheus
    sudo mv prometheus.yml /etc/prometheus/prometheus.yml
    sudo mv consoles/ console_libraries/ /etc/prometheus/
    
    prometheus --version
    ```
    
* Add Prometheus to the user, and change permission of the folders.
    
    ```bash
    sudo useradd -rs /bin/false prometheus
    sudo chown -R prometheus: /etc/prometheus /var/lib/prometheus
    ```
    
* Add Prometheus to systemd, so it runs in the background.
    
    ```bash
    sudo "[Unit]
    	Description=Prometheus
    	Wants=network-online.target
    	After=network-online.target
    
    	[Service]
    	User=prometheus
    	Group=prometheus
    	Type=simple
    	Restart=on-failure
    	RestartSec=5s
    	ExecStart=/usr/local/bin/prometheus \
        	--config.file /etc/prometheus/prometheus.yml \
        	--storage.tsdb.path /var/lib/prometheus/ \
        	--web.console.templates=/etc/prometheus/consoles \
        	--web.console.libraries=/etc/prometheus/console_libraries \
        	--web.listen-address=0.0.0.0:9090 \
        	--web.enable-lifecycle \
        	--log.level=info
    
    	[Install]
    	WantedBy=multi-user.target" > /etc/systemd/system/prometheus.service
    
    sudo systemctl daemon-reload
    sudo systemctl enable prometheus
    sudo systemctl start prometheus
    sudo systemctl status prometheus
    ```
    
* Prometheus should be up and running now!
    

### Loki

Get the latest release of Loki, unzip it and create a symlink for binary to local/bin.

```bash
sudo mkdir /opt/loki | cd /opt/loki
sudo wget -qO /opt/loki/loki.gz "https://github.com/grafana/loki/releases/download/v2.9.5/loki-linux-amd64.zip"


sudo gunzip /opt/loki/loki.gz
sudo chmod a+x /opt/loki/loki


sudo ln -s /opt/loki/loki /usr/local/bin/loki
```

Get Loki configuration files.

```bash
sudo wget -qO /opt/loki/loki-local-config.yaml "https://raw.githubusercontent.com/grafana/loki/v2.9.5/cmd/loki/loki-local-config.yaml"
```

Add Loki service to systemd.

```bash
sudo "[Unit]
Description=Loki log aggregation system
After=network.target


[Service]
ExecStart=/opt/loki/loki -config.file=/opt/loki/loki-local-config.yaml
Restart=always


[Install]
WantedBy=multi-user.target" > /etc/systemd/system/loki.service


sudo systemctl daemon-reload
sudo systemctl enable loki
sudo systemctl start loki
sudo systemctl status loki
```

Get Promtail, is the agent, responsible for gathering logs and sending them to Loki. We will get the latest release, and create a symlink for binary to local/bin.

```bash
cd /opt/loki
sudo wget -qO /opt/loki/loki-local-config.yaml"https://github.com/grafana/loki/releases/download/v2.9.5/promtail-linux-amd64.zip"


sudo unzip promtail-linux-amd64.zip
sudo rm -r promtail-linux-amd64.zip
sudo chmod a+x promtail-linux-amd64

sudo ln -s /opt/loki/promtail-linux-amd64 /usr/local/bin/promtail
```

Add promtail configuration.

```bash
sudo "[Unit]
Description=Promtail service
After=network.target


[Service]
Type=simple
ExecStart=/usr/local/bin/promtail -config.file /opt/loki/promtail-config.yaml


[Install]
WantedBy=multi-user.target" > /etc/systemd/system/promtail.service

sudo systemctl daemon-reload
sudo systemctl enable promtail
sudo systemctl start promtail
sudo systemctl status promtail
```

Loki should be up now, and should start ingesting logs from Promtail!

### Grafana

Get the latest release of Grafana, and unzip it.

```bash
sudo apt-get install -y apt-transport-https software-properties-common
sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

sudo apt-get update
sudo apt-get install grafana -y
```

The above steps have added a Grafana Server service to the systems, but we need to enable it.

```bash
sudo apt-get update
sudo apt-get install grafana -y
sudo systemctl daemon-reload
sudo systemctl enable grafana-server.service
sudo systemctl start grafana-server
sudo systemctl status grafana-server
```

* Add Prometheus as a data source in Grafana.
    
    * Go to your public IP, and open port **3000.**
        
    * This will open the Grafana login page for you.
        
    * Use "admin" as both username and password.
        
        ![](https://lh7-us.googleusercontent.com/2bxjrMjLPIhc5NmLeX7ZUgqwN-Z7pWb0JgFFBM3cPl4U9qK3kdbv6yZjbT778Qsqg_YsHLAdGWNnEJzYAxSY6ILA6hmhHLsWMd65rg3stI_lwjtx59STu8XTKDmsft45lFLigbgDyEzZ_p2ta2O9018 align="left")
        
    * Use the datasource open there, and add Prometheus in it, by pointing to [http://localhost:9090](http://localhost:9090), and save.
        
        ![](https://lh7-us.googleusercontent.com/Tb8Ybh0tY7F73o7suZ4yyMOOJ_LDQQiTnwJgrjUzJ0Y_K2Gv5Pc67wC6lCD03qe2V4yV7yNWqYCKfjewZTaKux16c2h1I8TN_EYVYEXFKRpwwwFExs3BpuR633eGWiXSAWVsBk7N6_gRCuZq_eAbMRA align="left")
        
    * You can follow a detailed description of the same here([https://grafana.com/docs/grafana/latest/datasources/](https://grafana.com/docs/grafana/latest/datasources/)).
        
* Add Loki as a data source in Grafana.
    
    * It follows the same steps as above, but when selecting the data source, select Loki, and point to [http://localhost:3100.](http://localhost:3100)
        
* [Grafana shoul](http://localhost:3100)d be up and running now! It should also have access to both Loki and Prometheus data.
    

## Step 3: Setup the microservice

For our test, we have selected [GoRubu](https://github.com/rv404674/goRubu) repo, because it already has Prometheus integrated, and was small and simple enough to make changes.

We also made some changes in the service, like adding our custom instrumentation, parameterising the config files path, adding different points of failures, and random response time to APIs. However, we won’t cover that in this blog.

#### Runtime Environment

* We will first set up the runtime environment for this project. Which requires Golang, Mongo and Memcached. Let’s install and run them.
    
    ```bash
    sudo apt install golang-go memcached make gnupg curl -y 
    
    curl -fsSL https://www.mongodb.org/static/pgp/server-7.0.asc | \
    sudo gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg --dearmor
    
    echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/7.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-7.0.list
    
    sudo apt-get update
    sudo apt-get install -y mongodb-org
    sudo systemctl daemon-reload
    sudo systemctl enable mongod
    sudo usrmod -aG sudo mongodb	
    sudo systemctl start mongod
    ```
    
* Let’s clone the repo and run it. We will compile it, so we can run it later.
    
    ```bash
    git clone https://github.com/rv404674/goRubu.git
    cd goRubu
    go run .
    go build
    ```
    

#### Run in background

* Add service to the systemd, so it can auto-restart and run in the background.
    
    ```bash
    sudo "[Unit]
    Description=backend-service
    After=network.target
    StartLimitIntervalSec=0
    [Service]
    Type=simple
    Restart=always
    RestartSec=1
    User=ubuntu
    ExecStart=/home/ubuntu/goRubu
    
    [Install]
    WantedBy=multi-user.target" > /etc/systemd/system/rubu.service
    
    sudo systemctl daemon-reload
    sudo systemctl enable rubu
    sudo systemctl start rubu
    sudo systemctl status rubu
    ```
    

#### Metrics & Logs Target

* Service is running in the systemd, so the logs are auto-collected for it by Promtail!
    
* Let’s add the service to the scrape config of Prometheus. Add the following to */etc/prometheus/prometheus.yaml*
    
    ```bash
    - job_name: "robu"
      static_configs:
        - targets: ["localhost:8080"]
    ```
    

## Step 4: Setup traffic simulation with [k6](https://k6.io/).

Now that all the services are running, we are ready to install and set up k6, to generate periodic load we will set up a cron job for it.

### K6

Let’s install k6.

```bash
sudo gpg -k
sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69

echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list

sudo apt-get update
sudo apt-get install k6
```

#### Load generation script

Let’s write a script to generate the load. Below is the javascript code for it. Let’s name it load.js

```javascript
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
        stages: [
                { duration: '15s', target: 200},
                { duration: '15s', target: 500},
        ],
}

export default function () {
        const shorten_url = "http://localhost:8080/all/shorten_url"
        const redirect = "http://localhost:8080/all/redirect"
        const payload = JSON.stringify({
                Url: "https://google.com"
        });

        const params = {
                headers: {
                        'Content-Type': 'application/json',
                        'Authorization': 'Bearer TOKEN'
                }
        };

        const res = http.post(shorten_url, payload, params);
        const res2 = http.post(redirect, payload, params);
        check(res, {'status was 200': (r) => r.status == 200})
        check(res2, {'status was 200': (r) => r.status == 200})
}
```

Let’s run the above script. You can read more about k6 [here](https://ramantehlan.github.io/blog/post/2023/k6/).

```bash
k6 run --vus 10 --duration 30s -o experimental-prometheus-rw load.jsk6 run --vus 10 --duration 30s -o experimental-prometheus-rw load.jsk6 run --vus 10 --duration 30s -o experimental-prometheus-rw load.js
```

#### Cron Load Generation

We will use crontab to set up a cron job. Let’s install it.

```bash
sudo apt install crong
sudo systemctl enable cron 
sudo systemctl start cron
```

Let’s write a load script, which we will trigger every minute. Let’s name it generate\_[load.sh](http://load.sh). This will trigger the k6 command, with our load.js file. It generates the load for 30 minutes.

```bash
#!/bin/sh


k6 run --vus 5 --duration 30s -o experimental-prometheus-rw ~/load.js > k6_logs.log
```

Let’s add our generate\_[load.sh](http://load.sh) to crontab.

```bash
crontab -e


// Add following line in the end of the file
// It sets the cron time, and the file to run.
* * * * * /usr/bin/sh ~/PATH/generate_load.sh
```

That’s all, this will generate a periodic and consistent load to our service.

### Conclusion

All the required services are active and running, which you can find in the systemctl.

![](https://lh7-us.googleusercontent.com/sT1lB_u-5owkuKC0Q6tkTG_2XUS92LFsn_9zSCpn6DWsIquEB4-OmwzeyaT0qlCpjU97fSMWLdsIyapQ5vK0vdHKxQDnP4zeBNerIqHF1re498yKn7nHbVSQlB7gpydLsARA7l_eSGee0Uef5T_azqs align="left")

Here is how the traffic looks in the Grafana.

![](https://lh7-us.googleusercontent.com/BY14UelYwDsoUjP1dlFDVuRJuL6o2866oGKmDbjb3RUunzv8w-jNRYoEmOu0ZALPM-o5z-k2NYKeC6dK6yYwRrdq8bDBT2Lr2dAIneZhQknpEBmrDtHT1OT09ykFXHPtctfyZFx0bwqk91PJLHhpjOI align="left")

Here is how the logs look in the Grafana.

![](https://lh7-us.googleusercontent.com/8H0IC20PfaBPUNeI4PMGuaCwJ94slPyZLrOI3fEudloyCGcZfmagbAtT-IP9B6F2yRIZFdYYHunRyjqUXi3rnYodJZijTUJK7o6jRT1kndHiUJ8HCYI0AVRAAPvTbbzSqbUWsCaCFk9DTAzL0NyFKbE align="left")

With that, you have your playground running in the VM, feel free to tune the service for the scenario you want, or maybe try adding multiple services, to test multi-service dependency.

Thank you for reading this blog, do drop any feedback or questions you might have!

Happy engineering :)