After you have working monitoring solution based on Prometheus/Grafana the next step is to get alerts for cases which you want to be informed.

kube-prometheus-stack brings alertmanager component which allows to handle alerts which are generated by prometheus or any other clients. It can route, suppress, group, de-duplicate alerts, but configuration has to be done via config-files (via operator if it’s running in kubernetes) so it’s not easy to manage and amend “on the fly”.

Alternatively Grafana itself (from the same kube-prometheus-stack) can generate alerts and moreover it can use Telegram Bot to generate alerts.

So plan is the following:

  • Reinstall kube-prometheus-stack without alertmanager
  • Register Telegram bot
  • Create Telegram channel and add access there for bot which was just created
  • Configure Telegram as Notification channel in Grafana
  • Enable alerts for desired metrics in Grafana using Telegram as notification channel for it

Reinstall kube-prometheus-stack without alertmanager

Add to kube-prometheus-stack-values.yaml the following:

cat >> kube-prometheus-stack-values.yaml
  enabled: false

And then upgrade kube-prometheus-stack with helm and confirm no alertmanager-prometheus-kube-prometheus-alertmanager-0 pod exist anymore

helm upgrade -n monitoring --values kube-prometheus-stack-values.yaml prometheus prometheus-community/kube-prometheus-stack

Release "prometheus" has been upgraded. Happy Helming!
NAME: prometheus
LAST DEPLOYED: Sat Feb 20 16:57:59 2021
NAMESPACE: monitoring
STATUS: deployed
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace monitoring get pods -l "release=prometheus"

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.

# helm history prometheus -n monitoring
REVISION	UPDATED                 	STATUS    	CHART                        	APP VERSION	DESCRIPTION     
1       	Sun Jan 17 18:04:37 2021	superseded	kube-prometheus-stack-12.12.1	0.44.0     	Install complete
2       	Sun Jan 24 12:29:47 2021	superseded	kube-prometheus-stack-12.12.1	0.44.0     	Upgrade complete
3       	Sun Jan 24 12:34:26 2021	superseded	kube-prometheus-stack-12.12.1	0.44.0     	Rollback to 1   
4       	Sun Jan 24 12:46:41 2021	superseded	kube-prometheus-stack-12.12.1	0.44.0     	Upgrade complete
5       	Sat Feb 20 16:57:59 2021	deployed  	kube-prometheus-stack-12.12.1	0.44.0     	Upgrade complete

# kubectl get pod -n monitoring
NAME                                                      READY   STATUS    RESTARTS   AGE
prometheus-prometheus-node-exporter-sl48d                 1/1     Running   0          33d
snmp-exporter-prometheus-snmp-exporter-795487bf7f-mdrsk   1/1     Running   0          33d
arm-exporter-48rps                                        2/2     Running   0          33d
prometheus-kube-prometheus-operator-7c657497cb-wd47n      1/1     Running   1          33d
arm-exporter-mgvq4                                        2/2     Running   2          33d
prometheus-prometheus-node-exporter-khb6w                 1/1     Running   1          33d
prometheus-kube-state-metrics-7b765d9f74-thnlr            1/1     Running   0          28d
prometheus-grafana-7d697d59f4-bpcq7                       2/2     Running   0          27d
prometheus-prometheus-node-exporter-z8smt                 1/1     Running   3          33d
arm-exporter-rmmc8                                        2/2     Running   4          33d
prometheus-prometheus-kube-prometheus-prometheus-0        2/2     Running   28         31h

Register Telegram bot

It’s really 5 minutes task as per How do I create a bot?, the only hard part is to find the name for bot which is not used by anyone yet. After bot is created you have to have its Bot API Token in format like 1522404850:AAHg80RJMSQLBsLD47jAQq3qtotyRI3MNbk.

Create Telegram channel and add access there for bot which was just created

Create new Telegram private channel and add your bot as Administrator (bot can’t be added as usual subscriber). If you don’t like to provide full list of permissions to channel for bot, just leave the right to post the message - it will be enough. Few minutes after bot is added to channel go to url https://api.telegram.org/bot1522404850:AAHg80RJMSQLBsLD47jAQq3qtotyRI3MNbk/getUpdates and you should get reply with details about created channel. ` {“ok”:true,”result”:[{“update_id”:867972158, “channel_post”:{“message_id”:5,”sender_chat”:{“id”:-1001469798833,”title”:”Alerts”,”type”:”channel”},”chat”:{“id”:-1001469798833,”title”:”Alerts”,”type”:”channel”},”date”:1611653459,”text”:”ping”}}]} ` Find chat id (-1001469798833 in example above) - it will be needed either on next step.

Configure Telegram as Notification channel in Grafana

In Grafana go to Alerting -> Notification Channel -> New Channel, specify name for new channel, BOT API Token, Chat ID and hit save. Telegram Notification Channel in Grafana

Enable alerts for metrics

Select Edit for metric which you want to get alerts for, select Alert tab and specify Rule, Conditions and Error Handling details. In example below checks will be done each 60 seconds for 5 minutes and if it hits 80 (or above) alert will be generated. If no data will be got or no response will be Alert will be generated either. Alert example in Grafana If you have only one notification channel it will be used as default for all alerts, if you have many - don’t forget to specify which one you want to use for each alert.

All Grafana alerts are available on Alerting dashboard.