Introduction

In this article, I will show you how I used Gotify to provide a very reliable information service for my IT team.

The messages are sent from our Icinga2 instance to the Gotify service. We are using the iGotify App on our mobile phones.

Whats needed?

1. Icinga2 Monitoring Instance https://www.icinga.com
2. Icinga Director to configure the Notification Rules.
3. Gotify Server https://gotify.net
4. igotify Notification Assistent https://github.com/androidseb25/iGotify-Notification-Assistent

Icinga Setup

First of all, we need an Icinga that monitors something. If the monitored service is in a state that we do not want, we can then trigger a notification accordingly.

Icinga Director is ideal for this purpose, as it is easier to create rules using the dedicated web interface. However, before we start with the rules, we need to create two commands of the type Notification. for doing this we must jump into a shell on the Icinga server:

cd /etc/icinga2/
curl -fssl https://willifix.net/blog/howto-configure-icinga2-notifications-with-gotify/service-by-gotify.txt > ./scripts/service-by-gotify.sh && chmod o+x ./scripts/service-by-gotify.sh
curl -fssl https://willifix.net/blog/howto-configure-icinga2-notifications-with-gotify/host-by-gotify.txt > ./scripts/host-by-gotify.sh && chmod o+x ./scripts/host-by-gotify.sh

you should now have 2 more files in /etc/icinga2/scripts please change the highlighted lines to your gotify server FQDN (line 46 in service-by-gotify.sh and line 30 in host-by-gotify.sh)

#!/bin/sh
# Notificate service events to gotify created by ♞ Raffael.Willems@im-c.de

# Environment variables
NOTIFICATIONTYPE=$ICINGA_NOTIFICATIONTYPE
HOSTDISPLAYNAME=$ICINGA_HOSTALIAS
HOSTSTATE=$ICINGA_HOSTSTATE
GOTIFY_APPTOKEN=$ICINGA_GOTIFYAPP

SERVICENAME=$ICINGA_SERVICENAME
SERVICESTATE=$ICINGA_SERVICESTATE
SERVICEOUTPUT=$ICINGA_SERVICEOUTPUT
NOTIFICATIONC=$ICINGA_NOTIFICATIONC

case $SERVICESTATE in
     WARNING)
          ICON="\ud83d\udd14"
          TITLE="PROBLEM"
          PRIORITY=5
          ;;
     CRITICAL)
          ICON="\u274c"
          TITLE="PROBLEM"
          PRIORITY=8
          ;;
     OK)
          ICON="\u2705"
          TITLE="RECOVERY"
          PRIORITY=3
          ;;
     *)
          ICON="\ud83d\udd14"
          TITLE="Notification"
          PRIORITY=5
          ;;
esac

if [ -n "$NOTIFICATIONC" ]; then
    NOTI="\u2139 ${NOTIFICATIONC}"
fi
# Build Message
MESSAGE="$ICON Service $SERVICENAME on Host $HOSTDISPLAYNAME is in state $SERVICESTATE : $(echo -n "$SERVICEOUTPUT" | tr -d '\n' | tr -d '"')"
curl    --silent \
    --show-error \
    -k \
    -X POST https://gotify.scn.imc/message \
    -H "Content-type: application/json; charset=utf-8" \
    -H "Authorization: Bearer $GOTIFY_APPTOKEN" \
    -d "{\"title\": \"$TITLE\", \"message\":\"$MESSAGE \n\n $NOTI\", \"priority\":$PRIORITY}" >/tmp/sbgout
echo "\n\n" >>/tmp/sbgout
echo $MESSAGE >>/tmp/sbgout

#MESSAGE="$ICON Service $SERVICENAME on Host $HOSTDISPLAYNAME is in state $SERVICESTATE : $SERVICEOUTPUT"
#curl -k "https://gotify.scn.imc/message?token=$GOTIFY_APPTOKEN" -F "title=$TITLE" -F "message=$MESSAGE" -F "priority=$PRIORITY" >/tmp/erg

#!/bin/sh
# Notificate Host events to gotify created by ♞ Raffael.Willems@im-c.de
# Environment variables
NOTIFICATIONTYPE=$ICINGA_NOTIFICATIONTYPE
HOSTDISPLAYNAME=$ICINGA_HOSTALIAS
HOSTSTATE=$ICINGA_HOSTSTATE
GOTIFY_APPTOKEN=$ICINGA_GOTIFYAPP

case $ICINGA_NOTIFICATIONTYPE in
     PROBLEM)
          ICON="\u274c"
          PRIORITY=8
          ;;
     RECOVERY)
          ICON="\u2705"
          PRIORITY=3
          ;;
     *)
          ICON="\ud83d\udd14"
          PRIORITY=5
          ;;
esac

# Build Message
MESSAGE="$ICON Host $HOSTDISPLAYNAME is $HOSTSTATE!"

curl --silent \
     --show-error \
     -k \
     -X POST https://gotify.scn.imc/message \
     -H "Content-type: application/json; charset=utf-8" \
     -H "Authorization: Bearer $GOTIFY_APPTOKEN" \
     -d '{"title": "'"$NOTIFICATIONTYPE"'", "message":"'"$MESSAGE"'", "priority":'"$PRIORITY"'}' >/tmp/erg_host

next step is to create a gotify.conf in /etc/icinga2/conf.d/ Folder which contains the Definition of our Notification Commands:

object NotificationCommand "service-by-gotify" {
  command = [ SysconfDir + "/icinga2/scripts/service-by-gotify.sh" ]
  env = {
    "ICINGA_GOTIFYAPP" = "$gotify_app$"
    "ICINGA_NOTIFICATIONTYPE" = "$notification.type$"
    "ICINGA_HOSTNAME" = "$host.name$"
    "ICINGA_HOSTALIAS" = "$host.display_name$"
    "ICINGA_HOSTSTATE" = "$host.state$"
    "ICINGA_HOSTOUTPUT" = "$host.output$"
    "ICINGA_SERVICENAME" = "$service.name$"
    "ICINGA_SERVICESTATE" = "$service.state$"
    "ICINGA_SERVICEOUTPUT" = "$service.output$"
    "ICINGA_NOTIFICATIONC" = "$notification.comment$"

  }
}
object NotificationCommand "gotify-host-notification" {
  command = [ SysconfDir + "/icinga2/scripts/host-by-gotify.sh" ]
  env = {
    "ICINGA_GOTIFYAPP" = "$gotify_app$"
    "ICINGA_NOTIFICATIONTYPE" = "$notification.type$"
    "ICINGA_HOSTNAME" = "$host.name$"
    "ICINGA_HOSTALIAS" = "$host.display_name$"
    "ICINGA_HOSTSTATE" = "$host.state$"
    "ICINGA_HOSTOUTPUT" = "$host.output$"
    "ICINGA_NOTIFICATIONC" = "$notification.comment$"
  }
}

reload the icinga config or restart the icinga2 daemon.

Gotify Setup (docker compose)

Now let's take a look at the gotify container. We need different Applications for the notifications, which we have to create accordingly in the gotify webinterface. here is the docker-compose yaml for creating a gotify instance with the needed iGotify-Notification-Assistent also:

services:
  gotify:
    image: gotify/server
    labels:
        - "traefik.enable=true"
        - "traefik.http.routers.gotify.rule=(Host(`gotify.scn.imc`) || Host(`{yourserverurl}`))"
        - "traefik.http.routers.gotify.entrypoints=websecure"
        - "traefik.http.services.gotify.loadbalancer.server.port=80"
        - "traefik.http.routers.gotify.service=gotify"
        - "traefik.http.routers.gotify.tls=true"
    environment:
      - GOTIFY_DEFAULTUSER_PASS={yourpass}
      - GOTIFY_SERVER_URL="{yourserverurl}"
    volumes:
      - "/srv/gotify/gotify_data:/app/data"
    networks:
        public:
  igotify-notification: # (iGotify-Notification-Assistent)
    image: ghcr.io/androidseb25/igotify-notification-assist:latest
    pull_policy: always
    environment:
      - GOTIFY_CLIENT_TOKEN=" {yourclienttoken}"  # create a client in gotify an add here the client token
      - GOTIFY_SERVERS="{yourserverurl}"
    restart: always
    volumes:
      - /srv/gotify/igotify-notification-data:/app/data
    networks:
        public:
    labels:
        - "traefik.enable=true"
        - "traefik.http.routers.gotify-api.rule=(Host(`api.gotify.scn.imc`) || Host(`api.{yourserverurl}`))"
        - "traefik.http.routers.gotify-api.entrypoints=websecure"
        - "traefik.http.services.gotify-api.loadbalancer.server.port=8080"
        - "traefik.http.routers.gotify-api.service=gotify-api"
        - "traefik.http.routers.gotify-api.tls=true"
        - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto = https"
networks:
    public:
        name: "public"
        external: true

Gotify should now be available. Please login to the webgui of gotify and create your desired Applications.

´´

Director Notification rulesetup

The Rule Setup is done in Icingaweb2 with a installed Icinga Director Module. Navigate to Director -> Commands -> External Commands and check that the created Notification Commands service-by-gotify and gotify-host-notification are recognized.

next step is to define a new Data Field of type Datalist and fill the list with all the created Applications and there corresponding Tokens.

1. Navigate to Icinga Director -> Provide Data Lists
2. Add a new Datalist
3. put the tokens in the key field and the Names in the Label field of that new list.

4. Next icinga Director -> Define Data Fields

now we can create our first Notification Template. The Template is needed to set the Application that fits the triggered Notification. the following screenshot shows the Host Notification Template. you must add more templates for each Application you want to filter later.

the Service Notification is as follows:

5. Director -> Users.

First Notification Rule

Now we can create the first Notification Rule. The Notification Rules are a powerful way to define when a Message should popup in Gotify. Here you can see the Host Notification.

The Setup of gotify is done and you can use third party apps as linked above for getting the notifactions on Apple or android phones.