I most recently moved my mails from the great guys at Uberspace to an own server, because I like owning things and running them by myself. This is no problem at all with hosting my own sync server for contacts, calendar and photos, but it is a problem with mail, because mail is critical infrastructure to me. So, although I like the thought of having my mail on a machine I own, I do not like the thought of my mailserver going broken without me noticing it.

As I run a bunch of online services for myself and for others, I already had an Icinga instance running to notify me if one of my services happens to go down. This Icinga instance already was configured to check whether my mailserver correctly responds on the SMTP and IMAP ports but this is not enough. This does not fire alarm, if there is a configuration mistake anywhere in the local mail transport (postfix), the spam filter (SpamAssassin), other filters (sieve), or the authentication system.

The only solution to this problem is to set up End to End monitoring for email. Most surprisingly, I failed to find a ready-to-use Icinga plugin for doing this. So this is what I came up with by myself.

For testing all of the steps in the flow of an email, you need a second mailserver you can regularly exchange mails with. You can use any mail account at one of the big mail hosters, but there is a chance they'll consider your testing mails as spam, so I just paired up with Ben, a friend of mine, to monitor each other's mailserver.

The basic idea is very simple: You have your mailserver (A) and your friend's mailserver (B) and on every run, the Icinga check script connects of the SMTP servers and sends one email each from A to B and from B to A, containing a timestamp. On the next run, it connects to both servers via IMAP, checks whether both mails arrived, deletes them and sends out new ones once again.

The check script is written in Python and requires at least Python 3.2 and the Python nagiosplugin. On Debian stable, just do a quick

apt-get install python3 python3-pip
pip-3.2 install nagiosplugin

Then, grab my script from Github:

wget https://raw.githubusercontent.com/raphaelm/monitoring/master/mail/check_mail_twoway
chmod +x check_mail_twoway
mv check_mail_twoway /usr/lib/nagios/plugins/

You can execute ./check_mail_twoway -h to get an overview of the command-line options and play around with them.

My Icinga configuration looks as follows:

define command {
    command_name    check_mail_twoway
    command_line    /usr/lib/nagios/plugins/check_mail_twoway $ARG1$ 
define service {
    use                 generic-service
    host_name           dannyl
    service_description Mail E2E
    check_command       check_mail_twoway!-s1 serverA \
        -a1 userA@serverA -i1 serverA -u1 userA \
        -p1 passwordA -s2 serverB -i2 serverB \
        -a2 userB@serverB -u2 userB -p2 passwordB

On the first run, the script will always report a failure, as it can not find any emails (as none have been sent out yet). I do not think there is a simple and perfect way to prevent this, please correct me if I'm wrong.

I'd love to hear if this works for you or to see any pull requests!

(And of course, my Icinga does not try to notify me via email if my email fails, but via SMS.)