In a previous blog post, I documented how to configure Zimbra’s DoSFilter in conjunction with a Failed Lockout Policy with the objective of blocking a bad actor’s IP address before locking out legitimate users from their own mailboxes.  While this works well for protecting Zimbra’s mailbox service, DoSFilter does not protect Postfix’s Submission port: 587/tcp.  In the absence of Two-Factor Authentication enabled on the account, bad actors can therefore try brute forcing a mailbox’s credentials on the submission port with no restrictions, hoping to be able to get full access to the mailbox.

This blog post shows you how to use fail2ban on your MTA servers, in conjunction with Ubuntu’s UFW (Uncomplicated Fire Wall) service, to protect the Zimbra Postfix submission port.  Even with this new protection in place, along with DoSFilter, we still recommend deploying Two-Factor Authentication.

Challenges:
Most articles I have seen regarding Zimbra and fail2ban discuss using fail2ban to protect all of Zimbra’s public-facing ports, using iptables.  That’s OK for a single Zimbra server, but when you have a multi-server hosting farm with separate MTA, Proxy, LDAP and Mailbox servers, and you are already running Zimbra’s fully supported DoSFilter, the only additional port protection needed is for Postfix’s Submission port.

Ubuntu out of the box deploys UFW, I already have a separate firewall system in front of Zimbra, and I’ve already deployed DoSFilter — so I only need fail2ban to work with UFW to block IP addresses that are trying to brute-force the Submission port.

How do you know you have a problem?  You’ll see entries like these in /var/log/zimbra.log:

Mar 5 16:01:34 my postfix/submission/smtpd[10686]: warning: unknown[202.5.53.69]: SASL PLAIN authentication failed: authentication failure
Mar 5 16:01:37 my postfix/submission/smtpd[10686]: warning: unknown[202.5.53.69]: SASL LOGIN authentication failed: authentication failure

You can do a whois on the IP and if it’s not an IP that should be trying to log in to your server, then you know for sure you have a problem.

Background and Initial Preparation Work:
Fail2ban has “filters”; files that contain regular expressions used to parse log files to identify bad actors trying to abuse the system.  Regular expressions are not my forte, so I am grateful to Manuel Garbin at Studio Storti in Italy (a company related to ZeXtras) for coming up with the elegant regex we use below.

Fail2ban also has “jails”: a collection of commands for each named “filter” that define the actions a bad actor needs to take get banned, what fail2ban is going to do about it (call UFW to ban the bad actor’s IP), and for how long the bad actor’s IP address will be banned. Sometimes one file will contain many jails; sometimes individual jails will be defined in individual files.

The specific mechanics of an action to be taken are documented in a fail2ban “action” file.

UFW needs some tweaking as well, to allow for Zimbra’s public-facing services as well as inter-server communication in a multi-server environment.  So let’s get started!  (Remember that this process needs to be repeated on every one of your Zimbra servers running the MTA service.)

Please note that all commands below are executed as the root user.

If you haven’t done so already, install fail2ban and ufw:

apt-get update && apt-get install ufw fail2ban

Configure UFW:
To simplify the config files and future administration, we define a new “application” for UFW, called “zimbra-submission”:

nano /etc/ufw/applications.d/zimbra-submission

The file should contain the following lines:

[zimbra-submission]
title=Zimbra Postfix Submission Port
description=For monitoring activity on port 587
ports=587/tcp

Check that UFW now knows about this new “application”:

root@zimbra:~# ufw app list | grep -i zimbra
zimbra-submission
root@zimbra:~#

Next, we need to configure UFW’s defaults.  We therefore need to allow all outgoing traffic; traffic inbound to Zimbra’s public-facing ports (the rules below are for a single Zimbra server) and; all traffic within the LAN networks on which your Zimbra servers live (to allow for inter-Zimbra-server traffic).  Note that if you allow POP3 traffic, or want to restrict Admin Console access to certain IPs, you’ll need to adjust the rules below accordingly.

ufw default allow outgoing
ufw allow ssh
ufw allow zimbra-submission
ufw allow 25/tcp
ufw allow 80/tcp
ufw allow 443/tcp
ufw allow 587/tcp
ufw allow 993/tcp
ufw allow 9071/tcp
ufw allow from 10.7.57.0/24
ufw allow from 10.8.0.0/19
ufw enable
ufw status numbered

Now that we are done configuring UFW, let’s configure Fail2ban:

Configure Fail2Ban:
First we need to configure the jail file:

nano /etc/fail2ban/jail.d/zimbra-submission.local

The jail file should contain:

[zimbra-submission]
enabled = true
port = 587
filter = zimbra-submission
logpath = /var/log/zimbra.log
maxretry = 3
findtime = 3600
bantime = 36000
action = ufw

Essentially, we are going to allow an IP to fail to authenticate on the submission port three times within an hour, and then after the fourth failed try we will ban the IP for ten hours.

Next we need to configure the filter file:

nano /etc/fail2ban/filter.d/zimbra-submission.conf

The filter file should contain:

# Fail2Ban Zimbra Submission filter configuration file
#
# Authors: Manuel Garbin, Studio Storti, https://studiostorti.com/ and
# L. Mark Stone, Mission Critical Email LLC, https://www.missioncriticalemail.com
# 7 June 2020
#
[Definition]
# We only want to use fail2ban with Submission issues as we use DoSFilter to protect mailboxd
#
failregex = postfix\/submission\/smtpd\[\d+\]: warning: .*\[<HOST>\]: SASL \w+ authentication failed: authentication failure$
ignoreregex =

If you are already managing SSH access via a firewall, you may want to turn off SSH access checking in fail2ban; SSH is the only jail that is enabled by default (at least in Ubuntu Server 16,04 and 18.04).

nano /etc/fail2ban/jail.d/defaults-debian.conf

Edit the “enabled” line in the file to read “false”:

[sshd]
enabled = false

Now that we have the jail and filter files configured, we need to configure the “action” file. There is already a ufw.conf action file; we just need to make sure that three variables already in the file read as follows:

...
actionban = [ -n "<application>" ] && app="app <application>" ufw insert <insertpos> <blocktype> from <ip> to <destination> $app
actionunban = [ -n "<application>" ] && app="app <application>" ufw delete <blocktype> from <ip> to <destination> $app
...
application = zimbra-submission
...

Now it’s time to restart fail2ban:

service fail2ban restart

Fail2ban Monitoring and How To Unban an IP Address:
Fail2ban ships with a command to check the status, i.e. which IPs have been banned, on a jail-by-jail basis.  This script below however will poll all of the jails and save you some time if you are using fail2ban for other things already:

nano ~/fail2ban-allstatus.sh

The file should contain:

#!/bin/bash
JAILS=`fail2ban-client status | grep "Jail list" | sed -E 's/^[^:]+:[ \t]+//' | sed 's/,//g'`
for JAIL in $JAILS
do
fail2ban-client status $JAIL
done

Make the file executable:

chmod +x ~/fail2ban-allstatus.sh

In our submission-only case, since we have only one jail active, we can instead just run:

root@my:~# fail2ban-client status zimbra-submission
fail2ban-client status zimbra-submission
Status for the jail: zimbra-submission
|- Filter
| |- Currently failed: 4
| |- Total failed: 49
| `- File list: /var/log/zimbra.log
`- Actions
  |- Currently banned: 1
  |- Total banned: 1
  `- Banned IP list: 89.111.33.223
root@my:~#

The statistics above tell us that 4 IP addresses have failed logins within the past hour; that 49 IPs have had failed logins, and that, at the moment, one IP is being blocked.

To unban an IP, just run:

fail2ban-client set zimbra-submission unbanip <ip_address>

 

Remember, fail2ban and UFW need to be installed on every one of your MTA servers!  To protect mailboxd in a single or multi-server environment, use DoSFilter with a Failed Lockout Policy.

Hope that helps,
L. Mark Stone
Mission Critical Email LLC
7 June 2020

The information provided in this blog is intended for informational and educational purposes only. The views expressed herein are those of Mr. Stone personally. The contents of this site are not intended as advice for any purpose and are subject to change without notice. Mission Critical Email makes no warranties of any kind regarding the accuracy or completeness of any information on this site, and we make no representations regarding whether such information is up-to-date or applicable to any particular situation. All copyrights are reserved by Mr. Stone. Any portion of the material on this site may be used for personal or educational purposes provided appropriate attribution is given to Mr. Stone and this blog.