In a previous blog post, I documented how to configure Zimbra’s DoSFilter in conjunction with a Failed Lockout Policy with the objective of blocking a bad actor’s IP address before locking out legitimate users from their own mailboxes. While this works well for protecting Zimbra’s mailbox service, DoSFilter does not protect Postfix’s Submission port: 587/tcp. In the absence of Two-Factor Authentication enabled on the account, bad actors can therefore try brute forcing a mailbox’s credentials on the submission port with no restrictions, hoping to be able to get full access to the mailbox.
This blog post shows you how to use fail2ban on your MTA servers, in conjunction with Ubuntu’s UFW (Uncomplicated Fire Wall) service, to protect the Zimbra Postfix submission port. Even with this new protection in place, along with DoSFilter, we still recommend deploying Two-Factor Authentication.
Most articles I have seen regarding Zimbra and fail2ban discuss using fail2ban to protect all of Zimbra’s public-facing ports, using iptables. That’s OK for a single Zimbra server, but when you have a multi-server hosting farm with separate MTA, Proxy, LDAP and Mailbox servers, and you are already running Zimbra’s fully supported DoSFilter, the only additional port protection needed is for Postfix’s Submission port.
Ubuntu out of the box deploys UFW, I already have a separate firewall system in front of Zimbra, and I’ve already deployed DoSFilter — so I only need fail2ban to work with UFW to block IP addresses that are trying to brute-force the Submission port.
How do you know you have a problem? You’ll see entries like these in /var/log/zimbra.log:
Mar 5 16:01:34 my postfix/submission/smtpd: warning: unknown[184.108.40.206]: SASL PLAIN authentication failed: authentication failure Mar 5 16:01:37 my postfix/submission/smtpd: warning: unknown[220.127.116.11]: SASL LOGIN authentication failed: authentication failure
You can do a whois on the IP and if it’s not an IP that should be trying to log in to your server, then you know for sure you have a problem.
Background and Initial Preparation Work:
Fail2ban has “filters”; files that contain regular expressions used to parse log files to identify bad actors trying to abuse the system. Regular expressions are not my forte, so I am grateful to Manuel Garbin at Studio Storti in Italy (a company related to ZeXtras) for coming up with the elegant regex we use below.
Fail2ban also has “jails”: a collection of commands for each named “filter” that define the actions a bad actor needs to take get banned, what fail2ban is going to do about it (call UFW to ban the bad actor’s IP), and for how long the bad actor’s IP address will be banned. Sometimes one file will contain many jails; sometimes individual jails will be defined in individual files.
The specific mechanics of an action to be taken are documented in a fail2ban “action” file.
UFW needs some tweaking as well, to allow for Zimbra’s public-facing services as well as inter-server communication in a multi-server environment. So let’s get started! (Remember that this process needs to be repeated on every one of your Zimbra servers running the MTA service.)
Please note that all commands below are executed as the root user.
If you haven’t done so already, install fail2ban and ufw:
apt-get update && apt-get install ufw fail2ban
To simplify the config files and future administration, we define a new “application” for UFW, called “zimbra-submission”:
The file should contain the following lines:
[zimbra-submission] title=Zimbra Postfix Submission Port description=For monitoring activity on port 587 ports=587/tcp
Check that UFW now knows about this new “application”:
root@zimbra:~# ufw app list | grep -i zimbra zimbra-submission root@zimbra:~#
Next, we need to configure UFW’s defaults. We therefore need to allow all outgoing traffic; traffic inbound to Zimbra’s public-facing ports (the rules below are for a single Zimbra server) and; all traffic within the LAN networks on which your Zimbra servers live (to allow for inter-Zimbra-server traffic). Note that if you allow POP3 traffic, or want to restrict Admin Console access to certain IPs, you’ll need to adjust the rules below accordingly.
ufw default allow outgoing ufw allow ssh ufw allow zimbra-submission ufw allow 25/tcp ufw allow 80/tcp ufw allow 443/tcp ufw allow 587/tcp ufw allow 993/tcp ufw allow 9071/tcp ufw allow from 10.7.57.0/24 ufw allow from 10.8.0.0/19 ufw enable ufw status numbered
Now that we are done configuring UFW, let’s configure Fail2ban:
First we need to configure the jail file:
The jail file should contain:
[zimbra-submission] enabled = true port = 587 filter = zimbra-submission logpath = /var/log/zimbra.log maxretry = 3 findtime = 3600 bantime = 36000 action = ufw
Essentially, we are going to allow an IP to fail to authenticate on the submission port three times within an hour, and then after the fourth failed try we will ban the IP for ten hours.
Next we need to configure the filter file:
The filter file should contain:
# Fail2Ban Zimbra Submission filter configuration file # # Authors: Manuel Garbin, Studio Storti, https://studiostorti.com/ and # L. Mark Stone, Mission Critical Email LLC, https://www.missioncriticalemail.com # 7 June 2020 # [Definition] # We only want to use fail2ban with Submission issues as we use DoSFilter to protect mailboxd # failregex = postfix\/submission\/smtpd\[\d+\]: warning: .*\[<HOST>\]: SASL \w+ authentication failed: authentication failure$ ignoreregex =
If you are already managing SSH access via a firewall, you may want to turn off SSH access checking in fail2ban; SSH is the only jail that is enabled by default (at least in Ubuntu Server 16,04 and 18.04).
Edit the “enabled” line in the file to read “false”:
[sshd] enabled = false
Now that we have the jail and filter files configured, we need to configure the “action” file. There is already a ufw.conf action file; we just need to make sure that three variables already in the file read as follows:
... actionban = [ -n "<application>" ] && app="app <application>" ufw insert <insertpos> <blocktype> from <ip> to <destination> $app actionunban = [ -n "<application>" ] && app="app <application>" ufw delete <blocktype> from <ip> to <destination> $app ... application = zimbra-submission ...
Now it’s time to restart fail2ban:
service fail2ban restart
Fail2ban Monitoring and How To Unban an IP Address:
Fail2ban ships with a command to check the status, i.e. which IPs have been banned, on a jail-by-jail basis. This script below however will poll all of the jails and save you some time if you are using fail2ban for other things already:
The file should contain:
#!/bin/bash JAILS=`fail2ban-client status | grep "Jail list" | sed -E 's/^[^:]+:[ \t]+//' | sed 's/,//g'` for JAIL in $JAILS do fail2ban-client status $JAIL done
Make the file executable:
chmod +x ~/fail2ban-allstatus.sh
In our submission-only case, since we have only one jail active, we can instead just run:
root@my:~# fail2ban-client status zimbra-submission fail2ban-client status zimbra-submission Status for the jail: zimbra-submission |- Filter | |- Currently failed: 4 | |- Total failed: 49 | `- File list: /var/log/zimbra.log `- Actions |- Currently banned: 1 |- Total banned: 1 `- Banned IP list: 18.104.22.168 root@my:~#
The statistics above tell us that 4 IP addresses have failed logins within the past hour; that 49 IPs have had failed logins, and that, at the moment, one IP is being blocked.
To unban an IP, just run:
fail2ban-client set zimbra-submission unbanip <ip_address>
Remember, fail2ban and UFW need to be installed on every one of your MTA servers! To protect mailboxd in a single or multi-server environment, use DoSFilter with a Failed Lockout Policy.
Hope that helps,
L. Mark Stone
Mission Critical Email LLC
7 June 2020
The information provided in this blog is intended for informational and educational purposes only. The views expressed herein are those of Mr. Stone personally. The contents of this site are not intended as advice for any purpose and are subject to change without notice. Mission Critical Email makes no warranties of any kind regarding the accuracy or completeness of any information on this site, and we make no representations regarding whether such information is up-to-date or applicable to any particular situation. All copyrights are reserved by Mr. Stone. Any portion of the material on this site may be used for personal or educational purposes provided appropriate attribution is given to Mr. Stone and this blog.