In a previous blog post, I documented how to configure Zimbra’s DoSFilter in conjunction with a Failed Lockout Policy with the objective of blocking a bad actor’s IP address before locking out legitimate users from their own mailboxes. While this works well for protecting Zimbra’s mailbox service, DoSFilter does not protect Postfix’s Submission port: 587/tcp. In the absence of Two-Factor Authentication enabled on the account, bad actors can therefore try brute forcing a mailbox’s credentials on the submission port with no restrictions, hoping to be able to get full access to the mailbox.
This blog post shows you how to use fail2ban on your MTA servers, in conjunction with Ubuntu’s UFW (Uncomplicated Fire Wall) service, to protect the Zimbra Postfix submission port. Even with this new protection in place, along with DoSFilter, we still recommend deploying Two-Factor Authentication.
Most articles I have seen regarding Zimbra and fail2ban discuss using fail2ban to protect all of Zimbra’s public-facing ports, using iptables. That’s OK for a single Zimbra server, but when you have a multi-server hosting farm with separate MTA, Proxy, LDAP and Mailbox servers, and you are already running Zimbra’s fully supported DoSFilter, the only additional port protection needed is for Postfix’s Submission port.
Ubuntu out of the box deploys UFW, I already have a separate firewall system in front of Zimbra, and I’ve already deployed DoSFilter — so I only need fail2ban to work with UFW to block IP addresses that are trying to brute-force the Submission port.
How do you know you have a problem? You’ll see entries like these in /var/log/zimbra.log:
Mar 5 16:01:34 my postfix/submission/smtpd: warning: unknown[18.104.22.168]: SASL PLAIN authentication failed: authentication failure Mar 5 16:01:37 my postfix/submission/smtpd: warning: unknown[22.214.171.124]: SASL LOGIN authentication failed: authentication failure
You can do a whois on the IP and if it’s not an IP that should be trying to log in to your server, then you know for sure you have a problem.
Background and Initial Preparation Work:
Fail2ban has “filters”; files that contain regular expressions used to parse log files to identify bad actors trying to abuse the system. Regular expressions are not my forte, so I am grateful to Manuel Garbin at Studio Storti in Italy (a company related to ZeXtras) for coming up with the elegant regex we use below.
Fail2ban also has “jails”: a collection of commands for each named “filter” that define the actions a bad actor needs to take get banned, what fail2ban is going to do about it (call UFW to ban the bad actor’s IP), and for how long the bad actor’s IP address will be banned. Sometimes one file will contain many jails; sometimes individual jails will be defined in individual files.
The specific mechanics of an action to be taken are documented in a fail2ban “action” file.
UFW needs some tweaking as well, to allow for Zimbra’s public-facing services as well as inter-server communication in a multi-server environment. So let’s get started! (Remember that this process needs to be repeated on every one of your Zimbra servers running the MTA service.)
Please note that all commands below are executed as the root user.
If you haven’t done so already, install fail2ban and ufw:
apt-get update && apt-get install ufw fail2ban
To simplify the config files and future administration, we define a new “application” for UFW, called “zimbra-submission”:
The file should contain the following lines:
[zimbra-submission] title=Zimbra Postfix Submission Port description=For monitoring activity on port 587 ports=587/tcp
Check that UFW now knows about this new “application”:
root@zimbra:~# ufw app list | grep -i zimbra zimbra-submission root@zimbra:~#
Next, we need to configure UFW’s defaults. We therefore need to allow all outgoing traffic; traffic inbound to Zimbra’s public-facing ports (the rules below are for a single Zimbra server) and; all traffic within the LAN networks on which your Zimbra servers live (to allow for inter-Zimbra-server traffic). Note that if you allow POP3 traffic, or want to restrict Admin Console access to certain IPs, you’ll need to adjust the rules below accordingly.
ufw default allow outgoing ufw allow ssh ufw allow zimbra-submission ufw allow 25/tcp ufw allow 80/tcp ufw allow 443/tcp ufw allow 587/tcp ufw allow 993/tcp ufw allow 9071/tcp ufw allow from 10.7.57.0/24 ufw allow from 10.8.0.0/19 ufw allow 123/udp ufw enable ufw status numbered
Lastly, UFW can block what it considers to be invalid packets but which are not, so we need to fix this:
Comment out these two lines:
# drop INVALID packets (logs these in loglevel medium and higher) # -A ufw-before-input -m conntrack --ctstate INVALID -j ufw-logging-deny # -A ufw-before-input -m conntrack --ctstate INVALID -j DROP
Save the file and then restart UFW:
service ufw restart && service ufw status
Now that we are done configuring UFW, let’s configure Fail2ban:
First we need to configure the jail file:
The jail file should contain:
[zimbra-submission] enabled = true port = 587 filter = zimbra-submission logpath = /var/log/zimbra.log maxretry = 3 findtime = 3600 bantime = 36000 action = ufw
Essentially, we are going to allow an IP to fail to authenticate on the submission port three times within an hour, and then after the fourth failed try we will ban the IP for ten hours (feel free to ban the IP for as long as you wish).
Next we need to configure the filter file:
The filter file should contain:
# Fail2Ban Zimbra Submission filter configuration file # # Authors: Manuel Garbin, Studio Storti, https://studiostorti.com/ and # L. Mark Stone, Mission Critical Email LLC, http://www.missioncriticalemail.com # 7 June 2020 # [Definition] # We only want to use fail2ban with Submission issues as we use DoSFilter to protect mailboxd # failregex = postfix\/submission\/smtpd\[\d+\]: warning: .*\[<HOST>\]: SASL \w+ authentication failed: authentication failure$ ignoreregex =
If you are already managing SSH access via a firewall, you may want to turn off SSH access checking in fail2ban; SSH is the only jail that is enabled by default (at least in Ubuntu Server 16,04 and 18.04).
Edit the “enabled” line in the file to read “false”:
[sshd] enabled = false
Now that we have the jail and filter files configured, we need to configure the “action” file. There is already a ufw.conf action file; we just need to make sure that three variables already in the file read as follows:
... actionban = [ -n "<application>" ] && app="app <application>" ufw insert <insertpos> <blocktype> from <ip> to <destination> $app actionunban = [ -n "<application>" ] && app="app <application>" ufw delete <blocktype> from <ip> to <destination> $app ... application = zimbra-submission ...
Now it’s time to restart fail2ban:
service fail2ban restart
Fail2ban Monitoring and How To Unban an IP Address:
Fail2ban ships with a command to check the status, i.e. which IPs have been banned, on a jail-by-jail basis. This script below however will poll all of the jails and save you some time if you are using fail2ban for other things already:
The file should contain:
#!/bin/bash JAILS=`fail2ban-client status | grep "Jail list" | sed -E 's/^[^:]+:[ \t]+//' | sed 's/,//g'` for JAIL in $JAILS do fail2ban-client status $JAIL done
Make the file executable:
chmod +x ~/fail2ban-allstatus.sh
In our submission-only case, since we have only one jail active, we can instead just run:
root@my:~# fail2ban-client status zimbra-submission fail2ban-client status zimbra-submission Status for the jail: zimbra-submission |- Filter | |- Currently failed: 4 | |- Total failed: 49 | `- File list: /var/log/zimbra.log `- Actions |- Currently banned: 1 |- Total banned: 1 `- Banned IP list: 126.96.36.199 root@my:~#
The statistics above tell us that 4 IP addresses have failed logins within the past hour; that 49 IPs have had failed logins, and that, at the moment, one IP is being blocked.
To unban an IP, just run:
fail2ban-client set zimbra-submission unbanip <ip_address>
October 2020 – Advanced Topic: Spotting and Dealing With Distributed Attacks:
Relying on DoSFilter and Fail2Ban alone isn’t enough when it comes to bad actors performing subtle probes from multiple IP addresses. This is why it’s really important to look at Zimbra’s Daily Mail Report every single day. Specifically, you want to look at the “Warnings” section towards the bottom, in the “smtpd” subsection. This is where all of the port 465/587 SASL LOGIN failures will be listed.
Bad actors who control an IP space will carefully try just a few brute force login attempts from a single IP address, and then move on to a different IP address in the same network that they control. It will look something like this in the Daily Mail Report:
Warnings -------- smtpd (total: 326) <snip> 2 unknown[188.8.131.52]: SASL LOGIN authentication failed: auth... 2 unknown[184.108.40.206]: SASL LOGIN authentication failed: auth... 2 unknown[220.127.116.11]: SASL LOGIN authentication failed: auth... 1 unknown[18.104.22.168]: SASL LOGIN authentication failed: auth... 1 unknown[22.214.171.124]: SASL LOGIN authentication failed: auth... 1 unknown[126.96.36.199]: SASL LOGIN authentication failed: auth... 1 unknown[188.8.131.52]: SASL LOGIN authentication failed: auth... 1 unknown[184.108.40.206]: SASL LOGIN authentication failed: auth... 1 unknown[220.127.116.11]: SASL LOGIN authentication failed: auth... <snip>
In such cases, what we do next is first, to browse to whois.com, plug in one of the IP addresses and see who owns the IP space. If it makes sense to send an email to the abuse contact, you should do that, and send them the snippet from the Daily Mail Report as proof.
If the outcome is not satisfactory, then what we do next is use a ufw command to ban the entire IP block outside of fail2ban’s control. Remember above, when you first set up fail2ban, one of the early steps was to run ufw commands to open up all of your Zimbra public-facing ports? We’ll do the same manual ufw programming thing here (except this time to block the bad actor’s network instead), like so:
First, run the following command:
ufw status numbered
…and get the number of the first ALLOW IN rule. Let’s say that number is “283”, for example to allow SSH inbound. We want to insert a REJECT IN rule just before the first ALLOW IN rule, and if we insert this new REJECT IN command with the same number as the first ALLOW IN rule, the insert function will just push the first ALLOW IN rule down the stack one position, like so for our example network above, where whois.com told us this was a /24 network:
ufw insert 283 reject from 18.104.22.168/24 to any
If/When the hosting provider who controls the IP space is able to rectify the situation, you can just remove the network from ufw’s ruleset. To do so, first run “ufw status numbered” again to find the current number of the REJECT IN rule you want to delete. The rule number will have changed given the fullness of time, because fail2ban will have added and deleted rules in between the time you added the REJECT IN rule manually and the time when you want to remove it. So let’s say the current number is “272” for the network we blocked above (i.e. fail2ban has deleted more expired REJECT IN rules than it has added), then you would just run:
root@mta3:~# ufw delete 272 Deleting: reject from 22.214.171.124/24 Proceed with operation (y|n)?
As you can see, ufw gives you a chance to confirm if this is really the rule you want to delete, and if so, just go ahead.
Remember, fail2ban and UFW need to be installed on every one of your MTA servers! To protect mailboxd in a single or multi-server environment, use DoSFilter with a Failed Lockout Policy.
Hope that helps,
L. Mark Stone
Mission Critical Email LLC
7 June 2020 – UPDATED 28 October 2020
The information provided in this blog is intended for informational and educational purposes only. The views expressed herein are those of Mr. Stone personally. The contents of this site are not intended as advice for any purpose and are subject to change without notice. Mission Critical Email makes no warranties of any kind regarding the accuracy or completeness of any information on this site, and we make no representations regarding whether such information is up-to-date or applicable to any particular situation. All copyrights are reserved by Mr. Stone. Any portion of the material on this site may be used for personal or educational purposes provided appropriate attribution is given to Mr. Stone and this blog.