At some point, you will suffer a Zimbra account breach and an email flood. This blog post will describe how to handle this situation.
Do I Have A Zimbra Account Breach?
Once a Zimbra account breach happens, if you are monitoring your mail queues, you will see the Deferred queue increase. Users will start to complain that they are not receiving emails. Single Zimbra servers (or the MTA servers in a multi-server farm) may experience 100% CPU utilization. Fortunately, Zimbra’s Admin Console and a few commands in an SSH session will quickly stop the bad actor in their tracks and return Zimbra to normal service.
Stopping The Breach
First, navigate to the Admin Console > Monitor > Mail Queues > Server > Sender address column. You are now looking at the Deferred queue, and the top-most Sender (or Senders) will be the account which has been compromised. Right-click on the user, and select “Hold” from the popup. This will move all of the email this compromised sender is trying to send out of the Deferred queue and into the Hold queue. It’s very fast, even when Postfix has thousands of emails to move.
Next, use the Admin Console to set that user’s account status to Locked. This works for new sessions, but not currently authenticated SMTP-Auth sessions (which is the most common account breach vector we see).
You will need now to open an SSH session to the MTA server(s), become the zimbra user and do zmmtactl restart which will terminate all running SMTP-Auth sessions.
Zimbra will still be trying to send the last batch of emails from this user, so go back to the Admin Console > Monitor > Mail Queues > Server, but this time click the Active queue tab. Now you can go to the Sender address column, right click on the sender again and put any emails in the Active queue from this user into the Hold queue.
Then, one last time, go back to the Deferred queue tab, right click on the impacted sender, and put any remaining Deferred emails from this sender into the Hold queue.
At this point, you’ve stopped the immediate breach and enabled Zimbra to return to normal operation.
Post-Breach Tasks
You may be tempted to delete the contents of the Hold queue, but don’t. The user was probably unaware that their account had been compromised and was trying to send legitimate email which is now in the Hold queue. Eventually, you’ll want to release those legitimate emails to be sent. Also, law enforcement, your cyber insurance company, the compliance department and others may need access to those emails, so best to keep them for now.
Assuming you are running Fail2Ban, go back to your SSH window and grep through /var/log/zimbra.log to obtain the IP address (or addresses) used by the bad actor. If the user had a compromised workstation, there may be only the user’s IP address to be found, but often, there may be dozens of IPs used by the bad actor and their colleagues. Once you have this space separated list of IPs you can manually ban them in Fail2Ban.
You may also want to grep through /opt/zimbra/log/mailbox.log and see if the bad actor was using the web client on the user’s workstation to send emails. This is rarer, but you’ll want to ban that IP address too. If that IP however turns out to be the WAN IP of that user’s office, you will shortly get a call from multiple users saying Zimbra is “down”. Not true! You can say that the user’s account was compromised and the office IP was used to send spam. Clearly there is an issue at the office, and once the infection is found (probably more than one infection in my experience…) and remediated, you can readily unban the office’s IP address.
Note: You can ask the office for their WAN IP, and then on your Zimbra server, do either like route -n | grep <office ip> or history | grep <office ip> to confirm that the IP has been banned.
Many jurisdictions (and your cyber liability insurance carrier) mandate prompt data breach reporting to the authorities. We recommend getting advice from your company’s legal department or an attorney to know what your specific reporting requirements are. Failure to report data breaches can have significant financial penalties, and some cyber liability policies we have seen won’t cover costs associated with a data breach unless the breach is properly reported.
You should also be prepared to see your IP(s) and email domains added to various block lists, and you will need to spend time contacting each such block list provider (as well as potentially Hotmail, Yahoo, Google and other major email service providers and ISPs) to get your IP(s) and domain(s) unblocked.
It Ain’t Over Until It’s Over
Note that if one workstation was compromised, chances are other workstations and accounts may be compromised, so be on the lookout for follow-on account breaches.
Pre-Breach Zimbra Hardening
We have a suite of techniques we use in addition to Fail2Ban to harden Zimbra systems. Nothing is guaranteed to prevent a breach, but various activity limits, throttling etc., 2FA, password and failed login lockout requirements can go a long way to stopping or at least slowing down a bad actor and giving you more time to deal with the breach — before your IP(s) and domain(s) are blocklisted.
Need Help?
Mission Critical Email is not a security shop. If you’ve had a breach, we are happy to work side-by-side with the security firm you have hired, law enforcement and your insurance carrier to perform tasks helpful to preserving evidence and restoring Zimbra to a good state.
We’d much prefer to get to know you before a breach has happened, where we can work together to harden your Zimbra system and (hopefully) prevent what would otherwise have been the next breach.
Don’t hesitate to reach out!
Hope that helps,
L. Mark Stone
Mission Critical Email LLC
26 February 2025
The information provided in this blog is intended for informational and educational purposes only. The views expressed herein are those of Mr. Stone personally. The contents of this site are not intended as advice for any purpose and are subject to change without notice. Mission Critical Email makes no warranties of any kind regarding the accuracy or completeness of any information on this site, and we make no representations regarding whether such information is up-to-date or applicable to any particular situation. All copyrights are reserved by Mr. Stone. Any portion of the material on this site may be used for personal or educational purposes provided appropriate attribution is given to Mr. Stone and this blog.