During yesterday’s Zeta Alliance call, Zimbra Patching, recent Zimbra exploits and their mitigation were a featured topic of discussion.  The Zimbra forums have some very active discussions about mitigating currently compromised systems, so I thought it would be timely to summarize the current status of the current compromise, Zimbra’s reactions and support stance, and helpful to share Mission Critical Email’s best practices when it comes to patching.

CVE-2019-9670 Exploit History Summary
In mid-March, Rene Otto, Zimbra’s Product VP, posted that Zimbra had been working closely with security researcher An Trinh to develop a security patch for Zimbra.  Zimbra then released Patches for Zimbra 8.7 and 8.8, as well as for Zimbra 8.6 — even though Zimbra 8.6 officially stopped receiving bug fixes and security patches in September 2018.

The Zimbra patch for this CVE prevents uninfected systems from becoming infected, but the patch does not include tools for removing the infection from already infected systems.  In the forum discussion thread on this, the malware is reported to be evolving, so as to avoid easy removal. One user has their own blog post kept up to date with various removal strategies.

At the same time, Zimbra posted a notice in the paid customer Support Portal which reads in part: “If you are not running the latest release with the latest patches, you are at risk of a compromised system. If your system is compromised, Zimbra Support recommends rebuilding the systems. It is your responsibility to perform the rebuild. Zimbra Support will not do system analysis or system rebuilds, but we will provide guidance. If you need assistance, please contact your Partner…”  We agree that a migration to a fresh Zimbra system, in this case, is the right way to go.

Patching Is a Risk Management Exercise
NIST, Spiceworks, Computer Weekly and plenty of others for years have correctly (IMHO) point out that patching is a risk management exercise.  Applying patches creates a risk of “breaking” functionality, especially interoperability with third-party applications.

In Zimbra’s case, Zimbra’s interoperability with third-party systems is almost always via established email/Internet protocols (e.g. IMAP, http, https etc.) or via Zimbra’s server extension and zimlet technology, both of which have well-defined programming interfaces.  In other words, the risk of a Zimbra patch “breaking” interoperability with third-party applications is extremely low. Further, even if a patch did so, that same avenue of interoperability, in the case of a Zimbra Security Patch, will be the same attack vector the malware author will use.  The tradeoff therefore between some temporary loss of functionality and an exploitable public-Internet facing system seems to favor strongly deciding to have some temporary loss of functionality.

In Zimbra 8.8, Zimbra moved away from the big meta-packages previously installed, in favor of a repository-based system of package distribution.  As a result, a single Zimbra server now has more than 200 individual repo-based software packages installed.  Upgrading one or a few packages is now much easier, and QA testing for new packages is also easier for Zimbra.  This new architecture IMHO further reduces the risk of patching Zimbra promptly.

Mission Critical Email Zimbra Patching – Recommended Best Practices
We recommend the following as a patching strategy for Zimbra:

  1. Get Backup MX. Contract for third-party Backup MX services if you are not already using a separate non-Zimbra email gateway in a separate data center.  The cost through a company like DNS Made Easy is $25 to $50 per year per domain.  In this way, if your Zimbra data center goes off the air, or Zimbra goes off the air from restarting Zimbra services or a reboot, you will not lose any inbound email.
  2. Automate Package Updates. Deploy unattended upgrades (Ubuntu) or similar on RHEL to cover security and version upgrades of operating system packages, but (for the moment) exclude Zimbra packages from these unattended upgrades.  Allow the unattended upgrades daemon to reboot the system (e.g. in the event of kernel update) during a defined maintenance window.  In the case of a multi-server Zimbra farm, stagger the system reboot timings to ensure at least one LDAP,  and if possible one MTA and one proxy server remain up and running at all times.
    1. At this writing, Zimbra’s packages do not automatically restart impacted Zimbra services, but the installation of updated Zimbra packages will output to the screen which Zimbra services need to be restarted.  This is why we exclude Zimbra packages from unattended upgrades presently.
    2. We have strongly encouraged Zimbra as a longer-term engineering goal to have Zimbra package updates gracefully restart their own impacted services, much the way an update to BIND9 gracefully restarts named.  We have further asked Zimbra in the interim to document in the Patch Release Notes which Zimbra services will need to be restarted on application of the Patch.
  3. Subscribe to Zimbra’s Announcements, to get email notifications of Zimbra patches, which, at this writing, appear to be released generally mid-month.  Additionally, if using a paid version, ask your Zimbra Partner to keep you apprised of Zimbra patches (Zimbra’s Partner Portal publishes a separate “Zimbra Releases” tab with dates a few months out of upcoming scheduled patches and releases).
  4. Read the Patch Release Notes promptly, and if the Patch contains any security fixes rated higher than “Minor”, we recommend patching Zimbra within 24-48 hours.
    1. Sneeky Tip: Zimbra publishes the Release Notes a few hours to a few days in advance of updating the repositories and officially releasing the patch.  You can read these release notes by manually altering the URL from the last set of release notes.  For example, the release notes for 8.8.12 Patch 1 live at https://wiki.zimbra.com/wiki/Zimbra_Releases/8.8.12/P1.  Change the “P1” at the end to “P2” and if the document exists, you are set.  Do keep in mind that the “sneek peek” you are looking at may get updated prior to the patch being officially released, but at least you’ll get a heads up to start planning.
  5. Stay On A Supported Zimbra Version. Don’t defer upgrading to a Supported Release!  And certainly upgrade before a release goes out of General Support!

 

Hope that helps,
L. Mark Stone
Mission Critical Email
1 May 2019

The information provided in this blog is intended for informational and educational purposes only. The views expressed herein are those of Mr. Stone personally. The contents of this site are not intended as advice for any purpose and are subject to change without notice. Mission Critical Email makes no warranties of any kind regarding the accuracy or completeness of any information on this site, and we make no representations regarding whether such information is up-to-date or applicable to any particular situation. All copyrights are reserved by Mr. Stone. Any portion of the material on this site may be used for personal or educational purposes provided appropriate attribution is given to Mr. Stone and this blog.