Security vs. Stability
One challenge facing Linux system administrators is the tradeoff between security and the potential loss of stability when considering whether or not to enable “unattended upgrades”. When enabled, the operating system self-updates and reboots as necessary whenever package updates are released.
In the case of public-facing Zimbra servers, this challenge is especially important, as current best practices comprise deploying most security patches as quickly as practicable to protect any public-facing server against hostile exploits.
And indeed, it has been our recommended best practice to enable unattended-upgrades on Ubuntu (and yum-cron on RHEL/CentOS) to do just that. The subscription version of Oracle Linux even allows for kernel updates without reboots.
Historically, this has been safe. When BIND9, OpenSSH or a similar service-based application is updated, the update package gracefully restarts the service as part of the update process. Even when the kernel has been updated, Linux servers continue to run with the old kernel loaded until the next reboot (and both Ubuntu’s and RHEL/CentOS’s updaters can be configured to automatically reboot servers at a specified time).
Zimbra 8.8.x Is Different From Earlier Zimbra Versions
This Zimbra series has moved away from standalone patches and operating system dependencies to its own repo-based installation and upgrades. Yes, you still run a PERL installer to get Zimbra installed initially, and yes, to upgrade to a later Zimbra version you’ll need to run that version’s standalone installer. But, once a specific version of Zimbra is installed, all updates, patches etc. are done the same way you’d deploy operating system updates (e.g. apt-get update; apt-get upgrade). And you’ll notice that the Zimbra installer does a lot of work on the front end to modify your operating system’s package repositories to be able to download the needed Zimbra packages from those repos, rather than installing .deb or .rpm packages bundled with the installer.
On a standalone Zimbra 8.8.x server, there are now more that 200 individual “zimbra-…” packages installed, and as Zimbra Patches (which are really bundles of zimbra-* packages) are being released now every few weeks, it would appear to make sense to enable unattended upgrades.
Well yes… and No. Zimbra is still new to the repo game and is proceeding deliberately. When an updated Zimbra package requires the restart of a Zimbra service, or a cache flush, the package updater lets you know by printing to the console that a service needs to restarted (as do the Release Notes), but the updated package (like zimbra-nginx) itself doesn’t restart that impacted service (zmproxyctl restart) automatically. And since some Zimbra service restarts result in configuration files getting rewritten, it is possible that updating a Zimbra package but not doing a needed service restart can result in a system error (we think we’ve seen this a few times already, but are not 100% sure at this writing). In any event, if Zimbra’s Release Notes and the package update itself both tell you to restart a service, it’s a good idea to do that anyway pretty much right after the package update is installed.
So now we are kind of stuck: on the one hand, we want to get security patches and bug fixes installed ASAP to avoid an exploit, but we want to keep Zimbra happy and stable. What should we do?
Solution: Hybrid Approach
The optimal solution at the moment is to continue to allow operating system packages to update themselves automatically, but for the System Administrator to continue to patch Zimbra manually, restarting indicated services as needed. When Zimbra starts bundling automatic service restarts with its upgrade packages, then it will likely make sense for the entire Zimbra server to upgrade itself automatically.
Both Ubuntu’s unattended-upgrades and RHEL/CentOS’s yum-cron packages contain the ability to exclude certain packages from automatic updates. All one needs to do is add a list of packages to be excluded in the proper block within /etc/apt/apt.conf.d/50unattended-upgrades (Ubuntu) or within /etc/yum/yum-cron.conf (RHEL/CentOS).
Fortunately, in providing a blacklist of packages to update, the Ubuntu config file defaults to allowing regular expressions and the RHEL/CentOS file allows wildcards. Otherwise, we’d have the burden of adding more than 200 packages to the the blacklist, and then keeping that list updated as Zimbra adds more packages, or renames some.
So, the Ubuntu blacklist section of /etc/apt/apt.conf.d/50unattended-upgrades should look like this:
// List of packages to not update (regexp are supported)
The /etc/yum/yum-cron.conf file on RHEL/CentOS should have a line added that looks like this:
exclude = zimbra*
Once you’ve made the necessary changes to those files, just restart the unattended-updates or yum-cron service and you should be good to go. Your operating system will stay up to date, and as soon as Zimbra releases a security or other Patch, you can install it manually at the next opportunity appropriate for your company.
One More Thing About Multi-Server Zimbra Environments…
Both unattended upgrades and yum-cron enable you to specify in their config files when to perform an automatic reboot. In a multi-server environment it is important to stagger the server reboots so as to avoid service interruptions, and also in the proper server order. LDAP servers first, then MTA and Proxy servers, then Mailbox servers.
If you have two or more LDAP servers for example, it’s a good idea to schedule their operating system reboots at least an hour apart, to give you some time to address any issue if a reboot goes not quite right and allow your other Zimbra servers to leverage the yet-to-be-rebooted LDAP server(s).
Similarly, redundant Proxy/MTA hosts’ reboots should also be staggered so as to avoid service interruptions.
Reboots for mailbox servers should be scheduled thereafter. If you have not yet migrated to the new continuous NG Backup and are still using the traditional Network Edition nightly backups, be sure to schedule mailbox server reboots at least an hour or so after your longest Full backup completes, to avoid having a partial (and generally unusable) Network Edition backup.
Hope that helps!
L. Mark Stone
Mission Critical Email
24 July 2018
The information provided in this blog is intended for informational and educational purposes only. The views expressed herein are those of Mr. Stone personally. The contents of this site are not intended as advice for any purpose and are subject to change without notice. Mission Critical Email makes no warranties of any kind regarding the accuracy or completeness of any information on this site, and we make no representations regarding whether such information is up-to-date or applicable to any particular situation. All copyrights are reserved by Mr. Stone. Any portion of the material on this site may be used for personal or educational purposes provided appropriate attribution is given to Mr. Stone and this blog.