Zimbra 10 Daffodil is the first majorly innovative release from Zimbra in a number of years. Daffodil has been General Availability since March 2023 and as we’ve been answering pretty much the same set of questions from customers consistently for months, albeit with a few recent changes, we thought it would be helpful to share at a high level what we’ve learned about Daffodil and how you can decide when to upgrade. (Spoiler Alert: “Now” is a good time to migrate to Zimbra 10 Daffodil in almost all cases…)
What’s Different About Daffodil?
Daffodil removes all of the NG code licensed from Zextras in favor of Zimbra-native code. With the exception of backups to S3 targets (coming in a future release), essentially all of the functionality provided by NG is now provided by Zimbra-native code. There is also some new functionality, like more granular, predefined delegated administration roles. As Zimbra now owns the entirety of the product’s code base, Zimbra have gone on record as saying that this change alone will speed development, including introducing new features, refactoring and continued security improvements.
Despite those under-the hood differences, Daffodil still includes the Classic (Ajax) and Modern (Preact) web client interfaces available in Zimbra 9. The Admin Console is also pretty much the same, aside from some UI changes to put all of what used to appear in the NG tab back in their respective logical locations.
Isn’t that A Lot of New Code? Shouldn’t We Be Worried About Lots of New Bugs?
No and No. Recall Synacor (Zimbra’s corporate owners) were Zimbra’s largest customer before they bought Zimbra. Synacor have provided customizations to their very large customers (hundreds of thousands and millions of mailboxes in a single Zimbra multi-server environment) for many years. Many of these customizations provide functionality previously provided by NG in the commercial product (like HSM support for S3 and other external volumes). We have been given to understand that this battle-tested code was simply updated and repurposed for Daffodil.
Further, although there is, as with any new release, new code in the product, Daffodil was released as “Early Access” in the Fall of 2022.
Also, as one of Zimbra’s Certified Trainers, I have so far put five groups of trainees through using Daffodil during the rigorous three-day classes, where we do installs/upgrades every day. No problems yet.
Let’s get specific about some of that seemingly-new-but-not-really code in Daffodil:
Zimbra 10 Daffodil Backups
Backups in Daffodil are realtime, just like NG, and are based on the Network Edition “Classic” backups that have been around forever. NG backups are similar to Apple Time Machine backups: a Full backup once at inception and then forever incrementals. Daffodil backups using the now-default “auto-grouped” method mix Full and Incremental backups each night, with continuously-written redologs providing the real-time backup capability. Enhancements regarding compression reduce storage consumption, and S3 as a backup target are scheduled for a future release. This architecture makes it easier to copy offsite point-in-time backups, without having to create a brand new External Backup in NG first.
NG Backup allows an administrator to backup everything in Zimbra, or, to exclude accounts by excluding one or more Classes of Service (and the accounts therein). Daffodil does this too, but also allows an administrator to exclude one or more domains from being backed up.
Restore modes between Daffodil and NG are comparable, except that Daffodil has an offline restore mode that in our experience works really fast, so can be used to lower RTO targets for customers sensitive to needing very short Recovery Time Objectives during Disaster Recovery.
NG restores, because they don’t rely on either Zimbra’s MariaDB databases nor Zimbra’s LDAP, provide more flexibility for doing things like consolidating two old mailbox servers onto a single new mailbox server (or vica versa), so with Daffodil you’ll need to move mailboxes after a disaster recovery restore if you want to consolidate (or expand) the number of mailstores.
Zimbra 10 Daffodil Mobile
Like NG Mobile, Daffodil Mobile includes a number of features we politely refer to as a “poor man’s MDM (mobile device management)” capability. You can implement ABQ (allow, block, quarantine) for mobile devices, enforce policies like longer alphanumeric passwords, and even wipe the mobile device from the Admin Console.
Zimbra 10 Daffodil Storage Manager (“SM”)
SM provides the HSM capabilities we are used to with NG, like supporting S3 volumes, Intelligent Tiering etc. We’ve been told that S3 throughput in Daffodil is higher than that in NG, but we haven’t yet had an opportunity to test this rigourously. As with both NG and Daffodil, you can lifecycle local and remote storage volumes with no downtime whatsoever. When I demonstrate this during the three-day Zimbra Administration Training course, most attendees were not aware of this capability. Someone always comments that they were dreading having to retire a SAN in the next quarter, and were expecting that they’d have to move Zimbra to entirely new virtual servers. It’s good that SM retains this very valuable feature set, which incorporates very handy NG features like Volume-to-Volume moves.
Zimbra 10 Daffodil Delegated Administration
NG Admin greatly simplifies creating and managing Delegated Administrators, at the cost of limited flexibility in adjusting rights and permissions of Delegated Admins. Daffodil uses the same, incredibly granular Access Control Entries framework that has been in Zimbra for years, but now offers a number of (soon to be growing) pre-defined roles for creating delegated admin accounts via check boxes in the Admin Console. The Wizards for creating hand-crafted Delegated Admin roles in Daffodil are a bit more helpful than I remembered them from before NG Admin.
If you are familiar with Windows Security Groups or other Role-based access controls, Daffodil’s Delegated Admin learning curve will be relatively short and not (terribly) steep. All you need to do is use the Wizard to create a Distribution List that has rights to access different portions of the Admin Console (so that members of the DL will only be presented with those portions of the Admin Console UI necessary for them to do what the role allows). After you create these “View” rights, the Wizard will then walk you through creating companion “Action” rights (technically, Access Control Entries) on the desired target(s). IOW, a delegated admin you want to be able to change passwords in two out of the five domains in your Zimbra system needs to have the ACEs created on those two domains, with the “View” rights added to the distribution list (sometimes called an “Admin Role” in the Admin Console UI).
Alignment between these View and Action rights, applied only to the Targets you wish (like one or more domains for which you want someone to be able to administer Distribution Lists) is what enables anyone you configure as members of this Admin Distribution List to inherit the rights on the DL. The final step, for anyone you want to be a delegated admin, is to make their account a member of the DL and set the account attribute zimbraIsDelegatedAdminAccount TRUE and you are done.
Aside from being able to create really granular delegated admin roles with this framework, recall that Zimbra distribution lists can be nested, so this provides for some permissions simplification when, for example, you have three tiers of Help Desk agents with somewhat overlapping permissions.
Briefcase and Drive
Drive, being part of NG, is gone in Daffodil, but Zimbra’s migration tool allows you to export Drive data. Briefcase supports versioning, check-out/check-in, and now in Daffodil, collaborative document editing without the need for a separate Docs server.
Text And Video Chat
Zimbra Connect, being part of NG, is also gone in Daffodil. Zimbra took a SaaS approach to text and video chat with Daffodil’s launch by doing a freemium deal with ImMail. For customers that care about data sovereignty, this isn’t acceptable, so Zimbra expects to release self-hosted chat functionality within Daffodil we are told in a later release.
Zimbra 10 Daffodil Product Lifecycle Changes
There is no more “LTS” edition, and the massive architectural changes that caused a lot of grief in the early 8.8.x series are also gone. Starting with 10.1, customers can choose with Daffodil to stay on a “no-new-features” upgrade path, receiving only bug fixes and security updates, or; they can choose an upgrade path that provides those same bug fixes and security updates, as well as modest new features and enhancements. Major architectural changes that for example have the risk of breaking things, as had been the case between Zimbra 8.8.9 and 8.8.15, will now result in a new Major version. By way of background, Zimbra numbering is Major.Minor.Micro, so Daffodil 10.0.5 for example will have had five rounds of bug fixes and security patches; Daffodil 10.2.5 will have had the same five rounds of bug fixes and security patches, but also two rounds of modest feature enhancements. Daffodil 11.0.0 will contain some significant architectural changes over Daffodil 10.x.x.
OK, OK… But Should I Migrate To Zimbra 10 Daffodil Now? – UPDATED 27 SEPTEMBER 2023
Absolutely yes, in most cases – provided you have the resources to do so expeditiously.
Zimbra 8.8.15 is End of General Support on 31 December 2023. So either you migrate to Zimbra 10 or upgrade to Zimbra 9, which goes End of General Support on 31 March 2024, so as to buy you an extra three months.
How To Migrate To Zimbra 10 Daffodil?
Technically, Zimbra supports doing a single-server in-place upgrade from either Zimbra 8.8.15 or Zimbra 9.0.0 to Zimbra 10, but only if you have never installed the NG modules. While supported, Zimbra is encouraging customers to instead migrate to Zimbra 10 using the Rolling Upgrade method. This is the supported path to get to Zimbra 10 for all (other) use cases.
FWIW we pretty much refuse to do in-place upgrades in most cases, because of the loss of rollback opportunities primarily. In-place upgrades are quite violent; they remove from your server all of the Zimbra packages and components on disk; destroy your LDAP databases and all of your customizations not stored in LDAP; as well as do in-place patching of your MariaDB databases. In other words, if an in-place upgrade goes sideways, you are almost guaranteed to have to execute your disaster recovery plan, and will need restorable backups to do so. As Zimbra have reported publicly that NG backups have not been restorable 100% of the time for Disaster Recovery purposes, we feel this is a needless exposure that trades convenience for conservative data safety. Hence our strong pushback when asked to do in-place upgrades versus migrations. (Every rule has an exception however, and we expect that early next year we will be comfortable doing in-place upgrades of fresh Zimbra 10 Daffodil servers to Zimbra 10.1.)
The Rolling Upgrade method is documented here, but essentially it works like this:
- Build a new LDAP MMR server if your existing MMR servers are not on the latest operating system or, if you have a single-server setup currently. The new MMR server will be built at the same version as your current Zimbra system.
- Do an in-place upgrade of the new LDAP MMR server to Zimbra 10 Daffodil and point all of your other Zimbra servers to use this LDAP server. Then, get rid of all of the non-Zimbra 10 Daffodil LDAP servers (or stop and remove the LDAP service from existing Zimbra 8.8.15 and Zimbra 9.0.0 servers).
- Add Zimbra 10 Daffodil Proxy, MTA and Mailbox servers and deploy all of your customizations (Spamhaus, invaluement, Fail2Ban etc.)
- Move proxy and mail flow traffic to the Zimbra 10 Daffodil Proxy and MTA servers.
- Use the Zimbra-provided tool to extract NG-specific attributes and from your existing mailbox servers, and redeploy those attributes in Zimbra 10 Daffodil format on the new Zimbra 10 Daffodil servers.
- Use the CLI to move mailboxes, Distribution Lists, etc.
- As your old Zimbra 8.8.15 and Zimbra 9.0.0 mailbox servers are evacuated, destroy them.
- Destroy the Zimbra 8.8.15 and Zimbra 9.0.0 Proxy and MTA servers, leaving only Zimbra 10 Daffodil servers running on the latest operating system of your choice.
There are of course a number of important details I’ve left out, like an incremental testing plan, moving the Logger host, etc.
The good news is that the Rolling Upgrade method has been supported by Zimbra since the beginning of time. It has always been permissible to run previous-version Zimbra mailbox servers in a Zimbra environment, provided however that the LDAP, Proxy and MTA servers are all at the latest Zimbra version.
Some Important Things To Consider:
- Customers running 8.8.15/9.0.0 with NG HSM, Centralized Storage on S3 and Intelligent Tiering will want to wait for the Zimbra Patch to be released in early October 2023. The upgrade/migration for this use case will be “blobless”, so all of your mail objects in S3 at lower-cost tiers will remain at their then-current lower-cost tier as part of the upgrade/migration, preserving your current low storage costs.
- Ubuntu 22.04 binaries should be available starting with Daffodil 10.1, now expected early in 2024.
- For LDAP MMR users doing Rolling Upgrades to new LDAP MMR servers who are worried about expanding the number of CSNs, this is no longer a concern: LDAP MMR now runs fine with lots of multiple CSNs. You can reduce the number of CSNs if you wish by writing a minor change to e.g the Description field of accounts/objects contained in the older CSNs. Doing so will cause the account/object data so updated to migrate to a newer CSN.
If you’d like help planning your Daffodil migration, please start the conversation by filling out the form:
Hope that helps,
L. Mark Stone
Mission Critical Email LLC
29 April 2023
Last Update 27 September 2023
The information provided in this blog is intended for informational and educational purposes only. The views expressed herein are those of Mr. Stone personally. The contents of this site are not intended as advice for any purpose and are subject to change without notice. Mission Critical Email makes no warranties of any kind regarding the accuracy or completeness of any information on this site, and we make no representations regarding whether such information is up-to-date or applicable to any particular situation. All copyrights are reserved by Mr. Stone. Any portion of the material on this site may be used for personal or educational purposes provided appropriate attribution is given to Mr. Stone and this blog.
