Zimbra 8.8 Network Edition includes three new features we have used to help customers slash storage costs by more than 60% while simultaneously increasing performance. For HIPAA Covered Entities and those concerned with GDPR compliance, these features leverage available encryption. It’s yet one more reason to upgrade to 8.8 sooner rather than later.
Backup Storage Savings
Zimbra 8.8 Network Edition ships with a new backup storage engine “BackupNG” (Backup Next Generation) that, unlike the previous Network Edition backup engine which ran nightly, provides continuous data protection, and which also performs deduplication and compression. It used to be the rule of thumb that if you had a one terabyte mail store, 30 days of mixed Full and Incremental backups would consume about 3.5 terabytes. BackupNG backups however, with the same 30-day retention policy, consume less than the mail store does. That’s right, 30 days of BackupNG backups in our experience to date can be expected to consume 60%-80% of the space consumed by the mail store itself.
Using our 1TB mail store example, that means total storage needs with Zimbra 8.8 BackupNG just went from (1 + 3.5) = 4.5TB to (1 + .8) = 1.8TB — a reduction in net storage consumed of 60%.
Mail Store Storage Savings
Zimbra has always shipped with a single-instance message store, so if a user emails a 30MB PowerPoint file to ten of their coworkers, the attachment only increases the mail store size by a little more than 30MB (for those of a technical bent, this is done by using hardlinks on the file system). Mail stores are server-specific and the deduping only works when mail is delivered, so if you moved those eleven coworker mailboxes to another Zimbra mailbox server, you’d see that your storage consumption went up by 300MB on the second server. You could run a deduplication utility, but the old utility “zmdedupe” relied on an object cache with a limit of 2,000 objects, so that older utility tends to miss mail blobs that could be deduplicated.
Zimbra 8.8 Network Edition ships with a new Storage module with a much more efficient deduplication engine. The efficiency improvements are based on the removal of the 2,000-object cache. Thus, on a busy server more mail blobs are deduplicated upon their initial arrival, and imported or migrated mailboxes are more efficiently deduplicated as well during the import or migration process.
It’s hard to quantify with specificity a range of storage space savings here, because the savings will vary a lot from company to company depending on things like whether coworkers email each other lots of large attachments or not. But I can say that with one customer I recently migrated to Zimbra 8.8, the mail store on their old Zimbra server was around 450GB, and after the migration and a deduplication, the mail store shrank to about 380GB.
HSM Storage Savings
“HSM” (Hierarchical Storage Management) is a Zimbra feature introduced years ago to allow large hosting providers to migrate older mail blobs and other objects to slower, less expensive storage. Consider that most end users want to save email forever, but only rarely look at emails older than a few weeks. All email servers are very disk I/O intensive, so you need fast (i.e. expensive) SAN primary storage for the current mail blobs, but you can safely move older mail blobs to slower (i.e. less expensive) SAN storage without adversely impacting the customer experience.
Zimbra 8.8 Network Edition ships with an enhanced HSM engine that supports Amazon S3 and other Object storage containers for HSM disk volumes. On Amazon, S3 object storage is much less expensive than the EBS block storage volumes needed for the mail store. Consider that because you pay for Elastic Block Storage volumes based on the size you provision, but you pay for S3 object storage based on how much you consume, you can save a lot of money by leveraging this feature if you host on AWS.
Even if you host on-premises, this feature can still help you save money as many newer low-cost storage appliances now support provisioning S3-compatible volumes.
One other thing… There are two pricing tiers for Amazon S3: the regular price, and the lower-cost “IA” (Infrequent Access”) price. Zimbra 8.8’s HSM storage module enables you to set a threshold such that larger mail blobs and attachments are moved to S3 IA storage — further reducing your storage costs.
Super-Secret Server Migration Trick (Shhhh…)
Let’s say you are running an older Zimbra, Kerio, Exchange or other email server with a 1 TB mail store, and you want to migrate to Zimbra 8.8 on AWS. I bet you are thinking you’ll need to provision an EBS volume greater than 1TB just to do the migration, and then afterwards run the new deduplication utility, and then after that run an HSM job to move older mail blobs out to S3 storage, and then after that run another deduplication against the S3 storage. But wait… you are then stuck paying for an expensive 1TB EBS volume on which you are consuming maybe 200GB. How to you solve that challenge?
The Simple Trick Solution: Provision a 300GB EBS volume for your 1TB mail store. But instead of waiting until after you’ve migrated everyone’s mailboxes (in this architecture you’d run out disk space long before then), you will run multiple deduplication and HSM migration jobs at the same time that you are importing emails blobs (e.g. via imapsync) or importing whole mailboxes. Yes, you have to pay attention during the import process to ensure you don’t run out of disk space, but this is data-safe and helps you avoid buying more EBS storage (for what would be forever) than you really need.
You’ll use this same process when life-cycling Zimbra mailbox servers to avoid customer downtime. (We hardly ever upgrade mailbox servers; we build fresh ones, move the mailboxes over, and then destroy the old mailbox servers. It’s a less risky way to “upgrade” Zimbra, and essentially eliminates end-user downtime during the upgrade as well).
A Small Note About HIPAA and GDPR…
Zimbra supports encryption in flight (even between components on a single Zimbra server) and Amazon supports encryption at rest, so it’s straightforward to secure your data in a compliant way when hosting on AWS, which is where we host our multi-tenant Zimbra hosting farm as well as a number of customer Zimbra environments.
HIPAA Covered Entities, other regulated entities and anyone requiring encryption should feel comfortable using an upgrade to Zimbra 8.8 as one method for better securing the data in your email system, as well as improving the security of end-user access to such data. If hosting on-premises, be sure to select SANs which offer self-encrypting disk drives at the hardware layer as well as encryption in the transport layer.
Zimbra have developed a slide deck covering GDPR compliance; it’s available at this link.
Hope that helps, and feel free to contact us if you’d like to explore how you can save money on storage with an upgrade to Zimbra 8.8.
L. Mark Stone
Mission Critical Email
23 May 2018
The information provided in this blog is intended for informational and educational purposes only. The views expressed herein are those of Mr. Stone personally. The contents of this site are not intended as advice for any purpose and are subject to change without notice. Mission Critical Email makes no warranties of any kind regarding the accuracy or completeness of any information on this site, and we make no representations regarding whether such information is up-to-date or applicable to any particular situation. All copyrights are reserved by Mr. Stone. Any portion of the material on this site may be used for personal or educational purposes provided appropriate attribution is given to Mr. Stone and this blog.