Zimbra servers make a lot of DNS requests, so having a locally available caching DNS server system for Zimbra to use is important. In helping customers with very active Zimbra servers, we have found that deploying BIND9 gets the job done when dnsmasq, Zimbra’s Unbound or even local Active Directory servers with the DNS role can’t keep up. This post will show you how to do this easily.
Which BIND9 To Use?
BIND9’s authors and maintainers, the Internet Systems Consortium, recommend installing BIND9 downloaded from them directly versus using the version of BIND9 shipped with your Linux distro. Their experience has been that many Linux distros do not update their BIND9 packages as quickly as is desired (if at all), so by downloading from the authors directly, you are ensured of getting the latest stable packages.
As of this writing, Ubuntu Server 22.04 has 9.18.18 as the latest version.
Given the foregoing, we recommend not installing BIND9 on your Zimbra servers, but instead to install multiple separate, BIND9-only servers on the latest stable operating system. In that scenario, ideally, you should use the BIND9 packages downloaded from ISC. But if you are prepared to rotate out your BIND9 servers and use the latest available supported distro (and confirm that their BIND9 packages are as up to date as Ubuntu 22.04 seems to be doing presently), then the ease of use of using your distro’s BIND9 packages may not be too big of a compromise–especially if the BIND9 servers to be used by Zimbra are isolated from the rest of your network. The other benefit of using ISC’s BIND9 packages is that you can purchase Support from ISC directly if you have an issue.
Architectural Decisions and Recommendations
We do our own Zimbra hosting on Amazon Web Services, and in line with AWS’s Well Architected Framework, we recommend deploying one Ubuntu 22.04 BIND9 server in each Availability Zone. In the example below, we will build three identical Ubuntu 22.04 BIND9 servers in each of three different Availability Zones, each on different subnets.
BIND9 has a very small footprint. Its cache size is memory dependent to avoid swapping, and it uses not a lot of CPU, so we typically start with a t3.micro instance (2 CPU cores and 1GB of RAM) and a single 32GB root disk. For safety’s sake, we add a 2GB swap file. These instances all-in cost a little under US$10/month each. But t2.micro (free tier eligible) instances we find also work unless your environment is very large.
If your Zimbra system delivers more than 500K emails/day, you may want to consider an instance size for your BIND9 servers with more RAM.
For simplicity’s sake, we will presume that you will use Ubuntu 22.04 servers with BIND9 installed from the Ubuntu repos. If you use a Red Hat-based distro, or ISC’s builds, please note the location of some files may be different, and some installations use chroot as well. (We have not tested this.) Your choice!
BIND9 will need one or more forwarders. (Forwarders are DNS servers that can perform DNS lookups on our BIND9 servers’ behalf, when the answer to a Zimbra server’s query is not already in the BIND9 server’s cache.) We use AWS’s Route 53 managed DNS service for hosting our zone files, so will use that service’s IP for the Zimbra servers’ forward and reverse zones in the setup example below, but, we’ll use Cloudfare and Google DNS for public resolution.
Whatever you use for a forwarder, the forwarder must resolve the Zimbra server’s forward and reverse lookups with the actual IP addresses of the Zimbra servers. In other words, if your Zimbra servers are NAT’d, the forwarder you use must resolve the Zimbra servers at their private IP addresses. Using caching DNS servers will reduce the load on your forwarder(s), but the forwarder(s) still need to be performant. If you are hosting your own forwarders (e.g. Active Directory servers), please ensure that the forwarders the AD servers use are also performant. In the BIND9 configuration files below, you’ll see we use local forwarders just for the Zimbra servers’ forward and reverse zones, and let BIND9 query Cloudfare and Google for everything else.
If you have been using dnsmasq and do not have a DNS server that will resolve the Zimbra servers at their private IP addresses, then you will need to add forward and reverse Zone Files to the BIND9 servers. Configuring Zone Files and Replication in BIND9 is well documented and can be added on to the configurations below. But, configuring BIND9 with Zone Files is beyond the scope of this post, which is about configuring BIND9 only as a caching DNS server for Zimbra.
Zimbra Configuration – Prebuild
The three BIND9 servers we will deploy will be named bind1.mydomain.com, bind2.mydomain.com and bind3.mydomain.com. Once these servers have been provisioned (no BIND9 configured yet) and you know their IP addresses, we should let Zimbra know about them. At the end of /etc/hosts on all of your Zimbra servers, add:
172.31.0.202 bind1.mydomain.com bind1 172.31.16.64 bind2.mydomain.com bind2 172.31.32.125 bind3.mydomain.com bind3
BIND9 Servers’ Buildout – Initial Preparation Work
First step is to deploy the 2 GB swapfile on your newly provisioned servers. Ubuntu on AWS is provisioned without any swap. As root you can run:
fallocate -l 2G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile && swapon --show && echo '/swapfile none swap sw 0 0' | tee -a /etc/fstab
With swap configured, we now need to configure the operating system to avoid using swap unless absolutely necessary (same as with Zimbra). As root, run:
nano /etc/sysctl.conf
At the end of the file, add the following two lines:
# Reduce Swappiness vm.swappiness = 1 # If hosting on AWS, uncomment the two lines below: vm.vfs_cache_pressure=200 vm.min_free_kbytes = 524288
Next step is to edit /etc/hosts on all three BIND9 servers to match /etc/hosts on your Zimbra servers (including the entries for the BIND9 servers themselves that you added in the previous section). In other words, at this point, /etc/hosts on your Zimbra servers and on the BIND9 servers should all be identical.
Edit /etc/hostname on each of the BIND9 servers to remove the existing entry and to replace it with the FQDN of each of the BIND9 servers (e.g. like “bind1.mydomain.com”).
Update the operating system and reboot. As root, run:
apt clean all && apt-get update && apt-get dist-upgrade -y reboot now
After the reboot is complete, install BIND9 and all of the companion packages. On Ubuntu Server 22.04, we did as root:
apt-get install bind9 bind9utils bind9-doc
BIND9 is now installed, but unconfigured. Let’s do that next.
BIND9 Servers’ Buildout – BIND9 Configuration Work
We have two BIND9 configuration files to edit: /etc/bind/named.conf.options and /etc/bind/named.conf.
For /etc/bind/named.conf.options, we need first to add an access control list to limit which hosts can query BIND9. A bigger issue is that big forwarders are often blocked by a number of the free blocklist providers referenced by SpamAssassin’s default configurations. If you grep /var/log/zimbra.log for like “URIBL_BLOCKED” then you know that the forwarder you are using is being blocked. To address this, we want our Zimbra servers’ forward and reverse zones to be handled by the AWS Route 53 forwarder, but we want all other queries made by BIND9 to go to Cloudfare and Google. We therefore need to specify in this file the forward and reverse zones, and let BIND9 know that it should use the AWS Route 53 default IP for those zones, but continue to use the specified forwarders for all other lookups; you see the “zone” sections below that do this for us.
For /etc/bind/named.conf, I like to turn up logging. In this way, if there are any issues (e.g. timeouts, lame server responses, however the forwarders are responding etc.), we’ll have that data captured in the log files. If you are using centralized syslogging, be sure to update the rsyslog configs on the BIND9 servers to ship these logs to your centralized syslog server.
As regards DNSSEC, Amazon Web Services Route 53 doesn’t allow you to configure DNSSEC on Private zones; their support team says DNSSEC is for public Internet use only. But when you turn on DNSSEC validation in BIND9, because Route 53 will mark the parent zone as secured, all lookups in the Private forward zone (and the Reverse zones) will fail. This means Zimbra servers won’t be able to find each other via DNS at all! We therefore need to exclude the zones for which we use Route 53 as forwarders from DNSSEC validation lookups
So here’s what /etc/bind/named.conf.options looks like:
acl goodclients {
// Consider tighter restrictions, e.g. instead of using subnets,
// list only the /32 IPs of your Zimbra servers. The networks
// below refer to each of the three AWS Availability Zones used.
172.31.0.0/19;
172.31.16.0/19;
172.31.32.0/19;
localhost;
localnets;
};
zone "missioncriticalemail.com" {
type forward;
forwarders { 172.31.0.2; };
forward only;
};
zone "31.172.in-addr.arpa" {
type forward;
forwarders { 172.31.0.2; };
forward only;
};
options {
directory "/var/cache/bind";
forwarders {
1.1.1.1;
8.8.8.8;
1.0.0.1;
8.8.4.4;
};
qname-minimization disabled;
recursion yes;
allow-recursion { goodclients; };
allow-transfer { none; };
allow-query { goodclients; };
auth-nxdomain no; # conform to RFC1035
empty-zones-enable no;
dnssec-validation auto;
validate-except
{
"missioncriticalemail.com";
"31.172.in-addr.arpa";
};
listen-on-v6 { any; };
};
And here’s what /etc/bind/named.conf looks like, after we break out logging into several different files for different kind os events:
// This is the primary configuration file for the BIND DNS server named.
//
// Please read /usr/share/doc/bind9/README.Debian.gz for information on the
// structure of BIND configuration files in Debian, *BEFORE* you customize
// this configuration file.
//
// If you are just adding zones, please do that in /etc/bind/named.conf.local
include "/etc/bind/named.conf.options";
include "/etc/bind/named.conf.local";
include "/etc/bind/named.conf.default-zones";
logging {
channel default_log {
file "/var/lib/bind/log/default" versions 3 size 20m;
print-time yes;
print-category yes;
print-severity yes;
severity info;
};
channel auth_servers_log {
file "/var/lib/bind/log/auth_servers" versions 100 size 20m;
print-time yes;
print-category yes;
print-severity yes;
severity info;
};
channel dnssec_log {
file "/var/lib/bind/log/dnssec" versions 3 size 20m;
print-time yes;
print-category yes;
print-severity yes;
severity info;
};
channel zone_transfers_log {
file "/var/lib/bind/log/zone_transfers" versions 3 size 20m;
print-time yes;
print-category yes;
print-severity yes;
severity info;
};
channel ddns_log {
file "/var/lib/bind/log/ddns" versions 3 size 20m;
print-time yes;
print-category yes;
print-severity yes;
severity info;
};
channel client_security_log {
file "/var/lib/bind/log/client_security" versions 3 size 20m;
print-time yes;
print-category yes;
print-severity yes;
severity info;
};
channel rate_limiting_log {
file "/var/lib/bind/log/rate_limiting" versions 3 size 20m;
print-time yes;
print-category yes;
print-severity yes;
severity info;
};
channel queries_log {
file "/var/lib/bind/log/queries" versions 150 size 20m;
print-time yes;
print-category yes;
print-severity yes;
severity info;
};
channel query-errors_log {
file "/var/lib/bind/log/query-errors" versions 5 size 20m;
print-time yes;
print-category yes;
print-severity yes;
severity dynamic;
};
channel default_debug {
print-time yes;
print-category yes;
print-severity yes;
file "named.run";
severity dynamic;
};
category default { default_debug; default_log; };
category config { default_debug; default_log; };
category dispatch { default_debug; default_log; };
category network { default_debug; default_log; };
category general { default_debug; default_log; };
category resolver { auth_servers_log; default_debug; };
category cname { auth_servers_log; default_debug; };
category delegation-only { auth_servers_log; default_debug; };
category lame-servers { auth_servers_log; default_debug; };
category edns-disabled { auth_servers_log; default_debug; };
category dnssec { dnssec_log; default_debug; };
category notify { zone_transfers_log; default_debug; };
category xfer-in { zone_transfers_log; default_debug; };
category xfer-out { zone_transfers_log; default_debug; };
category update{ ddns_log; default_debug; };
category update-security { ddns_log; default_debug; };
category client{ client_security_log; default_debug; };
category security { client_security_log; default_debug; };
category rate-limit { rate_limiting_log; default_debug; };
category spill { rate_limiting_log; default_debug; };
category database { rate_limiting_log; default_debug; };
category queries { queries_log; };
category query-errors {query-errors_log; };
};
After completing the above changes, let’s lint the configuration. As root run:
named-checkconf
If you get nothing back, that means all of your syntax is OK. It doesn’t necessarily mean you have a working configuration, but at this point you should restart BIND9 to find out for sure. As root:
service bind9 restart && service bind9 status
If BIND9 restarts OK, terrific! If not, you can look through /var/log/syslog for errors and address them.
BIND9 QA Testing
We still need to test that BIND9 resolves critical lookups correctly. By “critical” I mean that BIND9 performs successful forward and reverse lookups for all of your Zimbra servers, and is also able to resolve external hosts. To perform these tests, pick one of your Zimbra servers to use to test, let’s presume it’s proxy1.mydomain.com. On each and every BIND9 server, you can run:
dig @localhost proxy1.mydomain.com +short
You should get back just the private IP address of that Zimbra proxy server. Let’s presume the IP that comes back is 172.31.0.45. Using that IP address, let’s next QA reverse lookups:
dig @localhost -x 172.31.0.45 +short
You should get back the FQDN “proxy1.mydomain.com”. It’s a good idea to repeat the above two tests for all of your Zimbra server’s FQDNs, and, again, on all of the BIND9 servers.
Lastly, let’s confirm external lookups are OK, since we need these for Zimbra’s MTA to function:
dig @localhost www.yahoo.com +short
To eliminate the possibility of any networking/firewall issues between Zimbra and the BIND9 servers, I would suggest rerunning the above tests from each of your Zimbra servers, but instead of using “@localhost” in the dig command, use instead “@172.31.0.202” etc. to reference the IP of each of the three BIND9 DNS servers and confirm that every Zimbra server can perform lookups on every BIND9 server.
On the presumption that all of your BIND9 servers pass the above QA tests, we are now ready to configure Zimbra to use the three new BIND9 servers you have just built.
Configure Zimbra To Use BIND9
This step is pretty straightforward (usually). All we need to do is comment out in /etc/resolv.conf the existing nameserver entries pointing to the existing DNS servers we have been using, and add entries for our new BIND9 servers. If our new BIND9 servers have the IP addresses 172.31.0.202, 172.31.16.64 and 172.31.32.125, and our existing DNS server has the IP address 172.31.5.49, then we want to edit /etc/resolv.conf to look like this:
# Remove existing DNS server: # nameserver 172.31.5.49 # Add new BIND9 caching DNS servers: nameserver 172.31.0.202 nameserver 172.31.16.64 nameserver 172.31.32.125 search mycompany.com
As soon as you make the above changes on each of your Zimbra servers, Zimbra will start using the new BIND9 DNS servers.
If you need help with setting all this up, or with any other Zimbra issues, just reach out using the form below:
Hope that helps,
L. Mark Stone
Mission Critical Email LLC
29 March 2023
16 April 2024 – Updated to show enhanced logging options and use of Google and Cloudfare public DNS as forwarders
The information provided in this blog is intended for informational and educational purposes only. The views expressed herein are those of Mr. Stone personally. The contents of this site are not intended as advice for any purpose and are subject to change without notice. Mission Critical Email makes no warranties of any kind regarding the accuracy or completeness of any information on this site, and we make no representations regarding whether such information is up-to-date or applicable to any particular situation. All copyrights are reserved by Mr. Stone. Any portion of the material on this site may be used for personal or educational purposes provided appropriate attribution is given to Mr. Stone and this blog.
