Fail2Ban and PCRE Script Using Zimbra’s Daily Mail Report

Fail2Ban and PCRE Script Using Zimbra’s Daily Mail Report

Zimbra Tips

While Fail2Ban’s automation works great, we find that reviewing the “Recipient address rejected” section of Zimbra’s Daily Mail Report yields evidence of potential Advanced Persistent Threats as well as probes from senders whom you’d like to block.

Manually grepping through /var/log/zimbra.log* on the logger host to find the sender’s IP you’d like to block is time consuming, so we developed a script to automate much of the process.

Script Deliverables
First, the script will output a Fail2Ban command you can use to block IPs that are sending email to non-existent email addresses that you identified from the Daily Mail Report.  You’ll need to have already configured Fail2Ban on your MTA servers; you can use our guide to do this if you haven’t done so already.

Second, since we can’t really block the sending IP addresses for Google and Microsoft, the script will output a list of domains that are sending from Microsoft, Google (and other domains you wish to be excluded), so you can add those offending sending domains to your PCRE blocking file. The script will output these domains pre-formatted to add to /opt/zimbra/conf/sender_pcre.  If you haven’t yet configured per-domain blocking, you can follow our guide to do so.

Here’s what we do, presuming you already have a good working installation of Fail2Ban on your combined Zimbra Proxy/MTA servers (if you have separate Proxy and MTA servers you’ll need to use a different jail on the Proxy servers):

  1. Install the script below on the Zimbra logger host.
  2. Review the Daily Mail Report in a plain-text window (so you are copying email addresses and not mailto links).
  3. Scroll through the “Recipient address rejected” section for suspicious entries.  Here are some examples:
    1. enjoyohbzsukvadvent@customerdomain.com
    2. VbgiB3bO7sJl_6opqv8Oc4niV_43355@customerdomain.com
    3. Groups of emails indicating a bad actor may be trying to target a user for a whaling attack (legitimate email verification services BTW do not do this):
      m.stone@customerdomain.com
      m_stone@customerdomain.com
      m-stone@customerdomain.com
  4. Enter the emails into the script one at a time (the script will prompt you); enter “done” when finished.
  5. The script will then:
    1. Output the lines from from /var/log/zimbra.log.1.gz that have the email address you searched for;
    2. List all of the senders’ IP addresses.
    3. Output a Fail2Ban command you can run on your MTA, or Proxy/MTA servers as is (just change the jail when you run the command on your Proxy servers, if you have separate Proxy servers.

It now takes a keen eye and about ten minutes per day to manually evaluate the “Recipient address rejected” section of the Daily Mail Report and run the proposed output on the Proxy/MTA servers.

False Positives?  Once in a great while we will come across a customer’s user who wants to for example download some white paper, and feeds the publisher’s intake form a make-believe email address — on the customer’s domain — not realizing the publisher uses that email address to send a link to download the white paper.  Once or twice a year, we find we have banned one of the publisher’s sending IPs, and then some other customers who expect communication from that publisher (or the publisher) alerts us, and we remove the IP ban.  We have over the years educated our own customers not to use fake email addresses like that, so really this happens when we onboard a new customer and once in a while a user at the customer ignores our recommended best practices. This happens maybe once a year in our experience, but it might be a greater risk for you, so we thought we should mention it.

Note that the script will not ban Microsoft, Google and similar sending servers’ IPs.  Microsoft has publicly stated on the Mail Operators Mailing List that their policy is not to ban any outgoing email.  This is why, even if you have a Microsoft 365 account, you’ll find that most of your spam notifications are from Microsoft 365 tenants.  If you need to prevent the script from banning IPs of other senders’ domains, just add those domains in the line in Step 2 in the script body.

Here’s the script:

#!/bin/bash
#
# Copyright 2024 Mission Critical Email, LLC. All rights reserved.
#
# Step 0 - INSTALLATION AND USAGE:
# - Install this command to /opt/zimbra on the logger host.
# - Modify Step 2a. in the code to remove the subnets of your Zimbra server(s).
# - Run the command as the zimbra Linux user.
#

echo "Getting all domains on your system; please wait..."
echo ""

# Function to escape regex special characters in domain names
escape_domain() {
echo "$1" | sed 's/\./\\./g; s/-/\\-/g'
}

# Function to get list of Zimbra-hosted domains
get_zimbra_domains() {
local domains
mapfile -t domains < <(zmprov gad | sort)
echo "${domains[@]}"
}

# Store Zimbra domains in an array
zimbra_domains=($(get_zimbra_domains))

# Function to parse the log and execute the script logic
parse_log() {
local log_file=$1
echo ""
echo "Processing log file: $log_file"
# Step 1: Prompt user for text strings
echo ""
echo "Enter suspicious text strings/email addresses (one at a time) from"
echo "the 'Recipient address rejected' sections of the Daily Mail Report."
echo "(type 'done' to finish):"
text_strings=()
while true; do
read -p "> " input
if [[ "$input" == "done" ]]; then
break
fi
text_strings+=("$input")
done

# Check if no text strings were entered
if [ ${#text_strings[@]} -eq 0 ]; then
echo "No text strings entered. Exiting."
exit 1
fi

# Escape each text string to handle special characters
escaped_text_strings=()
for str in "${text_strings[@]}"; do
escaped_text_strings+=("$(printf '%q' "$str")")
done

# Combine text strings into a single pattern
grep_pattern=$(IFS='|'; echo "${escaped_text_strings[*]}")
# Debug: print the grep pattern
echo ""
echo "Grep pattern: $grep_pattern"

# First get all matching lines before filtering
all_matches=$(zgrep -E "$grep_pattern" "$log_file")

# Then get filtered output for IP processing
# Exclude domains we cannot ban and other false positive strings
zgrep_output=$(echo "$all_matches" | grep -Ev "Service unavailable|Sender address rejected: Access denied|127.0.0.1|saslauthd|microsoft|google|protection\.outlook\.com|antispamcloud\.com")

# Debug: print the zgrep output
echo ""
echo "zgrep output:"
echo "$zgrep_output"

# Step 2a: Parse the output for IP addresses
declare -A ip_addresses
while IFS= read -r line; do
# Extract all IP addresses from the line using grep with -o option
while IFS= read -r extracted_ip; do
# Remove the brackets from the IP
ip=$(echo "$extracted_ip" | tr -d '[]')
# Skip empty IP and ignore specific IP ranges (e.g. the IPs of your Zimbra servers).
if [[ -n "$ip" ]] && ! [[ "$ip" =~ ^10\.7\.57\.[0-9]+$ || "$ip" =~ ^10\.8\.[0-9]+\.[0-9]+$ || "$ip" =~ ^127\.0\.[0-9]+\.[0-9]+$ ]]; then
ip_addresses["$ip"]=1
fi
done < <(echo "$line" | grep -oE '\[[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+\]')
done <<< "$zgrep_output"

# Debug: print the filtered IP addresses
echo ""
echo "Filtered IP addresses:"
for ip in "${!ip_addresses[@]}"; do
echo "$ip"
done

# Check if no IP addresses were found
if [ ${#ip_addresses[@]} -eq 0 ]; then
echo "No applicable IP addresses found in $log_file."
fi

# Step 3: Output the fail2ban commands
if [ ${#ip_addresses[@]} -gt 0 ]; then
fail2ban_command="fail2ban-client set zimbra-smtp banip"
for ip in "${!ip_addresses[@]}"; do
fail2ban_command+=" $ip"
done
echo ""
echo "Here is the Fail2Ban command to run:"
echo "$fail2ban_command"
fi

# Step 4: Process domains from filtered entries
echo ""
echo "Domains to add to PCRE blocklist:"
echo "--------------------------------"

# Get the entries that were filtered out (containing microsoft, google, etc.)
filtered_entries=$(echo "$all_matches" | grep -E "Service unavailable|Sender address rejected: Access denied|microsoft|google|protection\.outlook\.com|antispamcloud\.com")

declare -A domains
declare -A suspicious_customer_domains
declare -A suspicious_domain_logs
while IFS= read -r line; do
# Skip entries with null sender or filter triggers
if [[ "$line" =~ "Sender address triggers FILTER smtp-amavis" ]] || [[ "$line" =~ "from=<>" ]]; then
continue
fi

# Extract from=<user@domain.com> pattern and get the domain
if [[ $line =~ from=\<[^@]+@([^>]+)\> ]]; then
domain="${BASH_REMATCH[1]}"
# Skip common domains, localhost, and Zimbra-hosted domains
if [[ ! $domain =~ ^(gmail\.com|outlook\.com|hotmail\.com|microsoft\.com|googlemail\.com|antispamcloud\.com|google\.com|docusign\.net|localhost|localdomain)$ ]]; then
if [[ " ${zimbra_domains[@]} " =~ " ${domain} " ]]; then
# This is a customer domain being used as a sender
suspicious_customer_domains["$domain"]=1
# Store the log line for this domain
if [ -z "${suspicious_domain_logs[$domain]}" ]; then
suspicious_domain_logs["$domain"]="$line"
else
suspicious_domain_logs["$domain"]="${suspicious_domain_logs[$domain]}"$'\n'"$line"
fi
else
# Not a customer domain, add to PCRE block list
escaped_domain=$(escape_domain "$domain")
domains["$escaped_domain"]=1
fi
fi
fi
done <<< "$filtered_entries"

# Output the domains in PCRE format
for domain in "${!domains[@]}"; do
echo "/${domain}/ reject"
done
echo ""

# Check if any customer domains were found in suspicious activity
if [ ${#suspicious_customer_domains[@]} -gt 0 ]; then
echo "SECURITY ALERT: Potential Account Compromise"
echo "----------------------------------------"
echo "Please check the following log entries. The following customer domain(s)"
echo "have been used as sender addresses for emails to non-existent recipients;"
echo "please check for any potential account exploitations:"
echo ""
for domain in "${!suspicious_customer_domains[@]}"; do
echo "Domain: $domain"
echo "Relevant log entries:"
echo "${suspicious_domain_logs[$domain]}"
echo ""
done
fi
}

# Initial processing with default log file
parse_log "/var/log/zimbra.log.1.gz"

# Step 5: Ask if the user wants to process more log files
while true; do
echo "Would you like to process another log file? (y/n)"
read -p "> " answer
if [[ "$answer" != "y" ]]; then
echo "OK then, we are done here! Exiting script."
break
fi

echo "Select a log file to process:"
echo "1) /var/log/zimbra.log.2.gz"
echo "2) /var/log/zimbra.log.3.gz"
echo "3) /var/log/zimbra.log.4.gz"
echo "4) /var/log/zimbra.log"
read -p "> " log_choice

case $log_choice in
1)
parse_log "/var/log/zimbra.log.2.gz"
;;
2)
parse_log "/var/log/zimbra.log.3.gz"
;;
3)
parse_log "/var/log/zimbra.log.4.gz"
;;
4)
parse_log "/var/log/zimbra.log"
;;
*)
echo "Sorry; that's an invalid log file selection. Exiting script."
break
;;
esac
done

If you’d like help with your Fail2Ban configuration, please use this form to connect with us!

 

 

Hope that helps,
L. Mark Stone
Mission Critical Email LLC
12 October 2024

The information provided in this blog is intended for informational and educational purposes only. The views expressed herein are those of Mr. Stone personally. The contents of this site are not intended as advice for any purpose and are subject to change without notice. Mission Critical Email makes no warranties of any kind regarding the accuracy or completeness of any information on this site, and we make no representations regarding whether such information is up-to-date or applicable to any particular situation. All copyrights are reserved by Mr. Stone. Any portion of the material on this site may be used for personal or educational purposes provided appropriate attribution is given to Mr. Stone and this blog.