Email Server With Postfix Dovecot and MailScanner (Part 3 - MailScanner and MailWatch)

Posted on   |   In   |  

I use MailScanner to scan emails for viruses, spam, phishing, malware, and other attacks against security vulnerabilities. Under the hood, MailScanner uses ClamAV(clamd) for virus scan, and uses Spamassassin to scan for spams.

MailWatch is the web UI frontend to manage MailScanner. I can manage qurantine and generate reports easily right in the web browser.

MailScanner come with a installation script you may use it for easy setup. It will install all compoenets once you made your selection.

ClamAV

Clam AV can be load in 3 different mode. Here is the explaination:

clamscan: most expensive CPU-wise, but involves no extra setup. This just executes the clamscan command-line tool. This causes the signature database to be re-read for each object scanned and can be pretty CPU intensive compared to the others.
clamav module: less expensive than clamscan CPU-wise, but needs the Mail::ClamAV perl module. This method loads a copy of the libclamav scanner library into MailScanner and keeps it resident, using it to perform scans without needing to re-read the signature libraries, etc. It can be somewhat touchy about what versions of Mail::ClamAV work with various versions of clamav.
clamd: less expensive than clamscan CPU-wise, but needs clamd running and is relatively new code. This causes MailScanner to connect to clamd’s socket and use that for scanning. Since clamd is already resident, there’s no need to re-read signatures. Since it’s using clamd, which comes with clamav, there’s no real version-compatibility problems like with the module, at least in theory

So I decide to run ClamAV as a daemon (clamd) for better performance.

ClamAVInstallation

Install ClamAV:

yum install clamav-server clamav-data clamav-update clamav-filesystem clamav clamav-scanner-systemd clamav-devel clamav-lib clamav-server-systemd

ClamAV will need unrar, it can be installed from rpmforge repository, so:

cd /usr/local/src/
wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el7.rf.x86_64.rpm
rpm -ivh rpmforge-release-0.5.3-1.el7.rf.x86_64.rpm 
yum install unrar

After install unrar, I disabled rpmforge repo:

sed -i 's/enabled = 1/enabled = 0/g' /etc/yum.repos.d/rpmforge.repo

freshclam

freshclam will update the virus signature database. To enable it and update:

sed -i -e 's/^Example/#Example/' /etc/freshclam.conf
freshclam -v

Also edit /etc/sysconfig/freshclam, comment out this line as:

#FRESHCLAM_DELAY=disabled-warn	# REMOVE ME

The MailScanner will call /usr/local/bin/freshclam to update the database, so make the proper link:

ln -s /usr/bin/freshclam /usr/local/bin/freshclam

clamd daemon

Enable clamd by editing /etc/clamd.d/scan.conf like this:

# Example
LogFile /var/log/clamd.scan
LogTime yes
LogSyslog yes
LogFacility LOG_MAIL
PidFile /var/run/clamd.scan/clamd.pid
LocalSocket /var/run/clamd.scan/clamd.sock

Create the log file:

touch /var/log/clamd.scan
chown :clamscan $_
chmod 0660 $_

Now enable and start the service:

systemctl  enable clamd@scan
systemctl  start clamd@scan
systemctl  status clamd@scan

Test clamd service

I download a virus file (Eicar Test) and send to ClamAV

# cd /tmp
# wget http://www.eicar.org/download/eicar.com 
# clamscan --infected --remove eicar.com
eicar.com: Eicar-Test-Signature FOUND
eicar.com: Removed.
 
----------- SCAN SUMMARY -----------
Known viruses: 4283601
Engine version: 0.98.7
Scanned directories: 0
Scanned files: 1
Infected files: 1
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 21.513 sec (0 m 21 s)

Spamassassin

Install Spamassassin”

yum install spamassassin

Update database:

sa-update

Enable snd start Spamassassin:

systemctl enable spamassassin
systemctl start spamassassin
systemctl status spamassassin

MailScanner

MailScanner work like this:

  1. As instructed, Postfix holds the mail upon receipt.
  2. MailScanner swoops in and scans the email in queue.
  3. MailScanner re queues the email and hands it over back to Postfix.
  4. Postfix processes the email as necessary and delivers the mail to recipient.

Install MailScanner

First stop and disable postfix. We will use MailScanner in the future.

systemctl stop postfix
systemctl disable postfix

Download MailScanner and install:

cd /usr/local/src/
wget https://s3.amazonaws.com/mailscanner/release/v4/rpm/MailScanner-4.85.2-3.rpm.tar.gz
tar zvxf MailScanner-4.85.2-3.rpm.tar.gz 
cd MailScanner-4.85.2-3
./install.sh

Start the installation. Answer Y to all questions except these three(they have been took care in the previous steps):

Do you want to install a Mail Transfer Agent (MTA)?

I can install an MTA via the Yum package manager to save you the trouble of having to do
this later. If you plan on using an MTA that is not listed below, you will have install 
it manually yourself if you have not already done so.

1 - sendmail
2 - postfix
3 - exim
N - Do not install

Recommended: 1 (sendmail)

Install an MTA? [1] : N
Do you want to install or update Spamassassin?

This package is recommended unless you have your own spam detection solution.

Recommended: Y (yes)

Install or update Spamassassin? [n/Y] : N
Do you want to install or update Clam AV during this installation process?

This package is recommended unless you plan on using a different virus scanner.
Note that you may use more than one virus scanner at once with MailScanner.

Even if you already have Clam AV installed you should select this option so I
will know to check the clamav-wrapper and make corrections if required.

Recommended: Y (yes)

Install or update Clam AV? [n/Y] : N

MailScanner Configuration

Edit /etc/MailScanner/MailScanner.conf

%org-name% = mydomain
%org-long-name% = mydomain Ltd.
%web-site% = www.mydomain.com
Incoming Work Group = clamscan
Incoming Work Permissions = 0640
Virus Scanners = clamd
Clamd Socket = /var/run/clamd.scan/clamd.sock
Clamd Use Threads = yes
MTA = postfix
Run As User = postfix
Run As Group = postfix
Incoming Queue Dir = /var/spool/postfix/hold
Outgoing Queue Dir = /var/spool/postfix/incoming
Use SpamAssassin = yes
SpamAssassin User State Dir = /var/spool/MailScanner/spamassassin

SpamScore Number Instead Of Stars = yes
Always Include SpamAssassin Report = yes
Log Spam = yes

Correct a permission to allow write for group clamscan:

chmod -R 770 /var/spool/MailScanner/incoming/

Edit /etc/MailScanner/spam.assassin.prefs.conf

envelope_sender_header X-mydomain-MailScanner-From

Edit /etc/MailScanner/virus.scanners.conf

clamd           /bin/false                           /usr

clamd permission

When clamd scan emails, I want pass --fdpass to it, so it won’t have permssion issue:

mv /usr/bin/clamdscan /usr/bin/clamdscan-cmd

Now create a new file /usr/bin/clamdscan

#!/bin/bash
/usr/bin/clamdscan-cmd --fdpass $@

Make it executable:

chmod +x /usr/bin/clamdscan

Postfix hold queue

Let postfix hold all mails for scan, add line at bottom of /etc/postfix/header_checks

/^Received:/  HOLD

Enable header check in Postfix, edit /etc/postfix/main.cf, uncomment this line 548:

header_checks = regexp:/etc/postfix/header_checks

Check MailScanner configration to see if there is any error:

MailScanner --lint

Spamassassin Plugins

With plugins, Spamassassin can detect spam and bulk email better with online resources.

First, I need open some ports on iptables needed by DCC, pyzor and razor. Add these rules to /etc/sysconfig/iptables in the INPUT chain and reload iptables:

### razor DCC pyzor ###
-A INPUT -p tcp --dport 2703 -j ACCEPT
-A INPUT -p udp --dport 24441 -j ACCEPT
-A INPUT -p udp -m udp --dport 1024:65535 --sport 6277 -j ACCEPT
###End of razor DCC pyzor ###

Edit /etc/mail/spamassassin/mailscanner.cf

# paths to utilities
ifplugin Mail::SpamAssassin::Plugin::Pyzor
pyzor_path /usr/bin/pyzor
endif

ifplugin Mail::SpamAssassin::Plugin::DCC
dcc_path /usr/bin/dccproc
dcc_home /etc/dcc
endif

ifplugin Mail::SpamAssassin::Plugin::Razor2
razor_config  /etc/mail/spamassassin/razor/razor-agent.conf
endif

Enable these in spamassassin. Edit /etc/mail/spamassassin/v310.pre

loadplugin Mail::SpamAssassin::Plugin::DCC
loadplugin Mail::SpamAssassin::Plugin::Pyzor
loadplugin Mail::SpamAssassin::Plugin::Razor2

DCC

Install DCC

yum install sendmail-milter
rpm -Uvh  https://www.mirrorservice.org/sites/dl.atrpms.net/el7-x86_64/atrpms/stable/DCC-1.3.145-25.el7.x86_64.rpm

Test

cdcc info

Pyzor

Add a line in /etc/mail/spamassassin/local.cf

pyzor_options --homedir /etc/mail/spamassassin/.pyzor

Install Pyzor:

rpm -Uvh ftp://mirror.switch.ch/pool/4/mirror/fedora/linux/releases/22/Everything/x86_64/os/Packages/p/pyzor-0.5.0-10.fc21.noarch.rpm
pyzor --homedir /etc/mail/spamassassin discover

Test

spamassassin -t -D pyzor < /usr/share/doc/spamassassin-3.4.0/sample-spam.txt

Razor

mkdir  /etc/mail/spamassassin/razor
razor-admin -create -home=/etc/mail/spamassassin/razor
razor-admin -register

Edit /etc/mail/spamassassin/razor/razor-agent.conf

razorhome	       = /etc/mail/spamassassin/razor

Test Razor2

spamassassin -t -D razor2 < /usr/share/doc/spamassassin-3.4.0/sample-spam.txt

Test

Check MailScanner configration again:

MailScanner --lint

Also check for SpamAssassin:

spamassassin -D --lint

Now restart services and check maillog to see if any error

systemctl restart clamd@scan
systemctl restart spamassassin
systemctl restart MailScanner

Now I can send some spam test email then check the maillog to see if it has need catched. Here are some test site:
[http://www.emailsecuritycheck.net/]
[https://www.mail-tester.com/]

MailWatch

Download code

cd /usr/local/src
git clone https://github.com/mailwatch/1.2.0.git

MariaDB database

Create a database with downloaded sql file:

cd /usr/local/src/1.2.0
mysql -uroot -p < create.sql

Create a DB user:

# mysql -uroot -p
Enter password:

MariaDB [(none)]> GRANT ALL ON mailscanner.* TO mailwatch@localhost IDENTIFIED BY 'MWpassword';
Query OK, 0 rows affected (1.27 sec)

MariaDB [(none)]> GRANT FILE ON *.* TO mailwatch@localhost IDENTIFIED BY 'MWpassword';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.05 sec)

Admin user

Create an Admin user gao:

# mysql mailscanner -umailwatch -pMWpassword

MariaDB [mailscanner]> INSERT INTO users SET username = 'gao', password = md5('mwpassword'), fullname = 'Gao', type = 'A';

Configure MailWatch

Move the mailscanner directory to the web server’s root

cp -r /usr/local/src/1.2.0/mailscanner /var/www/html/.
chown -R nginx:nginx /var/www/html/mailscanner/

Copy /var/www/html/mailscanner/conf.php.example to conf.php then edit the database setting:

define('DB_TYPE', 'mysql');
define('DB_USER', 'mailwatch');
define('DB_PASS', 'MWpassword');
define('DB_HOST', 'localhost');
define('DB_NAME', 'mailscanner');
define('DB_DSN',  DB_TYPE.'://'.DB_USER.":".DB_PASS."@".DB_HOST."/".DB_NAME);

define('TIME_ZONE', 'America/Vancouver');
// Log file location
define('MS_LOG', '/var/log/maillog');
define('MAIL_LOG', '/var/log/maillog');
// Date/Time settings
define('DATE_FORMAT', '%y-%m-%d');
define('TIME_FORMAT', '%H:%i:%s');

Now stop MailScanner

systemctl stop MailScanner

Edit /etc/MailScanner/MailScanner.conf

Always Looked Up Last = &MailWatchLogging
Detailed Spam Report = yes
Quarantine Whole Message = yes
Quarantine Whole Messages As Queue Files = no
Include Scores In SpamAssassin Report = yes
Quarantine User = root
Quarantine Group = nginx
Quarantine Permissions = 0660
Is Definitely Not Spam = &SQLWhitelist
Is Definitely Spam = &SQLBlacklist

Copy Perl module to CustomFunctions

cp /usr/local/src/1.2.0/MailScanner_perl_scripts/SQLBlackWhiteList.pm /usr/share/MailScanner/MailScanner/CustomFunctions/
cp /usr/local/src/1.2.0/MailScanner_perl_scripts/MailWatch.pm /usr/share/MailScanner/MailScanner/CustomFunctions/.

Edit /usr/share/MailScanner/MailScanner/CustomFunctions/SQLBlackWhiteList.pm

sub CreateList {
  my($type, $BlackWhite) = @_;
  my($dbh, $sth, $sql, $to_address, $from_address, $count, $filter);
  my($db_name) = 'mailscanner';
  my($db_host) = 'localhost';
  my($db_user) = 'mailwatch';
  my($db_pass) = 'MWpassword';
...

Edit /usr/share/MailScanner/MailScanner/CustomFunctions/MailWatch.pm

# Modify this as necessary for your configuration
my($db_name) = 'mailscanner';
my($db_host) = 'localhost';
my($db_user) = 'mailwatch';
my($db_pass) = 'MWpassword';
...

I also installed a perl module Encoding::FixLatin in CPAN:

# cpan

cpan[1]> install Encoding::FixLatin
...

Start MailScanner again, check /var/log/maillog for any error.

Nginx configuration

Create a new nginx virtual host configration file /etc/nginx/conf.d/mailwatch.conf

server {

  listen 80;
  server_name mailwatch.mydomain.com;
  return 301 https://$server_name$request_uri; # enforce https

}

server {

   listen          443 ssl;
   server_name     mailwatch.mydomain.com;
   root            /var/www/html/mailscanner;
   index           index.php;
   charset         utf-8;

   ## SSL settings
   ssl_certificate           /etc/letsencrypt/live/mydomain.com/fullchain.pem;
   ssl_certificate_key           /etc/letsencrypt/live/mydomain.com/privkey.pem;
   ssl_protocols             TLSv1.2 TLSv1.1 TLSv1;
   ssl_prefer_server_ciphers on;
   ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4";
   ssl_dhparam		     /etc/nginx/dhparams.pem;	
   ssl_session_cache         shared:SSL:10m;
   ssl_session_timeout       10m;
   ssl_ecdh_curve            secp521r1;

   add_header Strict-Transport-Security max-age=31536000;
   # add_header X-Frame-Options DENY;

   # auth_basic "Restricted area";
   # auth_basic_user_file /etc/nginx/passwd;

   location / {
      try_files $uri $uri/ index.php;
   }

   location ~ \.php$ {
        try_files $uri =404;
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include       fastcgi_params;
        fastcgi_pass  unix:/run/php-fpm/php-fpm.sock;
        fastcgi_index index.php;
   }

}

Mail queue

Configure mail queue directory:

chown -R postfix:nginx /var/spool/postfix/hold
chown -R postfix:nginx /var/spool/postfix/incoming
chmod -R 770 /var/spool/postfix/hold
chmod -R 770 /var/spool/postfix/incoming/

Test

Restart Nginx:

systemctl restart nginx

Now it’s time to go to https://mailwatch.mydomain.com/
Login as admin user “gao” with password “mwpassword”

Go to http://www.emailsecuritycheck.net/ and send myself some spam and virus, check MailWatch to see the result.

There are few tools came with MailWatch souce code in the "tools" directory. You may need to setup them following the instruction.

SpamAssassin Bayes

SpamAssassin Bayes can try to identify spam (or ham) by learning tokens. Here I configrue it with MailWatch installed.

Edit /etc/MailScanner/spam.assassin.prefs.conf

bayes_path /etc/MailScanner/bayes/bayes
bayes_file_mode 0660

bayes_ignore_header X-mydomain-COM-MailScanner
bayes_ignore_header X-mydomain-COM-MailScanner-SpamCheck
bayes_ignore_header X-mydomain-COM-MailScanner-SpamScore
bayes_ignore_header X-mydomain-COM-MailScanner-Information

Create the ‘new’ bayes directory, make the directory owned by the same group as the web server user and make the directory setgid:

mkdir /etc/MailScanner/bayes
chown root:nginx /etc/MailScanner/bayes
chmod g+rws /etc/MailScanner/bayes

Since I already some spam mails in quarantine, so I force a sa-learn

sa-learn --spam /var/spool/MailScanner/quarantine/20160420

After this I see few files have been generated in bayes home directory:

# ll /etc/MailScanner/bayes/
total 28
-rw-rw---- 1 root nginx    50 Mar 11 21:18 bayes.mutex
-rw-rw---- 1 root nginx 12288 Mar 11 21:18 bayes_seen
-rw-rw---- 1 root nginx 12288 Mar 11 21:18 bayes_toks

Test to see bayes work

spamassassin -D -p /etc/MailScanner/spam.assassin.prefs.conf --lint

shoud see these lines:

Mar 11 21:20:27.020 [6494] dbg: bayes: tie-ing to DB file R/O /etc/MailScanner/bayes/bayes_toks
Mar 11 21:20:27.022 [6494] dbg: bayes: tie-ing to DB file R/O /etc/MailScanner/bayes/bayes_seen
Mar 11 21:20:27.024 [6494] dbg: bayes: found bayes db version 3
Mar 11 21:20:27.025 [6494] dbg: bayes: DB journal sync: last sync: 0
Mar 11 21:20:27.026 [6494] dbg: bayes: not available for scanning, only 2 spam(s) in bayes DB < 200
Mar 11 21:20:27.026 [6494] dbg: bayes: untie-ing

It can also be verified in MailWatch in “Tools/Link” –> “Spamassassin Bayes Database Info”

Now restart Spamassassin and MailScanner and check maillog.

Now I have an email server with protections. Unforturnatly this is just a start. There are so many things need further configuration to improve the security of the mail system.

Quick links: