Mailhub Table Building

The mailhub tables should be rebuilt periodically to purge deleted aliases. The operation will typically take about 4 hours per mailhub and has a considerable impact on the performance of the X.500 server. Care should be taken to carry out the table rebuilds at suitable times to avoid too much impact on other operations e.g. student registrations. The instructions below should ensure the rebuilds happen as quickly as possible before resuming normal operations.

Restart the X.500 directory:

The X.500 directory process suffers from a memory leak so before starting the rebuild we need to ensure that it starts off with minimum memory usage. Perform the following operations as root on the X.500 server –

# /etc/rc2.d/S81eddy stop

This will shut down the main eddy directory process but leave the root process /opt/isode/sbin/isode.eddy -D /dirtables/root running. Once the main process has stopped find the process ID of the root process and kill it. Then restart the directory –

# /etc/rc2.d/S81eddy start

The registration client processes on the mailhubs should now be restarted. Run top on the X.500 server to monitor progress and restart the client on each of the mailhubs in turn using the command –

# systemctl start regclient

Wait until cpu idle returns to 90%+ before starting the client on the each mailhub in turn.

Now check that the openRA is still running on cvshost.qub.ac.uk and that the UNS and alias threads are running. The student thread will usually still be running in the qubRA process on the X.500 server. It is best to close down the thread whilst the table rebuild is running as sometimes large batches of delta files may be created that add significantly to the load on the server.

Start the table rebuild on the target mailhub:

First shut down the registration client on the mailhub –

# systemctl stop regclient

Now start the table rebuild as a background process –

# /home/exim/scripts/rebuild &

Now wait for around 4 hours until the job completes.

Tidying Up:

The memory leak in the directory process will mean that it has grown to 500Mb+ so the first thing to do is restart the directory according to the instructions above. Once the directory is running again start the registration clients on each of the mailhubs as above. Begin with mailhub that has just run the table build as that will take longer.

Now restart the student thread on the qubRA. You may need to start the qubRA process if it has crashed during the directory restart. Monitor cpu usage whilst the delta files are cleared down. This may take a while if significant numbers were created during the rebuild process.

Check the UNS and alias threads on the openRA on cvshost.qub.ac.uk and make sure the mailbox creation service is running on ex2k10-365a.ads.qub.ac.uk.

ClamAV on Mailhubs and SMTP Servers

ClamAV is used on the mailhubs and SMTP servers for content scanning of email messages. It is called via ACLs in the Exim configuration. The daemon is installed as a binary using yum. We also use additional unofficial signature definitions from SaneSecurity. The main files of interest are –

  • /var/clamav – directory containing signature definitions
  • /etc/clamd.d/scan.conf – main configuration file
  • /etc/freshclam.conf – signature update configuration
  • /etc/clamav-unofficial-sigs/master.conf – additional signatures

Configuration changes will require a restart of clamd using the command –

# systemctl restart clamd

Freshclam:

The main signature file is updated by the freshclam daemon according to the instructions in the /etc/freshclam.conf file. The daemon is set to use the default of checking for new signatures every two hours. The signature file is daily.cld located in the /var/clamav directory. There is also a main.cvd file that is no longer updated. It has been left in place as it does not seem to be causing any problems.

Unofficial Signatures:

The additional signatures are visible in the /var/clamav directory in various database formats. The ones to be used are defined in the file /etc/clamav-unofficial-sigs/master.conf. These should be reviewed frequently as some signatures become too aggressive. Check the signature listings at SaneSecurity. They list the signatures as low, medium or high risk of false positives. We only use low and medium risk signatures.

New signature files are checked for and downloaded hourly. This process is controlled by the clamav-unofficial-sigs and clamav-update scripts in /etc/cron.d.

Whitelisting:

Rules can be whitelisted by adding the definition to the file /var/clamav/local.ign2. This file is regularly rsync’d from mx1.qub.ac.uk to the other three mailhubs, so it only requires a single edit. The file on the SMTP servers needs to be updated separately.

Updates:

Updates to the Clam installation are made available via yum. Be cautious when updating Clam as changes may have been made to the configuration options. It is wise to update on one server and test before completing the others.

EXIM on Mailhubs

File Locations:

  • /usr/exim – distribution installation directory
  • /usr/exim/configure – main configuration file (same across all hubs)
  • /usr/exim/configure.local – local server specific configuration
  • /etc/exim – directory containing utility lists and databases

The master copy of the main configuration file /usr/exim/configure is held on mailhub mx1.qub.ac.uk. The file is rsync’d to the other three mailhubs every 5 minutes using the root cron job –

# /root/scripts/rsync_exim_conf.sh

This means that you only have to make configuration changes to the file on mx1.qub.ac.uk but Exim needs to be restarted on each mailhub to effect any changes. Restart Exim using the command –

# systemctl restart exim

The file /usr/exim/configure.local is particular to each mailhub and changes must be made on each server. Again, Exim needs to be restarted if any changes are made.

Utility Lists and Databases:

The filenames in /etc/exim fall into two different categories: they are either flat text files or dbm database files. The flat text files all contain addresses, domains or IP addresses. The names should be self-explantory and they are used principally in the ACL section of the Exim configuration for blacklisting or whitelisting. These files are also rsync’d from mx1.qub.ac.uk to the other mailhubs as part of the script referred to above.

The database files are –

  • exch_table – list of local Exchange aliases
  • o365_table – list of Office 365 aliases
  • mlist_table – list of aliases from the Sympa list server
  • sm_table – list of St. Mary’s student aliases
  • local_domains – list of local domains

These database files are principally used for routing mail. They are built separately on each mailhub from correspondingly named text files with the suffix .dat – e.g. exch_table is built from exch_table.dat.

The first 2 tables are generated on each mailhub by the script /root/scripts/hub_tables.sh that utilises perl subscripts to extract mail alias information from Active Directory using LDAP. The data for the first two files is initially saved as –

  • exch_table_tmp.dat
  • o365_table_tmp.dat

These files are first checked by the script to see if they are larger than the previous data files. If they are equal or smaller, the database files are not updated. This is a safety feature to avoid the possibility of missing data from the database files. You need to check these files, particularly when numbers of accounts are being removed. If you are sure that the smaller temporary file is valid then copy it to the main data file and build the database file from it e.g.

# cp exch_table_tmp.dat exch_table.dat

# exim_dbmbuild exch_table.dat exch_table

The mlist_table database file is built from the mlist_table.dat file that is pulled across from cmc-mlist.qub.ac.uk using scp.

The local_domains file is built from local_domains.dat and that file should be updated as required.

The sm_table file is built from sm_table.dat. The information for that file should come from St. Mary’s or QSIS and usually only requires attention annually.

Sympa Mailing List Server

The server is currently running on Centos 5. The server also requires apache, mysql (or mariadb) and exim to be installed. Main file locations are –

  • /home/sympa – live distribution directory
  • /etc/sympa.conf – main configuration file
  • /etc/mail/sympa_aliases – main alias file

The sympa service can be stopped and started using the init.d scripts e.g.

# /etc/init.d/sympa start

Admin Console:

Access the management console via a browser using the URL –

http://cmc-mlist.qub.ac.uk/sympa

You need to create a login the first time you access it using your email address as the account name. This should match the list manager address set in /etc/sympa.conf.

When you create a new list always use cmc-mlist.qub.ac.uk as the domain name. Sometimes when a new list is created and then the domain name is subsequently changed the alias entries for the list in /etc/mail/sympa_aliases may become garbled. If that happens edit the entries in the aliases file and restart sympa.

Mailhub Routing File:

The script /root/scripts/mx_alias.sh produces the file /home/sympa/tmp/mlist_table.dat that can be used for routing on the mailhubs. The script is run hourly and the mailhubs can retrieve the file using scp.

Switch SMTP Servers

The University SMTP service is handled by the two hosts smtp1.qub.ac.uk and smtp2.qub.ac.uk. The service is normally handled by smtp1.qub.ac.uk and smtp2.qub.ac.uk is available as a backup. The active SMTP server is configured with the IP address for smtp.qub.ac.uk set as a secondary address. Switching the service requires the secondary IP address to be removed from the current active server and added to the network configuration on the current passive server. A small change also needs to made to the Exim configuration on each of the servers.

Exim Configuration

It is best to update the Exim configuration in advance of making changes to the network configuration. The SMTP transport section in the configuration file looks like this –

remote_smtp_52:
driver = smtp
port = 52
connect_timeout = 1m
interface = 143.117.16.195

Comment out the interface line on the active server and uncomment it on the target server. Make the changes active by restarting Exim on both servers after the network configurations (see below) have been updated. Make sure that eximon is running on both servers so you can check whether the SMTP traffic has successfully switched.

Network Configuration

Display the current IP address configuration on the active server –

# ip a show dev ens192 | grep inet

inet 143.117.16.80/24 brd 143.117.16.255 scope global noprefixroute ens192
inet 143.117.16.195/24 brd 143.117.16.255 scope global secondary noprefixroute ens192:1
inet6 fe80::250:56ff:fe8c:c405/64 scope link

The secondary address is highlighted. Remove the secondary address from the active server as follows –

# ip a del 143.117.16.195/24 dev ens192:1

Add the secondary address on the target server as follows –

# ip a add 143.117.16.195/24 dev ens192:1

Make these changes in quick succession on the active and target servers and then restart Exim on each using –

# systemctl restart exim

Monitor SMTP traffic on each server using eximon. It sometimes takes a couple of minutes for connecting hosts to pick up on the change depending on the refresh rate of their arp tables.

Add Authorised Domain in Office 365

The authorised domains set in the local Exchange service do not get synchronised into Office 365, so they need to be added independently. Open a powershell window and start a session to Office 365 – you will also need to connect to the MsolService (see Connect to Exchange Online). Run the following command to add a domain –

New-MsolDomain -Name new.domain -TenantId eaab77ea-b4a5-49e3-a1e8-d6dd23a1f286 -Authentication federated

Note: The TenantId can be found in the Azure Active Directory admin centre under properties

Build Exim from Source on Centos 7

Pre-requisites:

#yum install gcc

#yum install libdb libdb-devel

#yum install opendmarc libopendmarc libopendmarc-devel

#yum install libspf2 libspf2-devel

#yum install libX11 libX11-devel

#yum install libXt libXt-devel

#yum install libXaw libXaw-devel

#yum install openldap openldap-devel

Install iconv library –

1. Download cert-forensics-tools-release-el7 rpm:
https://forensics.cert.org/cert-forensics-tools-release-el7.rpm

2. Install cert-forensics-tools-release-el7 rpm:
# rpm -Uvh cert-forensics-tools-release*rpm

3. Install libiconv rpm package:
# yum –enablerepo=forensics install libiconv

Create exim user and group

Exim Build:

Download exim source code tar file (see exim.org) and unpack it. Switch to the main distribution directory. The install instructions are in the README file.

Edit the makefile as instructed to include the following functions –

LDAP lookup
PCRE
Content scan
DMARC
SPF
TLS
TLS and plaintext AUTH
ICONV

Then –

# make
# make install

Upgrade Exim:

Download the latest distribution from exim.org and unpack it as above. Switch to the main distribution directory and copy the Local/Makefile from the current source distribution into the Local directory. The ‘make install’ process should detect the presence of an existing configuration file (/usr/exim/configure) and should not overwrite it. You may want to take a copy of the configuration file before installing as a precaution.

Then –

# make
# make install

This will replace the old version with the new one. You will need to restart the daemon for it to take effect i.e.

# systemctl restart exim

Repair Federation Trust Checklist

  1. Document the existing trust settings (federated domains, federation settings)
    • Get-FederatedOrganizationIdentifier | select -ExpandProperty domains (local)
    • Get-FederationTrust |fl (local)
    • Get-OrganizationConfig |fl (local)
    • Get-OrganizationRelationship |fl (local)
    • Get-SendConnector |fl (local)
    • Get-ReceiveConnector |fl (local)
    • Get-FederatedOrganizationIdentifier | select -ExpandProperty domains (Office 365)
    • Get-AcceptedDomain | fl (Office 365)
    • Get-OutboundConnector | fl (Office 365)
    • Get-InboundConnector | fl (Office 365)
    • Get-RemoteDomain | fl (Office 365)
    • Get-OrganizationConfig | fl (Office 365)
    • Get-OrganizationRelationship | fl (Office 365)
    • Get-FederationTrust | fl (Office 365)
  1. Force remove each federated domain from the federation:

Remove-FederatedDomain -DomainName o365mail.qub.ac.uk -Force

  1. Remove the federation trust:

Remove-FederationTrust “Microsoft Federation Gateway”

  1. Wait for AD replication
  2. Create a new federation trust:

New-FederationTrust -Name “Microsoft Federation Gateway v2” -Thumbprint “E866E662B3B5C57BE72DA541978BADB6ECED6E74”

  1. Update the trust organisation information:

Get-Federationtrust | Set-FederationTrust –RefreshMetadata

  1. Add o365mail.qub.ac.uk to the federated organization identifier:

Set-FederatedOrganizationIdentifier -DelegationFederationTrust “Microsoft Federation Gateway v2 ” -AccountNamespace o365mail.qub.ac.uk -Enabled $true

  1. Configure the required settings in the trust (as per the documentation created in step 1).
  2. Wait for AD replication
  3. Test the certificate and trust (Test-FederationTrustCertificate, Test-FederationTrust) – it can take 12-48 hours before the trust reports as being no longer expired!
  4. Add the federated domain back into the trust (this will involve generating domain ‘Proof’ entries and adding them to your external DNS, then waiting for DNS propagation):

Add-FederatedDomain -DomainName o365mail.qub.ac.uk

Add-FederatedDomain -DomainName ads.qub.ac.uk

Get-AcceptedDomain -Identity o365mail.qub.ac.uk |fl

Get-RemoteDomain -Identity o365mail.qub.ac.uk |fl

Get-FederatedDomainProof -DomainName o365mail.qub.ac.uk -Thumbprint E866E662B3B5C57BE72DA541978BADB6ECED6E74

    1. Add the DnsRecord TXT string to the external facing DNS for the o365mail.qub.ac.uk domain.

Managing Distribution Groups

Adding Members

Members can be added to a mail-enabled, universal group using powershell commands.

Extract details of the members from an existing group using the Get-DistributionGroupMember command. You can limit the information obtained to a single field e.g. PrimarySmtpAddress. The following command will do just that –

Get-DistributionGroupMember cmc-team |select PrimarySmtpAddress |Export-Csv cmc-addr.csv

The file it produces will be of the form –

#TYPE Selected.Microsoft.Exchange.Data.Directory.Management.ReducedRecipient

“PrimarySmtpAddress”
“a.manager@qub.ac.uk”
“f.bloggs@qub.ac.uk”
“j.soap@qub.ac.uk”

You need to produce a CSV file in the same format to add a list of members to a group. Import the CSV file to the group as follows –

Import-Csv cmc-addr.csv | foreach {add-distributiongroupmember -Identity cmc-team -member $_.PrimarySmtpAddress}

Note: If you have a list of staff IDs instead of addresses you need to use the SamAccountname attribute instead. The CSV file needs to be in the following format –

#TYPE Selected.Microsoft.Exchange.Data.Directory.Management.ReducedRecipient
“SamAccountName”
“1234567”
“7654321”
“1212343”

The import command in this case would be –

Import-Csv cmc-sam.csv | foreach {add-distributiongroupmember -Identity cmc-team -member $_.SamAccountName}

Hiding List Membership

Sometimes you do not want recipients of messages sent via mail-enabled groups to be able to expand the recipient list. Do the following to hide the recipients –

  1. Open Active Directory Users and Computers console.
  2. Click View Menu and select Advanced Features (to enable it)
  3. Find your Distribution List or Security Group (mail enabled) in the OU. DO NOT search for your group because if you open the DL/Group properties from search windows, it will NOT show the Attribute Editor which is the advanced feature).
  4. Double click to open the DL/Group to see the properties. Go to Attribute Editor tab.
  5. Find hideDLMembership attribute, double click to open and select True. Click OK twice to close the dialog boxes.

It takes a while to take effect the changes while Exchange generates Offline Address Book and Outlook installations retrieve it.

Copy Contents From One Office 365 Mailbox to Another

Open powershell and connect to Exchange Online (see earlier post).

You can only copy the contents from the source mailbox into a folder on target mailbox. Use the following command –

Search-Mailbox source-id -TargetMailbox target-id -LogLevel full

You will be prompted to enter a folder name. Once you have entered the name the command will run. This will probably take several minutes and a small report will be displayed at the end.

Close the session to Exchange Online