What is a rolling upgrade?

Rolling upgrade is the recommended method to upgrade a cluster with the least interruption to the end user. At the highest level, a rolling upgrade will upgrade the LDAP, Proxy, MTA then add a new mailbox server’s with the latest version. Once the new mailbox is ready for users, you will move users from the old to the new mailbox server.

This method is very flexible with the least down time and minimal effect to the users. Rolling upgrade gives you the ability to also add or move to new hardware, upgrade or move to new OS instances, customize and test the new experience before migrating users.

This document will provide steps through the rolling upgrade process using a hypothetical scenario that can guide you through the needed procedures to move your instance to new hardware and OS while upgrading and older version (8.6 P14, 8.7.11, 8.8.15, and 9.0) to 10.0 or latest ZCS version.

This is a document that will provide steps and show examples for a define setup and should be used as a reference. Every environment is different and it is always recommended to test within the lab so you can become familiar with your own setup, build a rolling upgrade plan and be familiar with the commands needed to perform the upgrade that meets your needs.
This document will not provide sizing recommendations. There are many variables that go into sizing and performing a rolling upgrade, if you need assistance or would like someone to perform the upgrade for you, please contact the Zimbra Sales to schedule time with a Partner or Professional Service (PS).

image

Environments Overview

The following is the hypothetical environment that we will upgrade:

  • Version: ZCS 8.8.15

  • RHEL 7.2

  • Approximately 11,000 mailboxes

  • Approximately 4 TB of mail data

  • 2 LDAP servers (Multi-Master)

  • 2 Proxy+MTA servers

  • 4 Mailstore servers

  • 1 Logger server

  • Running within VMWare

Upgrading

  • Version 10.0

  • RHEL 8

Upgrade Phases

While the actual schedule around the upgrade is different for each environment and each customer opinion is extremely significant, Synacor PS prefers to establish a starting point for the discussion, based on a best practice that’s been successful for these types of upgrades.

The Upgrade process should involve different types events (labeled as “Phases” below) based on the Zimbra server role. The proper upgrade order for any Zimbra multi-server environment is typically done in this order.

Phases Events

Phase 0

Preliminary Tasks

Phase 1

LDAP Tier Upgrade

Phase 2

LDAP Master Additions

Phase 3

Proxy+MTA Additions

Phase 4

Mailstore Additions

Phase 5

Traffic Cutover

Phase 6

Migration

In general, each of the phases must be analyzed with the following considerations:

  • Service impact (effect of the event on the current mail service).

  • Required technical procedures (exact steps necessary for the execution of the event).

  • Validation methods (how the execution can be checked for correctness).

  • Contingency plan (how to back out or implement a contingency should execution encounter unforeseen circumstances).

Best Practices

There should also be normal, accompanying best practices:

  • An established maintenance window before the LDAP Tier Upgrade, where any necessary restarts of hosts or Zimbra services or intensive processes can be executed in advance of the upgrade.

  • A time period that begins just before the LDAP Tier Upgrade and ends just after the upgrade where no other changes are permitted to the hosts, guest OS, the networks, the storage, on which the Zimbra VMs reside, or the Zimbra application itself (a standard blackout period so the only changes are that of the upgrade procedures).

  • An account modification/provisioning blackout period that starts just before the LDAP Tier Upgrade and terminates just after; thus in the event of a roll-back to the LDAP state just before the upgrade, there is the assurance of no lost LDAP data.

Often for end-users, simply issuing an email notification of the blackout period is necessary as well, as ZWC preferences mods or emailed contacts could potentially be lost in a roll-back scenario.

  • Full lab simulation been performed with the exact versions used.

  • The local backup of key configuration data needs to be created outside of /var/tmp so they are not automatically purged on reboot.

  • Sufficient backup and documenting of the method and associated files for any non-standard configuration outside of Zimbra (customizations to MTAs, Proxys, Mailstores, Branding, etc).

High-Level Upgrade Steps

Phase 0 - Preliminary Tasks

  • Validate connectivity to Zimbra Repo and Files

  • Pull Zimbra release files (.tgz files)

  • Backup certain files/directories

Phase 1 - In place upgrade existing ZCS 8.8.15 LDAP Servers to ZCS 10.0 or latest release.

  1. Upgrade LDAP Master first, than any other (Replica or Master)

    • Maintenance window required (some downtime, typically about 25 minutes per LDAP server).

    • Upon conclusion of this step, all 8.8.15 non-LDAP Zimbra servers will continue to interoperate with LDAP servers that have now been upgraded to ZCS 10.0.

  2. Enable Sync agreement in existing Master for MMR to first new RHEL8 Master.

    • Use the zmldapenable-mmr tool, adding MMR agreement to future RHEL8/ZCS 10.0 LDAP Master (only need the FQDN).

    • Modify zmlocalconfig ldap_master_url and ldap_url values appropriately.

    • Execute zmcontrol restart (1-2 minutes downtime).

Phase 2 - Add two new LDAP Masters (RHEL8/ZCS 10.0)

  1. Install/Config first RHEL8/ZCS 10.0 LDAP Master

    • Configure with reference to existing RHEL 7.2 LDAP Master.

    • Requires clean load of ldap.bak (zmslapcat/zmslapadd) from existing RHEL 7.2 LDAP Master.

    • No maintenance window is necessary.

  2. Install/Config second RHEL8/ZCS 10.0 LDAP Master

    • Configure with reference to the first RHEL8/ZCS 10.0 LDAP Master

    • Requires clean load of ldap.bak (zmslapcat/zmslapadd) from existing RHEL 7.2 LDAP Master.

    • No maintenance window is necessary

Phase 3 - Add two new RHEL8/ZCS 10.0 Proxy+MTA servers

  1. Install/Config new RHEL8 Proxy+MTAs (ZCS v10.0)

    • LDAP source will be the new RHEL8 LDAP Masters.

    • No maintenance window is necessary.

Phase 4 - Add new RHEL8/ZCS 10.0 Mailstores

  1. Install/Config at least all RHEL8/ZCS 10.0 Mailstores.

    • LDAP source will be the new RHEL8 LDAP Masters.

    • No maintenance window is necessary.

    • Add customizations.

Phase 5 - Traffic Cutover to RHEL8/ZCS 10.0 Proxy+MTA servers

  • Validate in advance the SMTP and client traffic can successfully route through the new Proxy+MTA servers for mailboxes on both ZCS 8.8.15 and ZCS 10.0.

  • Verify Login Page and Portal Functionality through the new Proxy+MTA servers.

Phase 6 - Migration

  • Add Migration Server (RHEL6 or RHEL7)

  • Test account Migration

  • Pilot group Migration (friendlies)

  • Production Migration

Low-Level Upgrade Steps

Phase 0 - Preliminary Tasks

These tasks should be executed immediately Prior to Phase 1

  1. Validate access to the public Zimbra servers from 8815LDAP01 and 8815LDAP02 as the root user.

    nc -zv files.zimbra.com 443
nc -zv repo.zimbra.com 443
yum search zimbra

  2. Obtain the ZCS 10 release for RHEL7 (as root), and download to 8815LDAP01 and 8815LDAP02 and unpack.

    cd /usr/src
wget https://files.zimbra.com/downloads/9.0.0_GA/zcs-NETWORK-10.0.0_GA_####.RHEL7_64.YYYYMMDD#######.tgz
tar zxvf zcs-NETWORK-10.0.0_GA_####.RHEL7_64.YYYYMMDD#######.tgz

  3. Take VMware snapshots of the existing environment (RHEL7/ZCS 8.8.15 servers).

  4. Setup firewall rules for all new RHEL8 servers.

Phase 1 - LDAP Tier Upgrade

Upgrading First Master (8815LDAP01)

  1. Preparation (after the start of Zimbra application and OS modification blackout period)

    1. Back up /opt/zimbra/conf directory on both LDAP hosts (as root)

      cd /opt/zimbra/conf; tar -czvf /var/tmp/ozc.tgz *

    2. Backup LDAP databases on Masters

      /opt/zimbra/libexec/zmslapcat /var/tmp
/opt/zimbra/libexec/zmslapcat -c /var/tmp
/opt/zimbra/libexec/zmslapcat -a /var/tmp

    3. Take VMWare Snapshot of VM.

  1. Execute Upgrade

    1. Disable Localconfig settings to get past the LDAP verification step

      zmlocalconfig -e ldap_starttls_required=false
zmlocalconfig -e ldap_starttls_supported=0
zmlocalconfig -e ssl_allow_untrusted_certs=true
zmcontrol restart
zmlocalconfig -s | grep -i bes_searcher
zmldappasswd -b <password>

    2. Run the Zimbra install.sh program from the new release (as root)

      cd /usr/src/zcs-NETWORK-10.0.0_GA_####.RHEL7_64.YYYYMMDD#######
./install.sh -skip-activation-check

    3. Enable multi-master replication to the first new RHEL8 Master then update the ldap_master_url and ldap_url to support LDAP redundancy.

Enabling multi-master replication is needed before installing a new server. If you are planning to use existing LDAP servers, you can skip this section. 
[zimbra@8815LDAP01]#./libexec/zmldapenable-mmr -r 101 -m ldap://10LDAP01.hostname.com:389/ (run zmldapmmrtool -q first)
[zimbra@ 8815LDAP01]# zmlocalconfig -e ldap_master_url="ldap:// ldap://8815LDAP01.hostname.com:389 ldap://8815LDAP02.hostname.com:389"
[zimbra@ 8815LDAP01]# zmlocalconfig -e ldap_url="ldap:// ldap://8815LDAP01.hostman.com:389 ldap://8815LDAP02.hostname.com:389"
[zimbra@ 8815LDAP01]# zmcontrol restart
Upgrading Secondary Master (8815LDAP02)

  1. Preparation (after the start of Zimbra application and OS modification blackout period)

    1. Back up /opt/zimbra/conf directory on all four LDAP hosts (as root)

      cd /opt/zimbra/conf; tar -czvf /var/tmp/ozc.tgz *

    2. Backup LDAP databases on Masters

      /opt/zimbra/libexec/zmslapcat /var/tmp
/opt/zimbra/libexec/zmslapcat -c /var/tmp
/opt/zimbra/libexec/zmslapcat -a /var/tmp

    3. Take VMWare Snapshot of VM.

  1. Execute Upgrade

    1. Disable localconfig settings to get past the LDAP verification step

      zmlocalconfig -e ldap_starttls_required=false
zmlocalconfig -e ldap_starttls_supported=0
zmlocalconfig -e ssl_allow_untrusted_certs=true
zmcontrol restart

    2. Run the Zimbra install.sh program from the new release (as root)

      cd /usr/src/zcs-NETWORK-9.0.0_GA_3924.RHEL7_64.20200331010312
./install.sh -skip-activation-check

    3. Then set localconfig parameters.

      [zimbra@ 8815LDAP01]# zmlocalconfig -e ldap_master_url="ldap://
ldap://8815LDAP02.hostname.com:389 ldap://8815LDAP01.hostname.com:389"
[zimbra@ 8815LDAP01]# zmlocalconfig -e ldap_url="ldap:// ldap://8815LDAP02.hostname.com:389 ldap://8815LDAP01.hostname.com:389"
[zimbra@ 8815LDAP01]# zmcontrol restart
If you have more than 2 master or replica ldap, repeat steps until all existing LDAP’s have been upgraded.

  1. Validate Phase 1

    • Check LDAP applications by verifying authentication against 8815LDAP01 and 8815LDAP02.

    • Validate Synchronicity between the two LDAP Masters.

      for i in \{8815LDAP01,8815LDAP02};do echo - ldap-$i -- && ldapsearch -xLLLH ldap://$i.{domain.com}:389 -s base contextCSN | grep context | awk '\{print $2}' | cut -c1-18 |grep 2021 ;done

    • Execute ldapsearch to count objects on the Master (must run zmlocalconfig -s | grep -i pass to get ldap password)

      for i in \{8815LDAP01,8815LDAP02}; do echo -- $i -- && ldapsearch -LLL -x -h $i.{domain.com}:389 -D 'uid=zimbra,cn=admins,cn=zimbra' -w XXXXXXX '(&(objectclass=zimbraAccount)(mail=*))' mail | wc -l; done

    • Monitor /var/log/zimbra.log cadence on Masters

      tail -f /var/log/zimbra.log

  2. Validate Mail

    • Login to ZWC and send email to internal (other Zimbra users) and outside (Yahoo/Gmail account).

    • Login with Thick client using IMAP, send mail to internal (other Zimbra users) and outside (Yahoo/Gmail account).

    • Validate Zimbra Mobile (phone-based send/receive).

    • Observe /opt/zimbra/log/mailbox.log on mailstores (verify operations are occurring)

    • Check zimbrCsrfTokenCheckEnabled - makes sure it is still FALSE (like current production), if set to TRUE issue zmprov -l mcf zimbraCsrfTokenCheckEnabled FALSE

  3. Phase 1 Revert Strategy

    • Shut down (shutdown now -r) for each of 8815LDAP01 and 8815LDAP02.

    • Revert snapshot taken in step 1c for each of 8815LDAP01 and 8815LDAP02

Maintenance Window Estimates for Phase 1
Estimated Upgrade Time Estimated True Outage Time Suggested Maintenance Window Time

25 minutes for each LDAP Master, an estimate of 75 minutes total

25 minutes

2 hours

Milestone - At this point, all RHEL7 LDAP servers are upgraded to ZCS 10.0 ie latest release.

Estimation time is based on all steps being verified within a test lab.

Phase 2 - LDAP Master Additions

The following steps will add two new master LDAP’s to the cluster which can be any supported OS but needs to be the version which you are upgrading to.

If you are using existing LDAP servers and not planning to add new servers, your LDAP’s are currently upgraded and this section can be skipped.
Adding 10LDAP01

  1. Execute a zmslapcat from 8815LDAP01 and copy the resulting ldap.bak file to /var/tmp on 2 new LDAP servers.

    /opt/zimbra/libexec/zmslapcat /var/tmp

  2. Validate access to the public Zimbra servers from your internal server using the root user:

    nc -zv files.zimbra.com 443
nc -zv repo.zimbra.com 443
yum search zimbra
cd /usr/src

  3. Obtain and prepare the ZCS 10 release for RHEL8 (as root) and Install ZCS 10.0

    cd /usr/src
wget zcs-NETWORK-10.0.0_GA_####.RHEL7_64.YYYYMMDD#####.tgz
tar zxvf zcs-NETWORK-10.0.0_GA_####.RHEL7_64.YYYYMMDD#####.tgz
cd zcs-NETWORK-10.0.0_GA_####.RHEL7_64.YYYYMMDD#####
./install.sh -skip-activation-check

    • Packages to Install

      • zimbra-ldap

      • zimbra-snmp

    • On the installation menu, choose "1" for common configuration

      • Change the LDAP master hostname to 8815LDAP01.

      • Change the LDAP admin password to that of the Zimbra LDAP admin password on 8815LDAP01.hostname.com.

    • On the installation menu, choose "2" for LDAP configuration

      • Choose "4" to change the replication type to mmr instead of replica.

      • The Server ID for this secondary master will default to 2, you must change it to a unique value (4).

      • Change "7" to match the replica password from 8815LDAP01.

      • Change "8" to match the postfix password from 8815LDAP01.

      • Change "9" to match the amavis password from 8815LDAP01.

      • Change "10" to match the Nginx password from 8815LDAP01.

  4. Updating 10LDAP01 is required if it cannot connect to itself.

    zmlocalconfig -e ldap_master_url="ldap:// ldap://10LDAP01.hostname.com:389 ldap://10LDAP02.hostname.com:389"
zmlocalconfig -e ldap_url"ldap:// ldap://10LDAP01.hostname.com:389 ldap://10LDAP02.hostname.com:389"
zmlocalconfig -e ldap_starttls_supported=0
zmlocalconfig -e ldap_starttls_required=false
./zmldapmmrtool -u -o 100 -t off
Zmcontrol restart (validate it connects to 8815LDAP01)

  5. Enable 10LDAP01.hostname.com for multi-master replication with 10LDAP02.hostname.com, but keep the agreement with 8815LDAP01.hostname.com, and appropriately modify zmlocalconfig settings.

    [zimbra@10LDAP01]#./libexec/zmldapenable-mmr -r 101 -m
ldap://10LDAP02.hostname.com:389/
[zimbra@10LDAP01]# ./zmldapmmrtool -u -o 101 -t critical
[zimbra@ 10LDAP01]# zmcontrol restart

  6. Reload the LDAP database using the ldap.bak copied in step 1

    1. As root make sure the ldap.bak is owned by Zimbra:

      chown zimbra:zimbra /var/tmp/ldap.bak

    2. Execute the following as a Zimbra user:

      zmcontrol stop
cd /opt/zimbra/data/ldap
mv mdb mdb.orig
mkdir -p mdb/db
mv accesslog accesslog.orig
mkdir -p accesslog/db
cd /opt/zimbra/libexec
./zmslapadd /var/tmp/ldap.bak
zmcontrol start

  7. Regenerate new 10-year Self-Signed Certs using the Zimbra user.

    cd /opt/zimbra/ssl
mv zimbra zimbra.old
cd /opt/zimbra/bin
./zmcertmgr createca -new
./zmcertmgr deployca
zmcertmgr createcrt -new -days 3650
zmcertmgr deploycrt self
zmcertmgr viewdeployedcrt
zmcontrol restart
tail -f /var/log/zimbra.log
zmprov -l ma zimbra.test2 zimbraPrefVoiceItemsPerPage 27 (some test account)
    NOTE: verify on 8815LDAP01 that it gets to change
zmlocalconfig -e ldap_starttls_required=true
zmlocalconfig -e ssl_allow_untrusted_certs=false
zmlocalconfig -e ldap_starttls_supported=1
zmcontrol restart
Adding 10LDAP02

  1. Validate access to the public Zimbra servers from your internal server using the root user:

    nc -zv files.zimbra.com 443
nc -zv repo.zimbra.com 443
yum search zimbra
cd /usr/src

  2. Obtain and prepare the ZCS 10 release for RHEL8 (as root) and Install ZCS 10.0

    cd /usr/src
wget zcs-NETWORK-10.0.0_GA_####.RHEL7_64.YYYYMMDD#######.tgz
tar zxvf zcs-NETWORK-10.0.0_GA_####.RHEL7_64.YYYYMMDD#######.tgz
cd zcs-NETWORK-10.0.0_GA_####.RHEL7_64.YYYYMMDD#######
./install.sh -skip-activation-check

    • Packages to Install

      • zimbra-ldap

      • zimbra-snmp

    • On the installation menu, choose "1" for common configuration

      • Change the LDAP master hostname to 10LDAP01.hostname.com.

      • Change the LDAP admin password to that of the Zimbra LDAP admin password on 10LDAP01.hostname.com.

    • On the installation menu, choose "2" for LDAP configuration

      • Choose "4" to change the replication type to mmr instead of replica.

      • The Server ID for this secondary master will default to 2, you must change it to a unique value (5).

      • Change "7" to match the replica password from 10LDAP01.

      • Change "8" to match the postfix password from 10LDAP01.

      • Change "9" to match the amavis password from 10LDAP01.

      • Change "10" to match the Nginx password from 10LDAP01.

  3. Reload the LDAP database using the ldap.bak copied in step 1

    1. As root make sure the ldap.bak is owned by Zimbra:

      chown zimbra:zimbra /var/tmp/ldap.bak

    2. Execute the following as a Zimbra user:

      zmcontrol stop
cd /opt/zimbra/data/ldap
mv mdb mdb.orig
mkdir -p mdb/db
mv accesslog accesslog.orig
mkdir -p accesslog/db
cd /opt/zimbra/libexec
./zmslapcat /var/tmp/ldap.bak
zmcontrol start

  4. Modify zmlocalconfig settings on 10LDAP02 to support the new LDAP replicas.

    [zimbra@ 10LDAP02]# zmlocalconfig -e ldap_master_url="ldap:// ldap://10LDAP02.hostname.com:389 ldap://10LDAP01.hostname.com:389"
[zimbra@ 10LDAP02]# zmlocalconfig -e ldap_url"ldap:// ldap://10LDAP02.hostname.com:389 ldap://10LDAP01.hostname.com:389"
[zimbra@ 10LDAP02]# zmcontrol restart

  5. Validate LDAP servers configuration and connectivity:

    • Validate Synchronicity between the two LDAP Masters.

      for i in \{8815LDAP01,8815LDAP02,10LDAP01,10LDAP02};do echo - ldap-$i -- && ldapsearch -xLLLH ldap://$i.hostname.com:389 -s base contextCSN | grep context | awk '\{print $2}' | cut -c1-18 |grep 2021 ;done

    • Execute ldapsearch to count objects on the Master (must run zmlocalconfig -s | grep -i pass to get ldap password)

      for i in \{8815LDAP01,8815LDAP02,10LDAP01,10LDAP02}; do echo -- $i -- && ldapsearch -LLL -x -h $i.hostname.com:389 -D
'uid=zimbra,cn=admins,cn=zimbra' -w XXXXXXX '(&(objectclass=zimbraAccount)(mail=*))' mail | wc -l; done

    • Monitor /var/log/zimbra.log cadence on Masters

      tail -f /var/log/zimbra.log

  6. Validate Mail flow and Client connectivity:

    • Login to ZWC and send email to internal (other Zimbra users) and outside (Yahoo/Gmail account).

    • Login with Thick client using IMAP, send mail to internal (other Zimbra users) and outside (Yahoo/Gmail account).

    • Validate Zimbra Mobile (phone-based send/receive).

    • Observe /opt/zimbra/log/mailbox.log on mailstores (verify operations are occurring).

      1. Check zimbrCsrfTokenCheckEnabled - makes sure it is still FALSE (like current production), if set to TRUE issue zmprov -l mcf zimbraCsrfTokenCheckEnabled FALSE.

  7. Revert Strategy

    • The addition of new ZCS 10 LDAP servers will not affect current production. To remove the new ZCS 10 LDAP servers, you would execute:

      zmprov ds 10LDAP01.hostname.com
zmprov ds 10LDAP02.hostname.com

State After LDAP Master Additions

image

Maintenance Window Estimates for Phase 2
Estimated Install Time (includes testing) Estimated True Outage Time Suggested Maintenance Window Time

45 minutes for each LDAP Master, an estimate of 90 minutes total

No outage

Maintenance Window Not Required (can be done any time)
Estimation time is based on all steps being verified within a test lab.

Phase 3: Proxy+MTA Additions

This phase will involve the creation of two new servers that contains the Proxy and MTA modules:

If you are planning to utilize your current Proxy and MTA servers, you can do an in-place upgrade. Please be aware that the server will be off line during the upgrade process. If you have a single Proxy and/or MTA setup, your system will be off line during the upgrade. If the cluster has more than one proxy and/or MTA, the remaining servers will handle the traffic until the upgrade is completed. Zimbra recommends reviewing, understanding and testing how your setup will handle this scenario before performing the upgrade.

  1. Validate access to the public Zimbra servers from your internal servers using the root user:

    nc -zv files.zimbra.com 443
nc -zv repo.zimbra.com 443
yum search zimbra
cd /usr/src

  2. Obtain and prepare the ZCS 10 release for RHEL8 (as root) and Install ZCS 10.0

    cd /usr/src
wget zcs-NETWORK-10.0.0_GA_####.RHEL7_64.YYYYMMDD#######.tgz
tar zxvf zcs-NETWORK-10.0.0_GA_####.RHEL7_64.YYYYMMDD#######.tgz
cd zcs-NETWORK-10.0.0_GA_####.RHEL7_64.YYYYMMDD#######
./install.sh -skip-activation-check

    • Packages to Install

      • zimbra-mta

      • zimbra-dnscache

      • zimbra-mta

      • zimbra-snmp

      • zimbra-proxy

      • zimbra-memcached

    • On the installation menu, choose "1" for common configuration

      • Change the LDAP master hostname to 10LDAP01.hostname.com.

      • Change the LDAP admin password to that of the Zimbra LDAP admin password on 10LDAP01.hostname.com.

  3. Apply Commercial SSL Certificates

    1. Copy commercial certs to /opt/zimbra/ssl/zimbra/commercial

    2. Set permissions/ownership for commercial certs

    3. zmcertmgr verifycrt

    4. zmcertmgr deploycrt

  4. Verify and/or settings the following configuration:

    a. Update zimbraMailTrustedIP
b. Update zimbraHttpThrottleSafeIps
c. Check zimbraMtaMyNetworks (subnet should already be included) at the server level
d. zmprov ms `zmhostname` zimbraMtaCommandDirectory /opt/zimbra/common/sbin
e. zmprov ms `zmhostname` zimbraMtaDaemonDirectory /opt/zimbra/common/libexec
f. zmprov ms `zmhostname` zimbraMtaMailqPath /opt/zimbra/common/sbin/mailq
g. zmprov ms `zmhostname` zimbraMtaManpageDirectory /opt/zimbra/common/share/man
h. zmprov ms `zmhostname` zimbraMtaNewaliasesPath /opt/zimbra/common/sbin/newaliases
i. zmprov ms `zmhostname` zimbraMtaSendmailPath /opt/zimbra/common/sbin/sendmail
j. zmprov ms `zmhostname` zimbraMtaSenderCanonicalMaps 'proxy:ldap:/opt/zimbra/conf/ldap-canonical.cf'
k. zmprov ms `zmhostname` -zimbraServiceEnabled antispam
l. zmprov ms `zmhostname` zimbraReverseProxyMailMode redirect
m. Check zmocalconfig for admin@<fqdn_of_this_host>, change to admin@hostname.com
n. zmlocalconfig -e ldap_master_url=”ldap://10LDAP01.hostname.com:389 ldap://10LDAP02.hostname.com:389”
o. zmlocalconfig -e ldap_url=”ldap://10LDAP02.hostname.com:389 ldap://10LDAP01.hostname.com:389”
The following two commands need to be ran on each proxy after installing the first two new mailstores. 
emsproxy01n
zmprov ms `zmhostname` zimbraReverseProxyUpstreamLoginServers emsmbs01n.hostname.com
emsproxy02n
zmprov ms `zmhostname` zimbraReverseProxyUpstreamLoginServers emsmbs02n.hostname.com

  1. Validate setup and configuration;

    1. Initial Checks of Zimbra Health

      zmcontrol status
for i in \{25,80,110,143,443,465,587,993,995};do lsof -i:$i; done
nc -zv mailstore 7025
nc -zv ldap 389
nc -zv other_proxy 11211

    2. Test/Validate routing (use command line tools and email clients)

      • Port 25

      • Port 587

      • Port 465

      • Port 80

      • Port 443

      • Port 110

      • Port 143

      • Port 995

      • Port 993

  2. Validate Mail:

    • Login to ZWC and send email to internal (other Zimbra users) and outside (Yahoo/Gmail account).

    • Login with Thick client using IMAP, send mail to internal (other Zimbra users) and outside (Yahoo/Gmail account).

    • Validate Zimbra Mobile (phone-based send/receive).

    • Observe /opt/zimbra/log/mailbox.log on mailstores (verify operations are occurring)

  3. Revert Strategy

    • The addition of new ZCS 10 Proxy+MTA servers will not affect current production. To remove the new ZCS 10 Proxy+MTA servers, you would execute:

      zmprov ds emsproxy01n.hostname.com
zmprov ds emsproxy02n.hostname.com

State After Proxy+MTA Additions

image

The production inbound client and mail connections are currently directed to emsproxy01 and emsproxy02, emsproxy01n and emsproxy02n are active and listening but not receiving active connections.
Maintenance Window Estimates for Phase 3
Estimated Install Time Estimated True Outage Time Suggested Maintenance Window Time

20 minutes for each Proxy/MTA, an estimate of 60 minutes total

No outage

Maintenance Window Not Required (can be done any time)

Estimated Post Install Time (configuration, tuning, testing)

Estimated True Outage Time

Suggested Maintenance Window Time

2 hours each proxy

No outage

Maintenance Window Not Required (can be done any time)

Phase 4 - Mailstore Additions

This step will involve the creation of new Zimbra mailstores and a new logger server. It is recommended to create all mailstores at the same time but not required.

For a rolling migration, phase 4 is required. Doing an in-place upgrade will stop the mail store and all users on the store will be off line during the upgrade. Also with the migration from 8.8.15 / 9.0 with NG modules you will need to ensure, you have utilize the migration tools for data some retention.
You will use the following steps t

  1. The following modification will configure the existing 8.8.15 proxies to utilize the 8.8.15 stores as the only login page.

    1. zmprov ms emsproxy01.hostname.com +zimbraReverseProxyUpstreamLoginServers emsmbs01.hostname.com

    2. zmprov ms emsproxy01.hostname.com +zimbraReverseProxyUpstreamLoginServers emsmbs02.hostname.com

    3. zmprov ms emsproxy02.hostname.com +zimbraReverseProxyUpstreamLoginServers emsmbs01.hostname.com

    4. zmprov ms emsproxy02.hostname.com +zimbraReverseProxyUpstreamLoginServers emsmbs02.hostname.com

  1. Validate access to the public Zimbra servers from your internal server using the root user:

    nc -zv files.zimbra.com 443
nc -zv repo.zimbra.com 443
yum search zimbra
cd /usr/src

  2. Obtain and prepare the ZCS 10 release for RHEL8 (as root) and Install ZCS 10.0 or latest release.

    cd /usr/src
wget zcs-NETWORK-10.0.0_GA_####.RHEL7_64.YYYYMMDD#######.tgz
tar zxvf zcs-NETWORK-10.0.0_GA_####.RHEL7_64.YYYYMMDD#######.tgz
cd zcs-NETWORK-10.0.0_GA_####.RHEL7_64.YYYYMMDD#######
./install.sh -skip-activation-check

    • Packages to Install

      • zimbra-snmp

      • zimbra-store

      • zimbra-apache

      • zimbra-spell

      • zimbra-converted

Only a single instance of Zimbra-Logger is needed. You can install logger onto a standalone or a mailbox store by selecting Y for zimbra-logger. Do not install more than one instance, it is not needed.

  • On the installation menu, choose "1" for common configuration

    • Change the LDAP master hostname to 10LDAP01.hostname.com.

    • Change the LDAP admin password to that of the Zimbra LDAP admin password on 10LDAP01.hostname.com.

  • While the Installation is occurring, using 10LDAP01 review the default COS and global config

    • Remove from default COS zimbraMailHostPool

    • Remove from zimbraReverseProxyUptreamLoginServers

    • Remove from zmprovReverseProxyLookupTarget

      1. Apply Commercial SSL Certificates (if necessary)

        1. Copy commercial certs to /opt/zimbra/ssl/zimbra/commercial

        2. Set permissions/ownership for commercial certs

        3. zmcertmgr verifycrt

        4. zmcertmgr deploycrt

      2. Validation

        1. zmcontrol status

        2. netstat -ntplu | grep java

        3. for i in \{7025,7026,7071,7110,7143,7993,7995,8080,8443};do lsof -i:$i; done

        4. nc -zv ldap 389

        5. nc -zv other_proxy 11211

      3. Tune Mailstore

        1. Reset Localconfig keys

        2. Reset MariaDB parameters

        3. Reset server config values

        4. Apply Zimlets

        5. Check zmocalconfig for admin@<fqdn_of_this_host>, change to admin@hostname.com

        6. Vm.swappiness setting

  • As root execute sysctl vm.swappiness=1

  • As root modify /etc/sysctl.conf, add the following lines

    # Zimbra settings
vm.swappiness = 0

    1. Mailbox Testing

      1. Create test mailbox in new mailstores

      2. Manually update new proxy to add new mailstores

      3. Access the mailbox through a new proxy (direct)

    2. Logger Configuration for emslog01n

      Edit rsyslog.conf (enable UDP 514)
Run zmsyslogsetup (as root)
Run zmloggerinit (as zimbra)
Run zmupdateauthkeys (as zimbra)
Run zmcontrol restart (as zimbra)
Run systemctl restart rsyslog (as root)
Validate zimbraloghostname still points to emslog.hostname.com

  • At this point we can manually paste in the bottom of /etc/rsyslog.conf on all ZCS 10 servers, the following:

    local0.*    @emslogn.hostname.com
local1.*    @emslogn.hostname.com
auth.*      @emslogn.hostname.com
local0.*    -/var/log/zimbra.log
local1.*    -/var/log/zimbra-stats.log
auth.*      -/var/log/zimbra.log
mail.*      @emslogtn.hostname.com
mail.*      -/var/log/zimbra.log

  • Restart rsyslog

    [root@emsproxy01n etc]# service rsyslog restart

  • Now the zimbra-stats and service up/down info will go to the emslog01n for the new version 10 servers only.

Actual cutover to use emslog01n is TBD, will require a change to zimbraloghostname (global config) and execution of zmsyslogsetup and restart of rsyslog service on all Zimbra servers.

  1. Validate Mail

    • Login to ZWC and send email to internal (other Zimbra users) and outside (Yahoo or Gmail account).

    • Login with Thick client using IMAP, send mail to internal (other Zimbra users) and outside (Yahoo or Gmail account).

    • Validate Zimbra Mobile (phone-based send/receive).

    • Observe /opt/zimbra/log/mailbox.log on mailstores (verify operations are occurring).

  2. Revert Strategy

    • The addition of new ZCS 10 Mailstore servers will not affect current production. To remove the new ZCS 10 Mailstores servers, you would execute:

      zmprov ds emsmbs01n.hostname.com
zmprov ds emmbs02n.hostname.com
zmprov ds emsmbs03n.hostname.com
zmprov ds emsmbs04n.hostname.com

State after Mailstore Additions

image

Overall End State (all Zimbra servers)

image

  • ZCS 10.0 side is completely built, but there is no impact on production - essentially idle.

  • You can now tune all ZCS 10.0 servers

  • Add test mailboxes to ZCS 10.0 mailstores

  • You can apply customizations

  • You can test scripts

  • You can execute isolated testing on the ZCS 10.0 side through the Proxy+MTAs

    • Preliminary testing using OpenSSL

    • Direct client connections for all secure POP, IMAP, HTTP and SMTP (25, 465, 587)

    • Test against test mailboxes on the ZCS 8.8.15 mailstores

Maintenance Window Estimates for Phase 4
Estimated Install Time Estimated True Outage Time Suggested Maintenance Window Time

45 minutes for each mailstore but can install in a parallel estimate of 120 minutes total

No outage

Maintenance Window Not Required (can be done any time)

Estimated Post Install Time (configuration, tuning, customizations, testing)

Estimated True Outage Time

Suggested Maintenance Window Time

4 hours each mailstore and logger

No outage

Maintenance Window Not Required (can be done any time)

Phase 5 - Traffic Cutover

  1. Validate in advance the SMTP and client traffic can successfully route through the new Proxy+MTA servers for mailboxes on both ZCS 8.8.15 and ZCS 10.0

    • Testing and Validation (initiate SMTP and client traffic at new MTA and Proxies, validate successful connectivity to ZCS 8.8.15 and 10.0 mailstores)

    • Verify Login Page and Portal Functionality through the new Proxy+MTA servers

  2. Execute Cutover

    • Review global config for zimbraMailTrustedIP and zimbraHttpThrottleSafeIps (adding in ZCS 10/RHEL8 IP addresses)

      • Issue zmmailboxdctl restart on all mailstores (ZCS 8.8.15 and ZCS 10.0)

      • Introduce first Proxy+MTA to production traffic by adding emsproxy01n to your external, inbound SMTP & HTTP/s relay.

      • Monitor/Evaluate

      • Remove one RHEL7.2/ZCS 8.8.15 Proxy+MTA server from Pool

      • Monitor/Evaluate

      • Repeat replacement procedures for the second Proxy+MTA

      • Soft Maintenance window recommended (execute upgrade during off-business hours)

      • Zmprov gacf zimbraReverseProxyUpstreamLoginServers should show all (ZCS 8.8.15 and ZCS 10.0) mailstores

      • Point all ZCS 8.8.15 servers to the new Logger

        [root@emsproxy01 etc]# perl -pi -e 's/emslog01/emslog01n/g' /etc/rsyslog.conf
[root@emsproxy01 etc]# service rsyslog restart
Redirecting to /bin/systemctl restart rsyslog.service

  3. As you add the new RHEL8/ZCS10 servers into production, you will need to update the Zimbra attribute zimbraThrottleSafeIps. At present you do not have your new production IPs within your list, but you can add them as well.

    • You don’t need to whitelist the LDAP or the migration server. So within the Phase 5 Cutover of traffic, we recommend adding the new production IPs to zimbraHttpThrottleSafeIPs, which can be done with a single request. The following is an example:

[zimbra@emsmbst02n ~]$ zmprov mcf +zimbrahttpthrottlesafeips xx.xx.xx.x1 +zimbrahttpthrottlesafeips xx.xx.xx.x2 +zimbrahttpthrottlesafeips xx.xx.xx.x3 +zimbrahttpthrottlesafeips xx.xx.xx.x4 +zimbrahttpthrottlesafeips xx.xx.xx.x5
+zimbrahttpthrottlesafeips xx.xx.xx.x6 +zimbrahttpthrottlesafeips xx.xx.xx.x7 +zimbrahttpthrottlesafeips xx.xx.xx.x8 +zimbrahttpthrottlesafeips xx.xx.xx.x9

  1. Because the mailstores will not pick up the change until you issue a zmmailboxdctl restart, you will have to do this on emsmbs01 - emsmbs04. We also have to do this on emsmbs01n - emsmbs04n, but these will not have any users on them (no impact). the zmmailboxctl restart only takes 30-60 seconds, so very, very minor, and minimized by the fact that you will have declared a maintenance window.

  2. Revert Strategy

    • Change the RRDNS to refer to emsproxy01 and emsproxy02

  3. After Phase 5 is complete

    • All traffic for ZCS 8.8.15 users will come through the ZCS 10 Proxy+MTA hosts

    • There will be a new Login Page (similar)

    • The ZCS 8.8.15 mailstores will continue to use ZCS 8.8.15 LDAP Masters

image

Maintenance Window Estimates for Phase 5
Estimated Cutover Time (testing included) Estimated True Outage Time Suggested Maintenance Window Time

90 minutes

10-15 minutes

Maintenance Window Required (because revert may be necessary)

Phase 6 - Migration

  • Prepare Directories

  • Migrate single account

  • Validate fidelity of mailbox

  • Migrate pilot group (friendlies)

  • Production Migration

    1. Prepare a migration list;

      1. Test accounts

      2. Pilot group of friendly users

      3. Migrate all users

Migration

<Need to obtain script form >

Maintenance Window Estimates for Phase 6
Estimated Migration Time Estimated True Outage Time Suggested Maintenance Window Time

TBA Days

No outage

Maintenance Window Not Required (can be done any time)

Migration Analysis

Building four new RHEL8/ZCS 10 mailstores in advance, it would provide the opportunity to execute parallel processing.

image

Key Assumptions

, The throughput rate that is typically achieved is about 20GB/Hour if using 2 threads between each source and destination mailstore. (But this is subject to change based on each environment). Although this will add some load on the Zimbra source and destination server it is typically undetectable by end users, however, to be safe - some customers opt to run the migration threads during non-business hours.

Using a single migration thread between the source and destination server drops the throughput down to around 12 GB/Hour, and the load increase on the source and the destination server is negligible which means the migration can be run through business hours with just a one hour break per 24 hours for remediation on any migration issues (typically solved by a simple retry of the migrated mailbox). Optionally, the migration can simply be run continuously until finished if using a single thread.